A Novel Real Coded Genetic Algorithm for Software Mutation Testing

Abstract

Information Technology has rapidly developed in recent years and software systems can play a critical role in the symmetry of the technology. Regarding the field of software testing, white-box unit-level testing constitutes the backbone of all other testing techniques, as testing can be entirely implemented by considering the source code of each System Under Test (SUT). In unit-level white-box testing, mutants can be used; these mutants are artificially generated faults seeded in each SUT that behave similarly to the realistic ones. Executing test cases against mutants results in the adequacy (mutation) score of each test case. Efficient Genetic Algorithm (GA)-based methods have been proposed to address different problems in white-box unit testing and, in particular, issues of mutation testing techniques. In this research paper, a new approach, which integrates the path coverage-based testing method with the novel idea of tracing a Fault Detection Matrix (FDM) to achieve maximum mutation coverage, is proposed. The proposed real coded GA for mutation testing is designed to achieve the highest Mutation Score, and it is thus named RGA-MS. The approach is implemented in two phases: path coverage-based test data are initially generated and stored in an optimized test suite. In the next phase, the test suite is executed to kill the mutants present in the SUT. The proposed method aims to achieve the minimum test dataset, having at the same time the highest Mutation Score by removing duplicate test data covering the same mutants. The proposed approach is implemented on the same SUTs as these have been used for path testing. We proved that the RGA-MS approach can cover maximum mutants with a minimum number of test cases. Furthermore, the proposed method can generate a maximum path coverage-based test suite with minimum test data generation compared to other algorithms. In addition, all mutants in the SUT can be covered by less number of test data with no duplicates. Ultimately, the generated optimal test suite is trained to achieve the highest Mutation Score. GA is used to find the maximum mutation coverage as well as to delete the redundant test cases.

1. Introduction

Software is used to control the entire hardware system. Due to this, it has become part of the most complex components and its use has grown more and more. Indeed, its growth has caused new algorithms and development methodologies to be implemented, and thus, software is being applied to a large number of different areas. What is more, software can also support symmetry as its use has been widespread across entire systems and independent content. Symmetry can be identified as an extraordinary characteristic that has been widely deployed in diverse research fields of computer engineering.

Software testing constitutes the process of executing a software product or a portion of it in a controlled environment with a given set of input (test cases) [1]. Moreover, it involves the execution and examination of a program or a software system in order to identify errors or faults. The main goal of software testing is to determine the errors in a complete software product (or in a component of it) to ensure a high probability that the corresponding software is correct [2].

Search-Based Software Engineering (SBSE) is an emerging field of research and practice in Software Engineering, where the term “search” refers to the metaheuristic search-based optimization techniques used. SBSE aims to reformulate problems in the area of software engineering, particularly in software testing problems, into search-based optimization problems. A comprehensive survey of SBSE is proposed in Ref. [3], where related research fields are organized into categories drawn from the corresponding ACM subject categories within the Software Engineering body of knowledge.

In software testing, mutants form simulated and artificially generated faults that behave in a similar way to the realistic ones [4,5]. In addition, mutants can be used for test data generation in a software testing activity [6]. These are created by systematic injection of faults using some predefined mutation operators [7]. Mutation testing is thus a form of white-box testing initially suggested in Ref. [8] and later explored by different researchers [9,10,11]. Execution of a test case (test inputs) against mutants results in the adequacy score of that test case, where this result is also called the Mutation Score (MS).

In recent years, a number of search-based algorithms, metaheuristic methods, Evolutionary Algorithms (EA) and optimization techniques have already been developed to automatically generate test data in the area of software testing. Different EAs like Genetic Algorithms (GA), PSO, ACO, Monarch Butterfly Optimization Algorithm (MBO), Slime Mold Algorithm (SMA), Moth Search Algorithm (MS), RUNge–Kutta method (RUN), Colony Predation Algorithm (CPA), Weighted Mean of Vectors (INFO), and Harris Hawks Optimization (HHO) have been proposed in unit level testing to generate optimized test data for path testing and these path coverage-based testing methods can only generate test data to traverse target paths [12]. However, these techniques may fail to detect all software faults because information about fault detection is not incorporated into the process of generating test data [13]. So, existing path coverage-based techniques may not always guarantee that the test data can effectively detect all faults in target paths. Hence, it is vital to implement the best of each test case through Mutation Analysis [14].

Monarch Butterfly Optimization Algorithm (MBO) employs the migration behaviors of butterflies where all monarch butterflies are located at two different lands. These lands are fixed and unchanged during the whole optimization process. A parameter p is used for calculation at the beginning stage of the search process [15]. The Slime Mold Algorithm basically refers to Physarum Polycephalum [16]. It is an eukaryote that inhabits cool and humid places. Also, the main nutritional stage is considered the Plasmodium; in this stage, the organic matter in slime mold seeks food, surrounds and secretes enzymes with the aim of digesting it. In the optimization process, the front end extends into a fan-shape, followed by an interconnected venous network that allows cytoplasm to flow inside. It can be used for multiple food sources at the same time to form a venous network connecting them. This metaheuristic algorithm is mainly used to solve many graph theory problems and also for generating different networks.

In addition, moths are like butterflies that belong to the order Lepidoptera. The main features of moths are Phototaxis, signifying movement of an organism towards or away from a source of light and Levy flights. Taking all the features of moths, a new kind of metaheuristic algorithm has been developed, called Moths Search Algorithm (MSA) [17]. The phototaxis and levy flights of the moths can be used to build up a general-purpose optimization method. Also, the Colony Predation Algorithm (CPA) is based on the co-existence of animals and mimics the supportive behavior of social animals as well as the predation strategy of hunting animals [18]. The different operations applied in CPA are communications and collaboration, disperse food, encircle foods, supporting the closest individual and searching for foods. Through these five processes, the global optimum solution can be achieved.

The RUNge–Kutta method (RUN) is an iterative process used to solve the numerical Cauchy problems for a system of ordinary differential equations. To solve the differential equations, both the arithmetic and the geometric mean are applied [19]. The most important strategy of Harris Hawks Optimization (HHO) is catching its prey in order to hunt in groups and to collaborate with the hawks, as opposed to other predators [20]. In this process, a number of hawks attack from different ways their chosen prey in collaboration with simultaneous delusion and approached in a controlled manner. The attack is desired to be completed in a few seconds and it continues until the hunting is successful or, on the other hand, the prey manages to completely escape. Finally, the hunting process is completed as the prey, which has low energy and has lost its defensive abilities, is easily hunted by the leader. HHO is a population-based optimization technique and it can be applied to any optimization problem with appropriate limitations and constraints. The optimum solution can be found in the prey itself.

Although there are various studies related to generating test data for path coverage, few researchers have proposed techniques to detect faults via path coverage-based test data. The purpose of this particular research is to generate test data for path coverage as well as to detect faults lying in the System Under Test (SUT). In the relevant literature, some of these methods not only generate test data with the aim of covering a single or multiple paths, but also attempt to find the mutation score of every test case for a specific SUT [21].

In this paper, we propose an SBSE algorithm that can effectively identify the maximum mutation score (coverage) by taking into account all the corresponding test cases, which are generated through path coverage-based testing. Initially, path coverage-based test cases are generated and then the proposed algorithm aims to find the mutation score for every test case by employing different mutation operators. In particular, a real coded Genetic Algorithm (GA) is proposed, which automatically generates test data to cover multiple paths at a time and further exercises the test data to achieve the maximum mutation score. The proposed algorithm is abbreviated as RGA-MS (Real coded Genetic Algorithm for maximizing the Mutation Score).

The rest part of the paper is organized in different sections as follows: In Section 2 some basic concepts on mutation testing are described along with some relevant studies. In Section 3, the steps of the proposed RGA-MS algorithm are outlined. Experimental setup along with the analysis of the experiment results are discussed in Section 4. The same Section also presents a comparative study of the proposed approach with other existing methods focusing on achieving the highest mutation score, and the threats to validity regarding the proposed technique. Finally, our research’s conclusion and future scopes are outlined in Section 5.

2. Background

In this section, an overview of mutation testing and GA fields of research is presented.

2.1. Mutation Testing

Mutation testing is a fault-based white-box software testing method. Its role is to evaluate and to improve the quality of test data by killing the active mutants present in the SUT [22]. During a mutation testing process, the SUT is initially tested with use of the test cases designed by the coverage-based testing. After the coverage-based testing is completed, mutation testing is performed. Using this technique, certain mutation operators are inserted into the code to mutate the program [23]. The mutated program is then tested against corresponding test cases previously designed to kill the mutants. If any test case detects a mutant, then this mutant has to be killed; otherwise, it is still considered as being alive [24]. Therefore, the main goal of mutation testing is to select the most efficient test data (i.e., those with efficient error detection capabilities) and to distinguish the initial program from the seeded mutants [25].

Even though test sets can be selected as well as generated by a tester, these manual practices are often complex and time-consuming, so the need for automated test case generation arises. Although test data generation is known to be a complex process, several researchers have proposed techniques for the automation of this task. In this field, a systematic mapping study, whose purpose is to identify test data generation approaches based on mutation testing, can be found in Ref. [26].

Furthermore, Search-based Mutation Testing uses meta-heuristic optimization techniques for mutation testing, like Genetic Algorithm, Hill Climbing, and other evolutionary approaches. The survey paper in Ref. [27] discusses the challenges and opportunities faced in applying search-based techniques for mutation testing. It is proven that the major challenge consists of the computational cost as well as the storage for a large set of mutants.

Mutation testing may overcome some limitations of other testing approaches, but it is often considered costly. Evolutionary algorithms can be used to reduce the cost of data generation in different testing methodologies. As a result, the authors in Ref. [28] introduced a novel strategy with the elitist genetic algorithm for generating efficient test input data in the context of mutation testing. Search-Based Software Testing (SBST) has also been applied to the generation of weakly adequate mutation-based test data [29,30]. Specifically, SBST was firstly used with the aim of killing mutants in Ref. [31].

Mutation testing is executed through a variety of different steps, which are summarized in Refs. [22,23,32]:

• Mutants construction regarding the program under test;
• Implementation of the test cases with the use of the mutation system to verify the program output;
• If the output is incorrect, then the test case detects a fault and the program must be alternated in order to restart the mutation testing process;
• If the output is found to be correct, then the test case is executed against each live mutant;
• After each test case has been executed against each live mutant, the remaining mutants are either equivalent or killed. An equivalent mutant always produces the same output as the original program, so there is no test case to kill it. However, for live mutants, new test cases must be created and the process is repeated until all mutants are killed;
• The efficiency of the mutation testing can be assessed by utilizing a higher mutation score, which is defined by the following Equation (1) [32].

$$MS=\frac{\mathrm{Total}\mathrm{Mutants}\mathrm{Killed}}{\mathrm{Total}\mathrm{Mutants}\mathrm{Present}\mathrm{In}\mathrm{The}\mathrm{Program}}\times 100$$

(1)

2.2. Real Coded GA

A Genetic Algorithm is initialized with a number of initial solutions, entitled initial population. Successively, it forms the new population or new off-springs from the old population based on the fitness value of the population towards the problem. The more the fitness value, the more they will be suitable to reproduce [33]. The process of a new off-spring generation starts with the selection of parents based on a specific selection process. After the parent selection process, the new off-spring is generated using two different operators, i.e., crossover and mutation [34].

The most critical task in software mutation testing is to generate test data, which could kill maximum mutants present in the SUT. In this case, the optimization problem is the achievement of the highest mutation score with a lesser number of test data. From literature, many evolutionary algorithms have already been used to solve this particular problem. So, considering mutation coverage as the test adequacy criteria, the GA-based method is proposed in this paper to identify the optimized test cases, which give maximum mutation score. GA is used in terms of automation process and objective function is formulated to find maximum mutation score for the SUT.

2.3. Related Work

In literature, different evolutionary algorithms have often been utilized to address the issues identified in mutation testing. A number of corresponding research approaches, from the domain of test case generation as well as optimization for mutation testing using different GA-based techniques, are presented in this section.

A hybrid GA (HGA)-based method for test case optimization, which combines the features of GA along with local search techniques, is proposed in Ref. [35]. This particular approach is employed in terms of test case generation and optimization regarding path coverage and mutation analysis. Different mutation operators for calculating the mutation score have been presented. From the experimental evaluation, the authors proved that the recommended HGA-based method generates both locally and globally optimal solutions with rapid convergence. Another work introduces a GA-based technique for mutation analysis through automatic test case generation [25]. The three following mutation operators, e.g., Condition Overall Replacement (COR), Condition Statement Replacement (CSR) and Branch Value Increment (BVI), have been used to kill the mutants. This particular GA-based approach reduces the number of test cases using the Fault Detection Matrix (FDM).

Authors in a similar work developed a GA-based method to solve their formulated model and then applied this technique to several real-world programs [36]. The experimental results confirm that the proposed approach can generate test data for traversing the target path and detecting faults in a SUT. The authors in Ref. [12] presented a novel method to generate test data for covering multiple paths at a time. The proposed method can not only simultaneously cover multiple paths but can also detect faults present in the SUT. A weighted GA-based model is constructed to solve their multi-objective optimization problem. This technique is compared with different methods, like the random method and the methods proposed in Refs. [36,37]. The reported results confirm that the proposed technique can efficiently generate test data for traversing target paths and detect maximum faults in the SUT.

A novel method for the automatic generation of test data has been pertained in Ref. [38]. The authors have used GA for mutant analysis to achieve the maximum mutation score, where the proposed GA-based approach achieves optimized test data without any redundancy. The reported results in mutation analysis can be benefited from the Delete operator. The authors in Ref. [39] introduced another GA-based technique for the automatic generation of test cases. The produced test suite is in addition implemented for mutation analysis, where the mutation score is computed according to the total number of mutants killed from a particular test suite. Specifically, only one program for computing the power value to perform the experiments, has been employed. The derived experimental results have demonstrated that the utilized method can achieve $100\%$ mutation score, also achieving better test cases. In another similar work [40], a GA-based method for optimizing software testing efficiency by taking into account path coverage and boundary coverage-based test data, has been introduced. A model that reveals faults and kills a mutant with use of genetic algorithms, is introduced in Ref. [41]. The source program along with the mutant are instrumented so that each unit’s input as well as output’s behavior can be tracked, where a checker module is employed in order to compare and track each unit’s output.

Another hybrid GA (HGA)-based method for the automatic production of test cases for mutation testing has been introduced in Ref. [42]. Authors utilized SUT data flow information for the computation of mutation scores. Moreover, the source code of a software program that generates prime numbers within a specific range for their experiments, has been utilized along with the development of a tool in $C\#$ to generate Control Flow Graph (CFG) for a particular program developed in C. A comparison of this approach against both random and GA-based methods has been performed and the results depicted that this particular HGA-based method generates a maximum mutation score. Moreover, the work in Ref. [43] suggests dummy variables introduction into some already defined constraints regarding the specification; the proposed method utilizes the constraints in input as well as output variables with the aim of guiding the generation of the test data. This is achieved on the contrary to the ordinary specification-based testing (SBT) methods that concern only constraints over input variables. This approach employs the conjunction of three techniques, e.g., GA, mutation testing, and SBT.

A novel two-way crossover and adaptable mutation method for generating better offspring was presented in Ref. [44]. Their introduced genetic algorithm is compared with both normal and random GA methods and proved to identify the optimal test cases in a smaller number of attempts with higher mutation scores, a fact that leads to reduced cost of computations. In Ref. [24], a novel approach for mutation testing called Particle Swarm Optimization along with Mutation Testing (PSO-MT) to generate test data for killing active mutants, was introduced. More to this point, more extensive programs in terms of benchmarks from the Software-artifact Infrastructure Repository (SIR) for their experiments, have been considered. The experimental results have been compared with GA-based results in test data generation and it has been found that PSO-MT achieved better than GA-based mutation testing in terms of computational efficiency. Specifically, five different benchmark programs for the experiments, e.g., triangle, quadratic equation, TCAS, Schedule 1 and 2, have been employed. The reported results have shown that the proposed method can achieve a maximum value of mutation score equals to $79.32\%$.

Dubbed Sentinel [45,46,47,48] is a novel multi-objective EA-based approach for automatic generation of test data with optimal cost reduction strategies for every program. This approach was evaluated by conducting a thoroughly empirical study from 40 releases of 10 open-source real-world software systems with the execution of 4800 experiments with the use of quality indicators and statistical significance tests. The reported results have depicted that strategies computed by the Sentinel approach outperform the baseline strategies in $95\%$ of the cases when considering larger sizes. Furthermore, the proposed method can automatically generate mutation strategies with reduced mutation testing cost and simultaneously without affecting its testing effectiveness in terms of mutation score.

An improved fitness evaluation along with the incorporation of elitism as well as a performance comparison with the existing techniques is performed in Ref. [49]. A GA variant, by effectively blending the advantages of mutation testing for non-redundant test suite generation, was implemented. This variant is followed by a novel fitness function that considers test case complexity in terms of time steps with high fault exposure. GAMuT is another method for generating a test suite for the testing of Finite State Machines (FSMs) by using a Genetic Algorithm and mutation testing [50]. A standard mutant dataset tool, called MutantBench, was employed in Ref. [51] where an Abstract Syntax Tree (AST) was used in conjunction with a tree-based convolutional neural network (TBCNN) for the classification of mutants with the aim of reducing the cost of mutation testing in software testing of android applications.

3. Proposed Algorithm for Mutation Testing (RGA-MS)

The proposed algorithm in this paper is designed to identify and extract a representative test suite, which achieves maximum path coverage. In the following, this test suite is exercised for mutation testing, i.e., to find the maximum mutation score. The algorithm is entitled Real Coded Genetic Algorithm for highest Mutation Score (RGA-MS) and is outlined in Algorithm 1.

Initially, the RGA-MS generates the test data for maximum path coverage by using the fitness, which is designed by considering the total number of paths covered by a chromosome. The fitness for path coverage is defined in following Equation (2) as it is calculated by the ratio of paths covered by a chromosome given the total number of paths.

$$f\left(x\right)=\frac{\mathrm{Paths}\mathrm{Covered}\mathrm{By}\mathrm{A}\mathrm{Chromosome}}{\mathrm{Total}\mathrm{Number}\mathrm{Of}\mathrm{Paths}}\times 100$$

(2)

In the following, the two best populations are selected and the maximum path coverage is calculated. If the maximum path coverage is achieved, then the optimized test suite can be printed, otherwise the mutation must be applied and the new chromosomes should be replaced with their duplicates. The generated test data are exercised to find the mutation score.

Different mutation operators are inserted in the source code of different SUTs, which are listed in

Table 1. SUT’s details.

Program Name	Description	LOC	Paths
P#1: TCP	Check the type of the Triangle	28	25
P#2: Max three	Find Maximum of three numbers	25	4
P#3: Evaluate $X^Y$	Determine the value of $X^Y$	27	3
P#4: Evaluate $X/Y$	Determine the value of $X/Y$	38	5
P#5: GCD	Returns the greatest common divisor	18	5
P#6: QES	Find the roots of Quadratic Equation	42	6
P#7: Calculate Days	Find Days between two Dates	153	18
P#8: Binary Search Tree	An ordered or sorted binary tree	119	16
P#9: Calculator	Find the results of arithmetic operations	60	9
P#10: Accept Saving	Banking process to deposit in saving account	24	8

.

Algorithm 1: Real Coded Genetic Algorithm for highest Mutation Score (RGA-MS)

1: input Set of basic paths for a specific SUT (Table 1)   2:            $GA$ parameters   3: output An optimized test data with maximum Mutation Score (MS)   4: Step 1: Create $Initial$ $Population\left(\right)$   5: Step 2: Initialize the Population as Test Suites (where Test Suite = number of Chromosomes)   6: Step 3: Calculate Fitness using Equation (2)                         /* Fitness Value */   7: Step 4: Select two best Populations   8: Step 5: do   9:               Average Crossover for selected Chromosomes         /* Maximum Path Coverage */ 10: while ($Feasible$ $Solution$ $Reached$) 11: Step 6: if ($Maximum$ $Path$ $Coverage$ $Achieved$) 12:               Goto Step 8 13:         else 14:               $Apply$ $Mutation\left(\right)$ 15: Step 7: do 16:               Replace the new Chromosome with the duplicate one 17: while ($Critical$ $Path$ $is$ $not$ $found$) 18: Step 8: if ($Maximum$ $Path$ $Coverage$ $Achieved$) then 19:               Print the optimized Test Suite with $100\%$ Path Coverage 20:         else 21:               Goto Step 1 22: Step 9: $Trace$$FDM\left(\right)$ 23: Step 10: Remove redundant test data covering same mutants 24: Step 11:$Calculate$$Mutant$$Score$$MS\left(\right)$ using Equation (1) 25: Step 12: Print optimized test data with highest MS

The proposed algorithm aims to identify the minimum test data that covers all mutants by eliminating the redundant test data. The redundant test data is traced using a Fault Detection Matrix (FDM) [25]. A fault-detection matrix contains sufficient information for finding minimal-length, fault-diagnosis test sets. The necessary condition is that any sub-matrix of this matrix should not contain equal rows. The flow of the proposed algorithm is shown in Figure 1.

4. Results

4.1. Experimental Setup

The proposed algorithm is first implemented on a particular case study, entitled as the Greatest Common Divisor (GCD), which produces the greatest common divisor of any two input numbers. The source code for GCD with two integers as input, namely m and n, is illustrated in Listing 1. Furthermore, the proposed algorithm is implemented on different SUTs, as listed in Table 1. Moreover, the Control Flow Graph of the GCD Function is illustrated in Figure 2.

Source code of the GCD program int GCD(int m, int n) int r; if(n>m) r = m; m = n; n = r; r = m while(r!=0) m = n; n = r; r = m return n;

The various mutation operators [38,44] used for mutation analysis are listed in

Table 2. Different operators used for mutation analysis.

Type	Description	Symbols
AOR	Arithmetic Operator Replacement	+, −, *, /, %
LOR	Logical Operator Replacement	==, !=
ROR	Relational Operator Replacement	>, <, >=, <=
NOR	No Replacement Operator	Operator present as it is
UOI	Unary Operator Insertion	++, − −
CDL	Constant Deletion	Delete a constant (if present)
ABS	Absolute Value Insertion	1 is replaced with 0 and vice versa

. Three different columns are listed, namely the type, the description, as well as the corresponding symbols. For example, for type equals to NOR, the description is “No Replacement Operator” and these operators present as they are.

Regarding the Control Flow Graph of the GCD function, which is presented in Figure 2, we notice that one extra path is added in order to perform mutation analysis, as in the GCD program, the number of linearly independent paths is small, i.e., three. So, the different paths for GCD are listed below:

• PATH 1: 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14;
• PATH 2: 1, 3, 4, 8, 9, 10, 11, 12, 13, 14;
• PATH 3: 1, 3, 4, 8, 9, 13, 14;
• PATH 4: 1, 3, 4, 5, 6, 7, 8, 9, 13, 14.

Regarding mutation analysis, some faults are inserted in the aforementioned source code. The proposed method aims to achieve maximum mutation score through the path coverage-based test data. The following

Table 3. Operators used in GCD for mutation analysis.

Line Number	Operator Present	Replace with Operator	Mutant Name
4	>	!=	M1
6	=	/ =	M2
8	%	/	M3
9	!=	==	M4
12	%	/	M5

introduces the details of mutants inserted to the GCD program. Concretely, for specific lines of the program, some operators can be replaced by others and the corresponding mutation will be utilized.

The fitness value proposed before is also considered for computing the maximum path coverage, as defined in Equation (2). Finally, the optimized path coverage-based test suite is further exercised for mutation analysis. So, the mutation score of each test data is calculated using, as previously, Equation (1).

Furthermore, regarding the genetic algorithm operation,

Table 4. Parameters used for mutation analysis.

SI. Number	Parameters	Values
1	Population size	1000
2	Input range	1 to 100
3	Encoding type	Real Encoding
4	Crossover Type and Rate	Average Crossover (AX), 0.8
5	Mutation Type and Rate	Insertion, Gaussian 0.02, 0.07
6	Generation	5

presents the parameters used for mutation analysis. These parameters have specific values, e.g., the population size equals 1000, and Gaussian is considered as the mutation type.

4.2. Result Analysis

The various steps of the proposed algorithm and the results obtained in GA operations are depicted in

Table 5. Initial population.

Test Suite Number	Test Data	Target Path	Fitness
1	(12,4), (8,27), (45,8), (9,44)	3, 1, 3, 1	0.5
2	(14,9), (23,8), (33,45), (14,5)	2, 2, 2, 3	0.5
3	(49,9), (7,33), (28,5), (39,8)	2, 1, 2, 2	0.5
4	(7,12), (6,18), (16,4), (9,42)	1, 4, 3, 1	0.75
5	(32,6), (44,16), (20,7), (17,12)	2, 2, 2, 2	0.25

,

Table 6. Best population.

Test Suite Number	Test Data	Target Path	Fitness
1	(12,4), (8,27), (45,8), (9,44)	3, 1, 3, 1	0.5
4	(7,12), (6,18), (16,4), (9,42)	1, 4, 3, 1	0.75

,

Table 7. New trait after crossover.

New Trait	Test Data	Target Path	Fitness
1	(9,8), (7,22), (30,6), (19,43)	2, 1, 3, 1	0.75

and

Table 8. Optimized test suite after mutation.

Test Suite	Test Data	Target Path	Fitness
(T1, T2, T3, T4)	(21,7), (14,9), (7,22), (6,18)	1, 2, 3, 4	100%

. Specifically, Table 5 presents the initial population for five particular test suites, where the corresponding test data, the target path, and the fitness value are also introduced. The fitness value ranges from $0.25$ to $0.75$.

In addition, Table 6 introduces the two best populations, which are test suites with number 1 and 4 having fitness values equal to $0.5$ and $0.75$, respectively.

After the crossover process, a new trait has been generated, with a fitness value equal to $0.75$, whereas the test data and the target path have different values than the 5 initial test data as presented in Table 7.

The optimized test suite that covers maximum path, after the mutation, is presented in Table 8. As before, the test data and the target path have different values than the 5 initial test data whereas the fitness value achieves percentage equal to $100\%$.

The FDM is traced out from the optimized test suite, which indicates the mutant coverage by individual test cases as shown in

Table 9. FDM for GCD.

Test Suite	T1	T2	T3	T4
Mutant
M1	1	0	1	0
M2	0	0	1	1
M3	0	1	1	0
M4	1	1	1	1
M5	0	1	1	0

. This table presents all the potential combinations between the five mutants and the four test suites.

Regarding the detection matrix, a value equal to 1 represents the case where a mutant is covered by the corresponding test case (and otherwise, for a value equal to 0). Next, the mutation score is calculated using Equation (1), and the mutation score for every test case is presented in

Table 10. Mutation score of test cases.

Test Case ID	Mutation Score (%)
T1	40
T2	60
T3	100
T4	40

. Finally, the redundant test cases, which cover the same mutants, are eliminated in order to find the optimum test data. We observe that the third test case achieves a mutation score equal to $100\%$, while the other three test cases have values equal to $40\%$ and $60\%$.

The proposed algorithm for mutation analysis has been experimented on different SUTs and executed in Dev C++ IDE-5.11 several times to obtain more accurate results. As a result, the complexity of the proposed GA-based approach for mutation analysis is equal to $O(gen\times pop)$, where $gen$ is the maximum number of generations and $pop$ is the number of populations.

The GA process has been executed several times with the same population scale, and the optimized test data are fetched. As aforementioned, Table 10 showed that T3 is the only test case that achieves the highest mutation score or full mutant coverage. The following Figure 3 illustrates the mutation score of each test case for the SUT GCD.

The same process has been applied to other benchmark programs listed in

Table 11. Highest mutation score of test cases for different SUTs.

Program Name	Test Case ID	Mutation Score (%)
P#1: TCP	T4, T5	85.71
P#2: Max three	T3	100
P#3: Evaluate $X^Y$	T2	100
P#4: Evaluate $X/Y$	T2, T5	100
P#5: GCD	T3	100
P#6: QES	T3, T4	100
P#7: Calculate Days	T4, T5, T11, T15, T18	71.42
P#8: Binary Search Tree	T3, T5, T8, T10, T11, T15, T16	85.35
P#9: Calculator	T3, T6, T8	95
P#10: Accept Saving	T4, T7, T8	100

. This table records the test case id for each program name and the corresponding mutation score. From recorded mutation scores, it is observed that duplicate test cases have the same mutation score for all the System Under Tests (SUTs). So, to remove the duplicates, an elitism operation has to be implemented in the GA method.

After identifying the test cases with the highest mutation score, it can be clearly seen that duplicate test cases achieve the same mutation score. So, through the elitism process, the optimized test case can be found. The optimized test cases with the highest mutation score are presented in the following

Table 12. Optimized test cases with the highest mutation score for different SUTs.

Program Name	Test Case ID	Mutation Score (%)
P#1: TCP	T4	85.71
P#2: Max three	T3	100
P#3: Evaluate $X^Y$	T2	100
P#4: Evaluate $X/Y$	T2	100
P#5: GCD	T3	100
P#6: QES	T3	100
P#7: Calculate Days	T4	71.42
P#8: Binary Search Tree	T3	85.35
P#9: Calculator	T3	95
P#10: Accept Saving	T4	100
	Average MS	93.748

.

The following Figure 4 depicts the optimized test cases with the highest mutation score for different SUTs.

4.3. Comparison with Related Work

Mutation analysis has been identified as a powerful testing method for revealing faults in SUTs. In literature, as aforementioned, several similar researches exist for the mutation analysis via path coverage testing strategies. Specifically, the authors in Refs. [24,35,36] have developed different techniques to cover maximum mutation coverage using the path aware approach. Specifically, authors have utilized different mutation operators to kill the active mutants present in the SUT. More to the point, the proposed approach also takes the path coverage test data as input for achieving the maximum mutation score.

However, we can only compare our work with the research proposed in Ref. [35] on the same domain in terms of mutation operators used to add mutants.

Table 13. Comparison with related work.

Algorithm	Programs	Operators Taken	Mutation Score (%)
HGA [35]	Academic Industrial based Programs	AOR, LOR, SOR, LOD, ASR	70
RGA-MS	Academic Programs	AOR, LOR, ROR, NOR, UOI, CDL, ABS	93.74

presents a comparative study of this previous related work with the proposed approach. Authors in Ref. [35] have taken different academic and industrial-based programs with various operators, and their proposed HGA method achieves a mutation score equal to $70\%$. On the other hand, our proposed GA-based approach achieves a mutation score equal to $93.74\%$, whereas the proposed RGA-MS method is implemented in several academic programs.

4.4. Threats to Validity

The evaluation of mutation scores in terms of test cases is implemented with the use of the path coverage-based test data. Therefore, it does not necessarily mean that the results can be extrapolated to the other coverage-based test data.

The proposed method is tested on some benchmark programs written in C/C++ language, and the target paths are found from CFG, which is independent of any programming language. The programs used for the experiments are small-scale as the flow graphs are hand coded. So, the approach requires tools that support the task of automatic flow graphs and path generation.

5. Conclusions and Future Work

The automatic test data generation regarding white box testing is considered vital in accomplishing both qualitative and reliable software. Although mutation testing is computationally costly, thus it can improve the test data quality in terms of higher mutation scores. In addition, it can enhance the reliability of the software by killing the active mutants present in the source code of a SUT.

In this paper, a Search-Based Software Engineering algorithm, which can effectively identify the maximum mutation score by considering all the corresponding test cases generated through path coverage-based testing, is proposed. Initially, path coverage-based test cases are produced. Using different mutation operators, the proposed algorithm aims to find the mutation score for every test case. Then, a real-coded Genetic Algorithm is utilized, automatically generating test data to cover multiple paths simultaneously and further exercising the test data to achieve maximum mutation score. This proposed Real coded Genetic Algorithm for maximizing the Mutation Score is called RGA-MS.

The proposed RGA-MS-based method can generate test data for both path testing and mutation testing. Real coded GA is applied to generate an optimized test suite that achieves maximum path coverage, and further, the suite is exercised to kill the mutants present in the SUT. The proposed method can achieve optimized test data with the highest mutation score.

Regarding future work, we think more tools could be developed towards having a standardized mutant dataset, as this will increase the research interest in other areas of mutation testing. Furthermore, more mutation operators like SDL, VDL, ODL, EHF, OBAF, NPDF, etc., should be used to implement the proposed technique further, and the framework could be further developed into an android application to ease access and increase the use of the model. Lastly, we could implement an additional number of experiments to ensure that our proposed method can be utilized in different kinds of programs.