Mutated Specification-Based Test Data Generation with a Genetic Algorithm †
Abstract
:1. Introduction
2. Related Work
3. GA with Mutated Specification
3.1. Description for GA
3.2. Mutation Testing
3.3. Mutated Specifications
- Reforming the original specification by introducing dummy variables into defining conditions so that test data that satisfy those reformed ones can trigger bad behaviors of the program;
- Finding appropriate concrete values through GA and assigning them to the output and dummy variables that occur in the reformed specification.
3.4. Chromosome Formation
3.5. Genetic Manipulations and Selection
4. Algorithm Summary
- Inject faults into the original program to obtain a set of program mutants;
- Use reformed specification as seed chromosomes. Each chromosome corresponds to a group of individuals that are generated by assigning concrete values to the output vector in the chromosome;
- Apply GA to each chromosome and select the best individuals (the best mutated specifications). According to the original specifications, determine whether or not a test case from a mutated specification (a constraint over inputs) kills the program mutants.
5. Case Study
5.1. Case Study 1: Mod
The Effect with Dummy Variables
- Version V1: Introducing dummy variables into only inequality relation;
- Version V2: Introducing dummy variables into both equality and inequality relation;
- Version V3: Putting no dummy variables in defining conditions.
5.2. Case Study 2: Gcd
The Effect without Dummy Variables
5.3. Complexity of Our Approach
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
Appendix A
Algorithm A1 GA to obtain mutated specifications. |
Inputs: the functional scenarios from the specification: Individuals: with concrete values Outputs: the reformed specification run(){ result = list() for in functional scenarios: spec = population = initial() while(not enough iterations){ one_step(spec) } best_individual = select_best_individual(population) reformed_specification = (spec, best_individual) result.append(reformed_specification) } one_step(spec) { # This function selects top 50% of the current population population = keep_good_individuals(population) do: father, mother = random_select_two(population) child1, child2 = crossover_operation(father,mother) child1, child2 = mutation_operation(child1,child2) population.put(child1,child2) until population increases enough do_valuation(population,spec) } do_valuation(population,spec){ for individual in population: datas = data_suite_from(individual,spec) statistic_sum = kill_program_mutants(datas) individual.value = Grade(statistic_sum) } |
References
- Wong, W.E.; Horgan, J.R.; London, S.; Agrawal, H. A study of effective regression testing in practice. In Proceedings of the Eighth International Symposium on Software Reliability Engineering, Albuquerque, NM, USA, 2–5 November 1997; pp. 264–274. [Google Scholar]
- Leung, H.K.; White, L. Insights into regression testing (software testing). In Proceedings of the Conference on Software Maintenance, Miami, FL, USA, 16–19 October 1989; pp. 60–69. [Google Scholar]
- Kazmi, R.; Jawawi, D.N.; Mohamad, R.; Ghani, I. Effective regression test case selection: A systematic literature review. ACM Comput. Surv. (CSUR) 2017, 50, 1–32. [Google Scholar] [CrossRef]
- Stocks, P.; Carrington, D. A framework for specification-based testing. IEEE Trans. Softw. Eng. 1996, 777–793. [Google Scholar] [CrossRef] [Green Version]
- Richardson, D.; O’Malley, O.; Tittle, C. Approaches to Specification-Based Testing; ACM: New York, NY, USA, 1989; Volume 14. [Google Scholar]
- Khurshid, S.; Marinov, D. TestEra: Specification-based testing of Java programs using SAT. Autom. Softw. Eng. 2004, 11, 403–434. [Google Scholar] [CrossRef]
- Hierons, R.M.; Bogdanov, K.; Bowen, J.P.; Cleaveland, R.; Derrick, J.; Dick, J.; Gheorghe, M.; Harman, M.; Kapoor, K.; Krause, P.; et al. Using formal specifications to support testing. ACM Comput. Surv. (CSUR) 2009, 41, 1–76. [Google Scholar] [CrossRef]
- Dokhanchi, A.; Hoxha, B.; Fainekos, G. Formal requirement debugging for testing and verification of cyber-physical systems. ACM Trans. Embed. Comput. Syst. (TECS) 2017, 17, 1–26. [Google Scholar] [CrossRef]
- Offutt, A.J.; Liu, S. Generating Test Data from SOFL Specifications. J. Syst. Softw. 1999, 49, 49–62. [Google Scholar] [CrossRef]
- Dick, J.; Faivre, A. Automating the generation and sequencing of test cases from model-based specifications. In Proceedings of the International Symposium of Formal Methods Europe, Odense, Denmark, 19–23 April 1993; Springer: Berlin/Heidelberg, Germany, 1993; pp. 268–284. [Google Scholar]
- Ed-Douibi, H.; Izquierdo, J.L.C.; Cabot, J. Automatic generation of test cases for REST APIs: A specification-based approach. In Proceedings of the 2018 IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC), IEEE, Stockholm, Sweden, 16–19 October 2018; pp. 181–190. [Google Scholar]
- Alrawashed, T.A.; Almomani, A.; Althunibat, A.; Tamimi, A. An Automated Approach to Generate Test Cases From Use Case Description Model. Comput. Model. Eng. Sci. 2019, 119, 409–425. [Google Scholar] [CrossRef] [Green Version]
- Jones, C.B. Systematic Software Development Using VDM; Citeseer: Princeton, NJ, USA, 1990; Volume 2. [Google Scholar]
- Larsen, P.G.; Battle, N.; Ferreira, M.; Fitzgerald, J.; Lausdahl, K.; Verhoef, M. The overture initiative integrating tools for VDM. ACM SIGSOFT Softw. Eng. Notes 2010, 35, 1–6. [Google Scholar] [CrossRef]
- Tran-Jørgensen, P.W.; Nilsson, R.S.; Lausdahl, K. Enhancing Testing of VDM-SL models. In Proceedings of the 16th Overture Workshop, Oxford, UK, 14 July 2018; pp. 7–22. [Google Scholar]
- Liu, S. Formal Engineering for Industrial Software Development: Using the SOFL Method; Springer Science & Business Media: Berlin, Germany, 2013. [Google Scholar]
- Liu, S.; Nakajima, S. Combining Specification Testing, Correctness Proof, and Inspection for Program Verification in Practice. In Proceedings of the 3rd International Workshop on SOFL + MSVL (SOFL+MSVL 2013), LNCS 8332, Queenstown, New Zealand, 29 October 2013; Springer: Queenstown, New Zealand, 2013; pp. 3–16. [Google Scholar]
- Wang, R.; Sato, Y.; Liu, S. Specification-based Test Case Generation with Genetic Algorithm. In Proceedings of the 2019 IEEE Congress on Evolutionary Computation (CEC), Wellington, New Zealand, 10–13 June 2019; pp. 1382–1389. [Google Scholar]
- Rapps, S.; Weyuker, E.J. Selecting software test data using data flow information. IEEE Trans. Softw. Eng. 1985, 367–375. [Google Scholar] [CrossRef]
- Weyuker, E.J. More experience with data flow testing. IEEE Trans. Softw. Eng. 1993, 19, 912–919. [Google Scholar] [CrossRef]
- Khedker, U.; Sanyal, A.; Sathe, B. Data Flow Analysis: Theory and Practice; CRC Press: Boca Raton, FL, USA, 2017. [Google Scholar]
- Pargas, R.P.; Harrold, M.J.; Peck, R.R. Test-data generation using genetic algorithms. Softw. Test. Verif. Reliab. 1999, 9, 263–282. [Google Scholar] [CrossRef]
- Girgis, M.R. Automatic Test Data Generation for Data Flow Testing Using a Genetic Algorithm. J. UCS 2005, 11, 898–915. [Google Scholar]
- Girgis, M.R.; Ghiduk, A.S.; Abd-Elkawy, E.H. Automatic generation of data flow test paths using a genetic algorithm. Int. J. Comput. Appl. 2014, 89, 29–36. [Google Scholar]
- Nayak, N.; Mohapatra, D.P. Automatic test data generation for data flow testing using particle swarm optimization. In Proceedings of the International Conference on Contemporary Computing, Noida, India, 9–11 August 2010; Springer: Cham, Switzerland, 2010; pp. 1–12. [Google Scholar]
- Biswas, S.; Kaiser, M.S.; Mamun, S. Applying ant colony optimization in software testing to generate prioritized optimal path and test data. In Proceedings of the 2015 International Conference on Electrical Engineering and Information Communication Technology (ICEEICT), IEEE, Dhaka, Bangladesh, 21–23 May 2015; pp. 1–6. [Google Scholar]
- Harman, M.; Jia, Y.; Langdon, W.B. Strong higher order mutation-based test data generation. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ACM, Szeged, Hungary, 5–9 September 2011; pp. 212–222. [Google Scholar]
- Papadakis, M.; Kintis, M.; Zhang, J.; Jia, Y.; Le Traon, Y.; Harman, M. Mutation testing advances: An analysis and survey. In Advances in Computers; Elsevier: Amsterdam, The Netherlands, 2019; Volume 112, pp. 275–378. [Google Scholar]
- Ma, Y.S.; Offutt, J.; Kwon, Y.R. MuJava: An automated class mutation system. Softw. Test. Verif. Reliab. 2005, 15, 97–133. [Google Scholar] [CrossRef]
- Derezinska, A.; Kowalski, K. Object-oriented mutation applied in common intermediate language programs originated from c. In Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, Berlin, Germany, 21–25 March 2011; pp. 342–350. [Google Scholar]
- Delgado-Pérez, P.; Medina-Bulo, I.; Palomo-Lozano, F.; García-Domínguez, A.; Domínguez-Jiménez, J.J. Assessment of class mutation operators for C++ with the MuCPP mutation system. Inf. Softw. Technol. 2017, 81, 169–184. [Google Scholar] [CrossRef] [Green Version]
- Jackson, D. Alloy: A lightweight object modelling notation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 2002, 11, 256–290. [Google Scholar] [CrossRef]
- Sullivan, A.; Wang, K.; Zaeem, R.N.; Khurshid, S. Automated test generation and mutation testing for Alloy. In Proceedings of the 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), Tokyo, Japan, 13–17 March 2017; pp. 264–275. [Google Scholar]
- Martins, E.; Sabião, S.B.; Ambrosio, A.M. ConData: A tool for automating specification-based test case generation for communication systems. Softw. Qual. J. 1999, 8, 303–320. [Google Scholar] [CrossRef]
- McMillan, K.L.; Zuck, L.D. Formal specification and testing of QUIC. In Proceedings of the ACM Special Interest Group on Data Communication; ACM: New York, NY, USA, 2019; pp. 227–240. [Google Scholar]
- Ali, S.; Iqbal, M.Z.; Arcuri, A.; Briand, L.C. Generating test data from OCL constraints with search techniques. IEEE Trans. Softw. Eng. 2013, 39, 1376–1402. [Google Scholar] [CrossRef]
- Jalila, A.; Mala, D.J. Automated optimal test data generation for OCL specification using harmony search algorithm. Int. J. Bus. Intell. Data Min. 2020, 16, 231–259. [Google Scholar] [CrossRef]
- Holland, J.H. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence; MIT Press: Cambridge, MA, USA, 1992. [Google Scholar]
- DeMillo, R.A.; Lipton, R.J.; Sayward, F.G. Hints on test data selection: Help for the practicing programmer. Computer 1978, 11, 34–41. [Google Scholar] [CrossRef]
- Luo, J.; Liu, S.; Wang, Y.; Zhou, T. Applying SOFL to a railway interlocking system in industry. In Proceedings of the International Workshop on Structured Object-Oriented Formal Language and Method, Tokyo, Japan, 15 November 2016; Springer: Cham, Switzerland, 2016; pp. 160–177. [Google Scholar]
- Liu, S. Formal Engineering for Industrial Software Development Using the SOFL Method; Springer: Berlin, Germany, 2004; ISBN 3-540-20602-7. [Google Scholar]
- Sen, K. Concolic testing. In Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering, ACM, Atlanta, GA, USA, 5–9 November 2007; pp. 571–572. [Google Scholar]
- Sato, Y.; Sugihara, T. Automatic Generation of Specification-Based Test Cases by Applying Genetic Algorithms in Reinforcement Learning. In Proceedings of the International Workshop on Structured Object-Oriented Formal Language and Method, Paris, France, 6 November 2015; Springer: Cham, Switzerland, 2015; pp. 59–71. [Google Scholar]
- De Moura, L.; Bjørner, N. Z3: An efficient SMT solver. In Proceedings of the International conference on Tools and Algorithms for the Construction and Analysis of Systems, Budapest, Hungary, 29 March–6 April 2008; Springer: Cham, Switzerland, 2008; pp. 337–340. [Google Scholar]
- Mackenzie, C.E. Coded-Character Sets: History and Development; Addison-Wesley Longman Publishing Co., Inc.: Boston, MA, USA, 1980. [Google Scholar]
- Basavaprasad, B.; Ravi, M. A study on the importance of image processing and its applications. IJRET Int. J. Res. Eng. Technol. 2014, 3, 1. [Google Scholar]
- Barannik, V.; Podlesny, S.; Tarasenko, D.; Barannik, D.; Kulitsa, O. The video stream encoding method in infocommunication systems. In Proceedings of the 14th International Conference on Advanced Trends in Radioelecrtronics, Telecommunications and Computer Engineering (TCSET), IEEE, Lviv-Slavske, Ukraine, 20–24 February 2018; pp. 538–541. [Google Scholar]
- Hur, T.; Bang, J.; Huynh-The, T.; Lee, J.; Kim, J.I.; Lee, S. Iss2Image: A novel signal-encoding technique for CNN-based human activity recognition. Sensors 2018, 18, 3910. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Chromosome | D-Chromo | Dummy Vars |
---|---|---|
The Best Individual of Chromosome | Grade |
---|---|
Original Specification | Grade |
---|---|
Chromosome | D-Chromo | Dummy Vars |
---|---|---|
∧ | ||
∧ | ||
∧ | ||
∧ | ||
The Best Individual of Chromosome | Grade | |
---|---|---|
Original Specification | Grade |
---|---|
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, R.; Sato, Y.; Liu, S. Mutated Specification-Based Test Data Generation with a Genetic Algorithm. Mathematics 2021, 9, 331. https://doi.org/10.3390/math9040331
Wang R, Sato Y, Liu S. Mutated Specification-Based Test Data Generation with a Genetic Algorithm. Mathematics. 2021; 9(4):331. https://doi.org/10.3390/math9040331
Chicago/Turabian StyleWang, Rong, Yuji Sato, and Shaoying Liu. 2021. "Mutated Specification-Based Test Data Generation with a Genetic Algorithm" Mathematics 9, no. 4: 331. https://doi.org/10.3390/math9040331
APA StyleWang, R., Sato, Y., & Liu, S. (2021). Mutated Specification-Based Test Data Generation with a Genetic Algorithm. Mathematics, 9(4), 331. https://doi.org/10.3390/math9040331