BlackBoxTestGen: An Automatic Black-Box Test Case Generation Framework

Abstract

Software testing is essential for software engineering practices, as it ensures that the final software product is reliable and satisfies all requirements before delivery. However, manually designing black-box testing test cases is time-consuming, inconsistent, and difficult to maintain in accordance with changing specifications. Therefore, this paper presents BlackBoxTestGen, an automatic framework that unifies three specification-driven black-box testing techniques, including rule-based Equivalence Class Partitioning (ECP), syntax, and state transition testing. The framework utilises a redesigned XML structure for test case generation to be shared among a data dictionary, decision tree, and state machine, used by each testing technique. The degree of testing coverage is accumulatively calculated during the test case generation process. The beneficial value of our proposed framework was demonstrated with the development of a web-based prototype tool. We rigorously evaluated its performance in terms of accuracy, computational efficiency, and scalability through a multidimensional approach. This included assessment by professional experts, algorithmic stress testing via parameter scaling, and application to close-to-realistic case studies. The results indicate that BlackBoxTestGen provides a robust integration of testing techniques. By automating the generation of compact and reproducible test cases, the framework substantially reduces manual effort and minimises drift between techniques.

1. Introduction

Software testing is a crucial stage in the software development life cycle [1]. It serves as a quality assurance mechanism by validating that the system aligns with a predefined requirement, identifying early-stage errors, and ensuring the accuracy of functionalities [2]. Effective testing strategies are essential to reduce defects discovered after deployment, which is critical given that the cost of fixing bugs increases dramatically the longer they remain hidden [3,4]. According to a 2022 report by the Consortium for Information & Software Quality (CISQ), poor software quality cost the U.S. economy approximately $2.41 trillion, largely driven by accumulated technical debt. Furthermore, empirical data confirm the ‘Rule of 100’, where defects found in production cost 100 times more to fix than those identified during the design phase [4]. For these reasons, a good software testing method is needed by software testing communities.

Black-box testing is a fundamental methodology that validates the correctness of the system against the Software Requirement Specification (SRS) without requiring knowledge of the internal code structure [5,6]. By employing standardised black-box testing techniques such as Equivalence Class Partitioning (ECP), decision table/tree, syntax testing, and state transition testing, testers can systematically maximise test coverage while minimising the number of redundant test cases. The effectiveness of this approach is evidenced by its successful application in various domains of practical software development including Point of Sale (POS) [7], messenger application [8], Internet of Things (IoT) [9], the e-Learning system [10] and the Enterprise Resource Planning (ERP) system [11]. However, modern software evolves quickly and often runs in various environments, making manual test generation slow, inconsistent, and prone to human error [12,13,14]. Furthermore, manual workflows may struggle to keep up with frequent changes, and the effort required to maintain suites grows disproportionately with system size [13]. To address these challenges, automated testing has become indispensable [15]. By integrating automation into deployment pipelines, testers can achieve repeatable execution, shorter feedback cycles, and improved coverage, ultimately allowing them to shift their focus from repetitive tasks to exploratory analysis [6,13,16].

Several frameworks and tools for automatic test case generation have recently been applied to the software testing community [6,13,16]. However, one primary challenge in modern software testing practices is the limited availability of free testing frameworks and tools that automate test case generation with a specific and appropriate technique. Most existing open-source tools such as Selenium [17], OpenScript [18], and xUnit.net [19] focus only on driving and exploring the automated execution of the system under test [15]. Consequently, testers must still manually design and inject test cases into these testing tools. Furthermore, achieving adequate functional coverage remains a critical aspect of software testing [1,20]. This requirement is still not fully addressed by current automated tools.

To fulfill these limitations, in our previous work [21,22,23], we introduced a series of open-source frameworks for automated test case generation, implementing various black-box testing techniques. This includes TestGen [21], SYNTest [22], and STATETest [23]. TestGen utilises ECP and decision tree techniques, called rule-based ECP, to categorise inputs based on decision rules. By processing XML-encoded requirements, including Use Case (UC) descriptions, business processes, and data dictionaries, the tool automates test case generation. This approach ensures full scenario coverage while significantly reducing testing time and costs. SYNTest employs syntax testing and regular expressions to parse XML-based UC descriptions and data dictionaries, automatically generating valid and invalid test cases from formally-defined input formats. To enhance the capability of the previous framework, STATETest was developed to automate test case generation specifically for state transition testing. The prototype parses XML files to identify system states and transitions, transforming them into a test case matrix and state tree. This mechanism automates the generation of single and sequence transition test cases, ensuring the scenario coverage. However, each framework relies on distinct principles and unique XML structures, creating significant challenges. This duplication results in redundancy and increases the chance of inconsistencies between the range value, the syntax rules, and the state logic. Furthermore, three separate prototype tools cause an increase in maintenance overhead when synchronising XML to reflect changing requirements.

Therefore, to strengthen comprehensive test coverage and systematic test case generation that addresses all critical aspects of the system behaviour, this article introduces BlackBoxTestGen, an automatic black-box test case generation framework that combines the strength of each black-box testing technique. The main contributions influenced by this framework are as follows.

• The proposed framework facilitates architectural unification via a shared XML basis. It integrates the existing automatic framework of TestGen, SYNTest, and STATETest into a single pipeline driven by a redesigned, centralised XML intermediate representation. As this shared XML simultaneously feeds a data dictionary, a decision tree, and a state machine, it seamlessly merges rule-based ECP, Regular Expression (RegEx) syntax, and state transition testing into a single-driven pipeline. This creates the benefit of the drift of the testing technique and manual stitching errors being eliminated across functional, format, and behavioral spaces. To ensure testing coverage across three testing techniques, the framework also enables the calculation of accumulative coverage.
• We developed a web-based prototype to demonstrate the effectiveness and usability of the proposed framework. The accuracy of the prototype was evaluated by professional software testing experts. A comparison was conducted between the test cases generated by the prototype and the expert benchmark results. These rigorous evaluations confirm the practical viability, significantly minimising manual engineering effort to deliver compact and reproducible test suites.
• We demonstrate the algorithmic optimisation of BlackBoxTestGen through two parts of evaluation experiments. In the first part, the parameters are scaled up to isolate the algorithmic complexity and structural overhead of each technique. These isolated stress tests prove that the framework performs computational resilience to handle extreme theoretical loads. Then, the integrated efficiency is evaluated across three diverse systems (HRMS, OSPS, and SSMS). This experiment confirms that our proposed integrated framework handles close-to-realistic scaling of variables and transitions with robust sub-linear resource growth and maintains stable peak memory usage.

The rest of the paper is organised as follows. Section 2 details the proposed BlackBoxTestGen framework and its underlying testing algorithms. Then, Section 3 presents the proof of concept to demonstrate the effectiveness and usability of the proposed framework through the developed prototype tool. In this section, the performance evaluation of the prototype in terms of accuracy, computational efficiency, and scalability is also explained. Section 4 discusses the study findings, limitations, and practical lessons learned, followed by a summary of conclusions and directions for future research in Section 5.

2. Materials and Approach

2.1. BlackBoxTestGen Framework

The BlackBoxTestGen framework was designed to support automatic test case generation, ensuring that tests covering all aspects of the system behaviour and input are created in the early stages of software development, as demonstrated in Figure 1. Since one of the fundamental challenges in automating test case generation from Use Case (UC) specification is maintaining traceability from high-level behavioural requirements to low-level data constraints, the proposed framework introduces a redesigned, unified XML structure that structurally binds the specification information, gathered during the requirement engineering stage, into a unified format that can trace back to UC. This includes the XML files of the UC data dictionary, the decision tree, and the state machine diagram. The test case generation engine consists of three main steps. This begins with (1) reading an XML file to check its syntax to ensure that it conforms to the definition. Then, (2) a pre-processing step is performed in which the variables with their corresponding information in each XML file are extracted and analysed. This results in the extracted information being transformed into the necessary test basis including data type and ranges, business rules/logic, and the format of input and state and transition. These are passed to generate test cases (3) by using different black-box testing techniques, rule-based ECP, syntax and state transition testing. The generated test cases cover all possible valid and invalid conditions, with coverage summaries, rendered for review, and available for export.

2.2. Pre-Processing Step

Figure 2 shows a workflow of the pre-processing step. After the variables with their corresponding information in each XML file are extracted and analysed, the process determines which black-box techniques are applicable based on the extracted data. It branches into three distinct checks. Firstly (2.1) if the range data exists, the framework proceeds to generate partitions. This step prepares the test basis for ECP testing techniques, dividing the input data into valid and invalid classes. Then, a further check is performed for the decision tree. If a decision tree exists, the business rules consisting of conditions and corresponding actions for decision are extracted. The consistency between the variable information extracted from the UC data dictionary and the business rules extracted from the decision tree is then verified, before identifying the relationship between conditions and actions. This is performed to ensure that the business logic is sound and aligned with the data definitions. Secondly, (2.2) if syntax is present, regular expressions are used to analyse the value of each attribute, creating the necessary syntax of input variables for test case generation by using syntax testing techniques. Lastly, (2.3) for the case that state information exists, the framework converts the extracted state and transitions from the XML file into a test case matrix and a state tree as test basis for state transition testing. The test case matrix is specifically utilised to create single transition test cases (covering valid and invalid transitions between adjacent states). Meanwhile, the state tree, which maps out the complete sequence of all possible transitions, is used to generate a sequence of transition test cases. If none of the three conditions (range, syntax, or state) are met, the framework cannot generate a test basis and will report the no-input-file error.

2.3. Test Case Generation Step

This section explains the detail of black-box testing algorithm implemented in our proposed framework.

2.3.1. Rule-Based ECP

The testing technique provides the extension of standard ECP which divides input data into partitions or equivalence classes specifically according to specific business rules, logic, or constraints rather than just a simple value range. A single representative value is selected to create a test case for each to reduce the total number of test cases.

Table 1. Examples of generated equivalence class partitions for the HRMS case study.

Variable	Partition	Input Data Range	Type	Test Data
NumStay	1	1–2	Valid	1
	2	3–7	Valid	4
	3	8–15	Valid	10
	4	16–30	Valid	21
	5	<1	Invalid	−1
	6	>30	Invalid	45
GuestType	1	Regular	Valid	Regular
	2	Silver	Valid	Silver
	3	Platinum	Valid	Platinum
	4	N/A	Invalid	N/A

Note: N/A = Not Available.

shows an example of the partitions of the NumStay and GuestType variables defined in the Hotel Room Management System (HRMS). The variable NumStay is divided into four valid partitions based on the discount promotion range and two invalid intervals (under 1 and over 30). The variable GuestType is separated into three valid inputs (Regular, Silver, and Platinum) and invalid value (N/A).

To generate test cases, our framework employs a strong robustness technique that enables two distinct generation methods. For valid test cases, a Catesian product is applied to all valid partitions to ensure the full coverage. In contrast, it generates invalid test cases using the single-fault method, in which a test case contains exactly one invalid value while all other input values remain valid. As illustrated in Table 1, this process generates twelve valid test cases (4 × 3) and three invalid test cases (single-fault method of three invalid partitions) from these two variables. When business logic is defined by the decision tree, the framework filters this generation result according to the specific condition rule mappings.

2.3.2. Syntax Testing

This technique generates the test case from input data that needs to verify the expected format defined in the system specification. The framework implements Regular Expression (RegEx), a sequence of characters that forms a search pattern. It is usually used in pattern matching, validation, or replacement. For example, [1–9][0–9]{1,2} represents a number between 10 to 999, where [1–9] ensures that the first digit is between 1 and 9 (to avoid leading zeros), and [0–9]{1,2} specifies that the number must contain 1 to 2 additional digits.

Table 2. Syntax and RegEx Mapping [22].

Type	Syntax	RegEx Mapping
1. Range of values	value1 .. value2	[value1–value2]
2. List of values	value1 |value2 |…	(value1 |value2 |…)
3. Iteration	min{item}max	item{min,max}
4. N times	n{item}n	item{n}
5. Optional value	[value]	value?
6. Sequence	item + item	(?:item)+
7. Literal values	“A”,“1”,“word”	(A),(1),(word)

shows the mapping of different syntax patterns with their corresponding RegEx, demonstrating how structured inputs are validated using syntax testing techniques.

Table 3. Examples of generated test cases for the email syntax case study.

TC#	Email Input	Type	Classification/Partition
1	test@gmail.com	Valid	Standard-Format
2	a1_b2@sub.domain.edu	Valid	Long-Domain-Multi-Dot
3	test@gmail.company	Invalid	TLD-Length-Exceeded
4	@com.gmail.test	Invalid	Syntax-Sequence-Error
5	test@gmail	Invalid	Omission-TLD
6	test@gmail.com#123	Invalid	Illegal-Char-Addition
7	a+b.c%d@x.tv	Valid	Special-Symbols-Local
8	123456789@abc.com	Valid	Numeric-Local-Part

shows an example of the test cases generated from the email syntax variable. This follows the approach that generates both valid and invalid test cases. Valid test cases are created strictly according to the format structure in which an email must follow the sequence of a local part (1–64 characters), the @ symbol, a domain name (1–63 characters), a literal dot, and a top-level domain (tld) (2–6 letters). All segments must follow the restricted character set of alphanumeric values and specific symbols, e.g.,  “_” or “%”. Invalid test cases are generated based on four distinct formats that violate the rules. This includes (1) introducing an invalid value, e.g., tld exceeding six letters, (2) substituting the sequence of syntax, e.g., starting with the “@” symbol, (3) omitting an element from the syntax, e.g., excluding dot and tld, and (4) adding an extraneous element, e.g., the “#” symbol that is not defined in an allowed set of symbol.

2.3.3. State Transition Testing

This technique generates test cases from the state machine diagram, where the input variable consists of states and transitions that respond to a change of states. It involves defining system states, identifying valid and invalid transitions, and constructing test cases to validate expected state changes. Two types of transition are considered to generate test cases: single-transition and sequence-of-transition. To facilitate the generation of test cases, the test case matrix demonstrating individual movements between states is created as shown in Figure 3. This includes a valid and an invalid transition. For example, TC002 represents the valid single-transition, indicating Vacant → Available after the room is cleaned. In contrast, TC010 is an example of invalid single-transition test cases, demonstrating Vacant → Occupied, allocating the dirty room to the guest. In state transition testing, invalid test cases exclude transitions from the state returning to the initial state, transitions reverting from the final state to the state, and self-loop transitions on either the initial or final states. These are indicated as the black shade in the test case matrix.

For generating sequence-of-transition test cases, the state tree, which maps all possible paths of transition, is transformed from the state machine diagram. An example of a generated tree is shown in Figure 4a, which is derived into three sequence-of-transition test cases (Figure 4b), such as $\mathtt{Initial}\to \mathtt{Vacant}\to \mathtt{Available}\to \mathtt{Occupied}\to \mathtt{Vacant}$.

2.4. XML Structure

To bridge the gap between high-level requirements and low-level test execution, standard XML files are used to encode the necessary input for the prototype tool developed. We utilised and customised the XML structure proposed in our previous work [21,22,23] to create a common structure that can be shared among the different black-box testing algorithms. This includes the data dictionary, decision tree and state transition metadata. All XML files used in case studies for prototype development and evaluation are available at [24].

2.4.1. XML Structure of Data Dictionary

Figure 5 shows the XML structure of data dictionary metadata (UCDataDic.xml). This XML file represents the data dictionary structure extended from the use case description in the requirement specification. It defines the testable variables including variable ID, variable name, data type, data scale, condition, and optional syntax patterns, which are used to generate test cases with ECP and syntax testing.

As can be seen in Figure 5, the structure of the XML data dictionary starts with the tag <Usecase>, which includes an id and name. In this example, it is defined for use case Calculate Room Discount for HRMS. This is followed by tag <Input> indicating the input variable. Inside this, there is tag <Varname> (the name of input variable used for testing) and <DataType> (data type of variable). There are two data types: Range and Nominal. Range refers to variables whose values are divided into intervals, indicated by the starting value (min) to the ending value (max) in tag <condition>. For example, the number of stays (NumStay) is divided into four ranges of values as 1–2, 3–7, 8–15, and 16–30. On the other hand, Nominal refers to variables that have distinct categories or labels. It stores a discrete value such as Regular, Silver and Platinum for each guest type (GuestType). Finally, tag <Output> defines the expected results corresponding to the data provided in tag <Input>. It uses the tag <Action> to indicate the value of this expected output.

Furthermore, the syntax of input can be defined in an attribute pattern in tag <Syntax> of input variables in the XML file of the data dictionary, as shown in Figure 6. The syntax is encoded in the form of regular expression. For example, the email syntax (Email) explained in Table 3 is transformed into a regular expression form as in lines 6–7. [a-zA-Z0-9._%+-]{1,64} corresponds to the local_part, while [a-zA-Z0-9.-]{1,63} and [a-zA-Z]{2,6} correspond to domain_name and tld respectively. \. represents the literal dot between domain_name and tld. Considering line numbers 14–15, it indicates the format of booking date (BookingDate) to be as dd-mmm-yyyy, e.g., 31-Jan-2024.

2.4.2. XML Structure of Decision Tree

An example of the XML structure of a decision tree is demonstrated in Figure 7. This file pairs data from the XML data dictionary to describe the decision conditions and the corresponding action choices within the system’s decision tree. Inside the XML, there is a tag <Decision> which describes the decision rules. Each rule consists of the decision conditions and the actions corresponding to the specified conditions, which are described by tag <Condition> and the <Action>, respectively. The refid attribute within both tags links to the id of the variable in the XML data dictionary that is used in the decision condition or action. For instance, Figure 7 represents the decision tree for Hotel Room Discounts (HRMSDiscount). It indicates the decision rule that if the number of night stays is in between 1 and 2 nights (as defined in line number 5) and the guest type is a regular membership (as defined in line number 6), he will offer the discount for the room charge to be 3% (as defined in line number 7). Since each rule can have more than one decision condition, each tag <Decision> can contain more than one tag <Condition>. However, each decision can lead to only one action path. Therefore, each tag of <Condition> will contain only one tag <Action>.

2.4.3. XML Structure of State Machine Diagram

Figure 8 illustrates an example of XML structure for the state machine diagram. The structure and notations were designed based on the State Chart XML (SCXML) standard defined by W3C [25]. The root element is the <stateMachine> tag. In this example, it represents the state machine for managing the Hotel Room Status within an HRMS used to evaluate the proposed framework. Tags <initial> and <final> define the initial and final states, respectively, while tag <state> defines any intermediate state. The name of the state is specified by the id attribute. As demonstrated in the example, five states are defined: Initial, Vacant, Available, Occupied, and Final. Inside the state tags, transitions are specified using tag <transition> to define how the state reacts to events. Attributes event and target in this tag define the occurrence of event, indicating the change of the current active state to the designated target state. For instance, in the Available state, the transition corresponding to the Check In event causes the state to change to Occupied.

2.5. Test Case Coverage Calculation

The testing coverage is calculated accumulatively as each case is produced, ensuring that the final suite addresses all possible valid and invalid scenarios. To achieve this, equations are provided by our framework to calculate test case coverages. Firstly, the total number of expected test cases is calculated as demonstrated in Equations (1)–(5) for each testing technique. Equation (1) illustrates how to calculate the total number of test cases of ECP. As the framework implements the strong robustness technique, the testing coverage is derived from the summation between the number of valid test cases from the entire cross product of valid partition classes ($C_i$) for the variable i and the number of invalid test cases from invalid cases with a single-fault ($I_i$) per input.

$$\#TotalT{C}_{ecp}=\left(\prod _{i=1}^n{C}_i\right)+\sum _{i=1}^n{I}_i$$

(1)

The rule-based ECP generates the same total number of test cases as the standard ECP. However, the application of the rules restricts the valid test cases to a subset $F_R$ as defined in Equation (2). This results in a larger number of invalid test cases, as shown in Equation (3). The new total number of invalid test cases is calculated from the combination of the failed rules (the original valid count minus the number of $F_R$), with the number of invalid test cases from a single-fault method.

$$\#TotalValidT{C}_{recp}=F_R$$

(2)

$$\#TotalInvalidT{C}_{recp}=\left(\prod _{i=1}^n{C}_i-F_R\right)+\sum _{i=1}^n{I}_i$$

(3)

For syntax testing, the total number of test cases is calculated from a single test case of correct format ($V_{syn}$) and four invalid cases for incorrect value ($I_{val}$), substitution ($I_{sub}$), omission ($I_{omi}$) and addition ($I_{add}$) cases, as shown in Equation (4).

$$\#TotalT{C}_{syn}=V_{syn}+(I_{val}+I_{sub}+I_{omi}+I_{add})$$

(4)

The total number of expected test cases ($\#TotalT{C}_{state}$) is calculated as the square of the number of states ($\#Stat{e}^2$), including all valid and invalid transitions. To align with state transition testing standards, we exclude transitions from any state to the initial state, from the final state to any state, and self-loops on the initial/final states. Thus, as shown in Equation (5), the final count is derived by subtracting twice the number of states ($2\times \#State$). from the square of the states.

$$\#TotalT{C}_{state}=\#Stat{e}^2-(2\times \#State)$$

(5)

After calculating the total number of expected test cases, the percentage of testing coverage is carried out as illustrated in Equations (6) and (7). The complete coverage of the test (100%) is reached when the number of generated test cases ($\#GenTC$) is equal to the total expected test cases ($\#TotalTC$).

$$\%Coverage={\displaystyle \frac{\#GenTC}{\#TotalTC}}$$

(6)

$$100\%Coverage=\#GenTC=\#TotalTC$$

(7)

Note: 100% of black-box coverage confirms structural model validation, not total code fault detection. To uncover a wider range of defects, specific fault injection scenarios are needed to be analysed and included in the invalid test cases.

3. Proof of Concepts

3.1. Case Study Formulation

Our proposed framework was evaluated for accuracy, effectiveness and efficiency by using three close-to-realistic case studies that consist of different sizes and various behaviours. To ensure these studies reflect real-world software complexities, the system domains were selected based on the high volume of complex requirements, as reported in our previous empirical study on real software projects [26]. These selected domains include the Hotel Room Management System (HRMS), the Online Shopping Process System (OSPS) and the Student Status Monitoring System (SSMS). Since raw real-world specifications lack the complete set of system models, the necessary structures were formulated and added to complete the automated generation. For example, detailed hotel discount business rules were integrated into the HRMS to complete its specification model. This formulation was a controlled necessity to ensure that each system provides a specification model that covers all criteria used to evaluate all black-box perspectives facilitated by the framework and tool. We selected the systems according to three criteria. First, each system must expose clear machine-readable specifications that can be transformed into tests, including ranges and nominals for ECP, a RegEx pattern for syntax testing, and a state machine for state transition testing. Second, systems should form a complexity ladder so that we can see how performance scales when variable counts, partitions, states, and transitions increase. Third, business rules should be realistic enough to apply decision mapping, boundary handling, and transition flows that occur in production software.

Table 4. Characteristics of case studies.

System	ECP		Business Rules		Syntax	State Transition
	#Var.	#Part.	#Rules	#Nested	#Var.	#States	#Trans.
HRMS	3	12	12	2	8	3	6
OSPS	4	19	48	3	5	6	11
SSMS	4	18	36	3	7	6	18

details the characteristics of these formulated case studies. HRMS is the simplest case study, having few ECP partitions considering that there are only 12 business rules containing two conditions. This case study also contains the lowest number of states and transitions compared to other case studies. OSPS and SSMS are more complex case studies with 4 variables with a similar number of partitions. OSPS has the highest number of rules (48) with three nested conditions, making its business logic the most extensive. All details of these case studies, including SRS and manually generated test cases, are available at [24].

To ensure comprehensive test coverage using syntax testing techniques, we defined syntax patterns based on common programming languages and database data types including number, text and list. These criteria were combined using the Cartesian product (a cross join) to yield all possible value combinations for the formulated case studies. HRMS defined the most syntax variables (8), followed by SSMS (7), whereas OSPS defined the fewest (5). This variation reflects that a system’s input format is determined by its required data integrity, constraints, and functional correctness. The higher variable count in HRMS, for example, requires a more complex input format such as guests’ email and book date as shown in Figure 6.

Furthermore, when considering criteria for evaluating the ability to generate test cases by using state transition testing technique, we ranked the case studies based on their range of state transition scenarios. SSMS is the most complex in terms of state transition, which consists of 18 transitions across 6 states, indicating the greatest number of possible moves between states. This is followed by OSPS (11 transitions across 6 states). HRMS is the simplest, implementing only 3 states with 6 transitions. The case studies were further ranked by the presence of loop transitions. For instance, HRMS contains no loop transition. In contrast, SSMS includes a loop transition in its Normal state. This demonstrates that students can maintain a normal status as long as their Grade Point Average (GPA) remains above 2.0 after completing each semester.

3.2. BlackBoxTestGen Prototyping

To demonstrate the effectiveness and usability of the BlackBoxTestGen framework detailed in Section 2, a web application prototype was developed. It was built on the JavaScript (Node.js and Next.js) and utilises MongoDB as the database tool. An example of the developed tool is demonstrated in Figure 9. It generates and visualises partitions (Figure 9a) from the UCDataDic.xml shown in Figure 5. The generated partitions consist of valid and invalid partition. For example, the variable NumStay consists of four valid partitions based on the specified min and max values (1–2, 3–7, 8–15, and 16–30) and two invalid partitions that are underflow (<1) and overflow (>30). Then, test cases are generated by using the Cartesian product of all valid range partitions of all variables, including NumStay and GuestType, which are cross-multiplicated (4 × 3). In the case that the decision tree exists, only the results of this cross product that match the condition in the decision rule are used as the final test cases. In HRMS, all partitions are used in the condition of decision rules. Therefore, there are 12 valid test cases generated as shown in Figure 9b. For invalid test cases, there are 3 invalid test cases generated via the one-invalid-class method, where each invalid partition is mapped to a valid base case.

Figure 10 shows the screenshot of test case generation resulting from the syntax testing technique. The test cases are generated from the pattern defined in the input variables (e.g., GuestType, Email, PhoneNumber, BookDate, RoomType, PaymentMethod, and CardNumber). It synthesises one valid and four common targeted violations including invalid value, omission, addition, and substitution. The complete set is available as a CSV for immediate use in automated or manual validation.

Finally, Figure 4a displays an example of a generated state tree from StateMachine.xml. This tree represents the expanded transition structure, spanning from the initial state to related states, ensuring coverage of all transitions in the state machine diagram. The results of the generation of the test cases are demonstrated in Figure 4b and Figure 11. Following the state transition testing algorithm, single transition test cases are generated from a test case matrix (Figure 3), while the sequence of transition test cases is derived from the generated state tree (Figure 4b).

3.3. Prototype Evaluation

This section presents the empirical findings from two types of prototype evaluation: the accuracy of the generated test cases, and the computational efficiency and scalability.

3.3.1. Accuracy Evaluation

The accuracy of BlackBoxTestGen was evaluated by conducting a comparison between the test cases generated by the tool and the benchmark results identified by the practical expert.

Experiment Preparation: A wide range of practical specialists with experience in software engineering and software testing of more than three years were selected. This included one senior software tester, two senior software developers, and one junior software tester (average 7 years of working experience). Each expert independently derived test cases using the testing technique implemented in the prototype tool (rule-based ECP, syntax, and state machine testing). To avoid the inherent subjectivity, a collaborative review meeting was held among experts to merge their individual test suites. Any conflict or disagreement regarding the redundant or non-overlapping test cases was resolved through a structured discussion during the meeting until reaching consensus on the most suitable final benchmark suite. Finally, these benchmark test cases were compared with the test cases generated by the tool to evaluate the performance of the prototype.

Evaluation Results: The accuracy of the developed prototype was measured by using precision (for correctness), recall (for completeness), and F-measure (for overall) metrics. The computation metrics were adapted from [27] as shown in Equations (8)–(10).

$$\mathrm{Precision}={\displaystyle \frac{\left|\mathrm{Tool}\mathrm{Identified}\cap \mathrm{Expert}\mathrm{Identified}\right|}{\left|\mathrm{Tool}\mathrm{Identified}\right|}}\times 100\%.$$

(8)

$$\mathrm{Recall}={\displaystyle \frac{\left|\mathrm{Tool}\mathrm{Identified}\cap \mathrm{Expert}\mathrm{Identified}\right|}{\left|\mathrm{Expert}\mathrm{Identified}\right|}}\times 100\%.$$

(9)

$$\mathrm{F-measure}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}\times 100\%.$$

(10)

Table 5. Evaluation results of BlackBoxTestGen.

Testing Technique	System	Number of Test Case		Precision	Recall	F-Measure
		Expert/Manual	Prototype
Rule-based ECP	HRMS	15	15	100%	100%	100%
	OSPS	51	53	96.23%	100%	98.08%
	SSMS	39	41	95.12%	100%	97.50%
Syntax Testing	HRMS	40	40	100%	100%	100%
	OSPS	25	25	100%	100%	100%
	SSMS	35	35	100%	100%	100%
State Transition	HRMS	18	18	100%	100%	100%
	OSPS	52	53	98.11%	100%	99.05%
	SSMS	58	60	96.67%	100%	98.31%

summarises the evaluation of the developed BlackBoxTestGen tool, developed in three distinct testing techniques, rule-based ECP, syntax, and state transition tests, in three different systems: HRMS, OSPS, and SSMS.

Taking into account the rule-based ECP evaluation, HRMS serves as the baseline, in which the decision space is small, and the tool exactly matched the set of experts (15/15 overlaps), generating 100% precision, 100% recall, and 100% F-measure. However, the precision dipped slightly for the more complex systems: 96.23% for OSPS and 95.12% for SSMS. This is because the prototype generated two additional test cases over a set of manual designs by experts. We discovered that two upper-bound out-of-range logical cases were added. Overall, the tool achieves perfect coverage of expert intent while systematically strengthening test cases in out-of-range cases, improving robustness with minimal overhead.

Considering the evaluation result for the syntax testing, it was in perfect agreement with the manual oracles, generating a 100% score of precision, recall, and F-measure in these three case studies. This outcome follows from the design of the syntax pattern where each input field is governed by an explicit pattern, and the generator must produce one conforming value together with a fixed set of violations, covering deterministic outcome categories (Valid, Invalid Value, Substitution, Omission, and Addition). Although specific invalid strings vary from the manual sets, they align in the same category and count. The tool, therefore, consistently reproduces specified cases and maintains comprehensive and reproducible coverage for all format constraints across all inputs.

Lastly, the evaluation result for the state transition testing highlights that HRMS achieved high accuracy due to its low complexity, consisting of only 3 states and 6 transitions. In contrast, as the OSPS and SSMS case studies are more complex, they yielded lower precision. After revealing the generated test cases, we found that the generated test case matrix resulted in all single transition test cases, including valid and invalid transitions, generated automatically by the tool for both case studies (the same 48 single transition cases, calculated from the same number of states—6 states) being matched with those designed manually by experts. However, as an increasing number of transitions affects the state tree for generating sequence transition test cases to be more complex, our generator engine discovered missing test cases for the sequence of transitions from hand-operation by experts (1 test case for OSPS and 2 test cases for SSMS). This led to precision scores for OSPS and SSMS of 98.11% and 96.67%, respectively. After the revision with the experts, the number of automatically regenerated test cases achieved an exact match with the expected manual test suite. The small precision dips in OSPS and SSMS reflect additional feasible paths discovered by BlackBoxTestGen exploration that broaden path coverage without contradicting the models.

Finally, at the end of the evaluation panel, the experts concluded that our proposed framework and the developed prototype provided concise and reproducible test cases. They also noted that it can significantly reduce manual effort and prevent drift between testing techniques.

3.3.2. Computational Efficiency and Scalability Evaluation

We also evaluate the efficiency of the developed prototype in terms of computational efficiency and scalability.

Experiment Preparation: Since the framework unifies three distinct testing techniques through a shared XML basis, evaluation experiments were conducted to ensure that they meet both the global structured overhead and the isolated algorithmic complexity. Two types of performance metrics were used: Total Test Generation Time ($T_{total}$) and Peak Memory Consumption ($M_{peak}$). $T_{total}$ is the end-to-end execution time from the XML being loaded and extracted ($T_{parse}$) to the final test cases generated by each testing technique ($T_{gen\left(recp\right)}$, $T_{gen\left(syn\right)}$, and $T_{gen\left(state\right)}$ for rule-based ECP, syntax, and state testing, respectively). This calculation is shown in Equation (11). $M_{peak}$ was measured from the maximum RAM allocated during the generation process.

$$T_{total}=T_{parse}+T_{gen\left(recp\right)}+T_{gen\left(syn\right)}+T_{gen\left(state\right)}$$

(11)

To evaluate computational efficiency and scalability, two types of experiments were conducted. The first is the evaluation of the isolated algorithmic complexity, where each testing technique was evaluated separately using its specific parameter scale. The second is the evaluation of the integrated algorithmic complexity, where all testing techniques integrated in the prototype were evaluated using the case studies. The experiment environments were run on a laptop equipped with an AMD Ryzen 5 5600H processor with Radeon Graphics running at 3.30 GHz, 16 GB RAM, and a Windows 11 64-bit operating system (x64-based architecture).

Isolated Algorithmic Complexity Evaluation Results: Figure 12 shows the comparison of the total generation time ($T_{total}$) for each isolated testing technique.

As can be seen in the figure, the rule-based ECP (Figure 12a) and syntax testing (Figure 12b) perform remarkably efficiently in test case generation. In rule-based ECP, the generation time takes only 1.936 ms, while parsing consumes 15.218 ms (approximately 88% of $T_{total}$) at the maximum load parameter scale (14 variables/100 rules). Similarly to this, syntax testing consumes only 1.175 ms in the generation time, while parsing takes 9.434 ms (approximately 89% of $T_{total}$). This is because the structural complexity of the input model (reading an XML file, extracting rules or syntax) affects the time to perform parsing, especially rule-based ECP, which requires cooperation between two XML files (data dictionary and decision tree). The state transition testing (Figure 12c) exhibits completely different performance results, which consume a small amount of $T_{parse}$ and $T_{gen\left(state\right)}$ at the lowest complexity (0.33 ms and 0.5 ms, respectively, at 5 states/15 transitions). However, the generation time (10.09 ms of $T_{gen\left(state\right)}$) significantly overtakes the parsing time (5.117 ms of $T_{gen\left(state\right)}$) on the largest scale (50 states/150 transitions) for roughly 66% of $T_{total}$. This is due to a compounded increase in the scale and complexity of the state transition models.

Our prototype also evaluated the maximum memory consumption ($M_{peak}$) of each isolated testing algorithm, as demonstrated in Figure 13. This memory matrix reveals the structural weight of each underlying testing algorithm. As can be seen in the figure, both rule-based ECP (Figure 13a) and syntax testing (Figure 13b) allocate approximately 80 MB at the initial scale of the smallest inputs (5 variables and 2 variables, respectively). However, state transition testing starts significantly heavier, consuming roughly 101.17 MB for only 5 states and 15 transitions (Figure 13c). This is because the data structures are required to represent a state machine and a necessary basis for test case generation in memory (state nodes, directed edge lists, test case matrix, state trees). This figure is significantly increased according to the incremental scale results in a high-RAM environment (168.0 MB in 50 states and 150 transitions). As syntax testing does not process complex semantic rules or interdependent state tracking, it is the most efficient memory-scalable technique among the three. These experimental results confirm that the peak memory usage by these testing techniques is strictly bounded. Even when performing under the heavy load of state transition testing, memory consumption remains low.

Furthermore, the relationship between total generation time and the volume of generated test cases was explored as shown in Figure 14. The results of trendline analysis ($R^2>0.99$) reveal three distinct patterns that illustrate the balance between generation speed and output quality. The rule-based ECP influences the highest computational cost per test case with exponential growth. As the cross product operation in ECP can result in a massive output volume when increasing the partitions of variables (from 13 to 121), it causes an exponentially high computation cost per test case (as in Figure 14a) due to heavy semantic parsing and validation. Syntax testing influences highly predictable linear scaling. Taking into account the trendline shown in Figure 14b, it follows a Power Series, the line of which is almost perfectly straight (the exponent value is nearly close to 1). Only RegEx uses it to generate test cases from the input syntax. This means that it does not need heavy semantic filtering (as in rule-based ECP) and does not face a combinatorial explosion (as in state transition). The state transition testing also follows a Power Series. However, unlike syntax testing, the trend line is concave down as illustrated in Figure 14c. As noted in this figure, 4349 test cases are generated in just 15 ms. Even though there is a considerable number of generated test cases when the number of states and transitions is scaled up, the execution remains exceptionally fast (under 20 ms). The results of the analysis of the total generation time ($T_{total}$) (as shown in Figure 12) and its trendline (as shown in Figure 14) prove that the execution time is highly reliable and predictable. As the input increases considerably, the scaling remains steady rather than suddenly spiking.

In short, the performance of the prototype in terms of computational efficiency and scalability is summarised in the form of Big O efficiency as follows. The standard ECP performs the exponential time complexity $O\left(V^k\right)$, where valid and invalid partitions are cross-multiplied together. The application of the rule-based approach to ECP allows Big O to be $O(V\times R)$, where V is the number of variables and R is the number of rules. On the other hand, the predictability and proportionality of Linear Power Series introduced by the syntax testing result in its Big O efficiency for both time and space complexities being $O\left(N\right)$, where N is the number of syntax nodes/variables. In the state transition testing technique, the state and transition parsing mechanism is fast; therefore, its Big O efficiency is measured as $O(S+T)$, where S is the number of states and T is the number of transitions. However, to satisfy the traversing coverage of the graph, the time and space complexity is measured towards $O\left(d^k\right)$, where d is the transition per state and k is the depth of the path.

Integrated Algorithmic Complexity Evaluation Results: In the previous section, computational efficiency and scalability evaluation for each isolated testing technique were presented. To move from theoretical complexity (O notation) to practical feasibility, we measured the efficiency of integration among the three testing techniques in three diverse systems (HRMS, OSPS, and SSMS) as explained in Table 4. It aims to demonstrate that our proposed framework is not only mathematically sound but also industrially viable.

The evaluation of the integrated framework across these case studies confirms a robust computational efficiency defined by sub-linear resource growth. As shown in

Table 6. Evaluation results of computational efficiency and scalability in case studies.

System	Total TCs	Tparse (ms)	Tgen(recp) (ms)	Tgen(syn) (ms)	Tgen(state) (ms)	Ttotal (ms)	Mpeak (MB)
HRMS	73	4.942	0.327	0.968	0.28	6.517	103.676
OSPS	131	7.781	0.567	0.208	0.413	8.969	104.117
SSMS	136	9.862	0.768	0.442	0.861	11.933	104.316

, while the complexity of the prototype increases, resulting in twice the number of test cases from 73 in HRMS to 136 in SSMS, the Total Execution Time ($T_{total}$) remains consistently under 150 ms. This shows that our proposed framework and prototype will not break the pipeline when the project grows from a small system to a larger enterprise system. The result also reveals that the primary computational overhead comes from the XML loading and extraction time ($T_{parse}$) rather than the algorithmic bottlenecks ($T_{gen}$).

Furthermore, peak memory usage ($M_{peak}$) remains highly stable. Although the number of variables and transitions were scaled up (as in Table 6), overall memory consumption only increased by approximately 0.6% (from 103.676 MB to 104.316 MB). This minimal increase indicates that the proposed framework and prototype successfully use the shared data structure, resulting in the overhead of increased complexity becoming nearly zero. These findings confirm the scalability of the framework. Its ability successfully avoids state space explosions and maintains predictable memory allocation, proving its practicality for increasingly complex OSPS and SSMS models.

4. Lessons Learned and Discussion

4.1. Discussion

Our main finding from the development of the BlackBoxTestGen framework that combines various black-box testing techniques is that it offers distinct benefits to validate complex software systems. Firstly, it provides comprehensive coverage of the system that addresses different dimensions of system behavior. This includes the robustness of logical conditions across the range of values, the integrity of the format, and the reliability of behavioral flow provided by rule-based ECP, syntax testing, and state transition, respectively. The feature-rich integration of these techniques influences the obvious benefit that the number of test cases is minimised, while the testing coverage is maximised. From our implemented experiences [21,22,23], we discovered that testing techniques employed single-value selection to generate test cases, covering all possible scenarios. This is also confirmed by the discussion suggested by [1,28].

Furthermore, the automation mechanism provided by the framework and tool eliminates the inherent limitation of manual test case generation in modern software development environments. Automated test case generation makes tests repeatable, faster to execute, and easier to maintain. This efficiency is essential to address the challenges of scalability in a more complex system. The high precision and recall in the evaluation results demonstrate that our developed automated framework and tool can achieve professional, expert-level accuracy. Similar to the suggestion of [6,14,16] regarding the automated testing tool, automation eliminates human errors that typically occur in manual testing, ensuring more reliable generation results and consistently accurate test cases.

4.2. Comparison to Related Approaches

Several frameworks and tools for automatic test case generation have recently been applied to the software testing community. This section synthesises and compares the reported capabilities, theoretical architectures, and methodological strengths of tools closely related to our BlackBoxTestGen framework in terms of capabilities and performance.

4.2.1. Comparison of Functionality and Capabilities

Table 7. Capability comparison BlackBoxTestGen with other testing frameworks.

Criteria	BlackBox TestGen (Proposed)	TestGen [21]	SYNTest [22]	STATETest [23]	UTG [33]	ProTest [30]	TESTONA [29]	IBM Rhapsody [31]
Testing Technique	ECP,       Syntax, State	ECP	Syntax	State	State	ECP	ECP	N/A
Source of Test Case Generation	SRS	SRS	SRS	SRS	SRS	GUI	GUI	Class and Sequence Diagrams
Test Case Generation	✓	✓	✓	✓	✓	✓	✓	✓
Test Case Coverage Analysis	✓	✓	✓	✓	✓	✓	✓	✓
Free (F)/Commercial (C)	F	F	F	F	N/A	C	C	C
Standalone (S)/Web-based (W)	W	S	W	W	S	S	S	S

Note: ✓ = Supported, N/A = Not Available.

demonstrates the comparison of capabilities of the framework categorised by input sources, generation techniques, and coverage capabilities. TESTONA [29] and ProTest [30] utilise ECP to derive tests from GUIs, while TestGen [21] applies rule-based ECP to functional logic and system rules defined in SRS. In contrast, Rational Rhapsody [31] generates cases from class and sequence diagrams. Syntax-based generation is supported by SYNTest [22], which uses regular expressions to model data dictionary patterns. This is similar to the research work proposed by [32], which applies RegEx to validate string-type input formats. For state-based behavior, STATETest [23] and UTG [33] automate test generation from state machine diagrams using XML structures. In contrast to STATETest, UTG is a legacy Windows-based application that relies on a dependent MagicDraw-specific XML schema. Despite these developments, most of the existing tools are standalone tools and focus on a single black-box technique. Given the multifaceted nature of modern software, a single-method approach is often insufficient for comprehensive validation of complex systems.

4.2.2. Comparison of Performance

This performance comparison in terms of usability and scalability, generation time, number of test cases produced, and coverage quality is shown in

Table 8. Performance comparison of BlackBoxTestGen with other testing frameworks.

Criteria	BlackBoxTestGen (Proposed)	Academic Standalones (TestGen [21], SYNTest [22], STATETest, [23])	UTG [33]	Commercial GUI-Based (TESTONA [29], ProTest [30])	Commercial UML-Based (IBM Rhapsody [31])
Usability and Scalability	High	Low	Low	Moderate	Moderate/Low
Generation Time and Performance	Highly Optimised	Fast execution	Slower	Delayed Workflow	Enterprise-Grade
Number of Test Cases Produced	Compact and Reproducible	Varies by scope	Highly Compressed	Dependent on Interface	Comprehensive
Coverage Quality	Multi-Dimensional	One-Dimensional	Incomplete	Restricted	High

. Firstly, regarding usability and scalability, the academic prototypes (TestGen, SYNTest, and STATETest) force the tester to switch between different testing frameworks. This consequently leads to inconsistency in synchronisation when the requirement specification changes. Commercial tools like IBM Rhapsody offer high scalability. However, expertise in UML class and sequence diagrams is required. Unlike Pro-Test and TESTNoNA, they operate primarily only at the GUI level, while our proposed framework offers test case generation to begin at the requirement level in the early phase of development. By unifying rule-based ECP, syntax, and state testing together with a shared XML structure, our BlackBoxTestGen provides high usability and scalability, eliminating inconsistency between testing techniques and further reducing manual overhead.

Secondly, performance in test case generation is considered, which typically relies on the complexity of the parsing algorithm and the underlying constraints engine. SYNTest and STATETest enable the lightweight mechanism. The application of Regular Expressions and XML state traversal results in negligible generation latency for small-to-medium case studies. However, the function minimisation utilised in the UTG tool results in the input being repeatedly modified. This causes a high computational cost and slows down generation when the complexity of the system increases. Compared with our BlackBoxTestGen that executes three algorithmic engines simultaneously, this influences a slightly higher baseline overhead. However, a shared XML basis still offers its high efficiency, which prevents redundancy in the same variable parsing multiple times.

Lastly, when comparing the volume and quality of the generated test cases, academic standalone prototypes influence the limitation of the single-paradigm. For example, STATETest ensures 100% of transition coverage but lacks data boundary flaws, while TestGen covers data boundaries but lacks behaviour states. Furthermore, UTG applies an Alternating Variable Method (AVM) for function minimisation to generate test data from state charts. Although this produces a high quality of test case generation, it fails to achieve 100% transition pair coverage (reaching only 54–67%) in their study. Unlike our proposed framework, the multi-dimensional coverage quality is provided by the cumulative calculation of testing coverage across rule-based ECP, syntax, and state machine testing. This results in a compact and reproducible test suite, ensuring high quality without the unnecessary volume of disconnected test sets.

4.3. Methodological Limitations

This section describes the limitations of our proposed framework and prototype as follows. One of the challenges in any specification-driven testing framework is the accuracy of the specification used as a source for test case generation. As BlackBoxTesstGen operates on test case generation based on the assumption of a complete and well-defined specification, it can only validate the syntax of XML and the consistency between the files as explained in Section 2. Therefore, it cannot resolve semantic ambiguities, missing requirements, or logical contradictions present in the original natural language documentation. To address this, software engineers need to handle natural language ambiguities and inconsistencies before the specification is transformed into the rigid XML schema. To detect ambiguity in natural language requirements, an automated Natural Language Processing (NLP) approach from our previous work [26] can be applied. Three levels of ambiguity detection, including lexical, syntax, and semantic, are processed before the design and implementation of the test case begins.

Furthermore, as noted in Section 2.5, since black-box testing inherently aims at evaluating both valid and invalid execution paths based on structural model behaviour, our proposed framework generates test cases covering all acceptable data partitions, correct syntax formats, and legal behavioral pathways. For maximising fault detection, the framework generates invalid test cases including out-of-range values, invalid syntax, and incorrect transition paths, based on a black-box perspective to deliver the error-handling mechanisms of the system under test. Therefore, the 100% coverage influenced by the framework does not inherently guarantee the detection of all internal implementation faults. It needs to specifically analyse and implement fault injection scenarios to facilitate the ability of the fault detection mechanism.

5. Conclusions and Future Work

This paper has presented an automatic black-box test case generation framework, BlackBoxTestGen, that provides a wide range of black-box testing techniques. It integrates our previously developed frameworks and tools, TestGen, SYNTest, and STATETest, to facilitate the choice of the most appropriate testing technique including rule-based ECP, syntax, and state transition testing based on the specific characteristics of various input types. A shared XML structure of the necessary test basis was redesigned to eliminate redundancy and inconsistencies between the UC data dictionary, the decision tree, and the state machine diagram used separately by each testing technique. This enables a single upload for an all-in-one test case generation mechanism that reduces the time for generation. The framework also provides an accumulative calculation of test case coverage to ensure that the generated test cases cover all possible testing scenarios. The results of the benchmark evaluation with practical specialists through three close-to-realistic case studies have shown that our proposed framework, with its corresponding developed prototype tool, achieved high levels of accuracy and precision. The experts also confirm that the integrated tool reduces the manual workload and prevents the testing technique from drifting. Furthermore, the stress testing evaluation reveals the resilience of the framework, which performs steadily with the predictable total execution time and peak memory consumption under an excessive increase in volume of input parameters. Finally, the framework also manages near-realistic scaling, which sustains sub-linear resource growth and keeps peak memory consumption completely stable.

The future direction of research will focus on expanding the scope of performance evaluation across other diverse real-world domains. To increase the ability of the fault detection mechanism, we intend to formally analyse and integrate fault injection methodologies into the generation pipeline. Furthermore, a plugin architecture for exposing an extension for a new testing technique into the prototype tool remains an ongoing research challenge.

6. Deployment and Availability

The tool, user manual, and case study resources (HRMS, OSPS, and SSMS) are available for public access at [24].