Test Case Generation Method for Increasing Software Reliability in Safety-Critical Embedded Systems

Finite-state machines (FSMs) and the W method have been widely used in software testing. However, the W method fails to detect post-processing errors in the implementation under test (IUT) because it ends testing when it encounters a previously visited state. To alleviate this issue, we propose an enhanced fault-detection W method. The proposed method does not stop the test, even if it has reached a previously visited state; it continues to test and check the points that the W method misses. Through various case studies, we demonstrated software testing using the W method and the proposed method. From the results, it can be inferred that the proposed method can more explicitly determine the consistency between design and implementation, and it is a better option for testing larger software. Unfortunately, the testing time of the proposed method is approximately 1.4 times longer than that of the W method because of the added paths. However, our method is more appropriate than the W method for software testing in safety-critical systems, even if this method is time consuming. This is because the error-free characteristics of a safety-critical system are more important than anything else. As a result, our method can be used to increase software reliability in safety-critical embedded systems.


Introduction
In recent decades, because of the groundbreaking development of computer engineering technology, automatic control systems have gradually substituted for humans. Initially, this change depended on the capability of hardware systems. However, with the growth of the semiconductor industry, better central processing units (CPUs) for computers were developed, and software became a more important part of such systems.
Accordingly, detecting a malfunction in software before its release has also become an increasingly important part of its development. This is particularly true for safety-critical systems, such as those in airplanes, trains, nuclear power plants, autonomous driving cars, and defense systems. This has led to the increasing need for software testing.
Among numerous testing methods, finite-state machine (FSM) has been widely applied in various fields. D. Lee and M. Yannakakis defined five fundamental problems in using FSM [1]. These are homing/synchronizing sequence, state identification, state verification, machine verification/fault detection/conformance testing, and machine identification. Conformance testing checks the equivalence between the FSM specification and the implementation under test (IUT). In other words, the state diagram of the FSM specification can be compared with the behavior of an implementation to verify whether the IUT is equivalent to the FSM specification in the conformance testing. This conformance testing is often used to test software. Numerous studies have been conducted in this field. Some studies involved reducing the testing size (length) [2][3][4][5][6], modeling FSM in various ways [7][8][9][10][11][12][13][14][15][16][17], and establishing a test criterion [18].
In this paper, we aim to reinforce the checking of the equivalence between the FSM specification and the IUT. We propose an enhanced fault-detection W method to generate test suites (sequences), analyze the testing time, and apply the method to case studies. Through testing time analysis and case studies, we demonstrate that system checking through the W method is not up to par with our method, even if our method is time consuming. This indicates that the consistency between design and implementation can be verified explicitly by our method, and our method is more appropriate than the W method in safety-critical systems where the error-free characteristic is more important than anything else. Moreover, we show that the difference in the number of errors captured by the two methods increases as the software becomes larger and more complex. This result indicates that our method is a better option for testing larger software. As a result, our method can be used to increase software reliability in safety-critical embedded systems.

Background
FSM and W method are widely used to test software. In this section, we briefly discuss the UML (Unified Modeling Language) [19,20], FSM, and W method as background techniques.

UML State Machine
An FSM can be represented by a state transition diagram. State-based languages, such as Statecharts [21], SDL (Specification and Description Language) [22], and UML are used for state transition diagrams. In this study, the UML state machine is used to represent FSM. Definition: The the state diagram of the FSM specification can be compared with the behavior of an implementation to verify whether the IUT is equivalent to the FSM specification in the conformance testing. This conformance testing is often used to test software. Numerous studies have been conducted in this field. Some studies involved reducing the testing size (length) [2−6], modeling FSM in various ways [7][8][9][10][11][12][13][14][15][16][17], and establishing a test criterion [18].
In this paper, we aim to reinforce the checking of the equivalence between the FSM specification and the IUT. We propose an enhanced fault-detection W method to generate test suites (sequences), analyze the testing time, and apply the method to case studies. Through testing time analysis and case studies, we demonstrate that system checking through the W method is not up to par with our method, even if our method is time consuming. This indicates that the consistency between design and implementation can be verified explicitly by our method, and our method is more appropriate than the W method in safety-critical systems where the error-free characteristic is more important than anything else. Moreover, we show that the difference in the number of errors captured by the two methods increases as the software becomes larger and more complex. This result indicates that our method is a better option for testing larger software. As a result, our method can be used to increase software reliability in safety-critical embedded systems.

Background
FSM and W method are widely used to test software. In this section, we briefly discuss the UML (Unified Modeling Language) [19,20], FSM, and W method as background techniques.

UML State Machine
An FSM can be represented by a state transition diagram. State-based languages, such as Statecharts [21], SDL (Specification and Description Language) [22], and UML are used for state transition diagrams. In this study, the UML state machine is used to represent FSM.

FSM and W Method
Here, we briefly describe the construction of an FSM from a specification and the generation of a test suite using the W method. For an equipment with the following requirement specification, the corresponding FSM is illustrated in Figure 2.

FSM and W Method
Here, we briefly describe the construction of an FSM from a specification and the generation of a test suite using the W method. For an equipment with the following requirement specification, the corresponding FSM is illustrated in Figure 2.   From the above FSM, the W method derives a tree "T" via breadth-first search or depth-first search as follows: 1. Set the root of "T" with the initial state of the FSM, "IDLE". This is level 1 of "T".
2. Suppose we have already built "T" to a level "k". The "(k + 1)"th level is built by examining nodes transferable from the "k"th-level nodes. A node at the "k"th level is terminated if it is the same as a non-terminal at some level "j", j ≤ k. Otherwise, we attach a successor node to the "k"th-level node.
In the process of making a tree in Figure 3, if one state was already visited or a final state was encountered, the state will be a terminal node. Five test sequences in Table 1 can be generated from the tree. From the above FSM, the W method derives a tree "T" via breadth-first search or depth-first search as follows: 1.
Set the root of "T" with the initial state of the FSM, "IDLE". This is level 1 of "T".

2.
Suppose we have already built "T" to a level "k". The "(k + 1)"th level is built by examining nodes transferable from the "k"th-level nodes. A node at the "k"th level is terminated if it is the same as a non-terminal at some level "j", j ≤ k. Otherwise, we attach a successor node to the "k"th-level node.
In the process of making a tree in Figure 3, if one state was already visited or a final state was encountered, the state will be a terminal node.  From the above FSM, the W method derives a tree "T" via breadth-first search or depth-first search as follows: 1. Set the root of "T" with the initial state of the FSM, "IDLE". This is level 1 of "T".
2. Suppose we have already built "T" to a level "k". The "(k + 1)"th level is built by examining nodes transferable from the "k"th-level nodes. A node at the "k"th level is terminated if it is the same as a non-terminal at some level "j", j ≤ k. Otherwise, we attach a successor node to the "k"th-level node.
In the process of making a tree in Figure 3, if one state was already visited or a final state was encountered, the state will be a terminal node. Five test sequences in Table 1 can be generated from the tree.  Five test sequences in Table 1 can be generated from the tree.

Proposed Method
In this section, we propose a modified method to supplement the W method. Moreover, we theoretically analyze the testing time of the two methods to determine the feasibility of the proposed method in real testing.
The terminologies used in this paper are defined as follows.
• Node: It is used as the corresponding expression of the "state" in the tree, i.e., a "node" is used in the tree as a "state" is used in the FSM.

•
Terminal node: This is present in W method. If one state has already been visited, the state becomes a "terminal node". The final state is also a "terminal node". • Initial entry status: This is the status of the state when it has been entered for the first time.

•
Re-entry status: This is the status of the state when it is re-entered after entering another state. • Post-processing: This is the processing that makes the re-entry status equal to the initial entry status.

•
Post-processing error(s): These error(s) are occurred by the reasons why post-processing is incorrect.

•
Test case: This has the same meaning as "test sequence".

•
Test set: This is the collection of test cases (sequences). It has the same meaning as "test suite".

Algorithm
The W method ceases to make a test sequence when it encounters a previously visited state. This mechanism fails to detect post-processing errors in the IUT. To illustrate, given a state A, let Ai be the initial entry status of A and Ar be the re-entry status of A. When a software is implemented, there are many cases in which Ai is not equal to Ar. This is because of the mistakes in post-processing such as memory allocation faults and faults for initializing variables. The mechanism of the W method does not test this point, which means that it fails to check the equivalence between an FSM specification and an IUT.
Based on the foregoing, we propose a modified method to improve the W method. The proposed method adds test paths from a terminal node to a specified final state. The added paths check whether the behavior of the software is the same in the initial entry and the re-entry, i.e., the deficiency of the W method. The algorithm of the proposed method is presented below. : n 0 is a node corresponding to the initial state s 0 ∈ SM 8: for each non-terminal leaf node n ∈ Tr 9: begin 10: for each outbound transition t of the state corresponding to n 11: begin 12: st = a target state of t 13: for each input i ∈ I ∈ t 14: begin 15: n' = a node corresponding to st, set n' is terminal node 23: end if 24: end for 25: end for 26: for each event sequence TC from n 0 ∈ Tr to a terminal node n ∈ Tr 27: begin 28: TS = TS ∪ TC 29: end for ///end /// In the above algorithm, lines 6 and 7 initialize the test set and the tree. The tree, Tr, consists of the root(init) node "n 0 ", node set "N" corresponding to the state, and edge set "E" corresponding to the transition.
Lines 8 to 25 are the main body of the algorithm. Among these, lines 10 to 24 add child nodes and edges for transitions. Lines 18 to 20 are the novel addition which enhances the algorithm. These lines add the shortest path from the terminal node in the W method to the specified final state. Lines 21 to 23 set the node as the terminal node in the proposed method if the added child node corresponds to a specified final node. Lines 26 to 29 collect all the sequences from the initial node to the terminal node in the proposed method and make a test set (suite).
Using the new algorithm, Figure 3 becomes the tree in Figure 4. and edges for transitions. Lines 18 to 20 are the novel addition which enhances the algorithm. These lines add the shortest path from the terminal node in the W method to the specified final state. Lines 21 to 23 set the node as the terminal node in the proposed method if the added child node corresponds to a specified final node. Lines 26 to 29 collect all the sequences from the initial node to the terminal node in the proposed method and make a test set (suite). Using the new algorithm, Figure 3 becomes the tree in Figure 4. The modified test sequences are shown in Table 2.  The modified test sequences are shown in Table 2.

Testing Time Analysis
The proposed method returns a longer test suite (sequences) than the W method. To compare the testing time of the two methods, theoretical analysis is performed.
Suppose we have an FSM as in Figure 5.

Testing Time Analysis
The proposed method returns a longer test suite (sequences) than the W method. To compare the testing time of the two methods, theoretical analysis is performed.
Suppose we have an FSM as in Figure 5. The tree derived using W method is illustrated in Figure 6. The tree derived using W method is illustrated in Figure 6.  The tree derived using W method is illustrated in Figure 6. In Figure 6, each state is denoted as "An" where "n" is varied from 1 to N. The level of the tree is denoted as "l" and is varied from 1 to N + 1.
To determine the testing time, we sum the tested nodes. In W method, the number of tested nodes, Tw, is given by the following equation.
The tree derived using the proposed method is shown in Figure 7. In Figure 6, each state is denoted as "An" where "n" is varied from 1 to N. The level of the tree is denoted as "l" and is varied from 1 to N + 1.
To determine the testing time, we sum the tested nodes. In W method, the number of tested nodes, Tw, is given by the following equation.
The tree derived using the proposed method is shown in Figure 7. In the proposed method, testing time is calculated using a different approach. In case of "An" terminal nodes: 1. The number of test nodes for a "An" in each level, Tpn(An, l), is the sum of the test nodes in W method and the added nodes in the proposed method. 2. If the number of "An"s in each level is Nt (An, l), the number of test nodes for "An"s in each level, Tpn(An, l), is Tpn(An, l) × Nt (An, l),. 3. The number of test nodes for "An"s in the tree, Tp(An), is the sum of Tp(An, l) where l is from 1 to N + 1. As a result, the testing time of the proposed method, Tp, is the sum of Tp(An) and is given by the following equation.
To obtain Tp(An) from Figure 7, we start with "A1" terminal nodes as follows. In the proposed method, testing time is calculated using a different approach. In case of "An" terminal nodes: 1. The number of test nodes for a "An" in each level, Tpn(An, l), is the sum of the test nodes in W method and the added nodes in the proposed method.

2.
If the number of "An"s in each level is Nt(An, l), the number of test nodes for "An"s in each level, Tpn(An, l), is Tpn(An, l) × Nt(An, l),.

3.
The number of test nodes for "An"s in the tree, Tp(An), is the sum of Tp(An, l) where l is from 1 to N + 1. As a result, the testing time of the proposed method, Tp, is the sum of Tp(An) and is given by the following equation.

5.
Level N + 1: Tpn(A1, N + 1) = 2N, Nt(A1, N + 1) = (N − 1)!), Tp(A1,N + 1) = 2N × (N − 1)! According to the above results, In the same manner, Therefore, Using Equations (1) and (6), we get the following results in Table 3 and Figure 8.  From the above, we know that the proposed method takes about 1.4 times longer than the W method. Nevertheless, the proposed method remains acceptable considering the enhanced effectiveness of the testing, especially in the safety-critical system. From the above, we know that the proposed method takes about 1.4 times longer than the W method. Nevertheless, the proposed method remains acceptable considering the enhanced effectiveness of the testing, especially in the safety-critical system.

Case Study
Four case studies are performed with the software of the safety-critical system. These are console software of the army missile defense system, a console software of the navy missile defense system, a satellite operation software, and a radar operation software. Among these, only the first case study is described in detail. The other case studies are briefly summarized.

Methodology
Case studies were performed through the following methodology.

1.
Construct an FSM specification from the requirement specification.

2.
Derive a tree and generate test suites using the W method and the proposed method.

3.
Test the implementation of the two test suites and compare the results. This step is repeated to check the reproducibility of the test.
The basic information of the case studies is shown in Table 4.

Constructing FSM
In this section, we show the FSM specification from the requirements specification. The requirements of the army missile console software are as follows. The constructed FSM is shown in Figure 9.  Figure 9.

Deriving Tree and Generating Test Suite
Two trees were derived from the above FSM. Figure 10 shows the tree using W method.  Figure 11 shows the tree using the proposed method.

Deriving Tree and Generating Test Suite
Two trees were derived from the above FSM. Figure 10 shows the tree using W method.  Req 7: Ready Monitoring, Firing Ready status monitoring  Req 8: Countdown Monitoring, Countdown status monitoring The constructed FSM is shown in Figure 9.

Deriving Tree and Generating Test Suite
Two trees were derived from the above FSM. Figure 10 shows the tree using W method. Figure 10. Tree derived using W method. Figure 11 shows the tree using the proposed method.  Figure 11 shows the tree using the proposed method. Two test suites were generated from the above trees as shown in Table 5 and 6. IdleAssignedWeapon Initializing  Fire Aborted Table 6. Test suite using the proposed method. Figure 11. Tree derived using the propose method.

No. of
Two test suites were generated from the above trees as shown in Tables 5 and 6. Idle→Assigned→Weapon Initializing → Fire Aborted Table 6. Test suite using the proposed method.

No. of Test Sequence
Test Sequence  Table 7 shows the requirements covered by the two test suites.

Test and Results
The configuration of the missile defense system is shown in Figure 12, which was used as the test configuration. The equipment used for testing were the real equipment, except for the launcher (with weapon), which was substituted with a simulator. While the test was processed, we checked the results through the graphic user interface(GUI) screen of the console. Electronics 2020, 9, x FOR PEER REVIEW 12 of 15 Figure 12. Configuration of the army missile defense system.
The test results from the two suites are shown in Table 8. The proposed method detected two errors more than that of the W method.
One error was detected in test sequence no.
The error is that the weapon (missile) was not assigned in the re-entry to the "Assigned" state. In the initial entry, missile assignment was performed successfully, but in the re-entry after cancelling the assignment, missile reassignment failed. This error is a fatal error resulting in the mission not being performed.
The other error was detected in test sequences no. 5 […Assigned…Fire EnableAssigned …Fire Enable…Lift off]. The original sequence of W method is […AssignedWeapon InitializingFire Ready Complete …]. The error is that the firing authority was permitted without checking for a password. In the initial entry to the "Fire Enable" state, password checking was performed successfully, but in the re-entry, the firing authority was given without checking for a password. This error is also a fatal error in that a missile can be fired by a person who does not have the authority. These errors appeared repeatedly when the same software is used without debugging the errors.
This case study indicates that our proposed method can more properly check the equivalence between the FSM specification and the IUT.
Other case studies were performed in the same manner. The test summary is shown in Table 9, Figure 13 and 14.  The test results from the two suites are shown in Table 8. The proposed method detected two errors more than that of the W method.
One error was detected in test sequence no.
The error is that the weapon (missile) was not assigned in the re-entry to the "Assigned" state. In the initial entry, missile assignment was performed successfully, but in the re-entry after cancelling the assignment, missile reassignment failed. This error is a fatal error resulting in the mission not being performed.
The . The error is that the firing authority was permitted without checking for a password. In the initial entry to the "Fire Enable" state, password checking was performed successfully, but in the re-entry, the firing authority was given without checking for a password. This error is also a fatal error in that a missile can be fired by a person who does not have the authority. These errors appeared repeatedly when the same software is used without debugging the errors.
This case study indicates that our proposed method can more properly check the equivalence between the FSM specification and the IUT.
Other case studies were performed in the same manner. The test summary is shown in Table 9, Figures 13 and 14. Electronics 2020, 9, x FOR PEER REVIEW 13 of 15  From Tables 4 and 9, we observed that the difference in the number of errors found by the two methods increases as the software becomes longer and more complex. This means that there are more post-processing errors, and our method becomes a better option for testing as the software becomes longer and more complex.

Conclusions
Testing is the most important part of software development, especially in safety-critical systems. Among numerous testing methods, FSM and the W method have been widely used.
However, the W method fails to detect post-processing errors in the IUT. To correct this deficiency, we proposed an enhanced fault-detection W method to generate test suites (sequences), analyzed the testing time, and applied the method to case studies. Through testing time analysis and case studies, we demonstrated that system checking through the W method is not up to par with our method, even if our method is time consuming. This indicates that the consistency between design  From Tables 4 and 9, we observed that the difference in the number of errors found by the two methods increases as the software becomes longer and more complex. This means that there are more post-processing errors, and our method becomes a better option for testing as the software becomes longer and more complex.

Conclusions
Testing is the most important part of software development, especially in safety-critical systems. Among numerous testing methods, FSM and the W method have been widely used.
However, the W method fails to detect post-processing errors in the IUT. To correct this deficiency, we proposed an enhanced fault-detection W method to generate test suites (sequences), analyzed the testing time, and applied the method to case studies. Through testing time analysis and case studies, we demonstrated that system checking through the W method is not up to par with our method, even if our method is time consuming. This indicates that the consistency between design and implementation can be verified explicitly by using our method. Furthermore, our method is more appropriate than the W method for software testing in safety-critical systems where the error-free From Tables 4 and 9, we observed that the difference in the number of errors found by the two methods increases as the software becomes longer and more complex. This means that there are more post-processing errors, and our method becomes a better option for testing as the software becomes longer and more complex.

Conclusions
Testing is the most important part of software development, especially in safety-critical systems. Among numerous testing methods, FSM and the W method have been widely used.
However, the W method fails to detect post-processing errors in the IUT. To correct this deficiency, we proposed an enhanced fault-detection W method to generate test suites (sequences), analyzed the testing time, and applied the method to case studies. Through testing time analysis and case studies, we demonstrated that system checking through the W method is not up to par with our method, even if our method is time consuming. This indicates that the consistency between design and implementation can be verified explicitly by using our method. Furthermore, our method is more appropriate than the W method for software testing in safety-critical systems where the error-free characteristic is more important than anything else.
Moreover, we showed that the difference in the number of errors captured by the two methods increases as the software becomes larger and more complex. This result inferred that our method is a better option for testing larger software. Thus, our method can be used to increase software reliability in safety-critical embedded systems.
In future studies, the proposed method will be applied to more case studies for further modification and generalization.