Innovative Cost-Effective Embedded System to Enhance ECU Firmware Quality Through Remote Hybrid Testing in Automotive Domain

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

(1) The motivation and contribution are unclear. Specifically, in the Abstract, you pointed out that "Traditional Hardware-in-the-Loop (HIL) systems, used for software quality assurance, are costly and not always readily available"; however, the main novelty of this paper is the new workflow as shown in Figure 3(b) in comparison with Figure 3(a). In my opinion, your workflow employed a pre-test method, which increases the work steps, and also has not solved the mentioned HIL issue. If you want to prevent the HIL problems, your proposed scheme should have the ability to enable the same functionality of HIL and not have the disadvantages of HIL. Unfortunately, I cannot find these features for your framework.

(2) Too many old materials have been presented, such as traditional HIL systems and their advantages/disadvantages (too many citations [1][4][5][6][7] on pp. 2-3); use of low-cost platforms like Arduino for ECU testing (on p. 3); application of the V-Model and PLC in automotive electronics development (on pp. 4-5); existing testing workflows and theoretical basis for optimization (on pp. 6-9); BIOComProP_ECU Platform and communication protocol (on pp. 16, 21); protocol based on UDS standard (on pp. 21-22). In summary, the main contents cover 25 pages from page 2 to page 26. However, as mentioned above, 11 pages are old materials, which accounts for 44% of the whole paper, a ratio that is too large for a journal paper.

(3) I'm not sure whether all the figures are necessary to be presented in this paper. In my opinion, if a figure is not shown, and there is no influence on the continuous story and the complete structure of this paper, then it can be deleted. In this way, the main contribution can be clarified.

(4) I don't know why you use different styles for presenting the contents in Table 3. Both horizontal and vertical words are shown in this table. The reading is inconvenient.

Author Response

Comment 1: The motivation and contribution are unclear. Specifically, in the Abstract, you pointed out that "Traditional Hardware-in-the-Loop (HIL) systems, used for software quality assurance, are costly and not always readily available"; however, the main novelty of this paper is the new workflow as shown in Figure 3(b) in comparison with Figure 3(a). In my opinion, your workflow employed a pre-test method, which increases the work steps, and also has not solved the mentioned HIL issue. If you want to prevent the HIL problems, your proposed scheme should have the ability to enable the same functionality of HIL and not have the disadvantages of HIL. Unfortunately, I cannot find these features for your framework.

Response 1: The purpose of the hybrid testing system is explained below. The HIL testing step has not been eliminated; however, there are numerous cases where software reaching the HIL stage exhibits severe quality issues. Due to the limited experience of some software developers, their products—although free of build or linking errors and formally qualifying for ECU testing on HIL—may fail to execute properly on the ECU. Execution on the ECU microcontroller can hang for various reasons, such as Multiple Task Activation (MTA), Memory Protection Faults, or simply inactive communication on CAN or FlexRay buses. These nonconformities become evident only after the ECU is programmed with the software. Moreover, HIL equipment is a scarce resource for embedded software developers because of its high cost and limited availability, requiring advance scheduling for testing. When access is finally granted, valuable testing time is often wasted on software containing critical bugs. TestBench emerged as a necessity following an analysis of the traditional workflow and its optimized counterpart. It is a mini-system for ECU handling that enables verification of the software’s basic state—whether it runs normally, operates without functional errors, and maintains communication—while also allowing fault injection on the communication bus. TestBench does not replace HIL; rather, it serves as a faster pre-testing method that is highly accessible to software developers. This device is intended for embedded software engineers working on the implementation of the AUTOSAR communication stack. They do not require the full complexity of HIL systems, which include sensor and actuator functionalities. In fact, communication stack testing typically uses less than 10% of HIL’s capacity. The tools for evaluating bus communication (CAN and FlexRay) are the same as those used on HIL—namely, Vector CANoe and Vector CANalyzer—and are installed on the remote computer connected to TestBench. The previous explanation details the aspects considered important by the authors.

Comment 2: Too many old materials have been presented, such as traditional HIL systems and their advantages/disadvantages (too many citations [1][4][5][6][7] on pp. 2-3); use of low-cost platforms like Arduino for ECU testing (on p. 3); application of the V-Model and PLC in automotive electronics development (on pp. 4-5); existing testing workflows and theoretical basis for optimization (on pp. 6-9); BIOComProP_ECU Platform and communication protocol (on pp. 16, 21); protocol based on UDS standard (on pp. 21-22). In summary, the main contents cover 25 pages from page 2 to page 26. However, as mentioned above, 11 pages are old materials, which accounts for 44% of the whole paper, a ratio that is too large for a journal paper.

Response 2: State of art will be reduced, and materials, mainly older than 2022, will be removed. Paragraphs 301-350 will be integrated into 352.

Comment 3: I'm not sure whether all the figures are necessary to be presented in this paper. In my opinion, if a figure is not shown, and there is no influence on the continuous story and the complete structure of this paper, then it can be deleted. In this way, the main contribution can be clarified.

Response 3: Agreed, the number of figures that are of limited relevance will be reduced.

Comment 4: I don't know why you use different styles for presenting the contents in Table 3. Both horizontal and vertical words are shown in this table. The reading is inconvenient.

Response 4: Agreed, the header of Table 3 will be modified, and all words will be displayed vertically.

Reviewer 2 Report

Comments and Suggestions for Authors

Dear authors, I have the following comments:

1. Please specify the advantages of the proposed system over the currently available commercial HIL systems beyond just cost-effectiveness.
2. Please specify the purpose and the originality of this research.
3. The expenditure detailed in paragraphs (194-199) must be supported by the necessary sources and financial analysis.
4. Part 2 generally describes the product life cycle, which is well illustrated by the V-cycle. However, the automotive industry's interest in the proposed budget-friendly Arduino-based system remains poorly documented.
5. The accuracy of the proposed system needs clearer illustration. The precision analysis requires significant improvement.
6. The test setup section lacks the actual test data obtained from the conducted automotive test.
7. To demonstrate the functionality of the suggested system, you might want to conduct an experimental comparison against a commercial HIL system.

I hope my comments will assist in enhancing the paper.

 

Author Response

Comments 1: Please specify the advantages of the proposed system over the currently available commercial HIL systems beyond just cost-effectiveness.

Response 1: Due to the limited experience of some software developers, the software products they create—although free of compilation errors and, in principle, eligible for ECU testing within a HIL environment—may fail to execute correctly on the microcontroller. In practice, execution on the ECU microcontroller can become blocked (hang) for various reasons, such as Multiple Tasks Activation (MTA), the occurrence of faults within the Memory Protection Fault class, or improper functioning of communication over CAN or FlexRay buses. The TestBench system is not intended to replace the HIL platform; rather, it provides a rapid preliminary check of the software’s ability to run on the ECU by establishing the power supply connection, enabling the debugger, and configuring communication with Vector CANoe/CANalyzer through RestBus simulation. The main objective is to prevent unnecessary occupation of HIL resources with defective software versions that can be validated in a simplified manner on TestBench to confirm whether they start and execute correctly on the ECU. TestBench is a simplified system that enables remote connection of the ECU to the power supply while also facilitating communication between the ECU and RestBus simulation through Vector CANoe/CANalyzer. Additionally, it allows monitoring of software execution on the microcontroller using the Lauterbach debugger connected directly to the device. The main advantage of the TestBench platform lies in its high availability, in contrast to the HIL system, whose accessibility is significantly more limited.

Comments 2: Please specify the purpose and the originality of this research.

Response 2: It is possible that this aspect was not presented with sufficient clarity in the paper. The purpose of this research is to identify a cost-effective solution to reduce delays in the delivery process of embedded software for ECUs used in the automotive industry. These delays are primarily caused by the stages of testing, defect correction, and software retesting on the ECU unit. The originality of this work lies in conducting a dedicated study on hybrid testing, identifying an optimal solution, and implementing it in a remotely operable test bench built around a mini-system for ECU handling, referred to as TestBench.

Comments 3: The expenditure detailed in paragraphs (194-199) must be supported by the necessary sources and financial analysis.

Response 3: Information about the financial cost of HIL systems is not public. Each HIL system is custom-built and customized according to the hardware configuration of the ECU it needs to test. Another criterion for determining the cost is depending on the testing destination: in development centers (HIL) or in production for individual testing of each ECU on the assembly line (EOL – End of Line or Final Test). To test a production ECU, from the CBC (Central Body Controller) class, a HIL with high complexity is needed to control a larger number of inputs and outputs of the ECU. The cost of such a system, configured for an ECU with 59 inputs and 98 outputs, was in 2010 at 110,000 Euros, of which only the switching part (Fault insertion relay matrix) cost 42,000 Euros and was provided by the CGS (Computer Gesteuerte Systeme GmbH) organization. A dSpace HIL system intended for R&D areas for ECUs that control vehicle suspensions (so with a much lower number of inputs and outputs than CBC), at the level of 2025 is 160,000 Euros, of which only the cost for the switching part (Fault insertion relay matrix) exceeds 90,000 Euros.

Comments 4: Part 2 generally describes the product life cycle, which is well illustrated by the V-cycle. However, the automotive industry's interest in the proposed budget-friendly Arduino-based system remains poorly documented.

Response 4: The TestBench system, based on the Arduino platform, is primarily intended for research and development activities, serving engineers involved in embedded software implementation to enable rapid verification of its execution on the ECU microcontroller. Arduino is employed as a simplified solution for controlling the relays that supply power to the ECU from the power source. These aspects are discussed in detail in Chapter 4.

Comments 5: The accuracy of the proposed system needs clearer illustration. The precision analysis requires significant improvement.

Response 5: The TestBench system does not require high measurement precision, as its primary role is to facilitate the connection of the ECU to the power source via relays K1, K2, and K3 (analogous to the Fault Matrix Relay in a HIL system) and to configure the power supply parameters. Voltage measurement accuracy is not critical, since these readings are used solely as feedback for the engineer monitoring the states KL15, KL30, and KL30_P. TestBench provides the capability to verify software functionality on the ECU, as previously described.

Comments 6: The test setup section lacks the actual test data obtained from the conducted automotive test.

Response 6: As previously mentioned, TestBench does not perform testing at the level of a HIL system; rather, it is limited to verifying that the software executes without deadlocks and that communication over the CAN/FlexRay bus is functioning correctly. This aspect is assessed by monitoring the communication using third-party applications such as CANoe and CANalyzer.

Comments 7: To demonstrate the functionality of the suggested system, you might want to conduct an experimental comparison against a commercial HIL system.

Response 7: Both systems provide ECU power supply capabilities, and both TestBench and the HIL platform can inject faults into the communication bus. Additionally, each system is equipped with the Lauterbach debugger, which is used to monitor software execution on the ECU microcontroller via the specialized Trace32 graphical interface. Furthermore, both systems include the necessary hardware components for performing RestBus Simulation, utilizing the graphical interfaces of Vector CANoe/CANalyzer applications to manage communication with the ECU. Both platforms also employ the USB-KLine interface for ECU reprogramming with software versions delivered by embedded software developers. However, TestBench does not incorporate the specific modeling capabilities of a HIL system nor the relay matrix required for comprehensive fault injection across all ECU inputs and outputs. This limitation is irrelevant to its intended purpose, as TestBench is not designed to replace the HIL system but rather to provide a preliminary verification of software execution within the microcontroller, ensuring that it runs correctly and without deadlocks.

Comments 8: I hope my comments will assist in enhancing the paper.

Response 8: THANK YOU!

Reviewer 3 Report

Comments and Suggestions for Authors

Dear Authors,

Your article presents TestBench, an integrated and cost-effective test system for verifying automotive electronic control unit (ECU) firmware. The solution integrates local testing with remote access and fault injection capabilities to complement expensive conventional HIL setups. The manuscript provides a clear overview of the development process, including design, architecture, communication protocol, and results. The authors demonstrate that their approach can reduce development costs and time while supporting engineer training. This topic is relevant to Applied Sciences readers as it links embedded systems, automotive electronics, and process optimization, and offers added value in both industrial and educational settings due to its Arduino-based hardware. The methodology is described clearly and in detail, including hardware, firmware, and communication protocols with supporting figures. The comparison between traditional and hybrid workflows (Tables 1-2, Figures 4-5) effectively highlights improvements in terms of time and productivity. Some weaknesses remain:

• Experimental validation is mainly qualitative; quantitative data comparing the accuracy and reliability of TestBench with standard HIL systems is needed, particularly with regard to measurement accuracy, repeatability, and timing.
• The originality is moderate, as similar tests on microcontroller-based ECUs have been explored. The main novelty lies in the integration and optimization of the workflow, which should be clearly stated.
• Methodological transparency: The technical details are rich, but there is no concise methodological section describing the design, tests, validation steps, metrics, scenarios, number of experiments, and limitations.
• Literature coverage: The references mainly cite the authors' previous work. The inclusion of more recent studies on low-cost automotive testing or remote HIL platforms would improve the context.
• Language and structure: The article is understandable but dense; sections 4.1-4.4 contain excessive detail that could be summarized or added to an appendix.
• Altogether, the conclusions correspond to the results, but the validation is not sufficiently robust to support claims of broad industrial application.

Constructive feedback for improvement:

1. Clarify the novelty and scope of application, specifying what distinguishes the proposed TestBench from alternative low-cost HIL or Arduino-based solutions.
2. Highlight whether the contribution is methodological, technological, or organizational.
3. Enhance validation by including quantitative data such as measurement accuracy, latency, and comparison with HIL standards, as well as information on repeatability and errors.
4. Add a subsection on the validation procedure, indicating tests, conditions, and indicators, and mention sources of uncertainty.
5. Reorganize the structure by moving hardware and code details to supplementary material and simplifying repetitive sections.
6. Update the literature with recent publications (2022–2025) on hybrid testing, open-source HIL systems, and ECU platforms, connecting them to the state of the art in automotive software testing automation.
7. Improve the English language for readability, and standardize and enhance the figures.
8. In the discussion, expand the analysis on the practical use of the TestBench in industrial settings and reflect on future developments such as scalability, cloud integration, and cybersecurity.

The paper is technically strong and relevant but needs clearer performance validation, better integration with current literature, and a more focused presentation before publication. With these revisions, it could significantly contribute to cost-effective automotive ECU testing and remote validation systems.

Author Response

Comments 1: Experimental validation is mainly qualitative; quantitative data comparing the accuracy and reliability of TestBench with standard HIL systems is needed, particularly with regard to measurement accuracy, repeatability, and timing.

Response 1: As previously mentioned, TestBench does not perform testing at the level of a HIL system; rather, it is limited to verifying that the software executes without deadlocks and that communication over the CAN/FlexRay bus is functioning correctly. This aspect is assessed by monitoring the communication using third-party applications such as CANoe and CANalyzer.

Comments 2: The originality is moderate, as similar tests on microcontroller-based ECUs have been explored. The main novelty lies in the integration and optimization of the workflow, which should be clearly stated.

Response 2: In addition to the main contribution by defining the optimized workflow, a simple system was designed and implemented that facilitates downloading SW to the microcontroller from the ECU, allows monitoring SW in the microcontroller using the debugger attached to the ECU, and monitoring communication on the CAN / FlexRay bus.

Comments 3: Methodological transparency: The technical details are rich, but there is no concise methodological section describing the design, tests, validation steps, metrics, scenarios, number of experiments, and limitations.

Response 3: The essential steps for performing a basic verification of the embedded software on the ECU are as follows:

1. The embedded software engineer establishes a connection to the bench hosting the ECU under evaluation via Remote Desktop Connection and launches the graphical user interface (GUI) (see Fig. 10).
2. The ECU is powered through the activation of relays K1, K3, and K2, while its current consumption is visually monitored for a short period to ensure proper operating conditions. Typically, the current does not exceed 500 mA.
3. To download the software into the microcontroller, relay K4 is closed to supply power to the USB-KLine interface. Using dedicated software tools, the compiled software is then flashed into the ECU’s microcontroller. Initiating the download process requires triggering the Bootloader execution on the ECU, which involves resetting the microcontroller by briefly interrupting and restoring its power supply. This is achieved by pressing the “Reset Pulse” button. Normally, the entire programming process takes only a few minutes.
4. The debugger is activated by closing relay K5, after which the Trace32 graphical interface is launched and the corresponding software symbols to be monitored are loaded. The software execution on the microcontroller is started via Trace32. Sensitive variables are observed in the watch window, such as task counters and other critical parameters relevant to the new software version under verification (see Fig. 13 C3). If the counters fail to increment, this indicates severe software issues (bugs), and the software must be returned to the developer for correction. This stage corresponds to T4 in Figure 2.b.
5. If all previous steps are successful, RestBus Simulation is executed using CANoe or CANalyzer. At this stage, the signals transmitted by the ECU are monitored (see Fig. 13 C2). Any inconsistencies or missing signals require reverting to stage T4 in Figure 2.b. Conversely, if all checks are satisfactory, the software is transferred to the HIL system for final testing. At this point, the likelihood of non-compliance is significantly reduced, and the number of “test fail” loops between the software developer and tester is minimized.

Comments 4: Literature coverage: The references mainly cite the authors' previous work. The inclusion of more recent studies on low-cost automotive testing or remote HIL platforms would improve the context.

Response 4: Research on low-cost platforms applied in the automotive domain remains limited. Most studies involving Arduino in this field appear to be primarily educational or laboratory-oriented projects. Nevertheless, some authors have introduced notable contributions addressing this topic.

Comments 5: Language and structure: The article is understandable but dense; sections 4.1-4.4 contain excessive detail that could be summarized or added to an appendix.

Response 5: We will optimize the respective description

Comments 6: Highlight whether the contribution is methodological, technological, or organizational.

Response 6: It should be noted that the authors of this scientific work contribute from a technical perspective by emphasizing improvements to all aspects identified as critical in the design, development, and utilization of the equipment presented in detail throughout the paper.

Comments 7: Improve the English language for readability, and standardize and enhance the figures.

Response 7: We will take the received suggestion into consideration and implement improvements accordingly

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you. All my comments have been well addressed.

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the answers.

Reviewer 3 Report

Comments and Suggestions for Authors

Dear Authors,

Thank you for your revised manuscript and detailed responses. I appreciate the significant improvements, especially the clear explanation of TestBench's focus on functional verification, reflected in updated Sections 5 and 6. The expanded testing procedure, concrete operational steps, and improved diagrams have enhanced methodological transparency and readability.

The paper now better highlights its practical contribution: a low-cost, remotely accessible tool that aids developers and reduces iteration time, a valuable addition to current ECU early-stage testing practices.

For further improvement, consider explicitly contrasting your work with existing low-cost or hybrid testing approaches and expanding the literature review. While language and structure are much clearer, another round of editing would enhance clarity.

Overall, the major concerns have been convincingly addressed; only minor edits remain to strengthen clarity and contribution positioning.