SW-VHDL Co-Veriﬁcation Environment Using Open Source Tools

: The veriﬁcation of complex digital designs often involves the use of expensive simulators. The present paper proposes an approach to verify a speciﬁc family of complex hardware/software systems, whose hardware part, running on an FPGA, communicates with a software counterpart executed on an external processor, such as a user/operator software running on an external PC. The hardware is described in VHDL and the software may be described in any computer language that can be interpreted or compiled into a (Linux) executable ﬁle. The presented approach uses open source tools, avoiding expensive license costs and usage restrictions.


Background
The verification of designs at the software, hardware and firmware level is one of the most important tasks to assure the correct functionality of a designed application. This can be especially critical in fields where a functional failure is costly in terms of economy or risk to human life, such as biomedical and aerospace industries.
At the firmware (also called gateware) level, there are multiple ways to verify the behaviour of a circuit, from the basic use of simulation tools to simulate specific parts of a design [1], or using these simulation tools following standard verification methodologies, such as UVM [2] or UVVM [3], to applying formal verification [4].
Simulation-based approaches allow the user to monitor the behaviour of his/her circuit when faced with given stimuli. This is also known as functional verification. When the complexity of the designs grows, testbenches also become more complex and thus require more time to be developed. For this reason, there is a high interest in the industry to reuse code for testbenches and streamlining the verification process. UVM and UVVM are simulation methodologies to describe testbenches and its components in a generic way, so that they can be reused for many different projects and allow automatic testing.
Formal methods are capable of proving that the properties of a given design hold, under specific conditions (such as inputs within an expected sequence of values), using mathematical methods, without the need to create complex testbenches.
As the complexity of the design increases, as can happen in the integration between software and hardware, the verification becomes more complex and therefore requires the use of more sophisticated tools to detect and correct errors in the application or design to be verified.

Problem of Interest
Nowadays, there are different tools to perform verification, such as Vivado Simulator [5] or Questa Advanced Simulator [6] for HDL simulation, which typically require the purchase of expensive licenses. Recently, free and open source software tools have emerged that are capable of simulating HDL designs with accuracy, speed and adherence to the HDL Standards that can rival many of the commercial tools (particularly the ones integrated with the software provided by FPGA vendors).
A revision of what can be done with these free and open source tools can be found in [7].
Basic simulation, without following any functional verification methodologies, has the advantage of requiring little engineering effort at first, but it does not scale well when testing complex modules or using complicated input sequences, requiring more and more engineer effort as complexity increases. For this reason, it is seldom used to simulate complete designs, but is instead used by the design engineers to make some quick verification of the most basic functionality of a module. When verification metrics are used, this verification method typically obtains poor results.
Functional verification methodologies improve on what can be done with simulation, by structuring testbenches so that engineering efforts are distributed between different testbench submodules which may also be reused. The main drawback of these methodologies is their steep learning curve: for example, if a VHDL designer wants to use UVM to verify a design, he or she must first learn Verilog, SystemVerilog and object-oriented programming, and then learn the UVM. Furthermore, the commercial simulators that support the versions of the HDL standards that these methodologies need (SystemVerilog in the case of UVM, and VHDL-2008 in the case of UVVM) are not distributed with the FPGA vendor tools and must be purchased separately. This makes designers, especially the ones whose target is an FPGA, not have the economic or time budget to use them. The reason being that these methodologies are mainly tailored for designs that are going to silicon, but designs whose target device is an FPGA (which would gain confidence if using these methodologies) often find that the designers do not need all the verification capabilities and their time/effort budget is not sufficient to warrant the use of these methodologies.
Formal methods are notably powerful for demonstrating circuit properties in small-to-medium design blocks. Due to the heavy mathematical load of these methods and the change in mindset that the verification engineer needs to undertake in order to use them, they typically require special training. As the complexity of the mathematical proofs increases exponentially with design complexity, these methods do not lend themselves well to checking full designs. In some cases, the formal models of the full designs may be simplified in order to reduce calculation time, which could create verification gaps if not done correctly.
Apart from the verification methods chosen, projects where integration between hardware, software and firmware is required often suffer from delays, which may risk the final project closure on time. A missing hardware or firmware part may block the progress of the software part and vice versa, resulting in a delay of the global project. The integration with external software is subject to the capabilities of proprietary simulators, and the users may be in a situation where they want to access a feature that is not covered by their current licenses. Furthermore, simulators that are provided with the typical FPGA vendor tools, which are the ones that the majority of users know how to use, do not support the VHDL or Verilog capabilities that these modern verification methodologies require. It would be desirable to have a verification environment whose capabilities were not limited by proprietary licenses, while being easy to use to verify a complex design where the integration of software and hardware is required.

Literature Survey
In the existing literature, there are several methods and tools to perform the verification of different kinds of designs or circuits. They can be classified into two categories: functional verification and formal verification.
Functional verification based on simulation can be done by stimulating the design under test with ad hoc test benches using a software simulation tool. Commercial simulation tools are used in [8], where genetic algorithms are used to define input patterns, which are fed to a VHDL simulator to drive the system to configurations that increase code coverage, improving the capabilities of the verification by simulation technique. In [9], the co-simulation of hardware and software for a system-on-chip is achieved by running the software part in a simulated hardware model. Co-verification allows verifying the complete final system, thus avoiding software development time delays due to missing hardware or mismatches between hardware architecture and software application.
Other approaches for functional verification, such as [10,11], use the standard methodologies, such as UVM or UVVM, to verify their designs. A recent study in FPGA verification trends in industry can be found in [12]. On the other hand, in academia, there seems to be a lower penetration of functional verification methodologies, and specific verification efforts are typically not mentioned in publications, except the ones that directly focus on verification [13][14][15][16]. Some researchers, such as [17], propose their own methodologies with reusable modules, resulting in testbench architectures similar to (but not compatible with) the standard methodologies.
If a higher level of confidence is needed, formal verification methodologies may be applied. This technique allows to mathematically prove or disprove properties of the circuit, without using simulations. A popular use is to check that specific error conditions will never occur in a module, given a set of restrictions in its inputs. For some circuits, another use case could be to compare the equivalence between the design under verification and a reference model that behaves as desired. A survey of techniques for formal verification for co-verification is collected in [18].
When the complexity of the designs increases, formal verification techniques are more difficult to apply. In these cases, approaches that combine different verification methodologies can be useful. Ref. [19,20] combine techniques of simulation and formal verification to reach a compromise between confidence and complexity.

Scope and Contribution of This Paper
This paper presents the design and development of a co-verification environment with both software and firmware components that allows to verify the functionality of an FPGA application that communicates with a software that runs on a PC, all using open source tools.
Although approaches like [9] also perform co-verification techniques, there is not a single one among the above proposed that performs co-verification where the software and hardware parts interact, but are independent of each other. In the case of [9], the software runs inside the simulated hardware, while in the proposed approach, the software runs on a PC and communicates with a firmware application running on an FPGA.
The proposed verification approach has a lower learning curve than standard methodologies, since it does not require learning a different language or constructing a very complex testbench in terms of transactions, drivers, monitors and agents.
Another advantage of the proposed approach compared to the rest found in the literature is the use of open source tools, which avoids licensing requirements that limit the capabilities of the software, allowing full access to the tool for the user.

Organization of the Paper
The paper is structured as follows: Section 2 describes the design under co-verification and how to construct the co-verification environment. Section 3 describes the test procedure. Experimental results obtained using the co-verification environment are shown in Section 4. Lastly, the discussion and conclusions are presented in Section 5. Figure 1 summarizes the proposed approach, test procedure and involved files and tools, which will be explained in the following sections.

Design under Co-Verification: The FTU-VEGAS System
The design under co-verification is the system logic of the FTU-VEGAS system, an FPGA-based fault injection platform developed by Universidad de Sevilla for the European H2020 project VEGAS. This fault injection platform is based on its previous version FT-Unshades2 [21], which allows to inject single event upsets (SEUs) on the user logic and configuration bits of an FPGA.
The architecture of the complete system can be seen in Figure 2. In the absence of any specific verification efforts, all the different elements of the platform must be available in order to test and verify the system logic, which is implemented in the software and firmware.
The elements that compose the design are described in the following sections.

The Hardware Device
The hardware side of the system is a custom PCI express dual-FPGA board. The two FPGAs are a Xilinx Virtex-5 and a NanoXplore BRAVE NG-MEDIUM. The first FPGA is called the Service FPGA, and the second one is called the Target FPGA. The purpose of the system is to study the behaviour under radiation-induced faults of the Target FPGA, which can be configured with different user designs. The Service FPGA must then contain all the necessary firmware to control the fault injection testing and associated on-board hardware: buses, communications, memory, etc. The hardware device also includes two on-board memories and a PCI express port to connect to a PC. The design and production of this hardware device consisting of a custom PCB with these two FPGAs require a considerable fraction of the project time, resulting in a bottleneck in the development process.

The FPGA Firmware
The firmware that runs in the Service FPGA is responsible for the low-level management of the fault injection testing and all on-board hardware. The different modules of the firmware exchange data using custom-made stream blocks that allow managing a flow of data whose width may change. The architecture of the firmware is shown in Figure 3, which also shows other on-board components that directly interact with it.
The main functional blocks in the firmware are: • Core: decodes commands from software, manages the other modules and sends responses to software. • Config: manages the configuration interface of the Target FPGA. • Vectors: manages the feeding of test vectors (stimuli and clock) for the user design.
Communicates with the SRAM controller to store the input and output vectors in order to minimize communications with the software. • SRAM controller: manages data transfer with one of the on-board SRAMs.

The Computer Software
The software runs on a computer to which the hardware is connected and interfaces with both the hardware and firmware of the system, interpreting and executing the user commands which can be read either from an interactive shell or from a script file. Most of the functionalities of the software require interacting with the firmware that is supported by the hardware. The software is called tntsh (test analysis tools shell), and its architecture is shown in Figure 4. The main functional modules of the software are: • tntsh: a tcl shell that interprets the user commands and can be used in batch mode. • libtnt-tcl: extends the tntsh functionalities by wrapping the C/C++ API of the internal modules, so that they can be accessed through the tcl shell.

Traditional Testing
Unit tests can be performed at all levels by building a test environment for every module developed, both at firmware and software source levels. Due to the complexity of the system, these unit tests may only cover small parts of the complete functionality. Many, if not most, of the functionalities of all system elements require the system to be at least partially complete, to cover all the functionality that depends on the interactions between the different system elements.
If no specific verification actions are taken, integration tests can only be run if we have the complete system, as shown in Figure 5. Waiting for the complete system to be available in order to begin a thorough testing of the system logic increases risk and may introduce delays in the project.
Hardware Firmware PCIe tntsh .tcl The integration tests use the Tcl API provided by tntsh to run a test script and compare the output with a model that is sometimes hand made, sometimes generated with an independent script, and sometimes generated with a previous version of the system and checked for errors. Unfortunately, they cannot be executed without the hardware.

Testing Software with No Hardware or Firmware
At some point in the development process, it was noticed that the software was getting ahead of the rest of the project, but could not be tested beyond the most basic unit tests because there was no platform to do so. The solution is then to provide a platform that does not require the same resources that the hardware does, as can be seen in Figure 6.  Since the communication protocol between software and firmware had been specified, it was possible to write a software module that behaved just like the firmware and hardware would. It could not be connected to a PCI express device, but since a PCI express device connected to a unix or unix-like server can become a device file in userspace, the presence of the device can be emulated by a pair of named pipes that connect the software and our virtual device.

Constructing the software virtual device
The software virtual device is a computer program that completes the following tasks: • Open the read and write pipes that connect it to the tntsh software.

•
While the pipes exist: -Read 1 byte from the input pipe.

-
Decode the read byte to obtain the command sent by tntsh.

-
Execute the command, reading from the input pipe any data needed.

-
Send the expected response to the output pipe.
• Exit cleanly, closing any open files and releasing any reserved memory.
If the software is well structured, multiple files can be shared between the software and the software virtual device, such as the header file with the command codes, communication protocol details, and internal data structures, which eases the construction of the software virtual device. Reusing these files is a tradeoff between development speed and verification capability.

Capabilities and limitations of the software virtual device
The software virtual device has many limitations: it does not actually simulate the Target FPGA; instead, it has a table where the bitstreams are copied and the values of individual bits can be queried and changed. For these reasons, only a subset of the software functions can be meaningfully tested. The simulation of the functionality inside the Service FPGA is not cycle-accurate, since no HDL code is being used. The emulation of the user design is also missing, since there is no simulation of any HDL. It does, however, handle all possible error conditions and returns the appropriate codes. Examples of these are sanity checks, such as reporting an error if the user tries to perform a single clock step without any vectors previously loaded. While this solution is interesting to test a number of software functions that need some interaction with the device, it cannot be used to verify any of the VHDL code of the firmware, since it is not involved in these tests. When enough firmware code is available, a firmware virtual device becomes preferable, making the software virtual device obsolete except for functional comparison purposes.

Testing Software and Firmware with No Hardware
As development advanced, the firmware for the Service FPGA grew, as did the required integration tests, but the development of the hardware lagged behind producing yet another bottleneck. With the software virtual device in mind, the same principles can be applied to remove the need for actual hardware in these tests. To achieve this, it would be desirable to have a simulation of the firmware HDL code that could manage the communication with the unix pipes. The components required to perform the integration tests in this case are shown in Figure 7.  Constructing the firmware virtual device In order to construct the firmware virtual device, a number of additions have been made to the firmware architecture. The modified architecture is shown on Figure 8, and it includes the following new elements: • Configuration interface simulation model: a simple model of the configuration memory of the Target FPGA that allows to write and read data from it. Compiling the first four modules with the rest of the firmware code, with GHDL [23], results in an executable file: the firmware virtual device. In case of testing any system features that need fault injection, the relevant environment variables must be set or passed so that the Python script is executed with the virtual device.

Capabilities and limitations of the firmware virtual device
The firmware virtual device solves the drawbacks that the software virtual device had: it includes a simulation model of the Target FPGA with fault injection capabilities, the simulation is cycle-accurate, and the user designs in the Target FPGA can be emulated. There is no possible verification gap due to code reuse, since all the code of the firmware virtual device is written in a different language than the software. The tradeoff is that the firmware virtual device requires longer execution times, since the VHDL simulation is more complex than the execution of the software virtual device.

Unit Tests
Before running the integration tests, it is always advisable to run all unit tests available. This saves time in case of an easier-to-detect error being inserted in the code, since it would not make sense to run the integration tests with erroneous code. The developed unit tests check the main functionalities of the different submodules of the system and have been automated using the GNU Make tool. Figure 9 shows the different components that must interface for every integration test performed:  • The software shell, which is the tntsh software of the project.

•
One or more virtual devices. •

One pair of named pipes per virtual device. •
A script that is executed in the shell (.tcl file). • A model output (.log) that is compared to the shell output (.log~).

Process
The procedure to run the tests is as follows: 1.
Create n pairs of named pipes, where n is the number of devices required to run the test (typically 1, but more are needed when testing multi-device functionality). These pipes will be used for communication between the virtual devices and the software shell.

2.
Launch n virtual devices, each one using a different pair of named pipes.

3.
Execute the script in the software shell. The script is the one responsible for opening the virtual devices through the names of the pipes. 4.
The output of the script is written to a file, e.g.,: testname.log5 .
When the software shell ends, the test environment must be cleaned: the processes for all virtual devices must be terminated and the named pipes used for communication must be removed. The Python part of the virtual device catches the SIGTERM unix signal in order to gracefully terminate the simulation, to avoid any possible data loss, such as code coverage files not being written. Table 1 summarizes the test procedure, which has been automated through the use of the GNU Make tool, showing the tools used to obtain the required components and perform the tests. As soon as it becomes available, the use of the firmware virtual device is preferred to the software one. Table 1.
Test procedure and software tools involved in test preparation, execution, debugging and automation. Step

Description Tool
Test preparation Step

Error Conditions
The test fails if one of the following conditions happen: • tntsh terminates with an error code or raises an exception.

•
The actual test output testname.log~and expected test output testname.log do not match.

•
The test does not terminate (see below).
Otherwise, the test is assumed to have succeeded. A potential pitfall for this system is that an error in the virtual device can result in a test that never ends if the software keeps waiting for a response that will never come. This can happen if the virtual device gives a shorter response than expected or is caught in an endless loop itself. A timeout can be implemented at several levels to prevent this: it can be implicit in the software shell itself-only in debug mode, in our case-or it can be implemented in the scripts that manage the test. Furthermore, assertions can be added to both software and HDL implementations of the virtual device to detect the most probable failure conditions. If a new failure condition is detected as a result of executing a new test, new assertions can be added as needed.

Debugging
The software virtual device can be debugged with any software debugger such as gdb, simply launching the application with it. Having launched this process, the rest of the test proceeds as usual, and it can be interrupted at any time by the user.
The GHDL virtual devices can be made to export waveform data. This can be done by passing the --wave GHDL option to the firmware virtual device executable. The generated files can be viewed with the open source waveform analyzer tool gtkwave.

Experimental Results
The co-verification environment, with the firmware virtual device, has been used as a verification platform for the firmware and software parts of the FTU-VEGAS fault injection system. A set of unit tests was performed to check the correct functionality of each element of the firmware and software parts. Afterwards, integration tests were executed, in order of increasing complexity. For each test, a comparison between the expected output and the actual output has been performed to detect differences between them, thus allowing to find bugs in the system and fix them. In addition, several fault injection campaigns have been performed using the virtual device, obtaining results as expected. These fault injection campaigns are also a type of integration test, since performing fault injection campaigns is the intended use of the system under co-verification, and in order to perform them, most of the functionalities of the system have to be used.
The code coverage metric has been obtained using GHDL and accumulatively measured after each test, so the increase in the metric with each new test can be measured. GHDL can generate line coverage metrics if compiled with the GCC backend. Figure 10 shows how the code coverage metric improves with each added test, where the green area corresponds to unit tests, the blue area to non-campaign integration tests and the red area to the fault injection campaigns performed. The final code coverage reached is 82.97% of the source code, which is a large quantity considering that no actions have been taken specifically to increase the code coverage metric. Further increases in this metric could be achieved by identifying and analyzing the unreached code, in order to develop unit or integration tests specially tailored to hit the missed lines.

Firmware Unit Tests
The set of firmware unit tests executed is summarized in Tables 2-5. The tests are executed in the order that they appear in the tables. An asterisk (*) in a test name in the Tables symbolizes a group of tests that result from executing the same test setup with different configurations. Nonetheless, all test names can be found in Figure 10. The coverage column in the Tables shows the accumulated line coverage metric up to the specific test.

Integration Tests
The set of Integration tests executed is shown in Table 6. As in the previous tables, the coverage column shows the accumulated line coverage metric up to the specific test.  Table 7 describes the designs for which a set of fault injection campaigns have been performed. For all designs, four campaigns have been made, using 1, 2, 4 and 8 devices in parallel. As in previous Tables, an asterisk (*) in a test name symbolizes a group of tests that result from executing the same test setup with different configurations, in this case with a different number of devices in parallel. As in the previous tables, the coverage column shows the accumulated line coverage metric up to the specific test.

Discussion
The proposed co-verification approach has been used to verify a complex design with a high level of confidence, obtaining very satisfactory code coverage results, in line with what can be expected using complex verification methodologies such as UVM or UVVM.
The main advantages of this approach are twofold: it bridges the efforts/results gap between simple ad-hoc testbenches and complex verification methodologies, and it does not require any proprietary software tool.
The approach has shown to be useful to methodically verify complex SW-VHDL co-designs, where the software and hardware parts interact, but are independent from each other. The approach is also fully automatable using any scripting or build tool such as GNU Make.
On the other hand, a number of simulation models for on-board hardware had to be developed to construct the firmware virtual device. Developing these simulation models is a tradeoff between test capability and verification effort, since each new simulation model added to the virtual device allows to test a bigger set of system functionalities. This is not a particularity of this approach, since at least basic simulation models of external interfaces or hardware are often required to perform high-quality functional verification.
Another advantage is that the integration tests are compatible with the physical hardware. This makes it possible to use the co-verification environment to reproduce bugs found while executing the system logic in the physical hardware, with the added debugging capabilities of a waveform viewer and software debugger.
A drawback of this approach is that it only supports VHDL code, since the GHDL simulator only supports VHDL simulation. While a firmware virtual device could be constructed with verilog code using verilog-only FOSS tools and simulators like Verilator or Icarus Verilog, currently there exist no free or open source simulators that support Verilog-VHDL mixed simulation, so this approach cannot be used yet in mixed-language designs.
One of the advantages of this approach is that it requires less effort to learn: the engineer can start adding tests as needed, adding different elements to the environment and virtual device architecture as soon as they become necessary. This reduces the learning curve as there is no need to have a complex testbench to begin performing tests. The proposed approach also allows mitigating delays introduced by a lack of hardware in complex projects.
Since the firmware under verification was highly structured by using custom stream interface blocks, the development of unit tests was made simpler, since the majority of the firmware modules use the same interface. In the case of a less structured design, unit test development would be expected to require more effort, depending on the complexity and number of module interfaces. This issue could be mitigated by writing less unit tests and focusing more on developing the firmware virtual device and integration tests, checking any gaps in the code coverage metric to identify untested functionality.
The advantages and disadvantages of different verification approaches, including the proposed approach, are summarized in Table 8.

Conclusions and Future Work
A co-verification environment has been proposed and developed to perform co-verification of the software and firmware of a complex system. This co-verification environment can be used in all the family of FPGA designs, which comprises FPGA designs that interact with software running on an external computer. This is a fairly common case of digital design that may not always require all the verification capabilities used in ASIC design, but would benefit from an easy-to-deploy verification methodology.
The co-verification environment has been successfully demonstrated and used to verify the FTU-VEGAS fault injection system. The use of the virtual device avoids bottlenecks in projects where the work of different development teams, which may evolve at different speeds, must be integrated.
The virtual device is developed using free and open source software tools, so it is not necessary to purchase licenses for its use, allowing the user to access all its capabilities for free. The proposed approach is extensible to multi-device tests, so that it can be used to co-verify multi-device systems. Furthermore, if a user wanted to co-verify a system with a huge number of devices running in parallel, fully taking advantage of the maximum number of processor cores in his/her computer, the cost in simulator licenses would be prohibitive if using any other different approach that required a commercial simulator.
Multiple bugs have been found and corrected in both software and firmware parts of the system without the need for hardware, by running unit and integration tests with the virtual device. The integration tests are fully compatible, without modification, with the real hardware.
The proposed co-verification methodology bridges the effort/results gap between basic simulation and ASIC-focused modern verification methodologies. Being fully applicable using free and open source software, it is available to companies, researchers and hobbyists. The methodology has also demonstrated that it can achieve very satisfactory values of code coverage.
Future work will include automating the generation of the firmware virtual device wrapper and developing abstraction layers for different communication protocols that do not get mapped to files in unix-like operating systems. Furthermore, synchronization schemas to use multiple computers to run simulations of a huge number of devices will be explored.

Conflicts of Interest:
The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Abbreviations
The following abbreviations are used in this manuscript: