VEHIOT: Design and Evaluation of an IoT Architecture Based on Low-Cost Devices to Be Embedded in Production Vehicles

Nowadays, the current vehicles are incorporating control systems in order to improve their stability and handling. These control systems need to know the vehicle dynamics through the variables (lateral acceleration, roll rate, roll angle, sideslip angle, etc.) that are obtained or estimated from sensors. For this goal, it is necessary to mount on vehicles not only low-cost sensors, but also low-cost embedded systems, which allow acquiring data from sensors and executing the developed algorithms to estimate and to control with novel higher speed computing. All these devices have to be integrated in an adequate architecture with enough performance in terms of accuracy, reliability and processing time. In this article, an architecture to carry out the estimation and control of vehicle dynamics has been developed. This architecture was designed considering the basic principles of IoT and integrates low-cost sensors and embedded hardware for orchestrating the experiments. A comparison of two different low-cost systems in terms of accuracy, acquisition time and reliability has been done. Both devices have been compared with the VBOX device from Racelogic, which has been used as the ground truth. The comparison has been made from tests carried out in a real vehicle. The lateral acceleration and roll rate have been analyzed in order to quantify the error of these devices.


Introduction
Given the high number of vehicle-crash victims, it has been established as a priority to reduce this figure in the transportation sector. For this reason, many of the recent research works are focused on including different control systems in commercial vehicles in order to improve their stability, comfort and handling [1][2][3]. These systems need to know in every moment the dynamics of the vehicle through variables such as longitudinal and lateral accelerations, yaw rate, roll rate, roll angle and sideslip angle, among others, when different maneuvers are performed, in order to actuate by means of the systems in the vehicle (brakes, steering, suspension) and, in this way, achieving a good behavior.
IMU sensors directly provide measurements of angular rates (yaw, roll and pitch rates) and accelerations (longitudinal, lateral and vertical accelerations) via rate gyroscopes and accelerometers. system was proposed for photovoltaic energy sources based on Raspberry Pi. The Raspberry Pi managed the data acquisition and controlled the system. In [27], the Raspberry Pi was used as a gateway for an efficient integration of wireless sensor networks with the Internet through IoT.
As can be seen, in previous works, these small computers are used for very specific works. In [28], the comparison of some of their inherent capabilities, such as connectivity and consumption, was performed under static conditions.
The novelty reflected in this work is the analysis of low-cost small computers and sensors under high dynamic conditions, by following specific best practices and considerations. The set of these devices is integrated with a specifically-designed IoT architecture oriented towards smart vehicles.
One of the main results of this research is the design of an architecture based on IoT, which integrates low-cost sensors and a small single-board computer to be mounted on a vehicle in order to carry out the estimation and control of vehicle dynamics, comparing the results with a high-end professional device. In this work, the low-cost sensor considered is an IMU due to it being the most common sensor used to estimate the vehicle dynamics. This research is focused on the comparison of two different low-cost systems (Raspberry Pi with IMU BNO0055 and Intel Edison with IMU LSM9DSO) in terms of accuracy, acquisition time and reliability. Both devices are going to be compared with the VBOX device from Racelogic, which will be used as the ground truth.
This article is organized as follows. In Section 2, the methodology is presented, including the experimental testbed design, experiments' definitions, and the data gathering and analysis, the experimental results and the calculation of the RMS error are presented in Section 3. Finally, in Section 4, the discussion and conclusion of the results and the method are exposed.

Methodology
In this section, firstly, the design of the experimental testbed defined to achieve the research goals is presented; then, the experiments to gather the data required are specified; and finally, the data gathered to analyze low-cost sensor kits' performance and reliability are presented, indicating the data analysis strategies for identifying relevant results and conclusions.

Experimental Testbed Design
The design of the experimental testbed considered for this research work is composed of three perspectives: hardware, software and communications.

•
Hardware perspective: The experimental testbed is an Internet of Things (IoT) architecture embedded in a vehicle. Although the complete architecture is packaged in a product that can be integrated in any vehicle, the Ford Fiesta was used.
To perform the comparative analysis properly, three kits of sensors were considered: -Reference or ground truth kit. This is composed of a VBOX 3i dual antenna data logger [29] having an IMU (Inertial Measurement Unit) connected and a dual antenna for incremental GPS from Racelogic. To accurately measure the roll rate, the two antennas form a 90-degree angle with respect to the traveling direction. For IoT kits' synchronization and data gathering, the VBOX controller is connected to a laptop embedded in the vehicle. The actual specifications of the ground truth kit hardware elements are detailed in Table 1.  [32]. The actual specifications of the Raspberry kit hardware elements are detailed in Table 2.  [34]. The actual specifications of the Intel kit hardware elements are detailed in Table 3. The IMU and the low-cost sensor kits were located in the gravity center of the vehicle considered in the testbed as is represented in Figure 1. These three kits were also interconnected using a WiFi router to manage the communications among them to synchronize the experiments and gather the information ready to be compared. According to [18], the accurate determination of IMU and controller position is essential for enhancing the precision of low-cost kits. • Software perspective: A software architecture was designed to gather the data provided by the sensor kits in a synchronized way and to provide the datasets necessary to analyze the precision and performance of each kit considered for this research work. The main components of this architecture are shown in Figure 2).
The Experiments Manager Componenthas the responsibility to synchronize the data-gathering of the kits taking part in the experiments. The specific classes included in this component are:

1.
Experiments Managerthat includes the user interface to let the researcher send a signal to the experimental kits for starting and finishing an experiment in a synchronized way.

2.
Communications Client that has the responsibility of managing the communication between the Experiments Manager and the Communications Server running in each experimental kit. The possible signals that can be sent from the Communications Client in the Experiments Manager and the Communications Servers in experimental kits are: 0, shutdown experimental kit; 1, keep running the experiment; 2, start the experiment; and 3, end the experiment and save the data in a file.

3.
Dataset Storage Manager that is in charge of taking the data coming from the kits and storing them in a CSV file. The information stored includes gyroscope and accelerator data gathered considering a sampling rate of 100 Hz.

2.
Communications Server that is in charge of receiving the signals from the Experiment Manager and sends the Experiment Launcher the order to start a new experiment. Moreover, this class sends the experiments' ending order to the Experiment Stopper class to process the information gathered during the experiment.

3.
The Experiment Manager class is in charge of creating an empty data structure in RAM memory when it receives the start experiment signal from the communications server and sending the data structure having the data gathered during an specific experiment to the Experiments Manager for storage purposes when it receives the end experiment signal.

4.
The Sensors Handler class has the responsibility of registering data items from sensors attending to the pre-configured sampling rate (100 Hz for this research work). The data are obtained from the gyroscope and accelerometer drivers that were obtained from GitHub [35,36].
The Raspberry Pi Software component has the same class structure as the Intel Edison Software Component, but it was developed in Python because the available drivers for low-cost IMU were only available in this language.
The VBOX Component is in charge of gathering the information provided from the Racelogic IMU sensor and GPS dual antenna data.
• Communications perspective: In order to provide homogeneity in communication among the different items that compose the solution and ensure synchronization when performing the different tests, a communications architecture has been defined.This architecture is shown in Figure 3. Given the nature of Racelogic VBOX devices, they need to be physically connected via a cable to the experiments managerand among themselves. However, both Raspberry Pi 3 and Intel Edison come with wireless communication interfaces that ease the connectivity among components and allow one to locate them in virtually any place of the vehicle without worrying about setting up specific data cables. Even more, the sensors used by these low-cost platforms are straightforwardly attached to the development boards by using the GPIO ports. By means of a wireless (802.11 g) access point, they can be connected to the experiments manager, which signals to them their operation mode via a TCP socket connection.

Experiments Definition
The hypotheses to evaluate during this research work are the following: • H1: The precision of low-cost sensor kits is similar to the precision provided by expensive experimental kits (i.e., VBOX-based kits). • H2: The performance and reliability of the low-cost sensor kits (i.e., Raspberry Pi and Edison Kits) is similar to the performance and reliability provided by expensive experimental kits (i.e., VBOX-based kits).
To evaluate the previous hypothesis, eight controlled experiments were executed (see Table 4. The experiments considered typical maneuvers such as J-turn and lane change maneuvers. These maneuvers are the most common used to test the vehicle's behavior. Furthermore, a long test simulating a general execution has been done. In the experiments carried out, the lateral acceleration and the roll rate are the variables that have suffered higher variation, and for this reason, these variables are considered in order to analyze the accuracy of different devices. The experimental tests have been carried out in Leganes (Madrid, Spain) on a commercial vehicle and a Ford Fiesta, as can be seen in Figure 4 in a period were the setting had no traffic restrictions interfering with the appropriate execution of the considered experiments.

Data Gathering and Analysis
The data obtained for the sensors considered for the previously-defined experiments were: For each experiment, the controller of each kit stored a CSV formatted file identifying the experiment and its execution date and time. The variables considered were the lateral acceleration measured by the accelerometer and the roll rate measured by the gyroscope included in each kit for the three axes. The measures were gathered according to the sampling rate stated for the experiments, which was 100 Hz. Figure 5 presents an example of the data gathered in each experiment. To determine the accuracy of the values obtained from low-cost kits, the root mean square (RMS) error was calculated. The maximum error comparing each point has also been calculated. This comparisons has been done using Racelogic VBOX as the ground truth. Section 3 presents the results obtained in the different tests.

Threats to Validity
In order to analyze the validity of the results obtained in this experimental work, several threats were considered prior to the experiments' execution:

•
Internal validity is the extent to which a causal conclusion based on the experiments defined is appropriate avoiding the introduction of systematic errors in the data used to determine the results and conclusions.

•
External validity is the extent to which the results of a study can be generalized to other situations.
(A) Internal validity: In this research, the only factor contributing to internal validity is related to the specific sensors used to configure each low-cost kit and the software components implemented to manage the required data: • The first threat was mitigated using two different kits for each type in order to prevent errors produced by sensors providing incorrect values. Even more, all the kits considered were tested in a static environment configuring the corresponding calibrations to assure that specific sensors included were providing appropriate data.
• Regarding software components, the possible threats were mitigated implementing an exhaustive unit testing process to ensure that each functionality properly processes the received values and the synchronization among devices is correctly implemented.

•
To verify the validity of the results, three similar tests for J-turn and double lane change maneuvers have been carried out.
(B) External validity: In the scope of this research, the factors that influence the external validity are related to the replication of this experiment. These replications must consider several relevant factors: sensors and controllers included in the experimental kits, their location in the vehicle and road conditions:

•
Regarding sensors and controllers, the possible threats were mitigated using sensors and controllers available on the market having regular features [37][38][39]. In this sense, the conclusions obtained are valid for the low-cost sensors currently available on the market, and as the technology is always improving the prior conditions, the conclusions can be used for future low-cost sensors.

•
Regarding vehicle conditions, the threats were related to the appropriate location of experimental kits in order to ensure equal conditions among them. This threat was mitigated creating a box to put the three sensor kits in a 3D printed box that was located in the vehicle gravity center. These issues are important to replicate this experiment properly in other vehicles. In this research, as the experiments have been carried out with a real vehicle, it has been difficult to maintain the same exact driving conditions related to steering wheel angle and velocity. In spite of this problem, similar tests for typical maneuvers have been carried out.

•
Regarding road conditions and experiments execution, the threats were related to the representativeness of the scenarios considered. This threat was mitigated considering a road without relevant slope variations and including different experiments with different types of directions, constant and variable speed.

Experimental Results
As is indicated in Section 2, a Ford Fiesta was used for this research (Figure 1). For the experimental tests, a total of eight driving maneuvers was carried out. Table 5 shows a summary of the successful tests. For both VBOX and Intel Edison, the percentage of successful tests is 100%. However, the percentage of successful tests for Raspberry Pi is 37.5%. This result together with the connectivity problems suffered previously in the performance of tests indicates a low reliability of Raspberry Pi compared with the other devices. In Table 6, the vehicle speed for the three successful tests in three devices is shown.  Figure 6)

Test 1: Roundabout
This first test is carried out with the vehicle taking a roundabout with a radius of 22 m (see Figure 7) at a constant speed on dry pavement. The car takes the roundabout with a velocity of 31 km/h. Figures 8 and 9 show the lateral acceleration and roll rate data, respectively, obtained from the lateral accelerometers and roll rate sensors contained in IMUs installed on Raspberry Pi (blue points) and Intel Edison (green). In order to prove the accuracy of these sensors, they have been compared with the data obtained from the IMU of VBOX, which is considered as the ground truth. It can been seen that the behaviors of the sensors in the three different devices are very similar.
To quantify the accuracy of the sensor, the norm, RMS (root mean square) and maximum errors have been calculated. The norm error as a function of time is calculated as follows [5]: where: φ GT represents the ground truth data, φ lc represents the low-cost sensor data and µ GT is the mean value of the ground truth data obtained during the period T . In Table 7, the error values are given. To verify the validity of the results, three similar tests for the J-turn maneuver have been carried out. To quantify the dispersion of data values, the standard deviation has been included for the RMS error (see Table 7). The results show that the errors are higher for the Intel Edison than the Raspberry Pi. Concerning the norm and RMS errors, the difference is about 1%, 0.01 g's and 3%, 0.02 • /s for lateral acceleration and roll rate, respectively. For maximum errors, the difference between them is higher (about 0.2 g's and 6 • /s, for lateral acceleration and roll rate, respectively). This is due to the Intel Edison being more sensitive to noise. In Figures 8 and 9, a high scattering for the Intel Edison IMU is observed.

Test 2: Double Lane Change
This second test is carried out with the vehicle doing a slalom maneuver at 79 km/h on dry pavement as shown in Figure 10. In this kind of test, the lateral acceleration varies very fast, so the sampling frequency of the devices and sensors can be checked. Figures 11 and 12 show the lateral acceleration and roll rate data, respectively, obtained from the lateral accelerometers and roll rate sensors contained in IMUs installed on Raspberry Pi (blue points) and Intel Edison (green). In order to prove the accuracy of these sensors, they have been compared with the data obtained from the IMU of VBOX, which is considered as the ground truth. It can been seen that the behavior of the sensors in the three different devices is very similar. Furthermore, it can be seen that the two devices are able to sample the signal sufficiently to show no differences with the Racelogic VBOX device.
To quantify the accuracy of the sensor, the norm, RMS and maximum errors have been calculated. In Table 8, the error values are given. To verify the validity of the results, three similar tests for the double lane change maneuver have been carried out. The standard deviation for the RMS error is shown in Table 8. Results show that the errors are higher for Intel Edison than Raspberry Pi. Concerning the norm and RMS errors, the difference is about 20%, 0.04 g's and 65%, 2 • /s for lateral acceleration and roll rate, respectively. For maximum errors, the difference between them is higher (about 0.5 g's and 57 • /s, for lateral acceleration and roll rate, respectively). This is due to the Intel Edison being more sensitive to noise. In Figures 11 and 12, a high scattering for Intel Edison IMU is observed; it also can be seen that the maximum error for roll rate on Intel Edison is 66.0722 • /s. This error is due to the noise that causes atypical data, as can be see in Figure 13.

Test 3: General Circulation
Compared to the tests described above, in this case, there are not only severe maneuvers, low and medium speed circulation and smooth movements are performed. At the end of the trial, two roundabouts (J-turn maneuver) and a Lane Change (LC maneuver) at a speed suitable for the testing environment have been made. It should be considered that in this test, low speed movements and high speed severe movements have been made.
This last test is carried out on the route shown in Figure 14. This test simulates a normal circulation behavior. Several curves were taken, and the vehicle was at the most appropriate speed for the road and the situation. The speed range on this test was from 15 km/h-75 km/h (see Figure 6). Figures 15 and 16 show the lateral acceleration and roll rate data, respectively, obtained from the lateral accelerometers and roll rate sensors contained in IMUs installed on Raspberry Pi (blue points) and Intel Edison (green). In order to prove the accuracy of these sensors, they have been compared with the data obtained from the IMU of VBOX, which is considered as the ground truth. It can been seen that the behaviors of the sensors in the three different devices are very similar.
To quantify the accuracy of sensor, the norm, RMS and maximum errors have been calculated. In Table 9, it can be observed that the errors are higher for Intel Edison than the Raspberry Pi. Concerning the norm and RMS errors, the difference is about 2%, 0.007 g's and 18%, 0.35 • /s for lateral acceleration and roll rate, respectively. For maximum errors, the difference between them is higher (about 0.3 g's and 5 • /s, for lateral acceleration and roll rate, respectively). This is due to the Intel Edison being more sensitive to noise. In Figures 15 and 16, a high scattering for the Intel Edison IMU is observed. However, the error on this test is lower than the other two. The reason is that the highest value for lateral acceleration was reached, so the influence of noise decreases compared to the measured value.

Discussion and Conclusions
The results can be used to design, implement and test an efficient, versatile and scalable low-cost hardware/software architectures able to be integrated on commercial vehicles. Even more, by using this sensor fusion approach with enhanced semantics, it may be possible to perform real-time estimation and control for more secure driving.
The following discussion is focused on the precision and performance similarity among precision and performance in both experimental low-cost and ground truth kits.

Precision
The obtained data show that low-cost sensors are more prone to noise. One reason is that their measurement range is higher than the VBOX ones. In case a noise reduction for the low-cost kit captured data were required, with the consequent approximation to the VBOX results, it may be possible to integrate filters via software, as the low-cost controllers can assume such a computational cost reducing the underutilization of their multiprocessing capabilities. Despite the noise influence, the average RMS error in BNO055 is 0.05 g's (0.4905 m/s 2 ) for lateral acceleration and 2 • /s (0.0349 rad/s) for roll rate, and the average RMS error in LSM9DSO is 0.07 g's (0.6867 m/s 2 ) for lateral acceleration and 3 • /s (0.0523 rad/s) for roll rate.
Furthermore, it has been found that low-cost accelerometers provide better precision than gyroscopes. A feasible justification is that the relative position among sensors guarantees the same acceleration measurement for the three devices. BNO055 has been proven to capture better results than LSM9DSO (with an approximate error of 0.02 g's (0.1962 m/s 2 ) and 1 • /s (0.01745 rad/s) less for BNO055). Notice that BNO055 is more expensive than the later one. As future work, it is planned to integrate BNO055 in the Intel Edison setup via the Sparkfun GPIO expansion board and also to integrate software estimators and controllers and analyze their performance in small computers using the information captured by the low-cost sensors.

Reliability
After concluding the experiments, a problem was identified regarding Raspberry Pi 3. A significant number of tests was not valid. It was concluded that most of these failed tests resulted from the lack of true parallelism in the Python main stack (to get true parallelism, it is necessary to call low level C routines [40]). Even more, BNO055 was attached to Raspberry Pi 3 in a circuit board without industrial-grade soldering connections, which can induct noise and additional impedances in the circuit.
Finally, VBOX can provide a sustained 100-Hz capture rate, as accelerometer and gyroscope sensors for the low-cost versions; however, given that the magnetometer is embedded and also initialized (but not used) with the previous two, the sampling rate decreases to 50 Hz. In any case, experiments determined that 50 Hz is enough sampling frequency to perform reliable experiments for the given case [41].

Factors to Replicate and Evaluate Experiments Using Low-Cost Sensors
Some important aspects have been determined in order to properly replicate and evaluate the experiments of the study: • Vehicle: The chosen vehicle perfectly fulfills the established requirements. This work is focused on commercial vehicles, so the tests must be carried out in one vehicle of these characteristics, in order to be exposed to the same control systems integrated in the vehicle (ESC, ABS, etc.).

•
Track: To take a proper measurement of sensors' reliability, the ideal scenario consists of having a test track without great camber and slope variations, as they can interfere with the collected data because of the lack of capacity to directly measure them.

•
Hardware: To obtain accurate values, it is necessary to properly fix the setup position inside of the vehicle. It must be as close as possible to the vehicle's center of mass. In addition, the relative position between the devices should be as small as possible, and the sensors must be aligned on the axis according to the characteristic to be measured (acceleration and angular velocity).
Regarding hardware, as indicated in [18], effective mechanical and software calibration of low-cost IMU devices is important to reduce the bias and noise in this kind of experimental kit. The use of VBox equipment was essential to complete this step by appropriately analyzing the reliability of low-cost sensor kits. Moreover, the integration at the hardware level assures higher reliability levels, avoiding kits without industrial-grade soldering connections. • Software: Regarding synchronization and capture software components, the preferred environment consists of using a well-proven and efficient programming language that allows one to take direct advantage of parallel hardware features, such as C or C++, in conjunction with efficient compilers that outcome lightweight and optimized binaries. This increases the results' reliability and the experiments' performance. Furthermore, it is important to define a unified test kit that allows one to verify, prior to starting the experiments, that all the elements in the setup are behaving properly.