3.1. Synchronized Working of the Sensors in the Android App
The Android application has been tested with four Android devices and with different number of sensors, sampling rates, and signals to obtain in each sensor. The first two Android devices were smartphones with basic features and running the version 2.3 of the Android operating system: Samsung Galaxy SCL (i9003) (Samsung, Seoul, Korea) and the Motorola Moto G (XT1032) (Motorola, Schaumburg, IL, USA). The former has a 1 GHz 32-bit Single Core Processor (TSMC, Hsinchu, Taiwan) and 478 MB LPDDR RAM and the latter has a 1.2 GHz 32-bit Quad Core Processor (TSMC, Hsinchu, Taiwan) and 1 GB LPDDR2 RAM. The third and fourth devices were a Wolder miTab ONE tablet running version 5.1 of the Android operating system with a 1.3 GHz Quad Core Processor and 1 GB RAM and a Samsung Galaxy J5 2017 smartphone with a Samsung Exynos 1.6 GHz 64-bit Eight Core Processor and 2 GB RAM running version 8.1 of Android. The aim of using these four devices was to study the constraints of the application as a function of the performance of devices not on the high end of the market.
The first test was the simplest and consists of using the ECG, EMG, GSR, and 9DOF sensors with a sampling rate of 10.2 Hz, 20-min duration, and synchronized transmission. The application worked correctly in the four devices without any interruption, delay, or block. As a result, it was proved that the application can communicate, represent, and store data from the 4 different sensors in a synchronized way.
The second test was with the four sensors for 20 min but with a sampling rate of 50.2 Hz for the ECG and EMG sensors and a different sampling rate of 10.2 Hz for the GSR and 9DOF sensors. While the Galaxy SCL suffered from several blocks before closing the application abruptly, the Motorola Moto G, Wolder miTab ONE, and Samsung Galaxy J5 were able to end the test.
The third test was also performed with the four sensors for 20 min but all of them working with a sampling rate of 50.2 Hz in a synchronized way. Neither the Galaxy SCL nor the Motorola Moto G worked properly. Therefore, with those two devices it is necessary to use the sampling rate of 10.2 Hz or reduce the number of Shimmer sensors. On the contrary, both the Wolder miTab ONE tablet and the Samsung Galaxy J5 smartphone worked correctly until the end of the test.
The fourth test was with the four sensors for 20 min, the ECG and EMG sensors at a sampling rate of 128 Hz and the GSR and 9DOF sensors at a sampling rate of 10.2 Hz. Similarly to the previous test, the Galaxy SCL and the Motorola Moto G did not work properly but the Wolder miTab ONE and Samsung Galaxy J5 worked properly until the end of the test.
We tested all the sensors individually with sampling rates ranging from 10.2 Hz to 128 Hz. All these tests were satisfactory as all the sensors worked properly without any interruption or block.
Lastly, we used the Samsung Galaxy J5 to make a performance testing of the app using the fourth sensors simultaneously, the ECG and EMG with a sampling rate of 128 Hz and the GSR and DOF with a sampling rate of 10.2 Hz. These rates for ECG, EMG, and GSR are suitable to monitor the physiological of a person. Regarding the 9DOF sensor placed on the steering wheel, the sampling rate of 10.2 Hz is suitable to obtain features to monitor driving performance. This final test lasted 60 min. The app starts up immediately as soon as the user taps on the app icon with no perceived delay. Once the app is running, it uses 13.5 MB of RAM memory. During the test, the battery temperature was below 31 ºC and the battery consumption after the 60 min was 5.5% of a full battery charge, being the capacity of the battery 3000 mAh. The app can run in parallel with other apps without any interference. Other apps such as WhatsApp, Gmail, and Samsung Gallery were used while our app was running in the background and it was retrieved to the foreground remaining in the same state that it was before bringing it to the background. While the app was running in the foreground, it was possible to select one of the four sensors that were sending data to observe their values and graphs in real time as described in
Section 2. The selection from one sensor to another was made smoothly. Regarding the DOF sensor, whose graphs and data cannot be seen altogether in the smartphone screen, the scrolling of the screen to see all the information was made smoothly. Besides, all the graphs with sensor data can be zoomed in and out smoothly by tapping on them.
When the service of the app receives the chain “stopall” from the PC simulator, the files with the data from all the sensors were closed. The transmission of the files from the app to the PC simulator is made through Transmission Control Protocol (TCP) socket. This socket is connection-oriented and guarantees the reliable reception of bytes in the PC simulator in the same order that they are transmitted from the Android app.
After the 60-min test, the files with the sensor data are quite big and it took 27 s to send the files through the socket by writing in a write buffer the lines read from the files, which were then read by a read buffer of the socket. That transmission time is not problematic for the application given that the driver simulator has stopped and the sensor data are sent to the PC simulator correctly. During the test, the Android smartphone and the PC simulator were connected to a LAN of 100 Mbps data transfer rate. If the LAN did not provide access, as the files are stored in the smartphone, they could be sent later as soon as the connection is recovered.
Through the simultaneous monitoring of the ECG, EMG, and GSR of a person, the application can be applied to many different environments such as patients at their homes or in hospital or people doing some physical work or sport, or, in general terms, while doing some activity which requires a certain level of physical or psychological effort such as driving. This monitoring of ECG, EMG, and GSR of a person could be completed properly with a sampling rate of 128 Hz for ECG and EMG and of 10.2 Hz for GSR, which requires a device with features similar or higher than the Wolder mi Tab ONE tablet. Thus, if the physical or psychological conditions are below some threshold, the user or some clinician can be sent a warning to act accordingly.
We aimed to use the Android app for the Shimmer sensors to monitor drivers in our simulator. The 9DOF sensor is placed on the steering wheel and a sampling rate of 10.2 Hz is appropriate. The same sampling rate of 10.2 Hz is adequate for the GSR sensor. For the rest of sensors (ECG and EMG), the proper sampling rate is 128 Hz. With these rates, the Wolder mi Tab ONE tablet and Samsung Galaxy J5 smartphone have been proved to work properly
3.2. Experimental Results with the Android App and Our Driving Simulator
In this section, we show the experiments carried out with the synchronized working of the driving simulator and the Android app for the Shimmer sensors. We made two sets of experiments. In the first set, we focused on the data extracted from the gyroscope and the variables computed by the Android app from these data. In the second set, we focused on the ECG, EMG, and GSR signals.
19 participants took part in the first set of experiments. From 4 out of the 19 participants, we did not use the gyroscope in their scenario routes so only data from the simulator were stored. From the remaining 15 participants, both the data from the gyroscope using the Android app and from the simulator were stored. The mean age of the participants was 26.68 years old with a standard deviation of 6.77 years old. The mean length of time for the participants having a driving license was 7.79 years with a standard deviation of 6.09 years. Only one of the 19 participants did not have a driving license although he or she was preparing to get it. The participants were asked to rate themselves with a number between 1 and 10 according to their experience in computer games and according to their experience in computer racing games. The mean value of the first ranked experience was 6.53 with a standard deviation of 3.43 and the mean value of the second ranked experience was 4.53 with a standard deviation of 3.23.
Each participant drove in four different scenarios of the simulator twice, the first time using automatic gear shift and the second time using manual gear shift. The four scenarios were developed as varied as possible and covering the different range of roads, traffic, and situations that have to be faced in real driving situations and require different driving skills. The first scenario was mainly urban with an interurban section consisting in a winding road with one lane per direction. The second scenario was interurban in a ring road in the surroundings of a big city and with a variable number of lanes per direction. The third scenario was urban in a through road on the outskirts of a medium-sized city. The fourth scenario was a monotonous route in a highway. The approximate time to complete the first, third, and fourth scenarios was 10 min and the approximate time to complete the second scenario was 20 min. As the participants completed each scenario twice, the approximate mean time to finish with all the scenarios was 100 min for each participant.
Before driving in the simulator, the participants had to make two questionnaires to have a subjective measure of their sleepiness at that very moment. These two questionnaires were the Karolinska Sleepiness Scale (KSS), and the Stanford Sleepiness Scale (SSS). Moreover, the participants made the Epword Sleepiness Scale (ESS), which evaluates the likelihood of falling asleep not at the time of making the questionnaire but in different real situations such as watching TV or being sat in a public place. Those questionnaires have been adopted in numerous research studies related to the clinical assessment of fitness to drive in sleepy individuals using a driving simulator [
39]. We aimed to relate the results obtained in these questionnaires with the driving performance computed with different variables.
The steering wheel adopted in the simulator was the Logitech G27 that also includes the gear lever and the clutch, brake, and accelerator pedals, making it possible a realistic driving experience in the simulator. The simulator scene is projected onto a big screen of dimensions 260 × 195 cm as shown in
Figure 9.
The simulator was configured to store the following vehicle data every second: instantaneous fuel consumption, speed, rpm, and position in the scenario. Moreover, the time and position in the scenario of all the committed traffic offences are stored. The implemented traffic offences were: collision with a pedestrian, collision with a vehicle, collision with a motorcycle, collision with a cyclist, collision with a still object, leaving the road, over-speed, under-speed, leaving a roundabout incorrectly, driving in a roundabout incorrectly, leaving a junction incorrectly, failing to comply with a stop sign, failing to comply with a yield sign, failing to comply with a traffic light signal, not respecting the safety distance, crossing a solid line in a road, not using the turn light in a turn, and not using the turn light while overtaking a car.
In the first set of experiments, the Android app for the Shimmer sensors stored the angular speed obtained from the gyroscope placed on the steering wheel. The angular speed was obtained from the gyroscope with a sampling rate of 10.2 Hz. From this speed, many variables were computed by the Android app as explained in
Section 2.3. For the data analysis, we used the software MATLAB (Natick, MA, USA), IBM SPSS (Armonk, NY, USA), and Excel (Redmond, WA, USA).
We first analyzed the influence of the type of gear shift on the speed and the variables extracted from the angular speed of the gyroscope. We aimed to test whether the gear shift has some influence on the driving style as, on the one hand, not all the participants have driven a car with the automatic gear shift in their driving history and, on the other hand, none of them is used to drive in a simulator.
We split the analysis into two parts, the first part grouping together the data from the two urban scenarios and the second part grouping together the data from the two interurban scenarios.
Figure 10;
Figure 11 show the mean speed of the 19 participants in the urban scenarios and in the interurban scenarios, respectively.
In the urban scenarios, there is some influence of the type of gear shift on the speed. 14 people drove faster with the automatic gear shift (participants #2, #3, #4, #9, #10, and #11 with more sensitive differences). Only four participants drove faster with manual gear shift but with a small difference with respect to automatic gear shift. Furthermore, for the interurban scenarios, the difference between the speed with manual gear shift and automatic gear shift is small, being the number of participants that drove faster with automatic gear shift equal to the number of participants that drove faster with manual gear shift. In the urban scenarios, gear shifts are more frequent so, logically, the driving differences using the two types of gear shift is larger in those scenarios.
We analyzed the relation between the gear shift type and the mean and the standard deviation of the angular speed obtained from the gyroscope for the urban and interurban scenarios.
Figure 12 and
Figure 13 show the mean angular speed of the steering wheel in the urban and interurban scenarios, respectively, of the 15 participants that used the gyroscope in the simulations.
With the exception of participant 13, the mean is quite similar with the automatic gear shift and the manual gear shift. The number of participants with a mean larger with the automatic gear shift is similar to the number of participants with a mean larger with the manual gear shift. The mean values in the urban scenarios are similar to the mean values in the interurban scenarios and, only for some participants, these values are larger in the interurban scenarios. The use of the manual gear shift did make participant #13 increase the mean angular speed because that gear shift distracted him or her from driving the car and he or she had to correct the trajectory with more motions of the steering wheel. Anyway, this lack of adaptation to the manual gear shift in the simulator was only present in that participant.
Figure 14 and
Figure 15 show the standard deviation of the angular speed of the steering wheel in the urban and interurban scenarios, respectively. In general, the value of the standard deviation is not influenced by the adopted gear shift. Whereas in the urban scenarios, eight participants have a larger standard deviation with automatic gear shift and six participants have a larger one with manual gear shift, in the interurban scenarios six participants have a larger standard deviation with automatic gear shift and seven participants have a larger one with manual gear shift. The value of the standard deviation is larger in the urban scenarios as the drivers have to make sharper turns in urban scenarios than in interurban scenarios.
We also analyzed the relation between the mean and the standard deviation of the angular speed of the steering wheel in the urban and interurban scenarios and the traffic offences committed. The time spent to finish a route varied from one participant to another as they drove at different speeds and some participants chose the wrong exit in some junctions and drive more distance than others. Because of this, we analyzed the number of traffic offences per second in each scenario instead of the total number of traffic offences. In
Table 1, the participants are ordered as a function of the decreasing number of traffic offences committed per minute in the urban scenarios and with the mean and standard deviation of the angular speed. In
Table 2, the participants are ordered with the same criteria but in the interurban scenarios.
It can be observed that there is no significant relation between the number of traffic offences per second and the mean and standard deviation of the angular speed for the urban and interurban scenarios.
Besides, we studied the percentage of mean values of the angular speed that lied within one of the five intervals considered: larger than 10°, between 10° and 7.5°, between 7.5° and 5°, between 5° and 2.5°, and lower than 2.5°. Studying these percentages for the users with larger and lower means, we did not find any significant data that could characterize them or relate them to the total number of traffic offences. Then, we analyzed the mean number of zero-crossing points per second in both the urban and interurban scenarios looking for some relation between them, the traffic offences, and the mean speed. We did not observe relations between these variables but what we did observe was that for the most participants, the mean number of zero-crossing points per second was larger for the urban scenarios than for the interurban scenarios, as shown in
Figure 16. In the interurban scenarios, the position of the steering wheel has to be modified less frequently than in the urban scenarios.
We conducted a detailed statistical study for each type of traffic offence in the urban and interurban scenarios looking for its correlation with the values of the mean and standard deviation of the angular speed obtained from the gyroscope and the speed, rpm, and mean fuel consumption in each simulation. In this study, first we used the Shapiro-Wilk test [
40]. This test is usually employed to verify if a set of data follows a normal or Gaussian distribution. If at least one of the two variables that one wants to correlate follows a normal distribution, the Pearson correlation coefficient can be used [
41]. For the Shapiro-Wilk test, the null hypothesis, which is aimed to be rejected, is that a sample
x1,
x2,
…,
xn comes from a normally distributed population. Once the test has been completed, if the
p-value is larger than α (significance level is set to 0.05), the null hypothesis is not rejected concluding that the data follow a normal distribution. Applying the Shapiro-Wilk test to the variables mean and standard deviation of the angular speed, speed, rpm and mean fuel consumption, we obtained
p-values of 0.23, 0.18, 0.45, 0.21, and 0.31, respectively. In all the cases, the values were larger than α (0.05) so that all the variables followed a normal distribution and we could correlate them with different types of traffic offences using the Pearson correlation coefficient.
The Pearson correlation coefficient is a measure of the linear correlation between two quantitative random variables and is independent of their measuring range. It is calculated as follows:
where
σXY is the covariance between
X and
Y,
σX is the standard deviation of
X, and
σY is the standard deviation of
Y. Its values lie within the interval [−1, 1], being 1 if there is a perfect positive linear relation between the variables, 0 if there is no linear correlation between them, and −1 if there is a perfect negative linear relation between them.
In our statistical study, we analyzed the data from driving with both automatic and manual gear shifts together. On the contrary, we have performed one independent analysis for the urban scenario and another one for the interurban scenarios. We took that approach as there are traffic offences that can only be present in the urban scenarios such as leaving a roundabout incorrectly and driving in a roundabout incorrectly as there are no roundabouts in the interurban scenarios in our simulator.
After calculating all the Pearson correlation coefficients between each one of the five variables (two obtained from the Android app for the Shimmer sensors and three obtained from vehicle performance in the simulator) and each type of traffic offence differentiating urban and interurban scenarios, we tested their statistical significance calculating the p-value. The p-value in this case is the probability of having obtained such values of the Pearson correlation coefficients supposing that the null hypothesis is true, that is, there is no linear correlation between the pair of variables. If the p-value is lower than α (typically 0.05), the null hypothesis is rejected and the statistical significance of the correlation value is verified.
Table 3 and
Table 4 show only the correlation coefficients that have led to
p-values lower than 0.05 for the urban and interurban scenarios, respectively. There are more significant correlations in the urban than in the interurban driving. The only valid correlation with a variable extracted from the gyroscope was the standard deviation of the angular speed in urban scenarios with the traffic offence of leaving the road. Hard turns of the steering wheel, which increase the value of the standard deviation of the angular speed and are typical of bad driving, are related to leaving the road in an urban environment. The other three verified correlations have included the traffic offence of not using the turn light in a turn, two cases in an urban scenario (for mean speed and mean rpm) and one case in an interurban one (for mean fuel consumption). In the urban scenarios, as the speed or the rpm increase, the probability of turning without using the corresponding traffic light also increases. Although it may seem that this traffic offence is not serious, it can cause dangerous situations or even traffic accidents due to the lack of information that the rest of the vehicles have. In the interurban scenarios, the variable correlated with the traffic offence of not using the turn light in a turn was the mean fuel consumption. It may seem that several variables, other than the mean fuel consumption, are more related to this traffic offence. However, the mean fuel consumption is representative of the driving style in interurban scenarios and was interestingly correlated with the traffic offence of not using the turn light in a turn in our experiments.
Following the study, we analyzed the relation between the number of traffic offences per second in the simulations and the length of time having a driving license, experience in computer games, and experience in computer racing games of the participants in the experiments. First, we verified that the number of traffic offences per second has a normal distribution with the Shapiro-Wilk test. Then, we obtained the Pearson correlation coefficient between the traffic offences per second and the three mentioned variables related to the driver experience. The p-value of these correlations was larger than 0.05 in the three cases so no significant correlation was found. A significant correlation between the number of traffic offences and the experience in computer games or computer car games would have meant a relation between this experience and the performance in the simulator. As the simulator has been developed to be as similar as possible to real driving and not to computer games, this lack of correlation can be considered a positive aspect of the simulator.
Next, we analyzed the relation between the subjective assessment of participants’ sleepiness through KSS, SSS, and ESS questionnaires and a measure of low vigilance in the simulations. We used a measure similar to the one proposed in [
34]. They related a lower number of zero-crossing points of the steering wheel to a less vigilant driver that does not make continuous direction corrections. They also related maximum values of the absolute angular angle with a drowsy driver that has to do particular hard turns.
From these measures, we considered a possible driving period of low attention if, in this period or the previous one, there was no zero-crossing point and a maximum value of the absolute steer angle has been obtained. We divided the simulations in periods of 5 s so that each period could be classified as having or not having possible low attention driving. With a sampling rate of 10.2 Hz for the gyroscope, there are 51 samples of zero-crossing and maximum value of the absolute angular speed in each period. We obtained the Pearson correlation coefficient between the number of low attention periods and the results obtained in KSS, SSS, and ESS. We also obtained the Pearson correlation coefficient between the number of low attention periods and the age, length of time having a driving license, and experience in computer racing games. The p-value of these correlations was larger than 0.05 for all except for the length of time having a driving license, whose Pearson correlation coefficient was 0.5594. The more experienced drivers had more low attention periods, which should be avoided in safe driving, in the simulations.
Finally, we studied the correlation between the number of traffic offences in all the scenarios and the results in the KSS, SSS, and ESS. The only Pearson correlation coefficient with a
p-value lower than 0.05 was with SSS, whose value was 0.5375. SSS is a scale of seven statements related to an increasing level of sleepiness. We found this valuable relation between a higher subjective assessment of the level of sleepiness and a larger global number of traffic offences although we did not find this relation with the number of low attention periods computed as in [
34].
In the second set of experiments, in which we focused on the ECG, EMG, and GSR signals, 6 participants took part. The mean age of the participants was 32.83 years old with a standard deviation of 16.82 years old. The mean length of time for the participants having a driving license was 20.83 years with a standard deviation of 14.85 years. They were required to drive in the scenarios twice, one time when they were rested and another time when they were tired. They were free to choose the time and day representative of these two physical states to drive in the scenarios.
Figure 17 shows the mean value of HRV obtained from the ECG sensor when the participants drove in the scenarios in rested state and distinguishing between the urban and interurban scenarios.
Figure 18 shows the similar information but when the participants drove in the scenarios in tired state. Most of the participants had bigger differences in the mean HRV, as a function of the scenarios (urban or interurban), in rested state than in tired state. For some participants, the mean HRV is bigger in the urban scenarios than in the interurban scenarios and for others is the other way round. That way, as there is a relation between the HRV and the physiological state, it can be asserted that in rested state most of the participants somehow adapted their physiological states depending on the requirements of each scenario while in the tired state the different scenarios did not make participants modify their physiological states significantly. There are big differences between participants so that an analysis to determine if the physiological or psychological state is suitable to drive would have to be individually specified from the learning of the relation between the HRV and the different scenarios and traffic states.
The electrodes of the EMG sensor were placed on the trapezius muscle. We chose that muscle as it is especially used while moving the steering wheel.
Figure 19 shows the mean value of EMG in mV (millivolts) for the six participants distinguishing between driving in rested state and tired state. For all the participants, the mean EMG was bigger in tired state so that the trapezius sends electrical impulses of bigger amplitudes as someone gets tired. The differences in the mean EMG in both states were very irregular depending on the participant so that a personalized threshold should be established to detect the transition from a physiological state suitable to drive to a tired state that would eventually lead to unsafe driving based on this parameter.
Figure 20 shows the mean value of GSR in kOhm for the six participants distinguishing between driving in rested state and in tired state. There are clear differences depending on the participants but two tendencies can be observed. On the one hand, participants #2 and #6 had a quite similar mean GSR in both states. On the other hand, the remaining participants had a clearly bigger mean GSR in tired state. For the latter, driving in tired state made them change the emotional state and so the skin resistance captured by the GSR sensor. Similarly to the ECG and EMG, a personalized analysis would need to be applied to establish a threshold in the GSR value to detect the transition from safe to unsafe driving based on this parameter.