Human – Machine Interaction in Driving Assistant Systems for Semi-autonomous Driving Vehicles

: Currently, the existing vehicle-centric semi-autonomous driving modules do not consider the driver's situation and emotions. In an autonomous driving environment, when changing to manual driving, human – machine interface and advanced driver assistance systems (ADAS) are essential to assist vehicle driving. This study proposes a human – machine interface that considers the driver's situation and emotions to enhance the ADAS. A 1D convolutional neural network model based on multimodal bio-signals is used and applied to control semi-autonomous vehicles. The possibility of semi-autonomous driving is confirmed by classifying four driving scenarios and controlling the speed of the vehicle. In the experiment, by using a driving simulator and hardware-in-the-loop simulation equipment, we confirm that the response speed of the driving assistance system is 351.75 ms and the system recognizes four scenarios and eight emotions through bio-signal data.


Introduction
Recently, there have been a few studies on human-machine interaction applied to autonomous vehicles [1,2]. An advanced driver assistance system (ADAS) is a system that assists drivers in driving in various ways. Until now, few studies on human-machine interaction for vehicle control systems using the driver's situation and emotion have been presented. Jeon et al. [3] researched the effect of drivers' emotional change on vehicle driving and control ability, and Izquierdo-Reyes et al. [4] designed vehicle control systems in a new aspect through research that analyzed driving scenarios and emotions for autonomous driving and driver assistance systems. Grimm et al. [5] presented studies on the interaction between a driver and a vehicle. The complementary and necessity of changing vehicle driving and control ability according to circumstances and emotions were confirmed in the previous study. However, detailed research is required for the advancement and integration of the human-machine interaction and the supplemented vehicle control module. Also, an accurate method and an analysis applicable to the existing vehicle system are necessary.
When conducting autonomous driving research with actual vehicles, a simulator in a virtual environment is often used to avoid problems such as human casualties, experimental equipment, and high cost. The 3D virtual simulator is not affected by limitations such as objects, weather, space, and experimental cost. Moreover, it makes it easier to set the scenario necessary for research by collecting various data. A Car Learning to Act (CARLA) simulator that does not have physical time and place restrictions is used when developing autonomous vehicles [6][7][8][9]. CARLA is being applied in multiple research fields using virtual vehicle driving simulators based on 3D game engines such as Unreal

Related work
In recent research on driver-machine interactions, the use of various bio-signals has seen a significant increase in applications. Sini et al. [19] used facial expression-based biosignals to smoothen the transition from manual driving to autonomous driving. Conveying passengers' intentions and emotions to the system provides driving decisions that are closest to the passenger's intention. Kamaruddin et al. [20] proposed a warning system for accident prevention using driver voice-based bio-signals. By comparing various driver behavior states (DBS) by applying the proposed method, it was confirmed that the existing vehicle control systems could be improved. However, there are many difficulties in switching vehicle control in semi-autonomous driving [21][22][23].
Du et al. [24] studied manual driving and the driver's emotions in situations by measuring the time to change vehicle control according to the driver's emotions in semi-autonomous driving. A study on the effect of high and low driver emotional states in terms of the control performance for manual driving was also conducted [25,26]. When the driver's emotions were positive, the concentration was high while driving, but the reaction speed was slow. When the emotional state was negative, it showed low concentrations and a high reaction rate [27]. Moreover, when the driver listened to happy music, the driving speed showed greater shifting and more degradation of steering control than the sad music [28]. According to the AAA Traffic Safety Foundation survey responses, 80% of drivers showed anger and aggression while driving [29]. Through the analysis of driving data, it was found that emotional states, including anger, sadness, crying, and emotional anxiety, increase the likelihood of a vehicle crash by 9.8 times [30], and when the emotional state is that of anger, cases of speeding and traffic rule violations and risky, aggressive driving increase [31,32]. It has been confirmed that accidents occur due to the driver's inability to control emotions while driving a vehicle [33], and because of the low accuracy of emotion recognition, it is necessary to combine human-machine interaction and driver assistance systems. Recently, autonomous vehicle technology has attracted a combination of human bio-signal-based emotion recognition [34,35] and subdivided vehicle control systems. A study on how to prevent and reduce accidents using driver behavior and emotion recognition is required to achieve perfect intelligent vehicle control for autonomous vehicles [36]. The research mentioned above is often conducted in a virtual driving simulator environment to reduce the risk of accidents in driver-based vehicle driving. In a driving simulator, it is easy to obtain reliable research data [10]. Bio-signals that are greatly affected by the surrounding environment are advantageous for studying the driver's condition according to the situation. Research is actively conducted to experiment without a real vehicle by linking the ROS middleware with the game engine. [37]. The former study confirmed that driving ability changes according to the driver's emotional state, but did not present a driver assistance system to which this was applied [30][31][32][33]. The latter study implements autonomous driving using a driving simulator, but it has a drawback in that it lacks driver assistance functions according to the driver's emotional state [34][35][36].

Proposed Method
This study proposes a human-machine interface that recognizes the driver's situation and emotions based on bio-signals to enhance the ADAS. The driving assistance system consists of an HMI, an ECU, and a controller. The HMI is based on the driver's bio-signals, and the situation recognition result is classified through the 1D CNN model. The ECU board transmits the result data extracted from the HMI, and the accelerator and brake values are measured by the controller during the manual control of the driving simulator. ROS manages the data received from the ECU and the simulator, processes the vehicle's throttle and brake values in ros_node, and transmits the data to the simulator to control the vehicle speed. A CARLA ROS bridge is used for data interworking with the CARLA server. If there are no HMI result data, the ECU board does not transmit data, and the vehicle is switched to manual operation; further, the vehicle is controlled using the controller. The overall flowchart is shown in Figure 1.

Semi-Autonomous Driving
The vehicle information and sensor data were managed using the ROS. The ROS bridge was used to link the CARLA simulator and ECU data. The data generated in the CARLA simulator are transmitted through the CARLA ROS bridge, and the received data are processed like various APIs for autonomous driving in each ROS node. Semi-autonomous driving is executed using the ROS message transmitted to the modules of the CARLA simulator. Figure 2 shows the control process of manual and autonomous driving, generated by the ROS rqt graph. Controlling client objects on the CARLA server and verifying the information is managed using ROS CARLA messages. Make_node is a ROS node that controls manual and autonomous driving simultaneously. Vehicle_con-trol_cmd messages are used when controlling a manually driven vehicle, and throttle, brake, and steering values are input as the corresponding message parameters to control the vehicle. Each parameter has a real value ranging from 0 to 100. The throttle and brake values are received from the physical accelerator and brake, and they are transferred to Make_node via the ECU. The akermann_cmd is used as a control message, and the throttle, brake, and steering values are used in vehicle_control_cmd in the carla_akemann_con-trol_ego_vehicle ROS node, which are controlled by Proportional Integral Differential (PID) by setting values (steering angle, steering anglevelocity, speed, acceleration, and jerk). The processed value is then transmitted to the virtual vehicle on the CARLA simulator as a ROS message.

Driver Assistance Systems
Driver assistance systems were configured based on HILS [38]. The ECU is responsible for controlling the virtual engine and vehicle and receives the HMI result data. The HMI result data are converted to messages via the CAN network. Figure 3 shows a control flowchart for semi-autonomous driving. The parameters of the message include the HMI result data, accelerators, and brake values. In manual operation, the driver receives the accelerator and brake values as ADC data through a controller composed of hardware used for control. This data controls the throttle and brake values of the virtual vehicle. In the control flowchart, autonomous driving is performed for scenarios 1, 2, and 3 based on the driver's driving ability. Situation 4 was executed by driver control with manual driving ability. As shown in Figure 4, CAN and serial interfaces were used for communication between the vehicle control hardware. A frame consisting of 3 bytes was used for serial communication between the HMI and ECU. The first index was the 0xFF value as the synchronization signal, the second index was the HMI result data, and the third index was the data end value, using line feed (LF). A total of 3 bytes of the CAN data frame were used for the controller area network between the ECU and driving simulator. Each byte contained different information: the first index data had HMI result data, and the second and third index data were the throttle and brake values, respectively. To implement the driver assistance system, the environment is configured as shown in Figure 5. The ECU received the result data processed by the HMI through serial communication and received the accelerator and brake values measured by the potentiometer from the controller to the analog-to-digital converter (ADC). The data were converted to a CAN frame and then transmitted to the driving simulator. It was then received through a socket CAN for data reception in the driving simulator. The vehicle configured in CARLA was controlled by determining driver control and autonomous control through the semi-autonomous driving API, which is the ros node.

Human-Machine Interface Using Emotion Recogntion
This study designed a system that recognizes the driver's situation when switching between manual and autonomous driving and prevents traffic rule violations and accidents. As mentioned above, the driver's driving ability is influenced by their emotional state. In addition, the driver's emotions are correlated with the surrounding situation, and they lack the ability to calm themselves when they feel frustrated or angry [39,40]. Inner emotions are represented as a two-dimensional arousal and valence domain [41] to control the vehicle based on the driver's situation, as shown in Figure 6. Figure 6a shows the case of decreasing the driver's speed recognition ability to prevent vehicle traffic accidents caused by excessive acceleration. When the driver is in a state of excessive happiness while driving, it can negatively affect their driving ability, and the subject may drive at a higher speed without focusing on the speedometer and speed control [42]. For an accurate comparison, Pêcher et al. [43] performed experiments on driving while listening to upbeat and soothing music. The results of measuring the vehicle's average speed and traction control system (TCL) confirmed that driving while listening to happy and exciting music resulted in driver distraction and weakened concentration. Figure 6b shows the case of decreasing the driver's cognition and coping ability when an unexpected situation occurs while driving. The emotions of anger that occur in the vehicle's external environment (traffic jams, quarrels with other drivers, etc.) can confirm that the driver's aggression, dangerous behavior, and the time it takes to crash increases [44]. Underwood et al. [31] investigated whether causes and factors related to anger while driving could affect the driver's behavior, and as a result of the experiment, driver's emotions (anger) related to social deviance and driving violation behavior and specific connectivity were configured. Figure 6c represents the case of decreasing the driver's situational judgment or recognition ability of the vehicle situation or driving negligence while driving. The problem of fatigue and drowsiness while driving a vehicle has been a major research area, and various investigations and experiments have been conducted [45]. Brown et al. [46] analyzed driver drowsiness based on EEG signals for drowsiness that occurs while driving. Dong et al. [47] derived the effect on the driver's condition and driving performance through real-time monitoring and classified the driving carelessness category into distraction and boredom based on the analysis of drowsiness, including bio-signals and physical signals. Figure 6d represents the case of maintaining driving ability in a normal state, where the driver is not affected by any emotions. It is normally converted to manual driving, and the driver changes the throttle value that directly controls the accelerator to drive the vehicle. When the control method is switched, the driver's situation can be normally reflected in the vehicle control, enabling stable driving. When the control is switched to manual driving, the driver's situation should be reflected in the vehicle control to enable stable driving. However, autonomic nervous system signals that humans cannot control have a few characteristic changes due to emotional changes, and several non-regular signals also exist. Therefore, existing studies have conducted emotion recognition by extracting features from raw bio-signals. Mantini et al. extracted bio-signal features using power spectrum density (PSD) [48], and Topic et al. used topography [49]. However, it is difficult to apply real-time emotion recognition because of the time delay required in the feature extraction process [50,51]. We used PPG [52] and GSR [53] bio-signals with specific regularity, which are easy to acquire in real time. The PPG signal is acquired by attaching an optical sensor to the driver's index finger. The extracted signal has a regular shape like ECG and includes various information such as blood pressure, volume change, and heart activity. The GSR signal is acquired by measuring the skin conductance on the middle and ring fingers. It includes information on the amount of change such as sweat secretion and body temperature according to the driver's body and emotions. In addition, we did not use the aforementioned feature extraction method through handcrafts. However, raw data are difficult to use immediately as training data and data for real-time emotion recognition, and a Butterworth filter is applied to remove low-frequency components of the data. High-order polynomial and moving average filters were used to reduce the baseline fluctuations and dynamic noise. After dividing the preprocessed bio-signals into short lengths of 1.1 s waveform units, data characteristics and learning were performed using a 1D CNN model.
An artificial neural network (ANN) consists of three layers: input, hidden, and output. However, it is difficult for the existing ANN to find the optimal value of the parameter, and it is often vulnerable to distortion due to movement and change [54]. The convolutional neural network (CNN), an improved model, is based on the weights and biases of the previous layer in the same way as an ANN, but it consists of a structure that extracts data features and understands rules. Therefore, in a recent study, a method of extracting and classifying signal features using a 1D CNN model for various voices and bio-signals was applied [55]. The configuration diagram of the multimodal 1D CNN used in this study is shown in Figure 7.

Experimental Environment
As shown in Table 1, the PPG and GSR signals of session1 of the MERTI-Apps [56] dataset were used as learning data. The data used in session 1 were measured by attaching electrodes to the index finger for the PPG signal and the middle and ring fingers for the GSR signal at 1Khz using BIOPACK's MP150 equipment. The first 5 s and the last 5 s were excluded to remove noise from the signal. A total of 32,000 segments, defined by 1100 samples, were extracted from the extracted waveform unit signal data. The range of labeling of the bio-signal data was −100-100. Table 2 shows the overall experimental environment, which includes a driving simulator PC configuring a virtual environment, vehicles, and ECU for communicating with HMI-PC and driving-simulator-PC, and an HMI PC for deriving 1D CNN model-based results after processing data received from bio-signal sensors. The Shimmer data acquisition device and HMI PC were connected to collect biosignal data to recognize the driver's emotions. The STM3204G-EVAL board, an microcontroller unit(MCU) board, was used to configure the ECU mounted on the driving simulator vehicle. The ECU uses the CAN protocol to transmit and receive data between the HMI PC and ego vehicles in a driving simulator. The overall Configurations are shown in Figure 8.
The experiment was conducted on the Town1 map provided by the CARLA simulator. Figure 9a is a picture of the Town1 map, and the part marked in blue represents the waypoints and routes the vehicle will track. The vehicle was driven on the road with a speed limit of 40 km/h, and the starting point was point A. After driving around the map once, the same point, A, was the destination. Figure 9b is a screenshot for vehicle monitoring made by Rviz of ROS, which is used to monitor camera and sensor data of the vehicle in real time. In evaluating driving ability according to a driver's situation, situational awareness occurs on a straight lane section on the route. We classified four scenarios according to the bio-signal data seized from the driver's hands. The first scenario was to recognize the decrease in the driver's driving ability for speed perception. When an event occured, the vehicle driving at 40 km/h was forcibly speed-limited to 20 km/h. The second scenario was to recognize the decrease in incident reaction ability, and when an incident occured, the vehicle that was driving at 40 km/h was suddenly stopped at 0 km/h. The incident was terminated when a bio-input signal other than the existing bio-input signal was received. The third scenario was to recognize the decrease in the ability to judge the driving situation. In the driving scenario, the vehicle was driving at 40 km/h, and when an event occured, the vehicle drove at 20 km/h, and after a certain period of time, the vehicle completely stopped at 0 km/h. The fourth scenario was to maintain driving capability. When an event occured, the driver's manual control was switched to directly control the accelerator, and the throttle value changed to drive the vehicle freely. Table 3 shows the accuracy and data size according to the scenarios.

Experimental Result
When the virtual vehicle received CAN data sent from the ECU, we measured the response time to take control of the vehicle in the event of a disengagement. For timing measurement, the response time was measured from the time taken for the driving simulator PC to control the vehicle by receiving the data in CAN format by converting the result data processed on the HMI PC into CAN data on the ECU board. In the experiment, four scenarios were performed, and the response times of start control and end control for each scenario were measured, respectively. For more accurate results, these experiments were repeated 10 times. Table 4 lists the average and total response times of the controls in the driver assistance system when the experiment of the four scenarios was repeated ten times. The average vehicle control response time was 351.75 ms, and the standard deviation was 12.13 ms, half of the aforementioned standard control time of 830 ms, showing a fast time. Also, as shown in Table 5, the data processing time for emotion recognition has a total average time of 66.1ms and a standard deviation of 4.1ms. Even if the vehicle control system and the emotion recognition module are used together, it was confirmed that total reaction time (417.85 ms) is within the stable scope of the commercial reaction time standard. To check the target speed of the vehicle in progress for each given scenario, Figure 10 shows vehicle (speed, throttle, brake) data values and control status values for each scenario. Figure 10a shows the target speed according to the total scenario, and Figure 10b-e show measured vehicle speed, throttle, and brake values for each scenario performed in the experiment. We can see that the throttle and braking data values work correctly depending on the vehicle speed appropriate for the scenario. When verifying the experimental results, it ensures that the results are similar to the target scenario and that the overall response time is within the standard scope. The experimental results confirmed the possibility of real-time vehicle control through the driver's situational awareness

Discussion
Using the results of the multimodal bio-signal-based 1D CNN model, a study on the driving assistance control module when the driver drives the vehicle was conducted on the simulator. Autonomous driving can also be used to develop an autonomous driving system through human emotions, and the interaction between the driver and the vehicle is necessary when manually controlling the vehicle and changing the mode.
When comparing previous studies based on our research results, Meshram et al. [36] proposed an architecture of semi-autonomous driving system combining with emotion recognition through human faces and only collected and classified data for the driver's four emotions. In contrast, we designed and implemented eight emotion based driving assistance systems using PPG and GSR signals and derived the experimental results in terms of response time. Izquierdo-Reyes et al. [4] in the study of the driver assistance system configuration presented the driver's emotion recognition using the EEG signal whereas they did not implement vehicle control method combined with emotion recognition. However, we have built a semi-autonomous driving system that connects HMI that recognizes the driver's emotions and driving simulator through ECU and CAN networks.
In the study of Dixit et al. [57], the average reaction time for controlling the vehicle was 0.83 seconds, which corresponds to the distribution of stable reaction time required by automobile vendors. Our experimental results showed stable vehicle controllability coping with situational awareness based on emotional recognition at 417.85ms. In previous studies, 830ms is required for the time required to control the vehicle when an event occurs. Therefore, the vehicle control response time for changing the driving mode was measured in the scenario, and it was confirmed that the response times for the start control and end control for each scenario event could be used in autonomous driving system. In addition, it is confirmed that there is no problem with adding various autonomous driving modules. It is possible to apply to actual vehicles as hardware and software semiautonomous driving modules with maintainability and availability using HILS and driving simulator.
Unlike robots, human emotions do not change rapidly. Therefore, the stored bio-signal data were used for accurate vehicle control in the simulator. If the system is studied for each situation using various autonomous driving sensors in the future, development and research will proceed in various driving assistance systems and autonomous driving systems. This study confirmed that the convergence of various fields and autonomous driving research is possible by approaching the proposed new module equipped with the driver's situation and emotion recognition as a center of interaction between passengers and vehicles rather than a vehicle-centered existing autonomous driving system module.

Conclusions
This study confirmed the possibility of a module for vehicle driving speed control based on multimodal bio-signals. To analyze the driver's emotions, we proposed a vehicle speed control and driving assistance system module using a 1D CNN model without input data of 1.1 s and without separate feature extraction to analyze the driver's emotions. The virtual city and vehicle environment were configured in the CARLA simulator server to configure an environment similar to a real vehicle. The ECU board was used to configure the same communication system as a real vehicle. In addition, CAN communication, which is widely used for in-vehicle communication, and situation scenario, and accelerator data were transmitted to the virtual environment vehicle, and the data managed by ROS middleware was monitored in real time to compare the measured and target values. The proposed prototype system shows stable performance of the average reaction speed of 351.75ms, the standard deviation of 12.13ms, the average processing time of 66.1ms, and the total processing time of 417.85ms. As can be seen from the experimental results, it can be confirmed that the proposed driver assistance system accurately achieves the target speed and vehicle control for each situation. In future research, we plan to study autonomous driving by integrating various autonomous driving sensors and systems such as automotive ethernet, bio-signals, and object recognition, which are advanced vehicle core technologies. Institutional Review Board Statement: This study was conducted by the IRB standard operating guidelines of the Institutional Review Board of Inha University. It was performed according to the approval number (170403-2AR) and the latest approval date (2020.12.29). Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Conflicts of Interest:
The authors declare no conflicts of interest.