A Station with Multiple Head-Mounted Displays for Learning Advanced Driver Assistance Systems

Advanced driver assistance systems (ADAS) are technologies that provide drivers with essential information or take over difficult and repetitive tasks. They contribute to improving road safety and increasing driving comfort. Apart from the technical development challenges, training and demonstration of ADAS in safe environments are important concerns for automobile manufacturers and suppliers. This paper presents the concept and prototypical implementation of an innovative training station for learning ADAS with driving simulators. The training station has a scalable and modular architecture, so that more than one driving simulator can be connected to a common instructor unit. Fully immersive visualization is provided by utilizing head-mounted displays for the participating driving simulators. The instructor unit consists of a computer with a developed software tool for session control, monitoring, and evaluation. Moreover, the instructor can use a head-mounted display and participate within the same virtual environment of a selected trainee. A simulation model for an autonomous driving system was implemented and a group of test persons were involved to show the usability and validity of the developed training station for ADAS learning and demonstration.


Introduction
Road safety and driving comfort are significant concerns in the automotive realm.Automobile manufacturers and suppliers develop new vehicle systems to reduce driving stress or to support drivers in critical traffic situations.Advanced Driver Assistance Systems (ADAS) present an example of such new vehicle systems.ADAS are innovative mechatronic systems that monitor the vehicle, the traffic environment, as well as the driving behavior [1].They provide drivers with information about the traffic and take over demanding driving tasks.In critical driving situations, these systems warn and may intervene actively to support the drivers, and hence, lead to increased road safety.Diverse sensor technologies and decision algorithms are developed to provide different levels of assistance [2].Some ADAS alert drivers to critical situations by means of optical, acoustic, and/or haptic feedback.Lane Change Assistance is an example of such systems [3]; it monitors the blind spot beside the vehicle and gives warnings if the driver tends to change the lane at the wrong moment.Other ADAS do not only recognize driving situations and warn the driver, but also intervene actively in order to prevent possible collisions.A common example of the latter type is Emergency Brake Assist [4], which applies full braking if driver fails to respond to obstacles in front of the vehicle.
Automobile manufacturers and suppliers have major technical challenges while developing and testing ADAS.However, training and demonstration of ADAS represent additional crucial concerns.These concerns contribute considerably to customer acceptance and market development.In particular, human-machine interaction constitutes one of the main aspects related to the introduction of such systems and customer acceptance.This covers not only the physical interface between drivers and assistance systems, but also drivers' understanding to operate these systems properly.Problems may arise when drivers feel out of control, if there are many arbitrary controls and actions, or if actions do not lead to the expected results.In addition to safety issues, training and demonstration of ADAS in real traffic environments are impractical and lead to constraining efforts and costs.Driving simulators offer a potent means for drivers training in general [5].Vehicle components can be real, virtual, or a combination of real and virtual elements.For ADAS training and demonstration in particular, driving simulators can be utilized to introduce the functionalities to drivers [5].Programs or scenarios are designed, so that drivers can experience safety and comfort benefits of different ADAS in safe virtual environments.Drivers can learn how to use system interface and properly perceive function indications.
However, future ADAS are becoming more complex and interconnected.This may negatively affect the driving behavior and performance in terms of reaction time and situation awareness [6].Therefore, training and demonstration of future ADAS should be carried out in more immersive virtual environments.These should provide realistic replicas of vehicle interior, user interface and functionality of assistance systems, and the surrounding traffic participants.Moreover, a capability for an accompanying supervision and instruction still should be provided.Thereby, the advantage of conventional drivers training, i.e., the presence of driving instructors in driving schools, shall not be sacrificed.This paper presents the concept and prototypical implementation of a new training station for learning ADAS with driving simulators.For an effective effort and time utilization, more than one driving simulator can be connected to a common instructor unit.A developed software tool is installed in the instructor unit for centralized control, monitoring, and evaluation.Full immersion is provided by using head-mounted displays (HMDs) as visualization systems for the participating driving simulators [7].HMDs have low cost and less space requirements in comparison to traditional display systems, like, e.g., screens and projectors.Moreover, they provide a full 3D-viewing and deliver user-dependent scenes.Figure 1 shows a scheme for the proposed ADAS training station.To show and validate the capability of the developed training station for ADAS learning and demonstration, a simulation model of an autonomous driving system was implemented [8].It presents a good example of a future ADAS that drivers must learn before its deployment in real traffic environments.Specifically, it must be clear who is in control at each moment, i.e., the human or the automated driver.The rest of this paper is organized as follows.Section 2 introduces the concept, architecture, and capabilities of the developed ADAS training station.Section 3 shows the main building blocks of the utilized driving simulators.Section 4 discusses the simulation model of the autonomous driving system as further development for a previous work [1].Section 5 presents the components and capabilities of the developed instructor software tool.Finally, Section 6 derives a conclusion and reveals the future work.

Architecture of ADAS Training Station
Future ADAS show higher degree of intervention and automation.A considerable part of driving responsibility is delegated to the assistance systems, while drivers gain more supervisory roles.For instance, autonomous driving systems will be capable of driving the vehicle independently [8].Although these systems are designed to reduce the burden on drivers, the complexity of user interface may grow with increasing automated functionalities.The human-machine interface of ADAS itself may demand some of driver's attention and introduce more cognitive load.Consequently, training with driving simulators must be adapted for more immersion and a capability for interactive supervision and instruction.Figure 2 shows the architecture of the developed ADAS training station using driving simulators.In this training station, more than one driving simulator can be connected to a central instructor unit.The instructor unit consists of a computer with a developed instructor software tool for session control, monitoring, and evaluation.In addition to the instructor software tool, there are instances of the visualization software of each of the participating driving simulators.Each driving simulator sends position and orientation information of its simulated vehicle and traffic participants to the instructor unit.In addition, the main signals of each simulated vehicle, like, e.g., steering wheel angle, acceleration and brake pedals, gear state, are sent to the instructor unit.By selecting a particular trainee through the instructor software tool, the instructor can monitor the vehicle signals and switch between different views for the driving scenario.Moreover, a supervision camera is directed to each driving simulator, so that the instructor can observe the gaze behavior of a selected trainee.Immersive 3D-viewing is provided by using head-mounted displays as visualization means for the participating driving simulators.The instructor can use another head-mounted display to accompany a selected trainee during the driving scenario interactively.That is, the instructor and the selected trainee can reside together within the same virtual environment.
The presented architecture has several advantages.The constitute systems communicate in a loosely coupled fashion, i.e., they have no knowledge about the internal operations of each other.This allows any driving simulator to be integrated with the training station, provided that an instance of its visualization software is available in the instructor unit.Moreover, the typical and necessary supervisory role of driving instructors is still provided.This is accomplished by sharing the same virtual environment between two head-mounted displays.Hence, driving instructors can step into the virtual driving scenarios of the trainees.Furthermore, instructors can train more than one driver at the same time, while they have centralized control and supervision capability.This in turn adds time and cost benefits regarding the utilization of driving simulators for training purposes.Figure 3 shows a prototypical implementation for the proposed concept of ADAS training station.In its current implementation, the developed ADAS training station involves two similar driving simulators connected to the instructor unit.The following section discusses the main building blocks of these driving simulators.As further development for the work presented in [1], a simulation model of an autonomous driving system is discussed in the section hereafter.

Utilized Driving Simulators
Two PC-based driving simulators without motion platform were build and integrated with the instructor unit [9].Each driving simulator has a commercial wheel-transmission-pedals set that provides low-cost, but reasonable, physical feedback and control cues.These driving simulators serve as virtual prototyping platforms for development, testing, and initial training purposes, specifically, in the field of Advanced Driver Assistance Systems (ADAS).A commercial head-mounted display is utilized in this work instead of conventional simulator display systems for more immersive 3D-viewing.Figure 4 shows the main building components of the driving simulator.Four main software models constitute the whole simulation environment of the PC-based driving simulator.With its modular structure, adaptation of model interfaces can be performed with minimum effort.Hence, each of the models can be exchanged or further developed separately with little or no knowledge of the implementation of other models.The following subsections present the underlying concept of each model.More detailed information about the implementation and a comprehensive discussion about the input-output relationships are provided in [9].

Visualization Software
The visualization software represents the main feedback cue of the driving simulator.It was implemented with Unity3D [11]; a development engine that provides rich and easy functionalities for creating interactive 3D environments.Realistic 3D models for the main vehicle and traffic participants were built.Moreover, city streets and highways can be generated; this is necessary for realistic and engaging driver training.Day-and nighttime drives can be performed and the driver can be subjected to different weather conditions, like, e.g., rain, snow, fog, etc. Figure 5 shows a sample screen shots of the 3D environment developed with Unity3D for day-and nighttime driving conditions.
(a) (b) In addition, realistic sound effects that accompany the 3D models are used to provide good acoustic feedback cues to the driver.The visualization software has been developed further to enable the use of head-mounted displays.These are 3D interactive displays that do not only allow free head motion, but also provide a good extent of body mobility [7].In contrast to typical display systems for driving simulators, the utilization of head-mounted displays delivers rich user-dependent scenes to the drivers.This characteristic is necessary for training with complex ADAS that require full immersion within the virtual driving environment.To present realistic vehicle dynamics and traffic interactions, the 3D models receive the position and orientation data from the vehicle physics and traffic simulation models presented in the following subsections.

Vehicle Physics Simulation Model
Modeling realistic vehicle dynamics is essential for driving simulation in general.The utilized vehicle model and the rest of models presented subsequently in this work were implemented with MATLAB/Simulink.The vehicle model produces the actual physical characteristics of the main vehicle and provides a total of 16 Degrees Of Freedom (DOF) [12].A nonlinear double-track model is used for modeling horizontal vehicle dynamics as shown in Figure 6 (a).This model is responsible for 3 DOF: longitudinal and lateral translational motions and a rotational motion around the vertical direction of the road.In the double-track model, the longitudinal and lateral velocities, as well as the yaw rate of the vehicle are described by a set of differential equations using Newton's law of motion and basic geometrical relationships [12].The vertical dynamics of the vehicle depends principally on suspension units at each wheel of the vehicle as shown in Figure 6 (b).The chassis of the vehicle is connected to four wheels through these suspension units.Each unit consists of a simple mass-spring-damper model [13].The units are constitutively connected through basic geometrical relationships.Each wheel has a relative vertical translational motion and a rotational motion around the lateral wheel axis.In addition, each of the front wheels has a relative rotational motion around its vertical axis.Moreover, sub-models for the engine, gearbox, tires and wheels, differential, steering, and braking supplement the vehicle physics simulation model.

Traffic Simulation Model
The traffic model is used to simulate the surrounding programmed vehicles and the road.It simulates realistic behavior of the traffic vehicles and their interactions.This is necessary to give realistic feedback cue to the driver on the one hand, and to demonstrate the benefits of ADAS functions on the other hand.The traffic simulation model consists mainly of four sub-models: driver model, road model, models of traffic vehicles, and a scenario manager.The traffic model receives current position, orientation, and speed of the main simulated vehicle from the vehicle physics simulation model; these are used mainly by the driver model to arrange for appropriate traffic flow.The main task of the driver model is it to achieve the desired traffic scenario without vehicle collisions.The road model is responsible for two tasks.The first task is to perform the necessary transformations from local coordinate system (s, t) to global coordinate system (x, y) used by the visualization software.The position of each object within the simulation environment is defined relative to road local coordinate system.However, the visualization software defines each object in 3D world relative to a global coordinate system.The second task of the road model is to define the friction and height of each point (s, t) of the road.The height values are required by the visualization software for appropriate objects positioning within the 3D world.Both friction and height values are required by the main vehicle model; they contribute to the calculations of horizontal and vertical vehicle dynamics respectively.Each traffic vehicle model consists of two sub-models: longitudinal direction vehicle sub-model and lateral direction vehicle sub-model.The longitudinal direction sub-model receives the desired s-speed from the driver model.It calculates the actual s-speed with a smooth transition, which results from a combination of a simple second-order system and a P-controller.The actual s-position of the traffic vehicle is then calculated by integrating the actual s-speed.Similarly, the lateral direction sub-model receives the desired t-position from the driver model.It calculates the actual t-position with a smooth transition, which results from a combination of a simple second-order system and a P-controller.The idea of the traffic vehicle model is to produce smooth and realistic, i.e., not abrupt, movements for the traffic vehicles.This is achieved through the transitional response of the second-order system to unit step inputs of the driver model.The traffic vehicles have to follow the predetermined longitudinal speed and lateral position given by the driver model.The scenario manager is used for arranging specific traffic situations, like, e.g., a sudden vehicle incursion from right.It observes the position and speed of the main simulated vehicle and moves the traffic vehicles according to a desired predefined scenario.According to the simulated vehicle systems or functions, arbitrary different traffic scenarios can be added to this model.The driver model receives the vehicle positions and speeds determined by the scenario manager model.According to the current traffic situation, the driver model decides whether to execute the orders of the scenario manager or to override them.Switching between the different scenarios can be performed during simulation runtime.

ADAS Simulation Model
The design of ADAS simulation model allows the integration of any prototypes of driver assistance systems.This simulation model consists of four sub-models: User interface, sensors model, controller module, and decision unit.The user interface model receives signals from the hardware control set to activate/deactivate different ADAS, as well as to set their parameters.The sensors model encloses sub-models for long-range radar and short-range radar.Moreover, it encloses a simplified camera model that resembles the road geometric design implemented within the traffic simulation model.The controller module contains a longitudinal controller and a lateral controller.The longitudinal controller is a cascaded speed-acceleration control loop system [14].A speed controller of a Proportional-Integral (PI) type constitutes the outer loop of the longitudinal controller.The speed controller generates an acceleration value required to obtain a desired speed command.The desired acceleration is forwarded to the acceleration controller, which constitutes the inner loop of the longitudinal controller and implements the inverse form of vehicle dynamics and drivetrain [15].The longitudinal controller provides the throttle angle or braking value to the vehicle model.The lateral controller handles the path following control problem, i.e., how to control the vehicle, so that it can faithfully follow a prescribed path [16].It is composed mainly of two further sub-models.Upon receiving a desired trajectory, a path following sub-model calculates the front axle force required to let the vehicle adjust its orientation.The path following sub-model is based on the feedback linearization control method [16].According to the vehicle dynamics, a steering calculation sub-model determines the steering angle, which corresponds to the desired lateral force.The lateral controller exports the steering wheel angle required to guide the vehicle in the desired direction to the vehicle model.The designed longitudinal and lateral controllers can serve a variety of ADAS functions.The decision unit contains the logic of different ADAS in form of sub-routines.It observes the driver's intention through the inputs delivered by the hardware control set.The decision unit also monitors the state of the main vehicle, i.e., position, orientation and speed, calculated by the vehicle model.Moreover, it gets the objects detected and filtered by the sensors model.According to the aforementioned inputs and the logic of the activated ADAS, a sub-routine determines whether the requests coming from the driver or the logic of the ADAS function must dominate in a specific driving situation.Sub-routines for emergency brake assistant and emergency steer assistant were developed and presented in a previous work [1].However, the general design of the controller module and decision unit makes it convenient to develop and integrate new ADAS functions.The following section discusses the autonomous driving simulation system implemented within the decision unit.

Interactive Simulation of Autonomous Driving
Autonomous vehicles are able to drive themselves as long as the input requirements of their logic are met [8].When an autonomous driving system reaches the limit of its capabilities, it has to disband or hand over the control to the human driver.Taking full control back from an autonomous system has to be trained and understood to avoid negative safety effects.A model for an autonomous driving system has been implemented and used as a validation example for the developed ADAS training station.A separate sub-routine was implemented within the decision unit of the ADAS simulation model.Figure 7 shows a flow chart for a simplified version of the autonomous driving logic.Upon activating the autonomous driving system, the sub-routine reads the current lateral position and orientation of the vehicle, as well as the road heading angle.A signal to activate the lateral controller is sent together with the difference heading between the vehicle and the road.Accordingly, the lateral controller adjusts the orientation of the vehicle.Moreover, the sub-routine checks whether a relevant object is detected in front of the vehicle by the sensors model.In case of a detected object, the longitudinal controller is enabled to maintain a constant distance to the object.Otherwise, it maintains an arbitrary constant speed.The sub-routine checks whether there is an ambiguous case.An ambiguous case can be programmed arbitrarily, so that it is invoked at certain time points or situations.The ambiguous case is meant to simulate the capability boundary of real autonomous driving systems, like, e.g., bad weather conditions.If it occurs, the sub-routine deactivates the lateral and longitudinal controllers.Moreover, it invokes optical and acoustic feedback, so that the human driver takes over the control.Additional disband measures can be added arbitrarily at this point.For example, a braking maneuver can be applied to bring the vehicle to a safe state.The sub-routine monitors the signals from the control set continuously, i.e., the acceleration and brake pedals, as well as the steering wheel, to check the eventual preference of the human driver to gain the control at any moment.Some automobile manufacturers have announced the development of self-driving vehicles.Different principles and possible technological capabilities for autonomous driving systems exist.However, the presented logic of the autonomous driving system is generic.That is, it has an initial set of capabilities and it can be adapted to resemble different real systems.The following section presents the design of the instructor software and its capabilities.

Design of Instructor Software Tool
The instructor software tool resides on the computer of the instructor unit.It was developed with MATLAB/GUI and designed so that it enables the instructor to control, monitor, and evaluate the training sessions.The instructor software tool consists of three main modules as illustrated in Figure 8.The instructor software tool receives the position and orientation information of the main simulated vehicle and the traffic participants from each participating driving simulator.Moreover, signals regarding the state of the vehicle and the assistance systems are received.In addition, the supervision cameras directed to each driving simulator send video signals to the instructor software.The following sub-sections discuss the concept of each module within the instructor software tool.

Control Modul
The control module allows the instructor to create user profiles.Different data items, like, e.g., name, gender, age, license type, can be registered.The control module is connected to a database, where users' data is saved.The instructor can view and edit the saved user profiles.Additionally, the instructor can start/stop or reset the training session of a selected trainee through the control module.

Monitoring Modul
The monitoring module allows the instructor to select a particular trainee and switch between different perspectives of the driving situation as shown in Figure 8.It forwards the signals coming from a selected driving simulator to the corresponding visualization instance.The concerned visualization instance in turn drives the head-mounted display of the instructor unit, so that the instructor can step into the virtual environment of a selected trainee.Moreover, the instructor can monitor various signals regarding the state of a selected vehicle, like, e.g., steering wheel angle, acceleration and brake pedals, gear state, ADAS functions state, etc.Through the video signals delivered by the supervision cameras, the instructor can observe the gaze behavior of the selected trainee.This gives crucial indications about the traffic situation awareness [6].

Evaluation Modul
The evaluation module saves the state signals of each simulated vehicle in a database.The data is sorted in sessions, which are associated to users' profiles created through the control module.The instructor can select a particular session and review the signal graphs.Maintaining these records allows the instructor to perform after-action analyses and discuss the results with the trainees.

Conclusion and Future Work
Advanced Driver Assistance Systems (ADAS) gain importance due to their safety and comfort features.The training station developed in this work offers a flexible solution to learn ADAS and to demonstrate their benefits to customers.The presented architecture takes the practice of drivers training in general and ADAS learning in particular to a new innovative dimension.On the one hand, driving instructors can participate within the virtual environments of the trainees.Hence, the traditional and necessary supervisory role of driving instructors is still provided.On the other hand, more than one participant can be trained simultaneously, while instructors have centralized control over them.This ensures better time, cost, and effort utilization for drivers training with future assistance systems.An interactive simulation model for an autonomous driving system was implemented in this work.It presents an example of potential assistance systems that drivers must practice initially in safe environments.A group of test persons was involved in the behavioral validation process for the autonomous driving system model [17].Hereby, an assessment of how drivers handle the system has been made.The test persons drove in scenarios with purposeful ambiguous driving conditions, where the autonomous driving system disbanded the control.The behavioral validation process showed how the test persons could reasonably handle system warnings and indications with very good learning curves.
In the next future steps, the presented ADAS training station will be developed further, so that the participating driving simulators can share the same virtual environment.The ability to create a virtual driving environment simultaneously accessed by two or more drivers allows a much closer approximation of reality, with its attendant risks and uncertainty.In addition, training with other sophisticated future systems, like, e.g., cooperative ADAS , can be performed.

Figure 1 .
Figure 1.Illustrative scheme for the proposed ADAS training station.

Figure 2 .
Figure 2. Architecture of the developed ADAS training station.

Figure 3 .
Figure 3. Prototypical implementation of ADAS training station with multiple head-mounted displays.

Figure 4 .
Figure 4.One of the utilized PC-based driving simulators and its main building components.

Figure 6 .
Figure 6.Modeling of the main simulated vehicle: (a) Double-track model for horizontal vehicle dynamics; (b) Suspension units model for vertical vehicle dynamics.

Figure 7 .
Figure 7. Simplified logic of autonomous driving simulation model.

Figure 8 .
Figure 8. Structure and main input/output signals of the instructor software tool.