Development and Testing of a Methodology for the Assessment of Acceptability of LKA Systems

Abstract

In recent years, driving simulators have been widely used by automotive manufacturers and researchers in human-in-the-loop experiments, because they can reduce time and prototyping costs, and provide unlimited parametrization, more safety, and higher repeatability. Simulators play an important role in studies about driver behavior in operating conditions or with unstable vehicles. The aim of the research is to study the effects that the force feedback (f.f.b.), provided to steering wheel by a lane-keeping-assist (LKA) system, has on a driver’s response in simulators. The steering’s force feedback system is tested by reproducing the conditions of criticality of the LKA system in order to minimize the distance required to recover the driving stability as a function of set f.f.b. intensity and speed. The results, obtained in three specific criticality conditions, show that the behaviour of the LKA system, reproduced in the simulator, is not immediately understood by the driver and, sometimes, it is in opposition with the interventions performed by the driver to ensure driving safety. The results also compare the performance of the subjects, either overall and classified into subgroups, with reference to the perception of the LKA system, evaluated by means of a questionnaire. The proposed experimental methodology is to be regarded as a contribution for the integration of acceptance tests in the evaluation of automation systems.

1. Introduction

This work consisted of formulating a specific test protocol for the assessment of the acceptability of a lane keeping assist (LKA) system and setting an “experimental design” that would allow its evaluation. The lane-keeping-assistance system keeps the vehicle within the lane during the driving, and it is inserted within that broad variety of active safety tools, named advanced driver assistance systems (ADASs). These systems are intended to assist the driver while driving, either by providing a real-time feedback and a warning and/or intervening directly while maneuvering the vehicle.

However, the multiplicity and complexity of the new ADAS may put excessive stress on the driver, disturbing him, or even cradling him into a false sense of safety [1,2]. The human–machine interface must thus be designed and optimized to allow the driver to always be able to intuitively understand signals and information from the vehicle [3,4].

This research offers the starting point for the integration of further tests in an LKA system test protocol, especially in terms of the approaching speed for a maneuver that involves the LKA intervention, and for the possibility of evaluating the driver’s adaptability, in terms of the driver’s perception of constant mastery of the vehicle even in LKA intervention phases.

2. Acceptability

The design of safety systems plays a key role in controlling or limiting the degree of behavioral adaptation [5,6]. It is evident that automation does not merely replace human activity, but it certainly changes the nature of tasks, often leading to unforeseen consequences for human performance and the relevant cognitive processes [7,8]. To design ADASs, researchers have proposed to learn from automation problems in other sectors of transport and take into account human factors, proactively rather than retroactively [9,10]. The “human factor” is now widely recognized by manufacturers and researchers who promote a «driver-centered» design of ADAS, although some uncertainties remain regarding the application of concepts and methods [11,12]. A ‘sensitive’ support system can be achieved only if there is a significant integration and recognition of the driver–vehicle interaction [13]. A major concern of automation is the “out-of-the-loop” performance, often documented by the difficulty of human operators in detecting system failures or automation defects when they do not actively (manually) control an action [14,15].

Acceptance is linked to usability characteristics of the system, leading to what is actually known with the terms “trust” or “confidence”, that it is a requirement for these systems to achieve the expected benefits [16,17].

Unluckily, in-vehicle systems’ acceptance depends upon drivers’ motivation and driving style, with high variability among the population and, at this stage, despite numerous studies on acceptance of driver assistance systems, neither a common definition nor a standardized measurement procedure seems to be available [18,19,20].

3. LKA

The LKA system is designed to help the driver to remain within the expected lane. Following the operating specifications acquired from the model installed on the Fiat 500x, LKA works at speeds between 60 km/h and 180 km/h. A camera, embedded in the rigid base of the rearview mirror allocated in the upper part of the windscreen, detects the lane markings ahead of the vehicle. Information acquired by the camera is combined with data of vehicle speed and trajectory to work out the time and distance before the marking line be crossed. When one of those parameters is lower than a critical value, the LKA system intervenes by applying a slight steering correction using the electric power steering system equipped on the car [21]. The amount of torque applied to the steering wheel is small and can be easily overcome so that the driver keeps the ultimate control of the vehicle [22,23]. If the system is not able to correct the drift from the lane (for instance, if the curve is too strong), the system may further warn the driver through a steering wheel vibration [24,25]. Corrective steering is suppressed if the driver activates the turn signal or if the power steering unit detects a strong steering. The intensity and activation threshold of the signals can be set by the onboard computer. The corrective intervention haptic system is also deactivated if no steering input is detected for a given time (“hands-off”), while a sound alert is issued. The LKA system is deactivated when ESP or ABS intervene.

Table 1. Intervention specifications.

Intervention Limitations
lane width	Width car +1 m ÷ 4.5 m
curvature radius	>125 m
lateral acceleration	<2.5 m/s2 in curve and 3 m/s2 in straight
steering angle	<50 degrees

shows the limit values within which the LKA system is enabled to intervene: the settings relating to the lane width and the curvature radius operate within the functional limits allowed by the camera; lateral acceleration and steering angle values higher than the limits described are recorded only in the case of active participation of the driver in driving and, therefore, related to a state of wakefulness.

4. The Criticalities Study

It is estimated that the LKA may be effective in about half of all accidents by lane departure that lead to death or serious injuries [26,27]. If all cars were equipped with such a system, this would be equivalent to a potential reduction of more than 5000 deaths and nearly 40,000 serious injuries every year in the EU 27 [28,29]. The effectiveness of these systems depends on the driver acceptance of these technologies, understanding of the system information, and adequate reaction [30,31,32]. On the basis of observations reported by testers and owners of vehicles equipped with such systems, critical issues have been found in performing some maneuvers, related to negligence or low reaction times required [33,34,35].

5. The Driving Simulator

Preliminary tests for the configuration of driving scenarios were conducted using the fifth level of expansion of the driving simulator R.A.P.I.D.S Mk5 (Reasonably Priced Immersive Driving Simulator Mark 5) of the University of Salerno. The layout of the simulator consists of a driving station with a moving platform equipped with a steering system and pedals, as well as a G-seat able to reproduce the sensations provided by the accelerations. Everything is assembled as shown in Figure 1; the instrumented workstation has been realized to reproduce the actual driving posture in a standard car and to comply with the basic rules of ergonomics. The display system consists of a video projector, with 720 p high-definition resolution, which shows images on a curved wall placed at a distance of 3 m from the driver’s eyes.

Virtual reality simulation allows detailed analysis of the trajectories traveled, with particularly meticulous sampling, to be able to make comparisons between the drivers and evaluate the dispersion of the trajectories, which represents an important quality indicator of the simulation system during operation [36,37].

6. Virtual LKA

The emulation of a real lane-keeping-assist was performed on a virtual model of a city car that was not yet equipped with LKA, a Fiat Panda. This allowed us to investigate the performance and to evaluate possible marketing. On the basis of specifications provided by models of the same manufacturer that already supplied an LKA system as Driving Advisor (Lancia Delta, 2008), Jeep Renegade (LaneSense, 2014), and Fiat 500x (Lane Assist, 2014), a system with similar functionalities was reproduced and applied to the Fiat 500x virtual model in a first phase to test the correspondence of the f.f.b. and intervention times of the simulator steering system with that of the real car. Subsequently, this virtual LKA model was used for simulator tests with the Panda virtual model (Figure 2). The next step was to properly configure rFactor simulation software to link it to the MoTeC i2 Pro telemetry one and to the other devices connected to the PC.

7. Methods and Processes

The purpose of the experiment is to work out data on how the LKA system is perceived by the final user and how the driver reacts to its activation in particularly critical scenarios, through the analysis of the responses to questionnaire and metrics generated by simulator’s telemetry, and derived from the vehicle model.

To allow the comparison of experimental groups with different characteristics, an experimental protocol for the simulator has been developed to have the opportunity of directly exploring the use of a LKA system in the first person and, especially, in a repeatable way to subjects with different driving experience.

The protocol uses a virtual scenario the driver is asked to follow. The paths are specifically designed to reproduce the lane change maneuvers where the LKA system takes action (Figure 3); that is, acceleration lane on the highway, overcoming a short obstacle (e.g., a passer, a hole, an oil slick), and overcoming a long obstacle (e.g., parked cars on the prohibited lane, queue of cyclists).

Telemetry allowed online recording of the operations, with a 20 Hz sampling rate, of data relating to driver input (steering angle and torque, accelerator and brake pedal positions), speed, and longitudinal and lateral acceleration. The three maneuvers were carried out in the editing phase and subsequently tested, to stay below the 0.3 G threshold value when entering to the maneuver. Hence, the 0.2 G value was chosen as the upper limit below which to consider the maneuver was completed and the stability was restored; this choice is also confirmed by observing the telemetry graphs in which it is noted that, in the straight path without obstacles, the car operates stably within the predefined range [−0.2 G, 0.2 G].

To evaluate the stability of the vehicle, the parameters on which attention was focused are those relating to lateral acceleration and steering angle [38,39,40], the same as those used by the control logic of the LKA system and by the ESP stability control system to communicate with the control unit. These parameters are recorded by telemetry software (Figure 4 and Figure 5). The ESP system is disabled in order not to affect the measurements obtained with the sole intervention of the LKA system.

In the graph of Figure 4, the lateral acceleration peaks that exceed the range [−0.2, 0.2] correspond to the values reached in carrying out the predetermined maneuvers. The steering value is represented as a percentage of the 360° of the round angle and is subsequently converted into degrees for data analysis.

Finally, the following metrics were acquired by telemetry:

• Distance (meters) needed to recover full control of the vehicle;
• Width “peak–peak” (% of the steering wheel oscillations);
• Max torque (N∙m) exerted on the steering wheel.

The next step was to run the test protocol to a significant number of users who were previously subjected to driving learning sessions lasting about 5 min. The purpose was to introduce the tester to the simulator guide, in order to let the subject become familiar with the controls of the simulator (steering wheel, pedals, and gear shift) and with the acceleration provided by the dynamic platform, partially reproducing the real acceleration of the vehicle.

In general, because the increase in age entails a slowdown in the processing of information and in reaction time, the performance results for the elderly have more variance between and within the drivers than for young people [41]. To avoid an excessively random distribution of data that would have prevented a trend from being detected, it was decided to choose a rather young sample. Globally, 21 subjects were involved in the experiment, 15 men and 6 women, aged between 22 and 40 years.

On the virtual test track, every driver performs six different simulated driving sessions, characterized by different speed limits (70, 100, and 130 km/h) and force feedback intensities (F.F.B.) (70 to 100%). They run five attempts for each combination. All the subjects perform the tests following the same sequence: from the lowest to the highest speed and, for each speed, first the lowest f.f.b. and then the highest one. This choice allows us to assess whether and how quickly the driver can adapt to the LKA system as driving conditions become more difficult to manage.

8. Results

The effects of behavioral adaptability are of main importance for all professionals dealing with road safety. While always perceived as helpful in some way to the individual, the impact of behavioral adaptability on road safety can be positive, negative, or neutral, and is influenced not only by the nature and relevance of a system change (vehicle, road, risk factors), but also by individual differences in user characteristics.

In [15], the factors incident in determining the intervention of a driving assistance f.f.b. are only the lateral position and the steering angle; in [26,42], the road orientation angle is also taken into account; and, in [27], weighting coefficients are adopted to determine the intensity of the steering assistance torque. In the design of an LKA system, it would be interesting to be able to identify and categorize different user-profiles and based on their behavioral adaptation to define and modulate the assisted intervention system.

The results were analyzed in terms of both an overall assessment of the performance of the subjects and partial analysis by aggregating classes of homogeneous subjects. First, the success percentage of each maneuver was detected according to the different speed and F.F.B. and, consequently, the influence that the same two g.d.l. produced on the outcome of the maneuvers. The results are reported in Figure 5. In each maneuver, the F.F.B. intensity does not substantially affect the outcome, while the speed increase has a remarkable effect on the success of the maneuvers, especially when the speed raises from 100 to 130 km/h. Figure 5 shows that the first maneuver is the most difficult to perform; while in the second and third maneuvers, the driver is ready to face a counter steering and has the hands firmly on the steering wheel, in the first maneuver, the subject tends to adapt to the trajectory without expecting an intervention by the LKA system.

An analysis of variance (ANOVA) confirmed the speed as a significant factor in determining the results achieved by the driver in terms of distance and steering wheel oscillations. For each maneuver, the figures below report the effect that speed and FFB intensity have on the two dependent variables distance and width (this latter in terms of degrees of the steering wheel oscillations).

Figure 6 shows the distances and the excursions, in degrees, of the steering wheel relating to the different steps of speed and F.F.B. intensity; both factors significantly influence the distance needed to recover the stability of the vehicle. In particular, regarding the first maneuver, the differences recorded at the different speeds are significant for each level of this factor, with a progressive increase in the overall distance of the maneuver as the speed increases. From the graph relating to the oscillations of the steering wheel, it is observed how the intensity variation of F.F.B. imposes a more onerous control and, on the other hand, the increase in speed allows the driver to restore control more easily. Even in the second maneuver, both factors clearly affect the distance needed to recover the stability of the vehicle, but only the effect of speed on the observed variable is significant; it is noticed that, at low speeds, 70 km/h, the system LKA is much easier to manage. While the increase in F.F.B. intensity produces an imperceptible two-degree increase in the overall excursion of the steering wheel, also here, as in the first maneuver, the increase in speed allows the driver to restore control more easily. In the third maneuver, the significant influence that speed has on the distance travelled at both levels of intensity of the FFB is also confirmed.

By evaluating the subjects’ performance, considering the first three repetitions valid for each maneuver and discriminating the results based on the speed and f.f.b. factors, we observe the training effect and progressive general ability to adaptability, which is even more evident if we take into account the overall of the three maneuvers performed. The histograms in Figure 7 are constructed as the sum of the averages relating to the three maneuvers to better highlight the desired learning effect.

Although behavioral adaptation may be an explicative variable for the observed dissimilarities between the expected and actual outcomes of safety interventions, there is currently no in-depth knowledge of this phenomenon and few models integrate the individual characteristics of the driver in the explanation of the behavioral adaptability.

The purpose of the study is to investigate whether an LKA system induces a behavioral adaptation in drivers required to perform specially configured maneuvers to trigger critical issues in the handling of the vehicle equipped with this system. A second target is to evaluate how the attributes of the system (f.f.b.) and the driver (confidence) contributed to the behavioral adaptation (acceptability) of the LKA intervention system. The rapidity of adaptation is a measure of the functionality and reliability of the device.

9. Survey

To measure the attitudes and behaviors of the tested drivers, the bipolar Likert scale has been used, as it allows us to discover the different levels of satisfaction. The range of answer options may also help to identify areas of possible improvements.

A survey was thus administered to all participants, after the test phase on the simulator, to let them express their opionions regarding the adequacy of the system, that is, how the driver perceived the intervention timing of the system and the subsequent reaction of the steering wheel; a low adequacy value is indicative of an intervention that is out of time and not coordinated with the need of the moment and an f.f.b. of inappropriate intensity, while a high value expresses the driver’s satisfaction in terms of timing and f.f.b. The usability was also evaluated, in terms of “easiness” and “learning”, that is, the time necessary to become familiar with the system (Figure 8); a low easeness value indicates the difficulty of driving with this system, while a low learning value indicates that it takes a long time to familiarize with the system. To investigate what effects this system may have on the driver’s behavior, the survey asked if the subject always felt the mastery of the vehicle and the possibility of performing any maneuvers beyond the intervention of the LKA system (high “control”), if the subject believes that this system would encourage more attention to situations unrelated to driving (referred to an as low level of “attention”) or drive more riskily (high level of “unsafe”), and lastly if the system would make driving more monotonous (high value of “monotony”).

In the end, an overall opinion was asked about the LKA system: (1) I would like to count on this system while driving: it can certainly control critical driving situations better than I can (“useful”); (2) I would like to rely on this system only after knowing how it works and being convinced of its reliability (“a little useful”); (3) my judgment is indifferent (“indifferent”); (4) I am skeptical about this system: I would only actuate it in situations where I can intervene if it fails (“enough useless”); (5) I would not trust this system while driving, I prefer to have control of the situation on my own (“useless”).

To evaluate the possible presence of relationships between the measured variables and the driver perception, the experimental population was sampled identifying two experimental groups, according to the perceived confidence in terms of maneuverability and control, and to the degree of general satisfaction defined as follows:

• Confidence/high satisfaction: Subjects with answer’s average higher than 5 on the Likert scale;
• Confidence/low satisfaction: Subjects with answer’s average lower than 5 on the Likert scale.

It was decided to include only those who assessed the LKA system decidedly positive among the subjects with high confidence to emphasize and verify if their opinion coincided with their actual greater mastery and adaptability to driving with this system.

Figure 9 and Figure 10 show that the best performance in terms of distance and oscillation are achieved by unsatisfied subjects, as much as by subjects stating a low level of confidence with the device (Figure 11 and Figure 12). The result could find an explanation in the perception of the LKA system as an aid by the less able subjects to adapt to the new system, and who thus declare to be satisfied with the help received and as an impediment by the ablest subjects to adapt who have not received any significant contribution from the system, and thus said they were dissatisfied. More detailed graphs are shown in Appendix A.

10. Conclusions

The pattern of data analysis carried out allows us to substantiate the characteristics of acceptability and adaptability through the use of the test protocol described above, which is still not envisaged by the regulations. By starting from the analysis of the obtained performance, it is clear that these two factors must be considered decisive in the design and testing phase of such an ADAS system and, therefore, must be investigated with long-term tests.

Even if, in a few decades, automated driving could become so reliable that humans will not be legally allowed to drive, at this stage, there are still several issues that need to be addressed in terms of human factors and acceptability, to evaluate how the driver will accept, face, use, and manage the critical situation, according to diversified levels of automation [43].

Experimental methodologies to assess drivers’ responses and capacities in critical situations still need to be developed and possibly to be adapted to prepare individuals for this huge step in the driving task concept and to ensure the expected positive consequences owing to automation for the road safety [42,44].

In this work, the changes in the driver’s trust in reply to changes in required tasks that concerned the LKA performances have been investigated. As hypothesized, the driver confidence on the LKA significantly decreases if they experienced critical situations, even partially showing the functional limitations of the device.

This suggests that changes in the driver’s cognitive processes may not become evident without a long time of interaction between drivers and the system itself. The results also point out that the interaction of drivers with the LKA system especially requires a substantial conscious control, in order to handle the ongoing actions of the system in the guide. This indicates that the results of this study should be complemented by long-term studies, in order to observe the relative impact of different types of human adaptation processes on the active workload of the drivers in the driving task.

The results above allowed to formulate a testing protocol not scheduled yet, although there are already several studies aimed at defining an adequate intensity of F.F.B. [45,46,47] and confirming its effectiveness. The developed test protocol integrates actual data with the perception expressed by the sample subjects. The statistical significance of the control variables obtained by the analysis of the maneuvers confirmed that it is possible to treat acceptability, which is a subjective parameter, as an objective variable.

11. Prospective Developments

The real challenge in product development is to find the “correct communication code” between man and machine, as well as to improve the transparency of the system, which is a contributing factor to reliability; if the driver fully understands the functionality of the system, his trust in the system increases, thereby increasing reliability and allowing the car manufacturer to investigate the active safety issues associated with it.

This study is intended as a methodological basis from which to move the next step towards a further development phase aimed at identifying any factors that modulate the perception of the LKA system by the user, as well as providing useful information in the design, development, and validation of LKA systems.

For this purpose, it will be necessary to integrate this analysis methodology with long-term follow-ups, allowing the observation of the impact of human adaptation processes on the active engagement in the task of driving vehicles equipped with ADASs.