An Objective Evaluation Method for Driver/Passenger Acceptance of an Autonomous Driving System for Lane Changes

Abstract

Reliable assessment methods of driver acceptance are needed due to increased interest in high levels of autonomous driving systems. Subjective evaluation methods have mostly been utilized to assess the acceptance of newly developed advanced driver assistance systems because acceptance varies depending on the individual. In this paper, an objective evaluation methodology of driver acceptance for an autonomous driving system was proposed based on objective measurable parameters in the case of automatic lane change situations. To this end, a massive driver–vehicle interaction database was utilized, constructed by a specially designed experimental program. The experiment was carried out with 19 selected drivers (9 experts and 10 novices), supposed as an autonomous driving system. The database consisted of not only various measurable parameters on control commands, vehicle behaviors, and relations with other vehicles but also subjective acceptances. To interpret the driver acceptance, objective parameter sets were derived by two different methods: a statistical significance test and an acceptance sensitivity analysis. Then, a modeling method based on stochastic estimation to evaluate driver acceptance was suggested as an objective evaluation method for the driver acceptance of an automatic lane change system. The data set of the expert drivers was only used for the acceptance evaluation modeling; the other data sets of the novice drivers were used for verifications for the suggested model. The estimation accuracies of the two different models using a significance test and sensitivity analysis were 90.2% and 99.5%, respectively. This objective method for acceptance evaluation can not only be expanded to other functions of an autonomous driving system but also to an entirely autonomous driving vehicle.

1. Introduction

In the last few decades, with increased interest in the technologies of autonomous vehicles, substantial efforts have been made in the research and development of various driver assistance systems as part of autonomous driving technologies, and such systems are now being introduced at a faster rate. Advanced Driver Assistance Systems (ADASs) such as Adaptive Cruise Control (ACC), Lane Keeping Aid (LKA), and Automatic Emergency Brake (AEB) have a positive impact on traffic safety [1]. It is expected that further development of Advanced Driver Assistance Systems, and eventually highly automated driving, will continue to increase traffic safety by reducing the impact of human error [2].

To advance the commercialization of fully autonomous vehicles, considerable efforts have been invested in the research of semi-automated driving systems such as an automatic lane change system. However, important issues remain including safety, reliability, passengers’ acceptance, etc. In particular, acceptance assessment is recognized as one of the most important issues [3,4]. Because customers decide to use or not use a system, most advanced technologies, such as an autonomous vehicle, will have no value if customers do not use a proposed system even when the system satisfies all requirements of safety and reliability.

In accordance with the increased interest in higher levels of automated driving technologies, a reliable assessment tool for driver acceptance is needed for their development and deployment which estimates and secures driver acceptance in a broad manner. Because acceptance is on an individual level, it can only be based on an individual’s personal attitudes, expectations, experiences, and subjective evaluation of a system and the effects of using it [3,5].

A number of studies on user acceptance have been conducted over the past decades. A standardized checklist was proposed for the assessment of the level of acceptance of a new vehicle technology [6]. The idea of technology acceptance was also defined in terms of usefulness, ease of use, effectiveness, affordability, and social acceptance [4]. Technology adaptation characteristics were studied based on gender differences [7]. A technology acceptance model was suggested based on subjective rating scales [8]. The aforementioned technologies are being developed not only for safety but also for convenience. Accordingly, drivers’ experience and acceptance of an Intelligent Speed Adaptation system were investigated [9]. Various studies on lane change assistance have been carried out to satisfy drivers’ acceptance [10,11,12,13,14].

In a literature review, referring to [3,4,5,6,7,8,9,10,11,12,13,14], almost all of the research was conducted based on subjective methods: questionnaire development, questions and/or rating scales, etc. Even though research results concerning acceptance have been presented, how it has been assessed and how the results have been obtained are not described in most studies. Previous studies mostly focused on measuring acceptance as various types of participants’ opinions on the target systems. These studies have suggested various types of methods to validate that their results are objectively reliable.

As seen in the literature review, acceptance has been considered with individuals’ subjective evaluation. Consequently, most studies have used moderate questions/or phrases to obtain individuals’ subjective opinions in accordance with a specific structure for the acceptance analysis, and the results were presented with statistics. Therefore, there are a few significant problems as follows. Acceptance is generally assessed after the target systems are realized at certain levels, for example, by performing a jury test with a prototype of the system. Target values or ranges for the evaluation are different in each case. Responses and evaluation results of participants may be dependent on the questionnaires used. The subjective assessment process is expensive and takes much time because the reliability of the evaluation results is also dependent on the number of participants. According to [4], there is also no consistency across studies as to what acceptance is and how to measure it.

Therefore, it is hard to assess the acceptance of systems in the design phase although everyone knows that it is desirable. If the acceptance can be accurately predicted as early as possible in the design process, different alternatives are not only able to be evaluated but also some obstacles may be identified and overcome.

To secure acceptance in the design phase, an objective evaluation method or index is needed. In general, an objective evaluation method has a significant advantage in quantifying some values to be assessed. The effects of the objective evaluation method can be increased for cases where assessment results are conventionally performed by some subjective methods such as the driver acceptance of an advanced driver assistance system. If acceptance is objectively evaluated in design phases, developers may have several advantages. For example, the design targets of a developing system can be clearly defined as driver acceptance based on objective and observable values, and the development period and cost can be substantially reduced.

Hence, in this study, the primary objective was the study of an objective evaluation methodology of passenger acceptance for autonomous driving systems with a representative autonomous driving system with an automatic lane change system, which is regarded presently as close to commercialization. The specific research objectives were to determine how objective parameters for describing driver acceptance can be derived and how driver acceptance can be evaluated with the derived objective parameters. It is assumed that the objective parameters must be simply measurable. The suggested objective evaluation method can be used for assessing driver/passenger acceptance of an automatic lane change system.

To study the objective acceptance evaluation method, a special database constructed in a previous study was used [15]. In Section 2, driver/passenger acceptance is defined and measurement strategy is discussed. Section 3 describes part of the special passenger-vehicle interaction database used throughout this study. Section 4 presents a novel objective evaluation methodology for passenger acceptance. Section 5 presents the objective parameters and objective evaluation models derived based on the suggested methods. Then, the verification results are presented.

2. Driver/Passenger Acceptance

2.1. Definition of Driver/Passenger Acceptance

Since acceptance is primarily influenced by individuals’ emotional factors, it is crucial to discuss and define what acceptance means in this study. In a previous study, a statement on the purpose of investigating the acceptance is shown: ‘Driver acceptance is the precondition that will permit new automotive technologies to achieve their forecasted benefit levels’; another statement addressing the need to determine whether drivers will accept and use the new technologies as intended is as follows: ‘Driver acceptance measurement also provides a means to estimate drivers’ interest in purchasing and using new technologies as a basis for estimating the safety benefit associated with its use [16]. Acceptance is seen as the link to usage in [6], and acceptance is explained as a predictor of the willingness to buy a system [17]. While many studies claim the importance of acceptance and assessing the acceptance of new technologies, there are many different ways of viewing acceptance. A statement on a problematic issue of the definition of acceptance is as follows: ‘While everyone seems to know what acceptability is, and all agree that acceptability is important, there is no consistency across studies as to what acceptability is and how to measure it’ [4].

The definition of acceptance identified in the literature was classified into five categories in a recent literature review [3]. The word “accept” is simply used for the first category: for example, ‘acceptance is the degree to which a law, measure or device is accepted’ [18]. The second category is defined concerning the satisfaction of the needs and requirements of users. For example, ‘basically the question of whether the system is good enough to satisfy all the needs and requirements of the users and other potential stake holders’ [19]. The third category is described as “the sum of all attitudes”. For example, ‘Acceptance refers to what the objects or contents for which acceptance is measured are associated to; what do those objects or contents imply for the asked person’ [8]. The fourth category is explained by “the will to use the system”. For example, ‘acceptance is the intention to adopt an application’ [20]. The fifth category focuses on “the actual use of the system”. For example, ‘the demonstrable willingness within a user group to employ information technology for the task it is designed to support’ [21].

In this study, if acceptance is not defined well, it is not only possible to assess acceptance, but also to build an assessment tool for an automatic lane change system. In order to achieve a study on an objective method of the acceptance for an automatic lane change system, acceptance is defined and suggested considering the target system as follows in the following statement.

“Acceptance is the classification to which an individual intends to use the system in his/her driving if the system is incorporated.”

This suggested definition of acceptance has the advantage of focusing on the individual perspective, both regarding the subjective evaluation of the system and the advantages of using the system. This provides the potential to realize the expected effects of the system. This definition also provides an opening for assessing a system in development by addressing the intention to use the system if the system was available as potential acceptance.

2.2. Measurement of Driver/Passenger Acceptance

Measurement of driver/passenger acceptance must be such an uncertain work because it is based on individuals’ emotional factors. However, measurement of those uncertain human factors has been researched in the fields of drivers’ mental workload. Thus, as well known, the solutions to measure drivers’ mental workload are classified into three categories: subjective measures, performance measures, and physiological measures [22].

One of the representative methodologies of subjective measures is a self-reporting method. Self-reporting methods have always been very attractive to many researchers. No one can provide a more accurate judgment concerning a mental load than the person experiencing it [23]. The self-reporting method is considered one of the best methods since this type of research comes closest to tapping into the essence of mental workload [24].

Performance measures are classified into three categories: primary-task performance measures, secondary-task performance measures, and reference tasks [23]. Primary task performance is a measure of the overall effectiveness of man–machine interactions [22]. For vehicle driving experiments, the number of errors made in tracking performance, speed of performance, or reaction time measures are frequently used as primary task performance measures. Secondary task measures can be taken when another task is added to the primary task. Reference tasks are standardized tasks that are performed ‘before’ and ‘after’ the task under evaluation, and they mainly serve as a validation instrument for trend effects.

The last type of measures represents those derived from the operator’s physiology. The advantage of physiological responses is that they do not require an overt response by the operator, and most cognitive tasks do not require overt behavior. Several disadvantages of physiological measures are mentioned in [25], i.e., the required specialized equipment and technical expertise and the critical signal-to-noise ratios. It is also stated that the operator’s physiology, a reflection of bodily functions, is further removed from operator–system performances than primary task performance.

Considering the measurement of driver acceptance concerning the above-described methodologies, in this study, two methods were employed to measure and investigate driver/passenger acceptance: the subjective measure and the performance measure. Thus, the vehicle experiment program was designed concerning those methods as follows. As mentioned above, the subjective assessment, self-reporting on acceptance, is supposed to be immediately coded between ‘Accept’ and ‘Reject’ considering the reliability of true values. Additionally, with respect to the performance measures, dozens of measurable objective parameters were continuously collected for the vehicle experiments. By investigating the relationships between the subjective assessment results and the measured objective parameters, objective methodologies for the evaluation of driver acceptance were herein studied.

3. Driver/Passenger–Vehicle Interaction Database

For this study, a special database was needed that included control command, vehicle behaviors, relations between vehicles, subjective assessment data of the driving performance, etc. To derive the objective evaluation method, a driver–vehicle interaction database constructed in a previous study was used. The database provides specific experiment results of passenger–vehicle interaction data conducted with 19 selected drivers, and it contains various types of measurable parameters and subjectively assessed acceptance values. For this study, the drivers served as control system for the automatic lane change system and were classified into two groups, i.e., expert and novice. The selection and classification conditions were as follows: more than three years of work experience as a regular chauffeur for the experts, and less than one month of driving experience with their vehicles for the novices. The expert group was expected to show stable and safe driving results with a comparably higher acceptance rate from passengers. The other group of novice drivers was expected to show unstable and dangerous driving results with comparably lower acceptance rates from passengers. For the classification of these two groups, it was mainly assumed that the driving characteristics of the expert drivers would be more desirable concerning driver/passenger acceptance.

In the experimental program, only the lane change situation was considered and drivers served as a control system of autonomous driving systems, particularly an automatic lane change system. Two passengers were designated to observe and evaluate lane change performance considering both vehicle behaviors and traffic relations. Throughout the experiment, the drivers drove and conducted lane changes not only autonomously but also as commanded. The database contains a massive amount of information on the results of subjective assessment and various objective data for lane change situations on an expressway, with vehicle speeds from 80 kph to 90 kph on average. As shown in Figure 1a, 19 drivers participated in this test: 10 of them were considered expert drivers; the other 9 drivers were considered novice drivers. Thus, 1823 events were collected through this test; 45% of those events were directed by the experimenter; the other cases were the results of the drivers’ own decisions. Figure 1b shows the ratios of lane change situations classified by certain conditions between the expert and novice drivers. Figure 1c shows the ratios between accepted and rejected lane change cases. It is seen that accepted cases were two times greater in occurrence than the rejected cases.

4. Objective Evaluation Methodology for Passenger Acceptance

4.1. Methodology of Objective Parameter Analysis and Selection

To study the objective parameters related to the subjective assessment results, a significance test and sensitivity values were used [15]. First, a significance test was performed on the parameters measured in relation to passengers’ acceptance during lane changes. A null hypothesis and alternative hypothesis are shown in Equation (1) and were tested. X and Y represent the sets of all parameters observed for accepted and rejected cases during lane changes, and E[X] and E[Y] are the statistically expected values of each set.

$$\mathrm{Hypothesis} \left\{\begin{array}{ll}{H}_0(null)& :\mathrm{E}[\mathrm{X}]=\mathrm{E}[\mathrm{Y}]\\ H_1(alternative)& :\mathrm{E}[\mathrm{X}]\ne \mathrm{E}[\mathrm{Y}]\end{array}\right.$$

(1)

The t-ratio for the significance test of each parameter in relation to lane change acceptance was calculated as shown in Equation (2):

$$\mathrm{t}=\frac{\overline{\mathrm{X}}-\overline{\mathrm{Y}}}{\sqrt{\frac{(n_{\mathrm{X}}-1)S_{\mathrm{X}}^2+(n_{\mathrm{Y}}-1)S_{\mathrm{Y}}^2}{(n_{\mathrm{X}}-1)(n_{\mathrm{Y}}-1)}}\sqrt{\frac{1}{n_{\mathrm{X}}}+\frac{1}{n_{\mathrm{Y}}}}}$$

(2)

where n_X and n_Y are the size of the samples, $\overline{\mathrm{X}}, \overline{\mathrm{Y}}$ are the sample means, and $S_{\mathrm{X}}^2, S_{\mathrm{Y}}^2$ are the variances of X and Y. Through this statistically significant test, in general, significant parameters in relation to subjective acceptance were drawn at a significance level below 5% (p-value < 0.05) as the objective parameters in order to represent driver acceptance.

The acceptance sensitivity, i.e., the sensitivity of parameters between the accepted cases and rejected cases, is defined in Equation (3).

$${\mathit{S}}_{\mathit{Acceptance}}=\mathrm{abs}\left[\frac{E[\mathbf{Y}]-E[\mathbf{X}]}{E[\mathbf{X}]}\right]\times 100(\%)$$

(3)

Only sensitive parameters showing an acceptance sensitivity greater than certain sensitivity levels were selected for the objective evaluation of acceptance. In this study, the cut-off sensitivity levels were designated at 30% and were designed by comparing the estimation results of the driver acceptance models by varying the acceptance sensitivity levels.

4.2. Stochastic Estimation and Evaluation of the Acceptance

A modeling method of the driver acceptance evaluation model based on a stochastic estimation is proposed to estimate and evaluate passenger acceptance with the suggested parameter sets; this modeling method fundamentally follows the well-known theory of binary logistic regression. By comparing the acceptance models developed based on the different objective parameter sets, the interpretation performances of each parameter set can be validated.

As stated in reference [26], regression methods have become an integral component of any data analysis concerned with describing the relationship between a response variable and one or more explanatory variables. In Figure 2, an example of stochastic estimations with a single variable is shown to show the suggested schematic methodology of the acceptance estimation for the objective evaluation. For this single-variable estimation model, it can be stated that the model is more deterministic when the absolute values of the constant (${\beta }_0$) and coefficient (${\beta }_1$) are higher. In this study, the subjective assessment results, the outcome variables, were recorded in binary with various continuous variables, which were the measured parameters on the vehicle behaviors and relationships. Additionally, specific parameter sets were derived for driver acceptance. Thus, the acceptance model was presented based on multi-variables. Therefore, acceptance can be objectively evaluated by the developed acceptance evaluation model with the measured values for the given parameter sets. Logistic regression models are the most frequently used models for analyses of this kind of binary data, and the experimentally interpretable model to describe the relationship between an outcome variable and a set of explanatory variables can be derived as follows.

In this study, dozens of explanatory parameters were used to describe emotional acceptance. To derive a multi-variable logistic regression model, a collection of p variables denoted by the vector ${\mathbf{x}}^T$ was considered and in Equation (4):

$${\mathbf{x}}^T=\left\{\begin{array}{cccc}{x}_1,& x_2,& \dots ,& x_p\end{array}\right\}$$

(4)

where it is assumed that each of the variables in the vector X^T are interval-scaled.

By defining the conditional probability, the outcome denoted by $\mathrm{Pr}\left(Y=1\left|\mathbf{x}\right.\right)=\pi (\mathbf{x})$, the logit of the multiple logistic regression model is given by Equation (5).

$$g(\mathbf{x})=\ln \left(\frac{\pi (\mathbf{x})}{1-\pi (\mathbf{x})}\right)={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_p{x}_p$$

(5)

Then, the logistic regression model based on conditional probability $\left(\pi (\mathbf{x})\right)$ is formulated by Equation (6).

$$\begin{array}{ll}\pi (\mathbf{x})& =\mathrm{Pr}\left(Y=1\left|\mathbf{x}\right.\right)\\ & =\frac{e^{{\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_p{x}_p}}{1+e^{{\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_p{x}_p}}=\frac{e^{g(\mathbf{x})}}{1+e^{g(\mathbf{x})}}\end{array}$$

(6)

Here, it is assumed that there was a sample of n observations of the pairs $\left({\mathbf{x}}_i,y_i\right)$, where i = 1, 2, ⋯, n. To fit the model in (6), the unknown parameter set of vector ${\mathbf{\beta }}^T$ and the p + 1 coefficients of each defined explanatory variable in Equation (7) were obtained.

$${\mathbf{\beta }}^T=\left\{\begin{array}{cccc}{\beta }_0,& {\beta }_1,& \dots ,& {\beta }_p\end{array}\right\}$$

(7)

Then, for the objective evaluation, the acceptance was discerned via outcome value of the model in (6) with respect to a critical value, 0.5, as shown in Equation (8).

$$\mathrm{Decision} \mathrm{criteria} \left\{\begin{array}{c}\mathrm{Accept}: \pi (\mathbf{x})\ge 0.5\\ \mathrm{Reject}: \pi (\mathbf{x})<0.5\end{array}\right.$$

(8)

Finally, the values of ${\mathbf{\beta }}^T$ were derived by using the maximum likelihood method [27,28]. A receiver operating characteristics (ROC) curve was also utilized for visualizing, organizing, and selecting classifiers based on their performances [29]. The ROC curve is an effective method for describing the performances of a stochastic classifier by varying the threshold value for the decision. The performances are presented with a true positive rate and false positive rate considering the predicted true values, which are defined in Equations (9) and (10) as follows.

$$\mathrm{True} \mathrm{Positive} \mathrm{Rate}=\frac{True Positive}{True Positive+False Negative}$$

(9)

$$\mathrm{False} \mathrm{Positive} \mathrm{Rate}=\frac{Fasle Positive}{False Positive+True Negative}$$

(10)

Here, the true positive rate means that the rate of the estimated values is correct in 1 for the true values coded as 1, and the false positive rate means that the estimated values are wrong in 1 for the true values coded as 0, respectively. A representative performance index of the ROC curve for a stochastic classifier is the area under the curve (AUC). When considering the ROC space, the performance of the classifier was worse when the AUC was closer to 0.5 and better when it approached 1.0.

5. Driver Acceptance Evaluation Model

5.1. Parameter Selection

This section is organized as follows. The objective parameters for this research are described and defined in Section 5.1.1. In Section 5.1.2, Section 5.1.3 and Section 5.1.4, the objective parameters for the modeling of the acceptance evaluation are presented based on the methodology described in Section 4.1. The specific data group among the entire database is subjective and was classified according to expert drivers performing lane changes only as commanded and lane changes in traffic. The criteria for the choice of the data set were as follows: ‘expert drivers’ for fine driving performance, ‘as commanded’ for the automatic lane change system considered in this study, and ‘with traffic’ for a more challenging condition.

5.1.1. Objective Parameters

In the interaction database, each lane change case is described with various measurable, objective parameters. Table 1 shows the parameters selected for this study, including several representative parameters related to the control command, vehicle dynamic behavior, and relative relations with surrounding vehicles with their descriptive statistical values.

To analyze the statistical significance and the acceptance sensitivity of the parameters associated with the spatial relationships between vehicles in terms of driver acceptance, the relative positions regarding three target vehicles are defined in Figure 3. Each surrounding vehicle was defined as follows: Target A represents the leading vehicle in the active lane, Target B represents the leading vehicle in the target lane, and Target C represents the following vehicle in the target lane.

Table 1. Parameters selected for the study, including several parameters related to the control command, vehicle dynamic behavior, and relative relations with surrounding vehicles.

Objective Factors			Description
First Level	2nd Level	3rd Level
Control Command	TPS	Minimum	Throttle position sensor
		Maximum
		Median
		Mean
		Standard Deviation
	$\delta$	↑	Steering wheel angle
	$\dot{\delta }$	↑	Steering wheel angular velocity
Vehicle behavior	$Vx$	Minimum	Longitudinal velocity
		Maximum
		Median
		Mean
		Standard Deviation
	$Vy$	↑	Lateral velocity
	$\dot{V}x$	↑	Longitudinal acceleration
	$\dot{V}y$	↑	Lateral acceleration
	$\ddot{V}x$	↑	Longitudinal jerk
	$\ddot{V}y$	↑	Lateral jerk
	$\psi$	↑	Yaw angle
	$\dot{\psi }$	↑	Yaw angular velocity
	$\ddot{\psi }$	↑	Yaw angular acceleration
	$\varphi$	↑	Roll angle
	$\dot{\varphi }$	↑	Roll angular velocity
Vehicle Relationships (Targets A, B, C)	TTC	Minimum	Time to collision
		Maximum
		Median
		Mean
		Standard Deviation
	TTCx	↑	Longitudinal time to collision
	TTCy	↑	Lateral time to collision
	relV	↑	Relative velocity
	relVx	↑	Longitudinal relative velocity
	relVy	↑	Lateral relative velocity
	D	↑	Distance
	Dx	↑	Longitudinal distance
	Dy	↑	Lateral distance

↑ means “same as above”.

Table 1. Parameters selected for the study, including several parameters related to the control command, vehicle dynamic behavior, and relative relations with surrounding vehicles.

Objective Factors			Description
First Level	2nd Level	3rd Level
Control Command	TPS	Minimum	Throttle position sensor
		Maximum
		Median
		Mean
		Standard Deviation
	$\delta$	↑	Steering wheel angle
	$\dot{\delta }$	↑	Steering wheel angular velocity
Vehicle behavior	$Vx$	Minimum	Longitudinal velocity
		Maximum
		Median
		Mean
		Standard Deviation
	$Vy$	↑	Lateral velocity
	$\dot{V}x$	↑	Longitudinal acceleration
	$\dot{V}y$	↑	Lateral acceleration
	$\ddot{V}x$	↑	Longitudinal jerk
	$\ddot{V}y$	↑	Lateral jerk
	$\psi$	↑	Yaw angle
	$\dot{\psi }$	↑	Yaw angular velocity
	$\ddot{\psi }$	↑	Yaw angular acceleration
	$\varphi$	↑	Roll angle
	$\dot{\varphi }$	↑	Roll angular velocity
Vehicle Relationships (Targets A, B, C)	TTC	Minimum	Time to collision
		Maximum
		Median
		Mean
		Standard Deviation
	TTCx	↑	Longitudinal time to collision
	TTCy	↑	Lateral time to collision
	relV	↑	Relative velocity
	relVx	↑	Longitudinal relative velocity
	relVy	↑	Lateral relative velocity
	D	↑	Distance
	Dx	↑	Longitudinal distance
	Dy	↑	Lateral distance

↑ means “same as above”.

The primary parameters are the time to collision (TTC), relative velocity (relV), and distance between vehicles (D); the definitions are given by Equations (11), (12) and (13), respectively.

$$TT{C}_i=\frac{\mathrm{abs}[P_{ego}-P_{\mathrm{Target}}]}{V_{ego}}$$

(11)

where TTC_i is the time to collision at the ith step; P_ego and P_{Target A,B,C} are the positions of the ego-vehicle and each target vehicle; and V_ego is the velocity of the driving vehicle.

$$rel{V}_i=V_{ego}-V_{TargetA,B,C}$$

(12)

where relV_i is the relative velocity between the driving vehicle and the target vehicles at the ith step and V_{Target A,B,C} is the velocity of each target vehicle.

$$D_i=\mathrm{abs}[P_{ego}-P_{TargetA,B,C}]$$

(13)

where D_i is the absolute distance between the ego-vehicle and the target vehicles at the ith step.

5.1.2. Acceptance in Relation to the Control Commands

The representative parameters of the control commands in relation to subjective acceptance were analyzed by their statistical significances and acceptance sensitivities. The parameter sets of the control commands are defined by Equations (14) and (15): X for the accepted cases and Y for the rejected cases.

$$\begin{array}{ll}{\mathbf{X}}_{Control}& =\left[\begin{array}{c}TP{S}^{Accepted}\\ SW{A}^{Accepted}\\ SW{V}^{Accepted}\end{array}\right]\\ & =\left[\begin{array}{ccc}\begin{array}{l}\min \left[TP{S}^{Accepted}\right]\\ ~n({\mu }_{\min \left[TP{S}^{Accepted}\right]},{\sigma }_{\min \left[TP{S}^{Accepted}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[TP{S}^{Accepted}\right]\\ ~n({\mu }_{\mathrm{SD}\left[TP{S}^{Accepted}\right]},{\sigma }_{\mathrm{SD}\left[TP{S}^{Accepted}\right]}^2)\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{l}\min \left[SW{V}^{Accepted}\right]\\ ~n({\mu }_{\min \left[SW{V}^{Accepted}\right]},{\sigma }_{\min \left[SW{V}^{Accepted}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[SW{V}^{Accepted}\right]\\ ~n({\mu }_{\mathrm{SD}\left[SW{V}^{Accepted}\right]},{\sigma }_{\mathrm{SD}\left[SW{V}^{Accepted}\right]}^2)\end{array}\end{array}\right]\end{array}$$

(14)

$$\begin{array}{ll}{\mathbf{Y}}_{Control}& =\left[\begin{array}{c}TP{S}^{\mathit{Rejected}}\\ SW{A}^{\mathit{Rejected}}\\ SW{V}^{\mathit{Rejected}}\end{array}\right]\\ & =\left[\begin{array}{ccc}\begin{array}{l}\min \left[TP{S}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\min \left[TP{S}_{}^{\mathit{Rejected}}\right]},{\sigma }_{\min \left[TP{S}^{\mathit{Rejected}}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[TP{S}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\mathrm{SD}\left[TP{S}^{\mathit{Rejected}}\right]},{\sigma }_{\mathrm{SD}\left[TP{S}^{\mathit{Rejected}}\right]}^2)\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{l}\min \left[SW{V}^{Accepted}\right]\\ ~n({\mu }_{\min \left[SW{V}^{Accepted}\right]},{\sigma }_{\min \left[SW{V}^{Accepted}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[SW{V}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\mathrm{SD}\left[SW{V}^{\mathit{Rejected}}\right]},{\sigma }_{\mathrm{SD}\left[SW{V}^{\mathit{Rejected}}\right]}^2)\end{array}\end{array}\right]\end{array}$$

(15)

Each set of parameters comprises the following descriptive statistical values: the minimum value, maximum value, median value, mean value, and standard deviation. A significance test and acceptance sensitivity analysis were performed for each parameter. Parameters for which the significance level was below 5% (p-value < 0.05) or the acceptance sensitivity was over 30% are presented in

Table 2. Comparison of the parameter selection results between the significance test and acceptance sensitivity analysis regarding the control commands.

Parameter		Unit	Descriptive Statistics				Significance Test (1)		Acceptance Sensitivity (%)
			Accepted		Rejected
			E[X]	Var[X]	E[Y]	Var[Y]	t	p-Value
$TPS$	standard deviation	%	7.400	30.410	9.954	14.011	1.497	0.141	34.525
$\delta$	minimum	°	0.535	1.824	0.994	1.977	1.020	0.313	85.787
$\stackrel{·}{\delta }$	median	°/s	0.001	0.000	0.000	0.000	−0.614	0.542	63.324

⁽¹⁾ for those selected by one method.

. As seen in the results, three parameters met the acceptance acceptance sensitivity criterion while no parameters met the statistical significance criterion. Additionally, Table 2 shows that no parameters were selected according to the historical effects on the acceptance in both the statistical significance test and the acceptance sensitivity analysis. Hence, the three parameters for the control commands selected from the acceptance sensitivity analysis are throttle open (TPS), steering wheel angle $\left(\delta \right)$, and steering wheel angular velocity $\left(\dot{\delta }\right)$.

5.1.3. Acceptance in Relation to Vehicle Behaviors

The representative parameters of the vehicle dynamic behaviors in relation to the subjective acceptance were also analyzed for statistical significance and acceptance sensitivity. The sets of parameters for vehicle dynamic behaviors are the same as those defined in Equations (16) and (17).

$$\begin{array}{l}{\mathbf{X}}_{Vehicle}^{}=\left[\begin{array}{ccccc}V{x}^{Accepted}& A{x}^{Accepted}& J{x}^{Accepted}& V{y}^{Accepted}& A{y}^{Accepted}\end{array}\right.\\ {\left. \begin{array}{ccccc}J{y}^{Accepted}& {\dot{\psi }}^{Accepted}& {\ddot{\psi }}^{Accepted}& {\varphi }^{Accepted}& {\dot{\varphi }}^{Accepted}\end{array}\right]}^T\\ =\left[\begin{array}{ccc}\begin{array}{l}\min \left[V{x}_{}^{Accepted}\right]\\ ~n({\mu }_{\min \left[V{x}_{}^{Accepted}\right]},{\sigma }_{\min \left[V{x}_{}^{Accepted}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[V{x}_{}^{Accepted}\right]\\ ~n({\mu }_{\mathrm{SD}\left[V{x}_{}^{Accepted}\right]},{\sigma }_{\mathrm{SD}\left[V{x}_{}^{Accepted}\right]}^2)\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{l}\min \left[{\dot{\varphi }}_{}^{Accepted}\right]\\ ~n({\mu }_{\min \left[{\dot{\varphi }}_{}^{Accepted}\right]},{\sigma }_{\min \left[{\dot{\varphi }}_{}^{Accepted}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[{\dot{\varphi }}_{}^{Accepted}\right]\\ ~n({\mu }_{\mathrm{SD}\left[{\dot{\varphi }}_{}^{Accepted}\right]},{\sigma }_{\mathrm{SD}\left[{\dot{\varphi }}_{}^{Accepted}\right]}^2)\end{array}\end{array}\right]\end{array}$$

(16)

$$\begin{array}{l}{\mathbf{Y}}_{Vehicle}^{}=\left[\begin{array}{ccccc}V{x}^{\mathit{Rejected}}& A{x}^{\mathit{Rejected}}& J{x}^{\mathit{Rejected}}& V{y}^{\mathit{Rejected}}& A{y}^{\mathit{Rejected}}\end{array}\right.\\ {\left. \begin{array}{ccccc}J{y}^{\mathit{Rejected}}& {\stackrel{·}{\psi }}^{\mathit{Rejected}}& {\ddot{\psi }}^{\mathit{Rejected}}& {\varphi }^{\mathit{Rejected}}& {\stackrel{·}{\varphi }}^{\mathit{Rejected}}\end{array}\right]}^T\\ =\left[\begin{array}{ccc}\begin{array}{l}\min \left[V{x}_{}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\min \left[V{x}_{}^{\mathit{Rejected}}\right]},{\sigma }_{\min \left[V{x}_{}^{\mathit{Rejected}}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[V{x}_{}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\mathrm{SD}\left[V{x}_{}^{\mathit{Rejected}}\right]},{\sigma }_{\mathrm{SD}\left[V{x}_{}^{\mathit{Rejected}}\right]}^2)\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{l}\min \left[{\stackrel{·}{\varphi }}_{}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\min \left[{\stackrel{·}{\varphi }}_{}^{\mathit{Rejected}}\right]},{\sigma }_{\min \left[{\stackrel{·}{\varphi }}_{}^{\mathit{Rejected}}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[{\stackrel{·}{\varphi }}_{}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\mathrm{SD}\left[{\stackrel{·}{\varphi }}_{}^{\mathit{Rejected}}\right]},{\sigma }_{\mathrm{SD}\left[{\stackrel{·}{\varphi }}_{}^{\mathit{Rejected}}\right]}^2)\end{array}\end{array}\right]\end{array}$$

(17)

Similar to those for the control commands, the parameter results for the vehicle behaviors are presented in

Table 3. Comparison of the results of selected parameters between the significance test and acceptance sensitivity analysis regarding the parameters associated with dynamic vehicle behaviors.

Parameter		Unit	Descriptive Statistics				Significance Test (1)		Acceptance Sensitivity (%)
			Accepted		Rejected
			E[X]	Var[X]	E[Y]	Var[Y]	t	p-Value
$Vx$	minimum	m/s	28.287	10.367	25.632	10.689	−2.489 *	0.016	9.387
	maximum	m/s	30.028	9.872	27.454	10.492	−2.465 *	0.017	8.574
	median	m/s	29.138	9.619	26.667	11.711	−2.356 *	0.023	8.478
	mean	m/s	29.148	9.756	26.606	11.144	−2.428 *	0.019	8.723
$\stackrel{·}{V}x$	minimum	m/s2	0.164	0.057	−0.195	0.115	−4.100 ***	0.000	18.831
	maximum	m/s2	0.559	0.068	0.300	0.161	−2.642 *	0.011	46.399
	median	m/s2	0.391	0.060	0.022	0.225	−3.584 ***	0.001	94.391
	mean	m/s2	0.384	0.054	−0.051	0.366	−3.743 ***	0.000	86.642
	standard deviation	m/s2	0.114	0.005	0.168	0.004	2.439 *	0.018	47.525
$\ddot{V}x$	minimum	m/s3	−0.476	0.097	−0.688	0.122	−2.001	0.051	44.520
	maximum	m/s3	0.525	0.228	0.687	0.238	1.020	0.313	30.748
	median	m/s3	−0.001	0.008	−0.063	0.015	−1.948	0.057	10,964.7
	mean	m/s3	0.009	0.010	−0.054	0.017	−1.807	0.077	492.66
	standard deviation	m/s3	0.001	0.000	0.002	0.000	1.570	0.123	207.065
$\stackrel{·}{\psi }$	minimum	°/s	0.003	0.000	0.006	0.000	1.213	0.231	92.947
	median	°/s	0.015	0.000	0.021	0.000	1.470	0.148	46.190
	mean	°/s	0.015	0.000	0.021	0.000	1.365	0.179	36.238
$\varphi$	minimum	°	0.976	2.052	0.603	0.844	−0.847	0.401	38.249
	standard deviation	°	0.443	0.065	0.629	0.367	1.550	0.128	42.135
$\stackrel{·}{\varphi }$	minimum	°/s	0.003	0.000	0.001	0.000	−0.787	0.435	46.556

⁽¹⁾ Level of significance: p < 0.05 *, p < 0.001 ***; light grey is for those selected by one method, dark grey is for those selected by two methods.

. As seen in the results, only five parameters of longitudinal velocity ($Vx$) and longitudinal accelerations ($\stackrel{·}{V}x$) met the significance test criterion. On the other hand, 16 parameters, associated with longitudinal acceleration ($\stackrel{·}{V}x$), longitudinal jerk ($\ddot{V}x$), yaw angular velocity ($\stackrel{·}{\psi }$), roll angular motion ($\varphi$), and roll angular velocity ($\stackrel{·}{\varphi }$), met the acceptance sensitivity criterion. Longitudinal acceleration parameters ($\stackrel{·}{V}x$) met the criteria for both the significance test and sensitivity analysis. It is inferred that the parameters for longitudinal acceleration are important parameters for the evaluation of acceptance. Moreover, no parameters related to lateral motion, i.e., lateral velocity, lateral acceleration, and lateral jerk, met the criteria of the significance test and sensitivity analysis. Therefore, it can be stated that longitudinal behaviors are more sensitive to lane change events than lateral dynamic behaviors, although it is generally believed that lateral dynamic behaviors are dominant in a driver’s acceptance of lane change situations.

5.1.4. Acceptance in Relation to Traffic

According to the surrounding vehicles defined in Section 5.1.1, the parameters concerning acceptance were analyzed for their statistical significances and acceptance sensitivities with respect to subjective acceptance. The subjected parameter sets are defined by Equations (18) and (19) for each target vehicle.

$$\begin{array}{l}{\mathbf{X}}_{\mathit{Target} A,B,C}^{}=\left[\begin{array}{ccccc}TT{C}_{A,B,C}^{Accepted}& TTC{x}_{A,B,C}^{Accepted}& TTC{y}_{A,B,C}^{Accepted}& rel{V}_{A,B,C}^{Accepted}& relV{x}_{A,B,C}^{Accepted}\end{array}\right.\\ {\left. \begin{array}{cccc}relV{y}_{\mathrm{A},\mathrm{B},\mathrm{C}}^{Accepted}& D_{\mathrm{A},\mathrm{B},\mathrm{C}}^{Accepted}& D{x}_{A,B,C}^{Accepted}& D{y}_{A,B,C}^{Accepted}\end{array}\right]}^T\\ =\left[\begin{array}{ccc}\begin{array}{l}\min \left[TT{C}_{\mathrm{A},\mathrm{B},\mathrm{C}}^{Accepted}\right]\\ ~n({\mu }_{\min \left[TT{C}_{A,B,C}^{Accepted}\right]},{\sigma }_{\min \left[TT{C}_{A,B,C}^{Accepted}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[TT{C}_{A,B,C}^{Accepted}\right]\\ ~n({\mu }_{\mathrm{SD}\left[TT{C}_{A,B,C}^{Accepted}\right]},{\sigma }_{\mathrm{SD}\left[TT{C}_{A,B,C}^{Accepted}\right]}^2)\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{l}\min \left[D{y}_{A,B,C}^{Accepted}\right]\\ ~n({\mu }_{\min \left[D{y}_{A,B,C}^{Accepted}\right]},{\sigma }_{\min \left[D{y}_{A,B,C}^{Accepted}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[D{y}_{A,B,C}^{Accepted}\right]\\ ~n({\mu }_{\mathrm{SD}\left[D{y}_{A,B,C}^{Accepted}\right]},{\sigma }_{\mathrm{SD}\left[D{y}_{A,B,C}^{Accepted}\right]}^2)\end{array}\end{array}\right]\end{array}$$

(18)

$$\begin{array}{l}{\mathbf{Y}}_{\mathit{Target} A,B,C}^{}=\left[\begin{array}{ccccc}TT{C}_{A,B,C}^{\mathit{Rejected}}& TTC{x}_{A,B,C}^{\mathit{Rejected}}& TTC{y}_{A,B,C}^{\mathit{Rejected}}& rel{V}_{A,B,C}^{\mathit{Rejected}}& relV{x}_{A,B,C}^{\mathit{Rejected}}\end{array}\right.\\ {\left. \begin{array}{cccc}relV{y}_{A,B,C}^{\mathit{Rejected}}& D_{A,B,C}^{\mathit{Rejected}}& D{x}_{A,B,C}^{\mathit{Rejected}}& D{y}_{A,B,C}^{\mathit{Rejected}}\end{array}\right]}^T\\ =\left[\begin{array}{ccc}\begin{array}{l}\min \left[TT{C}_{A,B,C}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\min \left[TT{C}_{A,B,C}^{\mathit{Rejected}}\right]},{\sigma }_{\min \left[TT{C}_{A,B,C}^{\mathit{Rejected}}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[TT{C}_{A,B,C}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\mathrm{SD}\left[TT{C}_{A,B,C}^{\mathit{Rejected}}\right]},{\sigma }_{\mathrm{SD}\left[TT{C}_{A,B,C}^{\mathit{Rejected}}\right]}^2)\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{l}\min \left[D{y}_{A,B,C}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\min \left[D{y}_{A,B,C}^{\mathit{Rejected}}\right]},{\sigma }_{\min \left[D{y}_{A,B,C}^{\mathit{Rejected}}\right]}^2)\end{array}& \cdots & \begin{array}{l}\mathrm{SD}\left[D{y}_{A,B,C}^{\mathit{Rejected}}\right]\\ ~n({\mu }_{\mathrm{SD}\left[D{y}_{A,B,C}^{\mathit{Rejected}}\right]},{\sigma }_{SD\left[D{y}_{A,B,C}^{\mathit{Rejected}}\right]}^2)\end{array}\end{array}\right]\end{array}$$

(19)

Table 4 presents the selected parameters and their significance levels and sensitivities. As seen in the results, only two parameters met the statistical significance test criterion for the leading vehicle in the active lane, Target A. On the other hand, 19 parameters for all the surrounding vehicles met the acceptance sensitivity criterion.

In the results of the surrounding vehicles for the lane change parameter, statistical significance was observed only for the leading vehicle in the active lane (Target A), while all of the surrounding vehicles (Target A, Target B, and Target C) were similarly sensitive to driver acceptance based on the acceptance sensitivity analysis. For all the surrounding vehicles, both time to collision (TTC) and the relative velocity (relV) were meaningful factors considering driver acceptance. On the other hand, no significant and sensitive parameters were found as related to the distance of the surrounding vehicles. It can be concluded that the role of parameters related to distances is already included in the time to collision (TTC). Additionally, few historical values were observed in either analysis, including the lateral time to collision (TTCy) for Target A in the significant test, the longitudinal time to collision (TTCx) for Target B, and the lateral time to collision (TTCy) for Target C in the sensitivity analysis.

Table 4. Comparison of the parameter selection results between the significance test and acceptance sensitivity analysis regarding vehicle relationships.

Parameter			Unit	Descriptive Statistics				Significance Test (1)		Acceptance Sensitivity (%)
				Accepted		Rejected
				E[X]	Var[X]	E[Y]	Var[Y]	t	p-Value
A	$TTCx$	maximum	s	0.269	172.598	0.752	943.830	0.068	0.946	179.044
	$TTCy$	minimum	s	−0.011	10.888	−1.245	2.103	−1.155	0.256	11,743.861
		standard deviation	s	0.178	1.008	0.743	0.711	2.383 *	0.023	317.775
	$relVx$	mean	m/s	−3.271	36.888	0.619	6.732	2.026 *	0.048	81.076
	$relVy$	maximum	m/s	0.141	0.717	0.425	3.978	0.850	0.399	201.364
		mean	m/s	−0.029	0.925	0.066	2.591	0.335	0.739	126.919
B	$TTC$	maximum	s	9.012	524.509	21.989	523.151	1.518	0.135	143.988
		standard deviation	s	1.292	28.789	4.229	26.036	1.562	0.125	227.424
	$TTCx$	minimum	s	−1.270	91.603	3.869	40.250	1.412	0.164	204.690
		maximum	s	2.820	309.389	10.483	167.218	1.227	0.226	271.747
		median	s	0.075	93.049	4.964	53.769	1.336	0.188	6535.908
		mean	s	0.174	105.859	5.522	64.751	1.396	0.169	3069.538
	$relVx$	maximum	m/s	−0.244	5.861	1.633	18.280	1.458	0.151	570.526
	$relVy$	maximum	m/s	0.015	37.588	−0.225	0.933	−0.417	0.679	1362.499
C	$TTCx$	minimum	s	−0.367	196.754	−5.056	1369.453	−0.410	0.684	1279.101
		median	s	4.123	153.344	20.982	1271.483	1.441	0.156	408.877
	$TTCy$	minimum	s	−2.701	23.580	−9.611	127.935	−1.422	0.164	255.864
		median	s	−1.881	21.257	−5.473	110.934	−0.865	0.392	190.902
		mean	s	−0.097	12.920	−3.118	60.352	−0.794	0.432	3119.044
		standard deviation	s	1.867	3.187	4.419	20.275	1.277	0.210	136.632

⁽¹⁾ Level of significance: p < 0.05 *, light grey is for those selected by one method, dark grey is for those selected by two methods.

Table 4. Comparison of the parameter selection results between the significance test and acceptance sensitivity analysis regarding vehicle relationships.

Parameter			Unit	Descriptive Statistics				Significance Test (1)		Acceptance Sensitivity (%)
				Accepted		Rejected
				E[X]	Var[X]	E[Y]	Var[Y]	t	p-Value
A	$TTCx$	maximum	s	0.269	172.598	0.752	943.830	0.068	0.946	179.044
	$TTCy$	minimum	s	−0.011	10.888	−1.245	2.103	−1.155	0.256	11,743.861
		standard deviation	s	0.178	1.008	0.743	0.711	2.383 *	0.023	317.775
	$relVx$	mean	m/s	−3.271	36.888	0.619	6.732	2.026 *	0.048	81.076
	$relVy$	maximum	m/s	0.141	0.717	0.425	3.978	0.850	0.399	201.364
		mean	m/s	−0.029	0.925	0.066	2.591	0.335	0.739	126.919
B	$TTC$	maximum	s	9.012	524.509	21.989	523.151	1.518	0.135	143.988
		standard deviation	s	1.292	28.789	4.229	26.036	1.562	0.125	227.424
	$TTCx$	minimum	s	−1.270	91.603	3.869	40.250	1.412	0.164	204.690
		maximum	s	2.820	309.389	10.483	167.218	1.227	0.226	271.747
		median	s	0.075	93.049	4.964	53.769	1.336	0.188	6535.908
		mean	s	0.174	105.859	5.522	64.751	1.396	0.169	3069.538
	$relVx$	maximum	m/s	−0.244	5.861	1.633	18.280	1.458	0.151	570.526
	$relVy$	maximum	m/s	0.015	37.588	−0.225	0.933	−0.417	0.679	1362.499
C	$TTCx$	minimum	s	−0.367	196.754	−5.056	1369.453	−0.410	0.684	1279.101
		median	s	4.123	153.344	20.982	1271.483	1.441	0.156	408.877
	$TTCy$	minimum	s	−2.701	23.580	−9.611	127.935	−1.422	0.164	255.864
		median	s	−1.881	21.257	−5.473	110.934	−0.865	0.392	190.902
		mean	s	−0.097	12.920	−3.118	60.352	−0.794	0.432	3119.044
		standard deviation	s	1.867	3.187	4.419	20.275	1.277	0.210	136.632

⁽¹⁾ Level of significance: p < 0.05 *, light grey is for those selected by one method, dark grey is for those selected by two methods.

5.2. Modeling of Driver Acceptance

In this section, two driver acceptance models using two different parameter selection methods are presented: the statistical significance test and the suggested acceptance sensitivity. To derive the models, a specific set of data was considered: classified as driven by expert drivers, performing lane changes only as commanded, and lane changes with traffic. The formulations of the models are defined in Equation (5) in Section 4.2. The vectors of the p input variable ${\mathbf{x}}^T$ and the p unknown parameters ${\mathbf{\beta }}^T$ are also the same as those expressed in Equations (4) and (7), respectively.

From the results of the parameter selection study, the two different input variable vectors are presented in Table 2 and Table 3 in Section 5.1; as a result, 11 input variables and their coefficients were used for one model based on their statistical significances, and 38 input variables and their coefficients were used for the other model based on the results of the acceptance sensitivity analysis, which is the suggested method in this study.

Then, the vectors of their coefficients (${\mathbf{\beta }}^T$) were designed by the maximum log-likelihood method described in Section 4.2 [27,28].

Table 5. Modeling results of the driver acceptance models using the two different parameter selection methods, the statistical significance test and the acceptance sensitivity analysis, for a specific data set classified as experts/commanded/with traffic.

Statistical Significance p-Value				$\mathbf{Acceptance} \mathbf{Sensitivity} \left({\mathit{S}}_{\mathit{A}\mathit{c}\mathit{c}\mathit{e}\mathit{p}\mathit{t}\mathit{a}\mathit{n}\mathit{c}\mathit{e}}\right)$
$\mathbf{Parameters} \left(\mathbf{x}\right)$		$\mathbf{Coefficients} \left(\mathbf{\beta }\right)$		$\mathbf{Parameters} \left(\mathbf{x}\right)$		$\mathbf{Coefficients} \left(\mathbf{\beta }\right)$
$x_1$	$\min \left[Vx\right]$	${\beta }_1$	−0.206	$x_1$	$\min \left[TPS\right]$	${\beta }_1$	−0.013
$x_2$	$\max \left[Vx\right]$	${\beta }_2$	−0.839	$x_2$	$\min \left[\stackrel{·}{\delta }\right]$	${\beta }_2$	−669.777
$x_3$	$\mathrm{median}\left[Vx\right]$	${\beta }_3$	−3.960	$x_3$	$\mathrm{median}\left[\stackrel{·}{\delta }\right]$	${\beta }_3$	−2.214
$x_4$	$\mathrm{mean}\left[Vx\right]$	${\beta }_4$	5.243	$x_4$	$\max \left[\stackrel{·}{V}x\right]$	${\beta }_4$	−1.374
$x_5$	$\min \left[\stackrel{·}{V}x\right]$	${\beta }_5$	−7.263	$x_5$	$\mathrm{median}\left[\stackrel{·}{V}x\right]$	${\beta }_5$	14.199
$x_6$	$\max \left[\stackrel{·}{V}x\right]$	${\beta }_6$	5.274	$x_6$	$\mathrm{mean}\left[\stackrel{·}{V}x\right]$	${\beta }_6$	−3.881
$x_7$	$\mathrm{median}\left[\stackrel{·}{V}x\right]$	${\beta }_7$	5.189	$x_7$	$\mathrm{SD}\left[\stackrel{·}{V}x\right]$	${\beta }_7$	−20.073
$x_8$	$\mathrm{mean}\left[\stackrel{·}{V}x\right]$	${\beta }_8$	0.233	$x_8$	$\min \left[\ddot{V}x\right]$	${\beta }_8$	0.148
$x_9$	$\mathrm{SD}\left[\stackrel{·}{V}x\right]$	${\beta }_9$	−19.741	$x_9$	$\max \left[\ddot{V}x\right]$	${\beta }_9$	5.142
$x_{10}$	$\mathrm{SD}\left[TTC{y}_A\right]$	${\beta }_{10}$	0.103	$x_{10}$	$\mathrm{median}\left[\ddot{V}x\right]$	${\beta }_{10}$	30.755
$x_{11}$	$\mathrm{mean}\left[relV{x}_A\right]$	${\beta }_{11}$	−0.142	$x_{11}$	$\mathrm{mean}\left[\ddot{V}x\right]$	${\beta }_{11}$	−26.580
		${\beta }_0$	−6.663	$x_{12}$	$\min \left[\ddot{V}y\right]$	${\beta }_{12}$	−1263.746
				$x_{13}$	$\min \left[\stackrel{·}{\psi }\right]$	${\beta }_{13}$	−127.343
				$x_{14}$	$\mathrm{median}\left[\stackrel{·}{\psi }\right]$	${\beta }_{14}$	−452.133
				$x_{15}$	$\mathrm{mean}\left[\stackrel{·}{\psi }\right]$	${\beta }_{15}$	569.911
				$x_{16}$	$\min \left[\varphi \right]$	${\beta }_{16}$	1.046
				$x_{17}$	$\mathrm{SD}\left[\varphi \right]$	${\beta }_{17}$	−2.377
				$x_{18}$	$\min \left[\stackrel{·}{\varphi }\right]$	${\beta }_{18}$	−148.541
				$x_{19}$	$\max \left[TTC{x}_A\right]$	${\beta }_{19}$	−0.012
				$x_{20}$	$\min \left[TTC{y}_A\right]$	${\beta }_{20}$	−0.179
				$x_{21}$	$\mathrm{SD}\left[TTC{y}_A\right]$	${\beta }_{21}$	−0.730
				$x_{22}$	$\mathrm{mean}\left[relV{x}_A\right]$	${\beta }_{22}$	−0.456
				$x_{23}$	$\max \left[relV{y}_A\right]$	${\beta }_{23}$	−0.901
				$x_{24}$	$\mathrm{mean}\left[relV{y}_A\right]$	${\beta }_{24}$	−0.103
				$x_{25}$	$\max \left[TT{C}_B\right]$	${\beta }_{25}$	−0.174
				$x_{26}$	$\mathrm{SD}\left[TT{C}_B\right]$	${\beta }_{26}$	0.707
				$x_{27}$	$\min \left[TTC{x}_B\right]$	${\beta }_{27}$	−1.151
				$x_{28}$	$\max \left[TTC{x}_B\right]$	${\beta }_{28}$	−0.156
				$x_{29}$	$\mathrm{median}\left[TTC{x}_B\right]$	${\beta }_{29}$	1.085
				$x_{30}$	$\mathrm{mean}\left[TTC{x}_B\right]$	${\beta }_{30}$	0.028
				$x_{31}$	$\max \left[relV{x}_B\right]$	${\beta }_{31}$	−0.165
				$x_{32}$	$\max \left[relV{y}_B\right]$	${\beta }_{32}$	0.604
				$x_{33}$	$\min \left[TTC{x}_C\right]$	${\beta }_{33}$	0.188
				$x_{34}$	$\mathrm{median}\left[TTC{x}_C\right]$	${\beta }_{34}$	−0.175
				$x_{35}$	$\min \left[TTC{y}_C\right]$	${\beta }_{35}$	0.087
				$x_{36}$	$\mathrm{median}\left[TTC{y}_C\right]$	${\beta }_{36}$	0.271
				$x_{37}$	$\mathrm{mean}\left[TTC{y}_C\right]$	${\beta }_{37}$	−0.283
				$x_{38}$	$\mathrm{SD}\left[TTC{y}_C\right]$	${\beta }_{38}$	0.139
						${\beta }_0$	309

presents the modeling results of the driver acceptance evaluation models using the different parameter selection methods, the two different sets of the input variables derived by the statistical significance test and the acceptance sensitivity, and their respective coefficient values. In addition, the effective ratios of the selected parameters for each method were analyzed using a Pareto chart as shown in Figure 4. It was observed that the parameters which were positioned higher had a greater impact on each parameter selection method. Specifically, the sensitivity analysis indicated that approximately 72% of the selected variables were from the top four parameters for the significance test, while the effective ratios of the top five parameters accounted for about 60% of the overall effect in the sensitivity analysis.

The performances of the two different models are compared in Figure 5, and the performance comparisons are summarized in several aforementioned indexes, as shown in Table 6. As seen in Figure 5a, the driver acceptance model using the acceptance sensitivity precisely estimated the driver’s acceptance. Considering the ROC curves in Figure 5b, the classification performance for the driver acceptance model using the suggested method showed a much better classification performance than that of the other driver acceptance model using the conventional method; the areas under the curves were 0.848 for the model using the statistical significance test method and 0.998 for the model using the acceptance sensitivity analysis method. Additionally, all the performance values of the model using the acceptance sensitivity analysis were much better than those of the other model in most of the performance indexes, as seen in the results: 0.256 vs. 0.064 for the standard error, 0.062 vs. 0.016 for the variance of the standard error, 23.905 vs. 3.822 for the sum of the squared error, 68.98% vs. 97.33% for the classification accuracy, and 0.848 vs. 0.998 for the area under the curve.

The estimation accuracy of the model using the statistical significance test between the rejected and accepted cases showed a considerable difference of about 22 percentage points: 53.57% for the rejected cases and 75.57% for the accepted cases. However, a smaller difference for the model using the acceptance sensitivity analysis with 94.64% for the rejected cases and 98.47% for the accepted cases was observed for the model using the acceptance sensitivity analysis. In those comparisons of the classification performance between the two different models, it was also observed that the driver acceptance model with the objective parameters selected by the suggested method was more robust for both the rejected events and the accepted events. The results showed that the classification accuracies for the rejected cases were generally worse than the accuracies for the accepted cases. This is due to the smaller size of the sample sets for estimating the sets of coefficients ${\mathbf{\beta }}^T$. Overall, the different pesrformance levels are based on the number of modeling parameters for each model. The estimation performance of the driver acceptance model could be improved by including a larger number of modeling parameters. This is because the model formulation is based on stochastic regression, which benefits from a more comprehensive set of parameters. However, even though a simple driver acceptance model with a smaller number of modeling parameters is desirable, it is not valuable if the estimation performance of the model with the smaller number of modeling parameters is insufficient. As seen in the results of the two different driver acceptance evaluation models, the driver acceptance model using the statistical significance test was insufficient to interpret the driver acceptance, meaning that the parameter selection method did not derive interpretable and important parameters well. However, the other driver acceptance evaluation model using the acceptance sensitivity analysis was interpretable for the acceptance, although the model had a comparably higher number of modeling parameters. Therefore, the parameter selection method of the acceptance sensitivity analysis can derive interpretable parameters, which are reasonably important to describe subjective driver acceptance.

Table 6. Performance comparisons of the driver acceptance models using the different parameter selection methods, the statistical significance test and the acceptance sensitivity analysis, for a specific data set classified as experts/commanded/with traffic.

Model Classification	Performance Index	Rejected (Sample Size: 56)	Accepted (Sample Size: 131)	Total (Sample Size: 187)
Driver Acceptance Model using Statistical Significance p-value	E[SE]	0.428	0.183	0.256
	Var[SE]	0.108	0.025	0.062
	SSE	16.267	7.638	23.905
	Accuracy (number of correct)	53.57% (30)	75.57% (99)	68.98 (129)
	AUC			0.848
Driver Acceptance Model using Acceptance Sensitivity $\left(S_{\mathit{Acceptance}}\right)$	E[SE]	0.107	0.046	0.064
	Var[SE]	0.028	0.010	0.016
	SSE	2.200	1.622	3.822
	Accuracy (number of correct)	94.64% (53)	98.47% (129)	97.33 (182)
	AUC			0.998

Table 6. Performance comparisons of the driver acceptance models using the different parameter selection methods, the statistical significance test and the acceptance sensitivity analysis, for a specific data set classified as experts/commanded/with traffic.

Model Classification	Performance Index	Rejected (Sample Size: 56)	Accepted (Sample Size: 131)	Total (Sample Size: 187)
Driver Acceptance Model using Statistical Significance p-value	E[SE]	0.428	0.183	0.256
	Var[SE]	0.108	0.025	0.062
	SSE	16.267	7.638	23.905
	Accuracy (number of correct)	53.57% (30)	75.57% (99)	68.98 (129)
	AUC			0.848
Driver Acceptance Model using Acceptance Sensitivity $\left(S_{\mathit{Acceptance}}\right)$	E[SE]	0.107	0.046	0.064
	Var[SE]	0.028	0.010	0.016
	SSE	2.200	1.622	3.822
	Accuracy (number of correct)	94.64% (53)	98.47% (129)	97.33 (182)
	AUC			0.998

5.3. Verifications of the Suggested Model

To verify the driver acceptance models developed in Section 5.2, another data group consisting of self-decided lane changes was utilized and the suggested models were developed based on the data group with the as-commanded lane changes. For this validation study, both expert and novice drivers were used in the data group for the self-decided lane changes. Additionally, the performance of the driver acceptance models was compared between the two different parameter selection methods.

Referring to the target data groups designated above, Figure 6 and Figure 7 show the estimated results and ROC analysis of the two types of suggested models, respectively. Table 7 and Table 8 present a summary of the performances for each data set in several indexes.

As seen in Table 7 and Table 8, the model derived from the sensitivity analysis showed better performances compared with the other model in most indexes for both the experts and novices. The comparisons of the classification accuracy were as follows: 78.31% vs. 96.79% for the experts and 68.27% vs. 99.52% for the novices. The AUC comparison results were as follows: 0.868 vs. 0.991 for the experts and 0.902 vs. 0.995 for the novices.

In the results presented in this section, the same aspects shown in Section 5.2 were observed, i.e., that the driver acceptance model derived with the acceptance sensitivity analysis showed a better performance in most of the indexes. Moreover, it can be stated that the acceptance model with the sensitivity analysis showed much better performances in estimation accuracy. Meanwhile, both the acceptance evaluation models with the two different methods of parameter selection were satisfactory in estimating acceptance when considering the AUC.

Table 7. Performance comparisons of the driver acceptance models using the different parameter selection methods, the statistical significance test and the acceptance sensitivity analysis, for specific data sets classified as expert drivers/self-decided/with traffic.

Model Classification	Performance Index	Rejected (Sample Size: 100)	Accepted (Sample Size: 149)	Total (Sample Size: 249)
Driver Acceptance Model using Statistical Significance p-value	E[SE]	0.342	0.230	0.275
	Var[SE]	0.095	0.035	0.062
	SSE	21.242	13.022	34.264
	Accuracy (number of correct)	61.00 (61)	89.93 (134)	78.31 (195)
	AUC			0.868
Driver Acceptance Model using Acceptance Sensitivity $\left(S_{\mathit{Acceptance}}\right)$	E[SE]	0.134	0.090	0.108
	Var[SE]	0.034	0.018	0.025
	SSE	5.150	3.910	9.059
	Accuracy (number of correct)	94.00 (94)	98.66 (147)	96.79 (241)
	AUC			0.991

Table 7. Performance comparisons of the driver acceptance models using the different parameter selection methods, the statistical significance test and the acceptance sensitivity analysis, for specific data sets classified as expert drivers/self-decided/with traffic.

Model Classification	Performance Index	Rejected (Sample Size: 100)	Accepted (Sample Size: 149)	Total (Sample Size: 249)
Driver Acceptance Model using Statistical Significance p-value	E[SE]	0.342	0.230	0.275
	Var[SE]	0.095	0.035	0.062
	SSE	21.242	13.022	34.264
	Accuracy (number of correct)	61.00 (61)	89.93 (134)	78.31 (195)
	AUC			0.868
Driver Acceptance Model using Acceptance Sensitivity $\left(S_{\mathit{Acceptance}}\right)$	E[SE]	0.134	0.090	0.108
	Var[SE]	0.034	0.018	0.025
	SSE	5.150	3.910	9.059
	Accuracy (number of correct)	94.00 (94)	98.66 (147)	96.79 (241)
	AUC			0.991

Table 8. Performance comparisons of the driver acceptance models using the different parameter selection methods, the statistical significance test and the acceptance sensitivity analysis, for specific data sets classified as novice drivers/self-decided/with traffic.

Model Classification	Performance Index	Rejected (Sample Size: 62)	Accepted (Sample Size: 146)	Total (Sample Size: 208)
Driver Acceptance Model using Statistical Significance p-value	E[SE]	0.357	0.152	0.213
	Var[SE]	0.112	0.030	0.063
	SSE	14.845	7.803	22.648
	Accuracy (number of correct)	56.45% (35)	73.29% (107)	68.27% (142)
	AUC			0.902
Driver Acceptance Model using Acceptance Sensitivity $\left(S_{\mathit{Acceptance}}\right)$	E[SE]	0.060	0.026	0.036
	Var[SE]	0.022	0.005	0.010
	SSE	1.576	0.837	2.413
	Accuracy (number of correct)	98.39% (61)	100.00% (146)	99.52% (207)
	AUC			0.995

Table 8. Performance comparisons of the driver acceptance models using the different parameter selection methods, the statistical significance test and the acceptance sensitivity analysis, for specific data sets classified as novice drivers/self-decided/with traffic.

Model Classification	Performance Index	Rejected (Sample Size: 62)	Accepted (Sample Size: 146)	Total (Sample Size: 208)
Driver Acceptance Model using Statistical Significance p-value	E[SE]	0.357	0.152	0.213
	Var[SE]	0.112	0.030	0.063
	SSE	14.845	7.803	22.648
	Accuracy (number of correct)	56.45% (35)	73.29% (107)	68.27% (142)
	AUC			0.902
Driver Acceptance Model using Acceptance Sensitivity $\left(S_{\mathit{Acceptance}}\right)$	E[SE]	0.060	0.026	0.036
	Var[SE]	0.022	0.005	0.010
	SSE	1.576	0.837	2.413
	Accuracy (number of correct)	98.39% (61)	100.00% (146)	99.52% (207)
	AUC			0.995

6. Discussion

This paper proposed an objective evaluation method for passenger acceptance of an autonomous driving system using a stochastic estimation methodology. In this study, an automatic lane change system was the target system as one of the representative autonomous driving systems.

In this study, the selection methods for objective parameters related to acceptance were defined in two ways: via a significance test and sensitivity analysis. Additionally, an acceptance evaluation model was proposed to determine acceptance with respect to the previously derived objective parameters. Based on the suggested methods of parameter selection and modeling, two types of acceptance evaluation models were derived and then compared and validated.

The validations used other data groups not used for the modeling. In the results, the model derived from the sensitivity analysis showed better performances compared with the other model in most indexes; the accuracy comparisons were 78.31% vs. 96.79% for the experts and 68.27% vs. 99.52% for the novices; the AUC comparisons were 0.868 vs. 0.991 for the experts and 0.902 vs. 0.995 for the novices, respectively. While both acceptance evaluation models using the two different methods of parameter selection were satisfactory in estimating the acceptance when considering the AUC, the acceptance model with the sensitivity analysis showed much better performances in estimation accuracy.

The differences in the performances are based on the number of modeling parameters used for each one. The estimation performance must be higher for a driver acceptance model with a larger number of modeling parameters because the model formulation is based on stochastic regression. However, even though a simple driver acceptance model with a smaller number of modeling parameters is desirable, it has no value when the estimation performance of a model with a smaller number of modeling parameters is insufficient. As seen in the cases of the verification results for the two different driver acceptance models, the driver acceptance model using the statistical significance test was insufficient to interpret the driver acceptance because of the smaller number of modeling parameters, meaning that the parameter selection method did not derive interpretable and important parameters well. On the other hand, the other driver acceptance model using the acceptance sensitivity analysis was interpretable, although the model had a comparably higher number of modeling parameters. This aspect means that the parameter selection method of the acceptance sensitivity analysis can derive interpretable parameters, which are reasonably important to describe subjective driver acceptance. Nevertheless, there are remaining tasks to be addressed, including conducting further parameter analyses and studies to represent smaller parameter sets. Additionally, a validation study is needed to compare subjective human assessments with a real automatic lane change system in order to evaluate the suggested methods.

The results of this study contribute to the development process of autonomous driving systems in several features: the design targets can be clearly defined by the proposed parameter selection methods considering driver acceptance. The driver acceptance can be objectively assessed by estimating the probability of acceptance based on certain objective values. The development period can be substantially reduced, and the system may potential for modifications based on objective feedback during the development process. The modeling method of this proposed driver acceptance evaluation model is applicable to the development of a personalized autonomous driving system.