The experiment took place in a fixed-based driving simulator, as shown in Figure 1
a. The GUI is shown in Figure 1
b, and was designed using Adobe Illustrator (Adobe Systems, San Francisco, CA, USA) and Processing 3.3 (Processing Foundation, Boston, MA, USA). It acted as a platform that facilitates information exchange between the user and the system. A simple design was used, avoiding as much unnecessary information and showing as few components as possible, while maintaining its essence: showing the speedometer, the fuel gauge and the system’s confidence level.
Confidence level of the system (i.e., the extent to which the system was certain about the decisions it took) was indicated with an icon in the interface showing either high (90%) or low (30%) confidence (see Figure 2
). A threshold was marked by a red line just above the 3rd bar (of a total of 10 bars) in the system confidence bar. The two levels of system confidence were chosen based on Helldin et al. [55
], and communicated to participants by telling them how certain the system was to drive itself.
A scenario was created consisting of an urban environment in which the simulator drove for five minutes, followed by a highway environment, where it drove for another five minutes. In accordance with the Gricean maxims, voice messages were played in the CUI condition at situations that demanded explanation. The content of these messages is included in the scenario description below.
The first five minutes of the scenario consisted of the AV driving in an urban environment. The route was a combination of two-lane roads and cobbled paths with pedestrians. In a typical session, the AV started on a straight stretch of a two-lane road, and stopped at a traffic light. When the light turns green, the AV gives way to a bicyclist (“I’m giving way to the bicyclist.”). By the time the bicyclist passes, the light turns yellow and the AV waits until it turns green again (“The light is about to turn red.”). Then, the vehicle turns right and drives on a curved road, taking a right at the end of it. After going straight for a while, the vehicle yields at a stop sign and then proceeds onto a cobbled road, slowing down as it does (“We’re on a cobbled road with pedestrians, I’m slowing down.”). At the end of the cobbled road, the AV turns left after yielding for a motorcyclist (“I’m yielding.”). Then, it drives on a two-lane road and waits at a traffic light. It yields for another motorcyclist when the light turns green, then turns left. In alternating between the confidence levels of the AV, the low confidence level was characterized by not giving the right of way to bicyclists and motorcyclists.
For the next five minutes of the session, the AV drove on a highway, mostly at a constant speed of approximately 100 km/h. While being on the left lane ready to overtake a car, it returned to the right lane to give way to a faster car (“I’m letting a faster vehicle overtake me.”). In the Low confidence condition, the vehicle exited the highway about two minutes after it starts, slightly going off the lane. In two cases, at this point, the vehicle crashed (“I’m terribly sorry about that, I hope you’re okay!”). On entering the highway again, the AV drove onto the shoulder of the highway at the end of the on-ramp, and slowed down to a halt. The vehicle then waited until the traffic on the highway lanes cleared (“I need the traffic to reduce before I can safely enter the highway.”), then entered the highway and kept driving.
The CUI was identical to the GUI, with an added layer of spoken messages played by the simulator at fixed points during the experiment. These spoken messages explained the behavior of the vehicle in instances where required. Since we wanted to focus on the evaluation of the effects of conversational intelligence, a Wizard-of-Oz approach was applied for the CUI instead of spending effort on the technical implementation of the CUI.
Voice messages were created using NaturalReader 14 (NaturalSoft Ltd., Vancouver, BC, Canada), which is a standard text-to-speech software that reads out loud content typed in a text-box. Participants were asked to choose a gender for the autonomous vehicle in which they would drive. Based on their answer, a male or female voice was used in conjunction with the interface.
Before the experiment, participants completed an online introductory questionnaire that consisted of demographic information, attitudes towards autonomous driving, and propensity to trust autonomous vehicles. Attitudes towards autonomous driving were measured with eight questions (a mixture of yes/no answers, multiple choice answers and Likert-scale judgments), extracted and adapted from Kyriakidis et al. [26
= 0.83). Propensity to trust autonomous systems was measured with six 5-point Likert-scale questions, adapted from Merritt et al. [56
During the experiment, multiple questionnaires were used to test the concepts of interest. To measure trust, a 7-point scale with 11 items was adapted from Jian et al. [57
= 0.92). Perceived intelligence was measured with a 5-point scale with five items, adapted from Warner and Sugarman [58
= 0.92). Likability was measured with a 5-point scale with five items, adapted from Monahan [59
= 0.88). Finally, Anthropomorphism was measured with a six-item binary scale, adapted from Ruijten et al. [60
= 0.91). Data on this scale were submitted to a Rasch model (for an overview, see [61
]). This model calculates so-called person ability scores: values that are calibrated to the probability of items being answered with “yes”. More specifically, an item that is unlikely to be answered with “yes” is given a higher weight than an item that has a high probability of being answered with “yes”. With this method, relative differences between the items are included in the calculation of a person’s score, and as such it is a closer representation of attributions of agency.
Participants signed up for the study online, after which they completed the introductory questionnaire that contained demographic questions, their preferred gender of the automated vehicle (used for determining the voice for the CUI), and attitudes of and propensities to trust AVs.
Upon arrival in the lab, participants read and signed an informed consent form. They then took a seat in the simulator, and drove it for three minutes to familiarize themselves with the vehicle before being driven around by it. This was done to induce a sense of realism because directly making them sit through autonomous driving in the simulator might make it seem like watching a video recording where the participants could potentially remove themselves from all responsibility.
Participants in the GUI condition were given an explanation about the elements in the interface by the researcher, whereas those in the CUI condition received the exact same explanation as a spoken message from the vehicle. Next, participants experienced the first ride, either with high or with low system confidence. At the end of the ride, participants left the simulator to complete the questionnaires.
They then experienced the second ride (with the other confidence level) and completed the questionnaires a second time. At the end of the experiment, participants were thanked for their contribution, debriefed, and paid for their participation.