The Value of Context-Based Interface Prototyping for the Autonomous Vehicle Domain: A Method Overview

Abstract

Before autonomous vehicles (AVs; SAE levels 4 and 5) become broadly available, acceptance challenges such as trust and safety concerns must be overcome. In the development of appropriate HMIs that will tackle these challenges, physical and social context play essential roles. Contextual factors thus need to be considered in early prototyping stages. Based on a qualitative semi-systematic literature review and knowledge from our research, this paper elaborates on the value of context-based interface prototyping in the AV domain. It provides a comprehensive overview and a discussion of applicable methods, including physical lab-based prototyping (mock-up, ride simulation with virtual and mixed reality, and immersive video), social context simulation (actors, enactment, items and props, and sound), wizard-of-oz, and experimental vehicles. Finally, the paper discusses factors affecting the impact of prototyping and derives recommendations for the application of prototyping methods in future AV studies.

1. Introduction: Autonomous Vehicles and the Nature of Prototyping

Driverless autonomous vehicles (AVs; SAE levels 4 and 5 [1]) promise to revolutionize our mobility systems. Before their broad introduction, however, severe challenges from a human–computer interaction (HCI)’s perspective need to be addressed to achieve acceptance by future passengers [2]. Prospective passengers will have to accept the unfamiliar—and potentially awkward experience—of driverless rides while being exposed to the decisions and actions of artificial intelligence (AI)-powered systems. Related work has identified trust as a major acceptance hurdle [2], along with concerns regarding safety, security, usability, accessibility, and comfort [3]. Human-centered design (HCD), based on extensive prototyping and evaluation of potential solutions, provides a promising approach to counteract these challenges. In a nutshell, the term prototyping describes the creation of (pre-final) representations of (or parts of) a product, system, or service [4]. Sayings such as, “If a picture is worth a thousand words, then a prototype is worth 10,000” ([5], p. 5), point out that prototypes not only show and tell, but make ideas, designs, and artifacts experiential [5]. Throughout the process of prototyping, the principle of “learning by doing” is essential [6]. We agree with the broad view of Thaler [7], who concluded that a prototype can basically be “anything that will move the process forward.”

In product design, a prototype usually refers to “a pre-production representation of some aspect of a concept or final design” ([8], p. 1). Service prototypes are used to simulate (already existing, (not yet) available, or new) service experiences and enable the consideration of relevant aspects of the real world environment [6]. In the field of user experience design, a prototype is referred to as something that “captures the intent of a design and simulates multiple states of that design” ([5], p. xii). Prototypes have a large impact on the success of design and development projects [8]. By making experiences tangible, prototyping reduces misinterpretations; can save time, effort, and money; and reduces the amount of waste created in the process [5]. As Camburn et al. [8] elaborate, prototypes can serve various objectives in product development, including, but not limited to, refinement, exploration, communication, learning, and—in terms of economic perspectives—cost or time reduction. Prototypes empower designers, researchers, users, and other stakeholders (1) to understand context and users’ experiences, (2) to explore and evaluate new approaches, and (3) to communicate ideas [9,10].

In HCI, two primary use cases of prototypes are often distinguished: supporting the generation (or exploration) of ideas about the design of interfaces and the evaluation of the quality of ideas, concepts, and solutions, especially in early development stages [11]. Often, prototypes are just considered for the latter. However, as Lim et al. [12] pointed out, prototypes can be a tool for “traversing a design space” to gain knowledge about the envisioned product or system and also serve as “manifestations of design ideas” [12]. Through the consideration of both use cases (generation and evaluation), prototyping becomes an essential component in the design process that supports informed decision making. Depending on their realization, most prototypes are typically limited in some way, e.g., in their implemented functionality or fidelity (i.e., level of detail [13]). Despite their limitations, prototypes can be used for several major activities, such as analysis, design, and evaluation [4] and are thus particularly valuable in the human-centered design of products, services, and interactive systems. Such are (always) used in a particular context [4]. Prototyping can enable the consideration of this context—which can relate to, e.g., physical, social, cultural, or organizational environments and influences [14]—and incorporate these crucial contextual components from early development phases.

In our research, we identified the lack of a discussion of the role and value of context and prototypes in the AV domain and the need for a comprehensive overview of applicable methods for context-based prototyping of human–AV interactions. In this paper, we therefore investigate the following research question: how might we efficiently prototype and evaluate human–AV interactions considering their dynamic context? We propose a meaningful integration of context in research, design, prototyping, and evaluation activities to achieve suitable HMI concepts for future AVs. Based on a qualitative semi-systematic review [15] of related work, we provide a comprehensive overview and discussion of practical considerations and applicable methods to consider physical and social contexts in AV HMI design. Contributing to a human-centered design of human–AV interactions, this paper provides researchers and practitioners with practical recommendations and an accessible and concise collection of suitable context-based prototyping methods for the AV domain.

2. Context-Based Interface Prototyping

We use the term context-based interface prototyping to describe the approach of prototyping human–machine interactions and respective HMIs in a contextualized setup (see Flohr et al. [10] and Hoggenmüller et al. [16,17]). In the following sections, we lay out the theoretical fundamentals and practical considerations regarding context and prototyping in HCI. We also provide an overview of applicable methods for context-based prototyping of (in-vehicle) human–AV interactions.

2.1. What Is Context?

The notion of context holds a variety of meanings and interpretations, even if we focus solely on the area of computer science and its subdisciplines [18]. In the following, we gather existing definitions to render what we consider to be an appropriate understanding for the HCI domain. Starting with a general description, the Cambridge Dictionary defines context in the sense of a “cause of event” as “the situation within which something exists or happens, and that can help explain it” [19]. Schmidt [18] also defines its understanding based on dictionary definitions and uses the term to “describe the environment, situation, state, surroundings, task, and so on” ([18], p. 193). Taking into account several varying definitions from related work, Trivedi and Khanum [14] also derived a rather broad definition and regarded context as “anything which has an effect on the human behaviour” ([14], p. 72). They distinguished cultural, organizational, technological, physical, and social context [14]. Situated within the HCI domain, we focus, similarly to Trivedi and Khanum [14], on the physical and social context and regard the technical aspects as a part of the physical.

2.2. Context in Human–Computer Interaction

As can be derived from the aforementioned general definitions, context is an essential component in HCI. In ISO 9241-110, the context of use is defined as “users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used” ([20], p. 6). This definition considers users, tasks, and equipment as part of the context which is “surrounded” by physical and social environments. Dey and Abowd [21] channeled their understanding of previous work into the following, more tangible definition, which we consider a proper understanding within the scope of this paper.

Context [in HCI] is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.

([21], pp. 3–4)

To consider context in the design and evaluation of systems that are not (yet) feasible or not available, prototypes incorporating this dynamic element can be used. Prototyping can help to understand and explore context and corresponding user experiences, ideas, and concepts [9].

2.3. Prototyping as a Means to Consider Context in HMI Design

In line with Thaler’s general perspective that a prototype can be “anything that will move the process forward” [7], prototyping can be regarded as ”interwoven with nearly all product, service, and systems development efforts“ ([8], p. 1).

Rendering a methodological, process-oriented perspective, Crabtree [22] distinguished four steps of prototyping: functional selection, construction, evaluation, and iteration. Through evaluation, prototyping enables feedback and communication between the use practice and the design process [22,23]. For most cases, Crabtree [22] differentiated three interrelated prototyping forms: exploration, experimentation, and evolution. In exploration, prototyping helps to understand context and—although eventually lacking large parts of the desired functionality—helps to foster ”cooperation between designers and end-users“ ([22], p. 131). Experimentation builds on the exploration to demonstrate and refine the prototype pragmatically but still includes user involvement [22,23]. The evolution phase is focused on the ”development and implementation of a stable prototype [… in] the target domain“ ([22], p. 131), and thus marks the transformation of the prototype into an actual product situated within its actual context of use.

Lim et al. [12] described prototyping as ”an activity with the purpose of creating a manifestation that, in its simplest form, filters the qualities in which designers are interested, without distorting the understanding of the whole“ ([12], p. 4). This points out that prototyping enables, on the one hand, one to explore (parts of) the final product considering the bigger picture (”the understanding of the whole”), which includes its environmental context. On the other hand, prototyping enables “filtering” to put the focus on particular aspects (“qualities”) of a product, system, or service in which the designer or the team is interested [12]. Filtering dimensions can be, e.g., the prototype’s appearance (e.g., in terms of shape, size, and color), considered functionalities, or the degree of interactivity (e.g., in terms of input and output behavior or feedback provision) [12]. Filtering enables the efficient investigation of design ideas without the need to implement the whole thing. Prototype manifestations can, according to Lim et al. [12], differ in three dimensions: material (i.e., the medium used to create the prototype), resolution (i.e., the prototype’s fidelity or level of detail), and scope (i.e., the range of what is included in the prototype).

Based on their extensive investigation of various prototyping approaches for urban robotic interfaces, Hoggenmüller concluded that the prototype of a product or system merges with the surrounding (prototyped) context “into one single manifestation” ([16], p. 210). This depicts the interdependence of interface prototypes and their (prototyped) surrounding physical and social environment. Context-based prototyping allows accounting for these circumstances. Besides creating more realistic experiences, it can also reveal requirements and constraints. In the AV domain, this could, e.g., refer to the readability of displayed information, the reachability of controls, or the compatibility of displays and controls. Hoggenmüller [16] illustrated the advantages of context-based (or “contextualized”) prototyping—where the envisioned prototype system, product, or service is situated in the (physical or virtual) context—by using a comparison to “decontextualized” prototyping (Figure 1). Contextualized prototyping does not only help to increase user’s immersion, but also supports designers (and other stakeholders) with envisioning the product, system, or service within the context [16]. With reference to Trivedi and Khanum [14] and Lacher et al. [24], we want to extend this view to also include the social context. Consequently, context-based prototyping can support the HMI design process from all different angles. Before creating (context-based) prototypes, however, a few things should be considered that we will elaborate in the next section.

2.4. Considerations and the Impact of (Context-Based) Prototyping

The “ultimate goal of prototyping” in the HCI domain is—according to Camburn et al. [8]—the enhancement of performance and user experience. Lim et al. [12] postulated as an economic principle of prototyping that “the best prototype is one that, in the simplest and the most efficient way, makes the possibilities and limitations of a design idea visible and measurable.” We want to highlight the aspects of simplicity and efficiency as core economic components. Based on the introduced definitions, we suggest expanding this principle to account for the various possible manifestations, places, and users; the different applications of prototypes in the HCD process; and the inherent importance of context. In doing so, we want to note that we do not see prototyping to be a procedure to create something “perfect” or “best.” Instead, we consider prototyping more of an iterative tool to achieve specific goals within the overall process. As such, it can provide a certain amount of (maximum) value but without achieving something like a state of perfection. In line with this, Camburn et al. [8] also pointed out that iterative prototyping directly causes an increase in performance and the ability to meet (difficult) requirements. The following statement summarizes these contemplations as a working definition for this paper. References to Lim et al. [12] and ISO 9241 [4] are highlighted in italics.

Prototyping in HCD is of greatest value when it most simply and efficiently supports achieving the goals of a particular activity, such as analysis, design, or evaluation. Prototyping can, for example, make ideas, concepts, and contexts visible, tangible, or measurable.

To achieve this, there is a wide variety of methods, materials, and tools available (Section 3). However, when it comes to their selection, it is important to consider that “everything is best for something and worst for something else” [25]. We render this to be crucial for successful (context-based) prototyping and agree with Buxton that the “trick is knowing what is what, for what, when, for whom, where, and most importantly, why” [25]. Similar to that, Dodge [26] lays out that how much a prototype can teach us depends on what, why, and how we are prototyping; and the when (i.e., the point of time in the process) and the amount of time spent to create the prototype significantly affect the impact on the (final) design. What, why, and how can direct refer to the notions of manifestations and filters introduced by Lim et al. [12]. Dodge [26] formalized their relationship with the time spent and the point of time in the process (when) in Equation (1).

$$\begin{array}{c}\hfill \frac{\mathrm{What}\times \mathrm{Why}\times \mathrm{How}}{\mathrm{Time}\mathrm{spent}}\times \mathrm{When}\underset{Dodge}{=}\mathrm{Impact}\mathrm{on}\mathrm{design}\end{array}$$

(1)

Although Buxton [25] did not explicitly refer to prototyping methods but to the choice of input devices, we want to note the close relation and the overlap to Dodge’s [26] impact of prototyping equation. In fact, we propose to extend Dodge’s [26] equation with the “missing” aspects of for whom and where. From a human-centered design perspective, for whom (i.e., for which audience or which particular stakeholders) we prototype is an essential consideration for creating prototypes. Similarly, where can be regarded as the representation of the product’s, system’s, or service’s physical context but also the prototype’s own physical location. Where we create, implement, or test a prototype significantly affects its manifestation and consecutive results. As a consequence, we propose to combine Dodge’s [26] equation with Buxton’s for whom and where [25] as laid out in Equation (2).

$$\begin{array}{c}\hfill \frac{\mathrm{What}\times \mathrm{Why}\times \mathrm{How}\times \mathit{For}\mathit{whom}\times \mathit{Where}}{\mathrm{Time}\mathrm{spent}}\times \mathrm{When}\underset{}{=}\mathrm{Impact}\mathrm{on}\mathrm{design}\end{array}$$

(2)

In line with Equation (2), Warfel [5] emphasizes that it is important to understand the audience (i.e., for whom) and intent (why) and to prototype only what you need. Furthermore, Warfel [5] points out that prototyping is a generative and iterative process which leads to the recommendation to prototype “early and often” ([5], p. 95).

We have already elaborated on the materials and the scope of prototypes—i.e., two of the three introduced dimensions of prototype manifestations described by Lim et al. [12]. Now, we want to complement this with a view on the third dimension, the resolution, or fidelity. In the HCI domain, fidelity is often referred to as “the extent to which a computer application or system reproduces visual appearance, interaction style and functionalities” [13,27]. In other words, fidelity describes the “level of detail” [13] or the degree to which a prototype represents the (actual, planned, or final) product, system, or service. In the above equation, fidelity is addressed by what, how, and where. Often, prototypes are described with the dichotomous categories “low-fidelity” or “high-fidelity” [27]. However, as Virzi et al. [28] and Warfel [5] pointed out, fidelity should be regarded as a continuum, not as a dichotomy. The required fidelity depends on the goal or the purpose that one wants to accomplish with a prototype [5]. Basically, this means that it depends on the answers to the questions that are part of Doge’s equation [26] and on the (requirements of the) target audience. With regard to the use of prototypes as (part of) simulations, Dahl et al. [13] distinguished three components that contribute to overall simulation fidelity: prototype fidelity, environment fidelity, and psychological fidelity. While we value this differentiation, we want to note—considering our adopted working definition of prototyping—that an environmental (or contextual) representation as part of a simulation can also be considered a form of a prototype. In general, higher fidelity often results in higher efforts to produce a prototype. Depending on the objective of, e.g., a study, it is vital to select an appropriate level of fidelity—since it can affect the accuracy of others’ interpretations [8] and may affect participants’ immersion, and consequently, their assessment [16]—and of course, economic aspects.

From an HCI perspective, AVs and related mobility (on-demand) concepts are still in an early development phase (when). Especially in this early stage, context-based prototyping can have a substantial impact (Equation (2)). In the following section, we provide a qualitative semi-systematic literature review [15] and discuss suitable methods to inform future research on what, why, how, and where context-based prototypes of AV HMIs can get created.

3. Methods for Context-Based Prototyping of In-Vehicle Interactions

The evaluation of new AV HMIs and concepts in early development phases with actual AVs and in real traffic is—similarly to the development of advanced driver assistance systems [29]—only possible with tight limitations. Aside from current technological constraints, this is primarily due to ethical aspects, especially regarding the potential danger when involving participants, other road users, and infrastructure. Context-based prototyping can help to face the problem of AVs being still in their technical infancy.

This section provides an overview of applicable methods for context-based interface prototyping [10,16] of in-vehicle interactions with AVs. Such methods enable researchers, designers, and other stakeholders to establish and experience contextualized setups of human–AV interactions to consider environmental factors in HCD activities, such as analysis, design, and evaluation. We focus on methods that support prototyping interfaces within their (intended) physical and social context. However, we do not focus on concrete interface prototypes. Within the scope of this paper, we neither discuss differences between HMI prototypes, e.g., in terms of the fidelity of sketches, wireframes, and high-fidelity visual design prototypes. Nor do we look at prototyping tools such as Sketch, Figma, Antetype, or Adobe XD. Interface prototypes are, especially with regard to the above-cited definitions, of course, a crucial part of the context of use. A detailed discussion of these is, however, beyond the scope of this paper.

Depending on the contextual situation, some methods are more suitable than others—for instance, regarding economic aspects. For example, for some AV scenarios, it might be required to have a dynamic high-fidelity representation of a ride through an urban environment. For others, it can be sufficient to have a static mock-up of a vehicle environment in a laboratory—e.g., to evaluate the general placement of display concepts within a vehicle. Often, a combination of several methods is used—e.g., Flohr et al. [10,30] used immersive video to create a dynamic ride simulation in combination with both interactive and video-based interface prototypes and actors that simulated the social context.

In general, prototyping methods need to be assessed and chosen depending on the aim or purpose of a particular project while considering their limitations. For example, while VR setups offer high degrees of flexibility, it still needs to be determined how participants’ perceptions differ from reality [31]. With reference to Bengler [32], Fuest [31] proposed that each method needs to get assessed with the three scientific quality criteria:

• Objectivity—the extent to which results are independent of any influences outside the matter of subject [33]. E.g., independence from influences of the investigator [31], test moderator, or analyzing and interpreting person [33].
• Reliability: The accuracy with which something is measured [33] or executed. E.g., for a wizard-of-oz study, the same driving style needs to get reproduced for each session [31,34].
• Validity: The extent to which a method actually measures or predicts what it is supposed to [33]. In HCI, one often distinguishes between internal validity—i.e., the extent of control of a study or method [35]—and external validity, which often refers to the generalizability of results [35].

Concerning the (external) validity of prototyping methods, and in particular, simulation methods, essential aspects are study participants’ immersion and sense of presence in the context, i.e., in the simulated environment or virtual world. A virtual world can be described as “an imaginary space often manifested through a medium” ([36], p. 8). The “sensation of being in an environment” ([36], p. 10) such as this virtual world is described by the term immersion. Creating immersion and the related experience of presence is a significant challenge in simulator studies [29] or—more general—in context-based prototyping. Sherman and Craig differentiated mental immersion as a “state of being deeply engaged” from physical immersion as “bodily entering into a medium [and the] synthetic stimulus of the body’s senses via the use of technology” ([36], p. 10). However, this “does not imply [that] all senses or that the entire body is immersed/engulfed” ([36], p. 10) at the same time. While sense of presence is often used as a synonym to immersion, Sherman and Craig assigned presence as equivalent to the state of mental immersion [36]. Similarly, Bubb [29] described presence as a cognitive state where users have the impression of being part of the virtual world and stated that this can be achieved through suitable design of the technology.

With these criteria introduced, the following sections provide an overview of the most frequently used context-based prototyping methods for human–AV interaction. We focus on the application for empirical studies on in-vehicle HMIs (and do not consider, e.g., questionnaire-based online studies or external HMIs—though some of the methods may also be applied for such study designs), but still, note that this collection is not exhaustive. We cluster the methods into the categories (1) static mock-up (including spatial interior and exterior representations), (2) ride simulation (with a focus on virtual and mixed reality and (immersive) video), (3) social context simulation, (4) wizard-of-oz, and (5) experimental vehicle.

Table 1. Overview of the discussed methods and their challenges, along with our recommendations for context-based prototyping of human–AV interactions.

Method	Challenges	Recommendations
Static mock-up
Static elements—that do not show (dynamic) changes over time—provide a spatial representation of AV interior and exterior components.	Weighing (study) requirements (e.g., regarding fidelity) and effort to construct a mock-up.	Use static mock-ups for spatial prototyping of AV interiors and exteriors; Combine with other methods to increase immersion, e.g., ride simulation.
Ride simulation
Simulating the dynamic (physical) context of riding in an AV using VR, MR, (spatial) sound, (immersive) video, or a combination of the mentioned approaches.	Achieving a sufficient level of fidelity for participants’ immersion in the simulation [29] that is required for a valid contextual representation; Coping with occurring simulator sickness symptoms [37,38].	Use VR and MR to prototype context that needs to respond to human input; Use real-world videos for (passive) high-fidelity simulation; Consider combining immersive video and VR/MR (e.g., [39]) to get the best of both worlds if required skills and resources are available; Use the checklist provided by [38] to design valid simulator studies and address issues such as simulator sickness.
Social context simulation
Simulating interactions and communication with others, e.g., co-passengers in shared AV rides. Prototypes can, e.g., make use of sound and actors [10] or enactment and props [40].	Incorporating social context into (semi-)controlled test environments can lead to adverse effects. E.g., people might feel uncomfortable with other (unknown) people present in certain situations, which might also lead to adverse effects in terms of simulator sickness [10].	Use sound as a baseline fo social context simulation. For instance, to simulate a shared ride, enhance the audio from the physical environment with noises of people entering or leaving the vehicle [10]; If social aspects are a crucial facet of a study, consider using actors and enactment to increase immersion and validity.
Wizard-of-Oz (WoOz)
Making participants believe they are riding in a real AV while a human driving wizard controls the vehicle [41]. WoOz can be used to prototype AVs on test tracks and on public roads.	Keeping up the WoOz deception throughout the conduct of the study; Coping with varying environmental conditions (e.g., weather, other road users) and ensuring comparability of test rides [34,41].	Use fitting cover stories [42] to maintain the deception of participants; Support the story with a consistent “AV-like” driving style (e.g., “like a professional limo driver” [43]) and fitting hardware [44].
Experimental vehicle
Vehicles with actual (but to some extent limited) automated driving capabilities [31].	Counteracting (technological) limitations that might not meet participants’ expectations (see, e.g., [45]) and affect their assessment, e.g., limited speed, restricted areas, presence of a safety driver.	Transparently inform participants about the vehicles’ capabilities (if not in conflict with the study design); If possible, use experimental vehicles that can perform the respective driving scenario without restrictions.

summarizes the methods’ key aspects and challenges and our recommendations for their practical application.

3.1. Static Mock-Ups

We use the term static mock-up to categorize methods that enable the inclusion of static contextual elements that do not show any changes over time, e.g., in their appearance. In terms of in-vehicle human–AV interaction, static components can, for instance, be used to spatially prototype the interior and the exterior of an AV. The construction of a mock-up can directly affect participants’ sense of presence [29].

To analyze user requirements regarding the design of shared AVs, Schuß et al. [40] used a tent-based mock-up to create an enterable prototype of an AV’s exterior. This also enabled them to situate participants of an empirical user study in a closed environment resembling the “pod’-like interior of a shared AV (Figure 2). Conventional chairs were used to resemble the seats of the AV (Figure 2ii). Similarly to that, Flohr et al. [10,30] used office chairs to do the same as part of an immersive video-based setup and used room walls, a movable whiteboard, and wooden pallets to create a rudimentary spatial mock-up (Figure 2iii,v).

While the mock-up of Schuß et al. [40] was constructed by the use of metal poles and a tent canvas, other materials such as paper, cardboard, and image prints could also be used to (re-)create similar setups. Static setups without additional components such as dynamic simulation, including the ones by Schuß et al. [40] and Flohr et al. [10,30], can be considered on the lower side of the fidelity continuum. However, exterior and interior prototypes also allow approaches with higher fidelity to provide a basis for the creation of sophisticated prototypes with the aim of resembling the final vehicle design (e.g., Figure 2v). Items, props, and physical controls such as emergency stop buttons or breaks can extend the physical immersion. They also enable one to investigate the match between digital displays and respective physical controls and to evaluate their compatibility or corresponding constraints. Furthermore, especially personal items can also increase social context simulation (Section 3.3).

3.2. Ride Simulation

Simulators enable researchers and designers to consider dynamic contextual factors in early development stages. In terms of driving and ride simulation, such dynamic factors might compromise, e.g., seeing a changing urban environment while looking out of a vehicle window during a ride through a city, sounds of the vehicle when accelerating, or the behavior of other road users, such as other vehicles or pedestrians. Depending on the setup and research aim, simulators offer, on the one hand, high controllability (e.g., of environmental conditions) and reproducibility (e.g., created simulations and test parameters can be easily transferred to other studies) [47,48]. On the other hand, they provide high flexibility in their manifestation and in terms of simplicity in data collection [47,48]. Furthermore, they allow the safe assessment of new systems and interfaces in early development phases [48]. With regard to the aforementioned quality criteria, simulators provide an excellent basis for the reliability of a study.

However, a major challenge of using simulators is the creation of a participant’s experience of presence in the simulated environment [29]. To achieve high presence perception, high-fidelity reproduction of visual, acoustic, haptic, and spatial stimuli is required [29]. As limitations regarding the realistic representation of these stimuli persist even in modern simulators, the validity of (automated driving) simulator studies remains an important research topic [38]. Furthermore, so called simulator-sickness symptoms, such as nausea, vertigo, sweating, or headaches [38], might occur while being in a simulated environment. Almallah et al. [37] found that women are more prone to simulator sickness than men and that older people experience more sever symptoms. Simulator sickness was found to be related to the sense of presence [37]. I.e., the more people are immersed in a simulation, the less likely is the occurrence of simulator sickness. According to Bubb [29], immersion depends on the reproduction quality of spatial and temporal stimuli that humans perceive with their sensory organs. Almallah et al. [37] also found that urban environments with close buildings and lower speed limits can increase participants’ sense of presence while simultaneously decreasing the occurrence of simulator-sickness symptoms. Hock et al. [38] provided a checklist to overcome typical challenges when conducting (driving) simulator studies.

As mentioned before, we focus on prototyping the physical and social context of in-vehicle human–AV interactions. By simulating these contextual aspects, we consider visual and auditory (noise, sound) impressions to be most relevant for (cost-effective) context-based prototyping, and will therefore put an emphasis on these. However, we want to note that other components, such as motion simulation and the inclusion of vehicle dynamics, might also be vital for some scenarios. Most common simulators used in automotive HCI research immerse study participants in a virtual world by using either computer-generated imagery (CGI; e.g., [49,50,51]) or (immersive) video (e.g., [10,30,39,52,53]). In the following, we discuss these approaches, their theoretical background, and their application in current HCI research.

3.2.1. Virtual and Mixed Reality

Being immersed in an alternate reality such as a virtual world is usually referred to as virtual reality (VR) [36]. VR allows investigations about how humans interact with computer-created worlds and simulations [54]. Milgram and Kishino describe a VR environment as “one in which the participant-observer is totally immersed in, and able to interact with, a completely synthetic world” ([55], p. 2). To describe how VR and associated concepts are related, they introduced a continuum between the real environment and the virtual environment, where they describe the space in between as mixed reality (MR) (Figure 3). MR is regarded as a state in which “real world and virtual world objects are presented together” ([55], p. 2). Subsets of mixed reality are augmented reality (AR; in which the real environment is supplemented with virtual (computer-generated) objects [56]) and augmented virtuality (“in which real objects are added to virtual ones […] and the surrounding environment is virtual” ([56], p. 34)). As can be seen in related work, e.g., in Azuma et al. [56], VR is often used as a synonym for virtual environments. Generally, both VR and AR, and augmented virtuality, can be used for simulation and context-based prototyping. For instance, Morozova [57] presented a “mixedUX” prototyping framework for usability testing in AR. While augmented virtuality is quite rarely used in the AV domain, some studies use AR to investigate new HMI concepts. Haeuslschmid et al. [58], for example, used AR to prototype interactions with a virtual avatar. However, most state-of-the-art driving/ride simulators use CGI-based VR as their basis.

In automotive simulators, popular hardware setups are CAVE-like [59] environments, head-mounted displays (e.g., HTC Vive, Oculus Rift, and Microsoft HoloLens), or compilations of three monitors (Figure 4). Currently, these methods are often applied to evaluate systems such as advanced driver assistance systems for non-driverless vehicles (i.e., SAE levels 0–3) or for teleoperation of vehicles in combination with video live-streams of their environment (e.g., [60]). Simulator studies in the automotive domain mostly refer to VR setups created with CGI (computer-generated imagery). However, an immersive virtual environment can also be created using real-world videos [10,39,52], into which we will take a more in-depth look in the following section.

3.2.2. Video and Immersive Video

Instead of CGI-based VR, it is also viable to use real-world videos as a basis for the simulation. For instance, Krome et al. [53] and Haeuslschmid et al. [58] used single videos from real-world traffic situations to provide a basic representation of the physical context of a ride through an urban environment for their HMI studies. Real-world footage can be enhanced with additional imagery or CGI—e.g., to prototype AR-based avatar concepts [58]. Multiple real-world videos can also be used to create a more immersive spatial simulation, similar to a CAVE-like [59] environment (see [52,61]). Kray et al. [52] called this approach immersive video. It features a high-fidelity audio-visual representation of real-world contexts and a high degree of control. Aside from using multiple cameras, it is also viable to use special equipment, such as 360° cameras.

Gerber, Schroeter, and Vehns [39] constructed, for instance, a 3D-printed camera rig to align three action cameras to capture 180° video footage. Since the CGI-based virtual environment of their pre-existing dynamic driving simulator with three front screens and a field of view of 180° lacked the required level of contextual fidelity and detail, they chose to use immersive video instead of CGI for their automated driving studies (SAE level 2 and 3) [39]. While their setup’s overall immersion was assessed to be high, Gerber et al. [39] stated that the sense of immersion was different compared to CGI-based studies, but that participants’ familiarity with the local environment supported their feedback quality.

Flohr et al. [10] presented a straightforward and low-budget adaption of an immersive video approach for the AV domain (SAE levels 4 and 5) based on the works of Kray et al. [52] and Gerber et al. [39], but without the need for special equipment such as camera rigs or sophisticated simulator setups (Figure 2iii).

3.3. Social Context Simulation

Simulating the social context means prototyping (aspects of) the context of interacting and communicating with other people. In shared AVs, and more generally, in public transportation, passengers face encounters with other people, such as co-passengers. Inevitably, they communicate with each other—even when they do not intend to do so since “one cannot not communicate” ([62], p. 30). This means that even though humans do not communicate verbally with others, they still communicate implicitly, e.g., through behavior, gestures, or mimics. Since people’s trust in a system does also depend on (their trust in) other people [24], and the presence of co-passengers can affect people’s wellbeing [10] and perceived security [40], considering the social context is a vital aspect for valid context-based prototyping.

3.3.1. Actors and Enactment

To derive design implications for human–AV interaction in an interview study, Schuß et al. [40] embodied participants with a user enactment [63] part in the user journey of using shared AVs for transportation. Odom et al. described user enactment as “fieldwork of the future” and as a method where “designers construct both the physical form and the social context of simulated futures, and ask users to enact loosely scripted scenarios involving situations they are familiar with, and novel technical interventions designed to address these situations” ([63], p. 338). First, Schuß et al. [40] told participants to imagine typical scenarios where they would ride with a shared AV in the future (e.g., traveling home from work, picking up kids from school). Then, they asked participants to enact and to interact with a static mock-up as if they would conduct a ride with a shared AV. In this shared ride, an actor joined the ride and mimicked a stranger with whom participants shared their ride [40]. The authors concluded that the enactment supported the consideration of the context of use [40]. Similarly to that, Flohr et al. [10] prototyped the social context of a shared ride, i.e., the interaction with fellow passengers, by using an actor who simulated another passenger getting on and off a shared ride. The results suggest that actors can increase participants’ immersion in the simulation but can also affect their wellbeing. Other approaches for social context simulation might encompass, e.g., the use of mannequins, puppets, drawings, or AR overlays to simulate the physical presence of others in a real environment.

3.3.2. Items and Props

Additional physical items can enhance the simulation of social context. For instance, Schuß et al. [40] let participants choose fitting props and items (e.g., backpacks, books, laptops, a stroller, or a baby doll representing a child) to take with them along the enacted journey in the shared AV (Figure 2) with the aim of enhancing the realism of the simulation and immersion. Besides using personal items for social environment simulation, physical items might also concern assistive devices such as wheelchairs or glasses that may be relevant for accessibility-related design and research activities.

3.3.3. Sound

Besides the visual environment, auditory aspects play an important role in terms of both the physical and the social context. This can encompass, e.g., driving noises, noises from other vehicles, or sounds of co-passengers. In an (immersive) video-based simulation, the sound footage of a driving video could serve as a reasonable basis. However, this might be meaningfully extended with additional sounds to simulate specific scenarios, including aspects of the social context such as noises of passengers getting on and off a shared AV, or sounds of opening and closing the vehicle’s door [10,30]. When used without visual (VR) simulation, sound simulations can either stand-alone or used in combination with mock-ups. In such cases, these can be regarded as a form of AR considering the above-discussed works of [36,55,56].

3.4. Wizard-of-Oz

WoOz studies allow for the evaluation of intelligent systems such as AVs prior to their availability [41]. They can go beyond the limitations of laboratory or test environments [64]. The general idea behind the WoOz method is to make participants believe that they are interacting with an intelligent artificial system. At the same time, their internal workings are, in fact, simulated by humans—the so-called wizards [65]. When using the method to prototype AVs, study participants are led to believe that a vehicle is driving (fully) automated while they are actually driven by a human driving wizard who controls it [41]. WoOz can be used to prototype AVs and corresponding interfaces in real-world environments, i.e., on public roads [42,64,66,67,68].

The past decade saw a significant increase in the popularity of WoOz within the automotive domain, e.g., to evaluate new HMI concepts [69] or to investigate non-driving related activities [42]. As a result, Bengler et al. [41] proclaimed the “renaissance” of WoOz [41] and provided an overview of typical WoOz settings. Those vary depending on the automation level of the envisioned system and the degree of participants’ (illusion of) control. Given that in AVs, passengers are only passive occupants, “classic” controls such as steering wheels and pedals are not required for (most) AV studies. Common setups typically position participants on the co-driver’s seat in the front [43,64,67] or in the back [42,66,68]) while physically separating them from the driving and interaction wizards. Karjanto et al. [66] and Detjen et al. [42], for instance, positioned study participants in the back of their WoOz vehicle and used an isolator wall with a mounted TV displaying the video stream of a webcam installed on the vehicle’s windshield. Inspired by their setups, we also created a video-based WoOz setup for our in-vehicle interaction studies (Figure 5; [44]).

While offering the advantage of relatively low limitations in terms of the physical context, WoOz poses methodological challenges. Concerning a study’s validity, it is essential to guide participants to believe in the WoOz illusion and to have the vehicle behave like an actual AV would do [34,64]. To achieve this, human driving wizards need to follow a pre-defined driving style strictly (e.g., like “a professional limo driver” [43]). This style must be consistently reproduced by the wizard(s) throughout all sessions and test rides to support the reliability of the study [34].

Cover stories [42] are used to create and maintain the WoOz illusion. In such cover stories, participants are told about the (simulated) capabilities of the AV, e.g., driving autonomously in an urban environment. Varying environmental conditions such as traffic density, the presence and behavior of other road users, weather, and lighting conditions, poses further challenges in terms of reliability [34]. Such variations might impact study goals and the comparability of test rides [41]. Bengler et al. [41], thus, proposed to include an assessment of the “comparability of test drives and the believability of the illusion” when conducting WoOz studies.

3.5. Experimental Vehicle

As mentioned before, “actual” AVs are still under development and currently only available with limitations; e.g., [45,70,71,72]. We use the term experimental vehicle [31] to describe vehicles with actual automated driving capabilities. Such experimental vehicles typically come with limitations, such as (maximum) speed limits, restrictions to specific test scenarios and tracks, and/or the need for constant surveillance through a human safety driver. Like WoOz vehicles, experimental vehicles can be used both on test tracks and in real traffic [31].

As this approach requires an actual vehicle and the technical expertise for implementing the desired scenarios, it is expensive—particularly compared to other prototyping methods [31]. Apart from the high cost, and since participants experience an actual ride in an automated vehicle that—potentially—takes place on a real (public) road, the method promises to offer high validity. Likewise to WoOz, reliability is impaired due to the dynamic and uncontrolled environment [31], e.g., regarding real traffic and weather. It needs to be considered that the limitations might also affect study participants’ evaluation. For instance, they might assess certain aspects, such as their trust in the system and safety perception, differently with the knowledge that the study is conducted in a restricted area or that there is a safety driver present. Furthermore, as, e.g., Nordhoff et al. [73] pointed out based on their results of an interview study with 30 participants experiencing a ride in an experimental vehicle on a campus in Berlin-Schöneberg: the experimental vehicle might not meet participants’ expectations—which might then again affect their assessment.

4. Conclusions

In this paper, we have argued that the consideration of physical and social context in the human-centered design of future systems is essential. This is particularly the case for new technologies such as AVs that face significant acceptance hurdles. We postulate that applying context-based interface prototyping—i.e., prototyping HMIs in a contextualized setup—enables the creation of interfaces that can counteract these challenges. Based on a qualitative semi-systematic review of related work, we discussed suitable prototyping methods for (in-vehicle) human–AV interactions—including lab-based mock-ups, and (VR-, video-, sound-based) simulations, wizard-of-oz setups, experimental vehicles, props, and social context simulations. The comprehensive method overview and derived recommendations (Table 1) provides a helpful resource for future work and contributes to the human-centered design of suitable AV HMIs.

The impact that a prototype can have on the final product’s design depends on various factors comprising the what, why, how, for whom, where, when, and time spent in prototyping. We pointed out the relations of these factors (Equation (2)) with a special emphasis on the context-based design of AV HMIs. The choice of a particular prototyping method (or a combination of multiple approaches) depends on several considerations affecting these factors, such as the desired fidelity, the relevant stakeholders, and the purpose of the prototype. In addition, further requirements can be associated with the reliability and validity of the methods, and associated costs and potential risks. Although methods differ in the general effort, and consequently, the time spent to create a prototype, one cannot say that a particular method is best across the board. Instead, an appropriate prototype needs to be chosen by carefully weighing the factors mentioned.

Future research should investigate the matter and “weight” of the discussed factors for different design activities and comparatively examine the benefits and downsides of the respective prototyping methods for specific AV scenarios in a standardized manner.