Hands Up! Investigating Intrinsic Gestures of Persons with Intellectual Disabilities for Inclusive Automated Traffic

Abstract

As traffic automation grows, research into communication between automated vehicles (AVs) and pedestrians is increasing. Since communication is particularly challenging for persons with intellectual disability (PID), we investigated which gestures are used during street crossings and whether they differ between PID and persons with no diagnosed disability (PnDD). We conducted a virtual reality study with $N=70$ participants ($n=38$ PID, $n=32$ PnDD), observing participants’ movement and gestures in a road-crossing scenario and having them rate their presence, user experience, and the AV’s behavior. PID descriptively gestured less often than PnDD (33.3% vs. 18.8% of trials without a gesture), though this difference was not statistically significant. The increased learning effect of using gestures improves crossing time for PID. We identified three primary gestures (Greet, Barrier, Stop) that both groups equally favored. Our work contributes to making bidirectional communication with AVs more inclusive by providing insights into gestures used by PnDD and PID and identifying indicators of subconscious learning effects when using gestures.

1. Introduction

Road traffic is a ubiquitous phenomenon [1,2], which is expected to transform as automated vehicles (AVs) will be more common [3,4]. With a crash-rate of $14.56$ for AVs in comparison to $1.89$ per one million vehicle miles for human-driven vehicles in 2023 [5], data shows that vehicle automation still needs major improvements for mass-adoption. Especially in uncertain situations, the crash rate of AVs is up to $5.25$ higher [6]. The absolute number of AVs is constantly rising, so encounters between vulnerable road users (VRUs), such as pedestrians [7], and AVs will become common. Current data backs the assumption that the number of accidents between AVs and VRUs will rise similarly [8]. Such encounters create ambiguous right-of-way situations in which the AV must communicate its intent—either yielding to the other road user or signaling that it will proceed first. Vehicle communication relies heavily on visual, auditory, or complex cues, referred to as external human-machine interfaces (eHMIs) [9,10,11]. These cues may compromise interaction for groups with support needs, such as children or persons with disabilities.

With up to $17\%$ of the global population living with disabilities [12], inclusion is largely considered a human right [13]. This also includes participation in individual mobility. In the USA, the risk of a person using a wheelchair being involved in a lethal road accident is “[…] roughly 36 percent higher than a pedestrian on foot” [14]. Current traffic infrastructures disproportionately endanger marginalized groups, leading to a higher prevalence of traffic accidents among persons with an intellectual disability (PID) [15,16]. But alongside persons with motor disabilities, people with vision, hearing, or intellectual disabilities are also among the most vulnerable persons [17,18,19,20]. Therefore, communication concepts between AVs and other road users must be designed inclusively.

Intellectual [21] and cognitive disabilities [22] are related concepts that encompass a wide range of serious difficulties in concentrating, remembering, or making decisions due to physical, mental, or emotional conditions, reflecting heterogeneous underlying causes. These can manifest from mild to severe characteristics and stem from various diagnoses, e.g., autism spectrum disorder, learning disabilities, Alzheimer’s disease, acquired brain injury, or mental illness [23]. In the literature, cognitive disability is a broad term that encompasses a variety of developmental disorders, intellectual disabilities, and neurodiversities. Intellectual disabilities specifically refer to neurodevelopmental conditions characterized by limitations in intellectual functioning. On the other hand, neurodiversity describes the natural variations in human cognition, highlighting them as an essential aspect of human diversity [24,25]. Given the substantial overlap between intellectual and cognitive disabilities, we are using the terms synonymously and refer to them collectively as PID, adhering to the social model of disability [26] by characterizing the experienced impairments rather than the individual disability. While the kinds, severity, and characteristics are highly heterogeneous, PID share common support needs regarding mental workload and memory [27,28]. Although not as apparent as vision or motor deficits, intellectual disabilities are prevalent, with rates of 1–3% in the general population [29,30], but are not adequately represented in science [31]. This blind spot for the support needs of PID in future research-driven developments, such as city planning, traffic infrastructure, and vehicle construction, fuels the exclusion of marginalized communities.

Road traffic is a very versatile and complex setup, where increasing urbanity and traffic density have strong impacts on mental workload (MW) in vehicle operators and pedestrians [32,33,34,35]. Furthermore, traffic-specific conventions must be learned (e.g., right-hand driving, traffic lights, road signs). For persons with learning disabilities among PID, this poses a particular challenge [36]. So rather than introducing new symbols and conventions that must be learned, novel developments could be more inclusive by leveraging intrinsic human affect [37], thereby enabling immanent learning.

To date, little research has investigated encounters between AVs and PID. The few existing studies come from different areas, including mobility research [38], visual impairment studies [39,40], and PID-focused work [41,42]. Additionally, most research has evaluated communication between AVs and VRUs as one-way. This means that while data support an increased risk due to the ambiguous behavior of VRUs [43], the communication from the VRU to the AV, for example, via gestures, remains largely unexplored.

In this work, we differentiate PID from persons with no diagnosed disabilities (PnDD). In doing so, we acknowledge that people do not experience the same daily-life barriers, allowing us to differentiate these groups based on their accessibility needs. We follow the Universal Design approach, which suggests that systems should be designed to be accessible for PID and PnDD equally [44,45]. The current study contributes to this by evaluating the bidirectional communication between VRUs and AVs in an exploratory between-subjects mixed-methods study with $N=70$ participants ($n=38$ PID, $n=32$ PnDD) to investigate how people with and without intellectual disabilities interact with AVs. In the remainder of this paper, we begin with a brief review of related work. We then present the key research questions guiding this study, along with several high-level methodological decisions. After that, we describe the empirical experiment conducted to investigate these questions. Finally, we summarize our findings, discuss them, and suggest directions for future research. As categorized by Wobbrock and Kientz the main contributions of this work to the field of the HCI research community are as follows [46]:

(a)

Artifact or System and Method: We created an accessible human-in-the-loop experimental setup for Wizard-of-Oz conduction in virtual reality (VR) based on a Unity simulation of a street-crossing scene, which we will make openly available upon acceptance.

(b)

Empirical study: Our results indicate that gestures can significantly enhance communication with AVs. Using the gesture set developed by Brand and Schmitz [47], we identified three primary gestures used by PID when interacting with AVs in our experiment: “Greet”, “Barrier”, and “Stop”.

2. Related Work

2.1. Interpersonal Communication

Interpersonal communication has been examined since 1949 [48]. Interactional views treat people as alternating senders and receivers [49]; for example, in the Sender, Message, Channel, Receiver model [50]. Transactional accounts extend this with factors such as social and cultural context [51], and DeVito’s interactive model foregrounds context, feedback, and noise [52]. Furthermore, it is well elaborated that listening and feedback are central to effective exchange [53]. These principles have been used to ground AV and pedestrian communication, where eHMI design spaces explicitly integrate such factors, including affective signals [54]. Because feedback underpins successful interpersonal exchange, we examine how the addition of feedback shapes communication between AVs and pedestrians.

2.2. External Communication of AVs

To substitute driver–pedestrian interaction, many modalities have been proposed, including exterior displays [55], LED strips [55,56], vehicle motion patterns [57], projections [58], auditory or tactile cues [59], multimodal combinations [59], and infrastructure support [60]. Early studies on simple one-vehicle, one-pedestrian scenes highlighted benefits. For instance, Löcken et al. compared six eHMI variants, with Smart Road based on Umbrellium rated highest [61,62]. Debate continues about the necessity of external communication [63], yet eHMIs can help convey intent [64]. When messages and vehicle kinematics conflict, people prioritize behavior over the eHMI [64]. Current work addresses risks of overtrust [65] and limits to scalability [66]. Distance-contingent cues can improve understanding and willingness to cross [67]. Accessibility-focused work demonstrates the value of including persons with vision disabilities in the design for everyone and highlights the need for additional audio, which can also benefit sighted pedestrians [39]. We aim to contribute to the field of inclusive AV–pedestrian interaction by gathering insights into the particular needs of PID.

Beyond pragmatic messaging, researchers increasingly study hedonic, pleasure-oriented communication [68,69], including politeness and respect [69] and traffic prosociality [68]. Sadeghian et al. ask how mindful and prosocial AVs should act and communicate [68] (p. 205). Their evaluation of on-road light projections found such communication to be trusted and perceived as clear, adequate, and effective, aligning with prior findings [10,39,59,61,67,70]. Effects depend on one’s role and situation; pedestrians rated messages less favorably than drivers and judged communication more positively in vulnerable situations such as unsigned crossings [68]. Lanzer et al. compared Chinese and German samples related to polite strategies at a crosswalk and when a pedestrian blocked the lane, polite messages increased acceptance in both samples and increased compliance only in the Chinese sample [69]. Most eHMI work remains unidirectional from vehicle to pedestrian, motivating studies on bidirectional exchange [68,69], which we aim to follow up on.

User motion inside an AV can also interact with exterior messaging. Colley et al. showed that passenger gesticulation can lower willingness to cross despite clear explanations. That intention-based eHMIs reduced confusion compared to command messages [71], which could be an important lever to support the needs of PID by reducing MW [36].

2.3. Gestures, Anthropomorphism, and Inclusive Populations

Pedestrian gestures at the curb can increase driver yielding [72]. Anecdotal evidence suggests that PID gesticulate toward AVs even when they are aware that no driver is present [42], echoing a broader tendency to humanize technology. Although PID are heterogeneous, common accessibility barriers for this demographic relate to high demands on working memory [73,74,75,76,77], reduced performance under high workload [74,77,78], and complex informational processing [79]. Because urban traffic imposes a substantial cognitive load (CL) that often overlooks these cognitive profiles, it motivates researchers to work with this group.

Anthropomorphism benefits mutual interaction with and trust in AVs [80,81,82], consistent with the well-documented humanization of pervasive technologies [83,84]. Building on this, several works address gesture-based pedestrian-to-vehicle interaction. Feedback to pedestrian gestures can raise acceptance of AVs [40]. Gesture-driven stopping was explored in a VR testbed, which also led to the surfacing of participant insecurities when they were performing predefined gestures [85]. Human-like interactions can simplify the interaction with AVs for pedestrians [86] and the adjustment process to novel technologies [87], while also impacting MW [87]. This strongly motivates the investigation of human-like interactions with technology.

Landry and Akhloufi showed in their literature survey that current pedestrian intention predictions rely on trajectory calculations, with gesture recognition remaining challenging [88]. Research on traffic gestures spans codified rules, proposed vocabularies, and recognition, while intrinsic pedestrian choice has only recently been studied. Officer signal catalogs exhibit limited universality across countries at intersections, stronger alignment in work zones, and meanings that often follow affordances [89]. Design work proposes gesture sets for human vehicle interaction [90] and an analysis of social media suggests distinct classes of gestures towards AVs [91]. Recognition studies identify common pedestrian gestures with keypoint methods, but emphasize detection over mapping the repertoire and its scenario dependence [92]. Recent evidence has elicited and observed pedestrian gestures across nine scenarios in both virtual and real settings, resulting in an eighteen-gesture taxonomy that consistently identifies types across settings and exhibits lower expressiveness in natural traffic. This validates virtual reality as a data source and exposes context effects relevant to VR interpretation [47]. With high validity for VR studies with high-presence participants [93], VR setups are a proficient, affordable, and easily usable setup.

2.4. Research Gap and Implications

We identified a research gap concerning bidirectional communication between AVs and VRUs, particularly for persons with disabilities. Given that humans tend to humanize automated technology and that gestures are a common communication strategy, we designed a study to investigate how people use gestures as communication cues with AVs. As intrinsic communication strategies are affective signals, we furthermore investigated how these signals differ between PID and PnDD and how positive feedback on these signals leads to imminent learning effects for both groups.

3. Materials and Methods

Based on the identified research gap, we designed an observational between-subject VR study to be conducted with PID and PnDD, investigating the following research questions (RQs):

• RQ1: How do pedestrians gesture and vocalize towards AVs unprompted, and are there differences between PID and PnDD?
• RQ2 How do pedestrians intrinsically use gestures in terms of position, crossing time, and repetition effects, and are there differences between PID and PnDD?

3.1. Methodology

Our study used a mixed-methods approach to observe whether and which gestures are performed without prompting when crossing a street with heavy traffic in VR. The investigation of learning effects was designed by repeating the road-crossing task after completing a questionnaire on the system’s user experience (UX) to separate the trials. In the end, we collected subjective data on the AV’s prosocial behavior.

3.2. Apparatus

As data about persons with disabilities is subject to especially stringent legal protection [94], and the study was conducted across multiple locations (see Section 3.6), the setup was designed to be mobile and fully usable on-premises.

The system that was set up for the experiment was composed of multiple hardware and software systems (see Figure 1): The VR headset (Pico 4; https://www.picoxr.com/global/products/pico4; Accessed: 5 May 2026) ran the scenario, which included the voice recognition module. The headset’s movement data and transcribed speech were transmitted to the broker running on the experiment laptop. Simultaneously, the video stream of the image displayed to the participant was sent to the experiment laptop for storage and processing. The broker also published the received data to all clients. For gesture recognition, we followed the methodology of Shyam et al. [95]: A high-definition (HD) camera continuously recorded the participant’s motions using OpenCV (4.6.0.66), which were translated into skeleton landmarks using Mediapipe (0.8.10.1). The position of the landmarks was evaluated using a custom-trained model based on Keras (2.9.0) and TensorFlow (2.9.2) to identify the preselected gestures from related work. The model was trained on 7503 images of 9 people performing 8 gestures from [92], all wearing the VR headset used in the experiment. This set was used because the more elaborate gesture set by Brand and Schmitz had not yet been published at the time of model training. The resulting model was released on Hugging Face (https://huggingface.co/daHaimi/body-poses-with-pico4-headset; Accessed: 8 July 2025). The classification result was finally sent to the broker. We also added the option to manually mark a timestamp when the participant visibly performs a gesture. The gestures were noted down by description and scribbles, and subsequently classified based on the more recent dataset from Brand and Schmitz. These were used for later evaluation to prevent data corruption due to detection errors and classification. Furthermore, a web server was running on the experimental laptop, hosting questionnaires as HTML pages for Wi-Fi access. These were displayed on a tablet PC and presented to the participants. The results were submitted to the experiment laptop and stored alongside the broker data. The orchestration command-line interface (CLI) facilitated software management and the remote administration of the VR headset.

Figure 1. System architecture of the experimental setup displaying the components deployed on the VR headset and on the experiment computer with all connections and protocols between the systems. The components on the experiment laptop provide all the functionality for managing the VR headset, recording data, and hosting questionnaires on the tablet PC. All hardware is connected via a dedicated Wi-Fi network.

For experiment conduction, a flat and empty area of at least $2\times 5$ m was marked (see Figure 2 and Figure 3). At one end in the middle, two strips representing the starting points for the feet were marked, maximizing the available real space. About 2 m into the projected path, the participants encounter the curb in VR. As we expected participants to perform gestures here, we focused the camera on this area. The scene was designed for right-hand traffic, so participants facing the approaching cars would turn left. Thus, we placed the setup to the left of the participant’s projected path, so that participants would perform gestures towards the camera rather than away from it.

Figure 2. Experimental setup with the marked area, starting point, the camera, and technical setup at the side, and the virtual areas taken into account.

Figure 3. Photo of the experimental setup with highlights on the starting point (A), the technical setup at the side (B), and the camera (C). The projected path is drawn in green.

3.3. Virtual Reality Scenario

A pedestrian crossing scenario in an urban environment in VR was selected to capture participants’ intrinsic gestures. A first-person view of the scenario is presented in Figure 4. Participants stood on the right side of a two-lane urban road, and were tasked with crossing it. To increase presence, people talking and walking, as well as roadside objects like trees, were added. The participants’ starting point was placed 2 m from the roadside, facing directly toward the target position. The target was marked with a transparent blue mark on the floor and a floating blue ball positioned over it to enhance visibility. An endless stream of AVs operating in an active automation mode as described in Section 3.2 drove on the right lane at a maximum speed of $50{\displaystyle \frac{\mathrm{km}}{\mathrm{h}}}$. The distance between the cars was set to be random but always less than 15 m, making a fast crossing without stopping the upcoming vehicle unsafe. Before the initial run, participants received the instructions described in Section 3.4. When participants performed a gesture during the trial, the next car that was still away in a safe braking distance of 15 m, the vehicle was yielding and exposing an external human–machine interface, similar to [41] with a green glowing hood as visual and a spoken text message telling “I am yielding” as an auditory cue.

Figure 4. Screenshot from the VR display of the scenario. (A) highlights the stream of incoming AVs, and (B) is the target marker. The expected trajectory is marked with a green arrow.

3.4. Study Procedure

The study was conducted with one interviewer and one experimenter. The interviewer was seated at a table face-to-face with the participant, while the experimenter was positioned at the computer station adjacent to the $2\times 4$ m area designated for the experiment. The experimenter set up the individual runs, while the interviewer handled only the questionnaires; this setup was chosen to optimize the study’s flow, though it could technically be replicated with a single experimenter. However, the use of multiple interviewers can reduce priming effects [96], which is beneficial for repeated tasks. Each participant was asked to provide the previously filled-out informed consent form. They then entered their demographic data on a tablet. For all digital forms, the interviewer offered support, while allowing participants to begin individually [97]. Then, participants were guided to the experimental area where they received instructions from the experimenter. During that, the interviewer digitized the Extended Barthel-Index (EBI) for the PID. The experimenter introduced the following statement to the participant:

Today, pedestrians can communicate their intentions to drivers with gestures. In the future, cars will be automated and thus have no driver. They will, however, be able to detect other road users. In the simulation you will be set into, all cars are automated. Your objective is to reach the other side of the road and the blue-marked area without colliding with a car. It is not important how fast you are, but how safely you cross. We are always right beneath you so that you can move freely without any collision hazard. If you do not feel well, we can stop at any moment.

The experimenter calibrated the headset and supported mounting it to the participant’s head at the starting position. The participant then familiarized themselves with the virtual lobby. After the familiarization, the experimenter started the gesture recognition system and the application remotely. If a participant made no progress within 45 s [98], they were reminded that even if AVs recognize them, they still do not know that they want to cross to induce a call-to-action. The wording was chosen to avoid any explicit hinting at any point in time, e.g., “The vehicles do not know that you want to cross”. After an additional 45 s, the pass was stopped. Participants then filled out the Technology Usage Inventory-Immersion (TUI-IMM) with the interviewer before performing a second pass with the VR study. The second pass was introduced as “almost the same scene with the same objective” to frame a separate investigation. The process was identical to the first pass. After the second pass, the participant completed the User Experience Questionnaire-Short (UEQ-S) and Situational Prosocial and Aggressive Behavior Perception in Traffic Scale (SPAT) with the interviewer. In the end, the interviewer asked some traffic-related open questions and collected open feedback (see Section 4.5).

3.5. Measurements

We collected both objective and subjective measures. From the VR headset, we collected gesture and movement data and transcribed spoken texts. To rate the severity of disabilities, PID completed an EBI questionnaire [99] filled out by a caregiver or other close person before the study commenced. We asked caregivers or close relatives to complete the EBI, as it requires a third-person assessment of a participant’s need for assistance with activities of daily living. This allows for classification of the severity of the cognitive disability by ICD-10-GM U51 (https://gesund.bund.de/en/icd-code-suche/u51; Accessed: 5 May 2026). As subjective data, we measured the quality of VR immersion using the TUI-IMM sub-scale [100], the perceived UX using the UEQ-S [101], and the perceived behavior using the SPAT [102]. All questionnaires have been adjusted for improved accessibility by applying the measures from Haimerl and Riener [103].

3.6. Participants

Eligible PID were recruited through the collaboration with multiple sheltered workshops. We described our study setup, including the space requirements, and submitted the informed consent forms, along with a paper version of the EBI, before the experiment. We acquired $n=40$ PID. $n=32$ PnDD was recruited from the university and work environments.

The initial sample consisted of $N=72$ participants. $n_E=40$ (13 female, 27 male, 0 non-binary) PID with an average age of $M=36.02$, $SD=16.39$ participated. $n_C=32$ (6 female, 26 male, 0 non-binary) PnDD with an average age of $M=36.72$, $SD=11.96$ participated. Employment in sheltered workshops in Germany requires an assessment by the ICF framework as an entry requirement [104]. This framework emphasizes functioning and contextual factors, so many of them lack an unambiguous medical diagnosis [105]. With all participating PID being classified by the ICF framework, all were able to communicate and understand simple language. Following the social model of disability [26], we did not classify clinical diagnoses but the degree of cognitive functioning. From all PID, $n=33$ filled out the EBI with a mean of $54.55$ ($SD=23.06$), (12 with no or mild, 21 with medium, and 0 with severe ICD-10-GM U51 classification). We did not measure the EBI of PnDD.

4. Results

Objective data had to be removed from P15 due to a software malfunction, and all data from P35 due to VR-sickness [106]. Both participants were PID, resulting in a final sample of $n_E=38$ PID. For measures with effects on repetition only, we had to exclude eight participants who only performed a gesture during one of the trials (P6, P18, P34, P39, P40, P45, P55, P56), resulting in $N_{rep}=62$ ($n_E=32$, $n_C=30$).

4.1. Data Analysis

Data analysis was performed using R (4.5.3) and the statistics package from Colley [107]. All packages were up to date as of April 2026. For non-parametric data, we used the ARTool (ART) by Kay et al. [108]. Assumptions for each test were verified, and Holm’s method was used for multiple-comparison correction.

4.2. Crossing Time

We measured the durations from the participants’ initial movement until they reached the roadside, then until they started to cross the road, and then until the target was reached. Participants needed an average of $M=31.76$ ($SD=36.39$) seconds from initial movement until they reached the target. The ART found a significant interaction effect of impairment × whether a gesture was performed × #Trial on overall duration ($F(1,58)=6.81$, $p=0.012$, ${\eta }_p^2$ = 0.11, 95% CI: [0.01, 1.00]); see Figure 5. This indicates an imminent learning effect that significantly improves crossing times for PID and is only present when a gesture is performed.

Figure 5. IE of impairment × whether a gesture was performed × #Trial on overall crossing duration.

4.3. Gestures

We analyzed whether performing a gesture influences the crossing time. The ART found no significant effects ($F(1,58)=1.52$, $p=0.223$); without a gesture, participants took $M=32.80$, $SD=20.32$ s; with a gesture, participants took $M=32.02$ and $SD=41.80$ s.

We analyzed whether PID or PnDD are more likely to perform gestures intrinsically. We fitted a logistic mixed model (estimated using ML and Nelder–Mead optimizer) to predict whether a gesture was performed with impairment (formula: whether a gesture was performed ∼ impairment). The model included participant_id as a random effect (formula: ∼1 | participant_id). The model’s total explanatory power is substantial (conditional $R^2=0.99$) and the part related to the fixed effects alone (marginal $R^2$) is of <0.001. The model’s intercept, corresponding to impairment = FALSE, is at $9.30$ ($95\%$ CI [$5.69$, $12.90$], $p<0.001$). Within this model, the effect of impairment is statistically non-significant and negative (beta = $-0.98$, $95\%$ CI [$-4.65$, $2.69$], $p=0.601$; Std. beta = $-0.98$, $95\%$ CI [$-4.65$, $2.69$]). Standardized parameters were obtained by fitting the model on a standardized version of the dataset. Then, $95\%$ confidence intervals and p-values were computed using a Wald z-distribution approximation. Descriptively, PID performed no gesture in $33.3\%$ of trials, compared to $18.8\%$ for PnDD (Table 1); this difference was not statistically significant.

Table 1. Absolute and relative occurrences of gestures performed by both participant groups.

Gesture	PID Abs	PID Rel	PnDD Abs	PnDD Rel
(NONE)	25	33.3	12	18.8
Attention
Greet	22	29.3	19	29.7
Raise Hand	-	-	2	3.12
Maneuver
Barrier	13	17.3	14	21.9
Slow Down	1	1.33	2	3.12
Stop	11	14.7	14	21.9
Negotiation
Thumbs Up	-	-	1	1.56
Pointing
Over There	3	4.0	-	-

We analyzed the position at the moment of gesturing as a distance in meters from the reference point perpendicular to the road. The ART found no significant effects of disability ($p=0.447$) nor of repetition ($p=0.947$). PID had a distance of $M=14.18$ m ($SD=0.50$ m) while PnDD had $M=14.27$ m ($SD=0.52$ m), indicating a very close and common positioning.

For the $N_{\mathrm{rep}}=62$ participants who performed gestures in both runs ($n=47$), we measured the durations from the participants’ initial movement until they first performed a gesture. The ART found a significant interaction effect of impairment × #Trial on overall duration ($F(1,45)=18.70$, $p<0.001$, ${\eta }_p^2=0.29$, $95\%$ CI: $[0.12,1.00]$); see Figure 6. This indicates that there is an inherent learning effect that AVs will react to certain gestures and that the effect is more pronounced for PID.

Figure 6. IE of impairment × #Trial on time to gesture.

We classified the gestures using the set from Brand and Schmitz [47], which includes the four classes: Attention, Maneuver, Negotiation, and Pointing. We detected seven different gestures from all four classes: Greet and Raise Hand from Attention; Barrier, Slow Down, and Stop from Maneuver; Thumbs U from Negotiation; Over There from Pointing. The occurrences of detected gestures are shown in Table 1; the absolute values are shown in Figure 7.

Figure 7. Absolute occurrences of gestures performed by both participant groups.

We fitted a Poisson mixed model (estimated using ML and the Nelder–Mead optimizer; see Figure 8) to predict n with impairment and gesture type (formula: n∼$\mathit{impairment}\times$ gesture type). Standardized parameters were obtained by fitting the model on a standardized version of the dataset. Then, $95\%$ confidence intervals (CIs) and p-values were computed using a Wald z-distribution approximation. The model included participant_id as a random effect (formula: ∼$1|participant\_id$). The model’s intercept, corresponding to impairment = FALSE and gesture type = (NONE), is at $-0.98$ ($95\%$ CI $[-1.55,-0.42]$, $p<0.001$). We only report statistically significant effects. Within this model:

Figure 8. Predicted gesture frequencies from a Poisson generalized linear mixed model. The model includes fixed effects of Disability and Gesture Type and their interaction, with a random intercept for participants. Points and lines represent model-based expected counts; differences between groups across gestures indicate the interaction effect.

• The effect of gesture type [Raise Hand] is statistically significant and negative (beta = $-1.79$, $95\%$ CI $[-3.29,-0.29]$, $p=0.019$; Std. beta = $-1.79$, $95\%$ CI $[-3.29,-0.29]$)
• The effect of gesture type [Slow Down] is statistically significant and negative (beta = $-1.79$, $95\%$ CI $[-3.29,-0.29]$, $p=0.019$; Std. beta = $-1.79$, $95\%$ CI $[-3.29,-0.29]$)
• The effect of gesture type [Thumbs Up] is statistically significant and negative (beta = $-2.48$, $95\%$ CI $[-4.52,-0.44]$, $p=0.017$; Std. beta = $-2.48$, $95\%$ CI $[-4.52,-0.44]$)

Vocalizing

We transcribed texts participants vocalized towards AVs. Only $n=5$ participants were speaking, none used the same wording, and different cues were used as displayed in Table 2.

Table 2. Table of all detected transcribed vocal cues from participants classified into participants’ disability and message class.

Participant	PID	Text	Class
P5	Yes	“Hey!”	Information
P26	Yes	“Stopp!”	Maneuver
P42	Yes	“I am going now!”	Intention
P43	Yes	“So, let’s go!”	Intention
P60	No	“Hello!”	Information

4.4. Questionnaires

Following the first trial, participants completed a TUI-IMM questionnaire to assess their sense of presence in virtual environments. The mean score for PID was $M=23.23$, $SD=3.77$, while for PnDD, it was $M=23.38$, $SD=3.15$. The reference values from Kothgassner et al. [100] (App. B) provide a mean of $M=15.13$, $SD=6.60$, suggesting the perceived immersion of both participant groups was above average. The ART found no effects on the TUI-IMM score ($F(1,69)=0.01$, $p=0.922$).

After the second trial, participants were asked to complete a UEQ-S questionnaire to assess their user experience with the gesture recognition and reaction system. The experimental group scored $M=1.88$, $SD=1.20$, while PnDD scored $M=2.32$, $SD=0.48$. The ART found no effects of disability ($F(1,69)=0.61$, $p=0.438$) nor of whether a gesture was performed ($F(1,69)=0.51$, $p=0.479$) on the UEQ-S score.

Lastly, participants completed an SPAT questionnaire regarding the yielding vehicle’s behavior toward them. PID scored $M=5.94$, $SD=1.33$, while the PnDD scored $M=5.83$, $SD=0.82$. The ART found no significant effects of disability ($F(1,69)=3.63$, $p=0.061$) nor of whether a gesture was performed ($F(1,69)=0.23$, $p=0.634$) on the SPAT score.

We performed ART on all subscales of the UEQ-S and the SPAT, but no significant effects were found (see Table 3).

Table 3. Results of the UEQ-S and SPAT questionnaires by disability with ART statistics.

Scale	PID		PnDD		ART
	M	SD	M	SD	F(1, 69)	p
UEQ-S	$1.88$	$1.20$	$2.32$	$0.48$	$0.61$	$0.438$
Pragmatic Qual.	$1.79$	$1.37$	$2.43$	$0.53$	$1.83$	$0.181$
Hedonic Qual.	$1.97$	$1.16$	$2.22$	$0.69$	$0.00$	$0.995$
SPAT	$5.94$	$1.33$	$5.83$	$0.82$	$3.63$	$0.061$
Socialness	$6.01$	$1.43$	$5.85$	$1.14$	$3.73$	$0.058$
Awareness	$5.96$	$1.41$	$5.98$	$0.83$	$1.67$	$0.200$
Predictability	$5.74$	$1.56$	$5.31$	$1.47$	$3.22$	$0.077$

4.5. Open Feedback

Following the last SPAT, we invited participants to share their impressions and report any observations in brief, complementary interviews. Instead of using a structured questionnaire, we asked guiding questions such as how they commute to work, how they experienced the interaction with the automated vehicle in VR, and whether they could imagine using an automated vehicle in the future. Notes collected by hand during the interviews were later analyzed in MaxQDA (Version 26.0.0) to develop codes and themes based on the methodology of qualitative content analysis [109,110]. From the initial sample ($N=72$), we received feedback from $n=26$ participants ($n=14$ PID; $n=12$ PnDD).

In general, opinions about automated vehicles varied among participants. While some ($n=3$ PID) rated them positively, others ($n=3$ PID) considered them unusual or untrustworthy.

$N=8$ ($n=7$ PnDD; $n=1$ PID) participants specifically provided feedback regarding the use of eHMI. The helpfulness of the large green area on the hood of the vehicle was highlighted by several participants (e.g., [P64, P66, P69], all PnDD), which complies with the findings of [41]. For example, [P70] (PnDD) had a deuteranope deficiency [111], but still emphasized the helpfulness, stating that “the meaning could be derived from the context”. Aside from that, $n=2$ (PnDD) reported the absence of a driver or eye contact, leading to a lack of feedback on the vehicle’s intention.

Some participants gave feedback about using gestures to stop the vehicle during the experiment. [P6] (PID) reported that he did not recognize that he was gesticulating. [P55] (PnDD) was only aware after the experiment that he was building himself up with his hands to the hips to get more visible for the car. This gesture was not previously observed in related work. In the taxonomy of Brand and Schmitz [47], we classify it as an attention gesture. [P54] (PID) reported he was deliberately not gesticulating as “[…] the car will not react anyway”, suggesting distrust in the technology.

Further, [P63, P65, P70, P71] (all PnDD) raised concerns about cultural differences in gesture use across cultural areas.

5. Discussion

In the following section, we will discuss our RQs based on findings from our investigation of the intrinsic gestures of PID.

Regarding RQ1, our results in Section 4.3 show a descriptive but statistically non-significant tendency for PID to gesture less. This differs from the assumption in [42], which suggested that PID might still gesture toward AVs even if clearly no driver is present. Most participants who did not use gestures waited for a gap between the vehicles and then rushed across the street. One possible explanation is that, even when prompted, it is unclear whether the AV can perform gesture recognition, as evidenced by some comments in the open feedback (Section 4.5), possibly connected to a lack of trust in automation [112]. Another explanation could be, that in the context of automation, trust can be divided into initial trust and dynamic trust [113]. Initial trust is based on a person’s existing knowledge and past experiences with similar systems before using an automated system. The use of gestures of our participants, therefore, may have been shaped by earlier, negative experiences with mobility. However, there was no significant difference in crossing time between participants who used gestures and those who did not.

We refrain from a mechanistic interpretation of this non-significant descriptive pattern. We note only that prior work on gestures and cognitive load [114,115,116] would predict either increased or decreased gesturing under high CL depending on whether gestures offload demand or require mental simulation capacity; the present data cannot distinguish these accounts, and most of that literature concerns co-speech gestures in children rather than communicative gestures toward vehicles.

Regarding gesture types, we identified commonalities in the performed gestures across groups. The most common ones were “Greet” from the “Attention” class, and “Barrier” and “Stop” from the “Maneuver” class. This suggests that there is no uniform intrinsic type of transmitted message, but rather that people tend to try to catch attention or convey a command individually. However, we could identify only three major gestures, all of which were used equally across participant groups. Regarding RQ1, this indicates that both PID and PnDD intrinsically use the same kind of gesture. Interestingly, during our short follow-up interviews, participants highlighted concerns about cultural differences in gestures across the respective cultural areas. According to Dong et al. [117], gestures convey different meanings and levels of suitability across various cultures, reflecting each culture’s norms and social expectations, while Tran et al. [91] find that cultural differences in gestures are rather subtle. While gesture-based interaction is feasible for both passenger control and vehicle-to-human communication in automated driving, existing research acknowledges the influence of culture on gesture adoption and interpretation. Still, it lacks the systematic cross-cultural comparisons needed to characterize specific cultural differences or to inform culturally adapted design [118,119], especially for PID. Additionally, Rasouli et al. [120] demonstrate that pedestrians primarily use implicit forms of communication, and that both the manner and frequency of this communication are strongly influenced by the surrounding context. As a result, the gestures produced by our participants, and consequently the identified gesture types, may reflect local gestural practices and therefore may not be generalizable to other populations or cultural contexts.

Regarding unprompted vocalizing, out of all participants, only five used vocal cues (see Section 4.3). Although four out of these were PID, there is no statistical significance that could help answer the vocalizing aspect of RQ1.

This restraint may be connected to the ambient noise, which produces continuous acoustic stimuli, known to lower the stimulus to speak [121,122]. A similar effect was shown for acoustic communication from AVs to pedestrians: Some papers are in favor of acoustic signaling [123,124,125], while others are rather opposing this [41,126]. In terms of universal design, acoustic cues should not be the sole communication method, but rather a mandatory modality for persons with vision disabilities or others who cannot rely solely on visuals [127]. Supporting studies investigate interfaces that adapt to the specific needs of VRUs [128] and to the situational context [129]. Our findings align with those of Meo, who found that current eHMI lack personalization because they rely on a one-size-fits-all message format. This limits their effectiveness for pedestrians with diverse needs, and adaptive solutions remain rare [130].

For RQ2, we found no influence of gestures on crossing time or position while gesturing, nor did PID differ in these terms from PnDD. This indicates that disabilities have little to no influence on the position on the roadside when communication with AVs commences.

We attempted to identify the immediate learning effects of gesturing by repeating the trial after a break. This break was designed to be threefold to minimize priming bias [131]: (1) Participants were taking off the VR headset and actively asked to walk some steps to leave the experimental area to disrupt the impression walking in VR produces. (2) Participants were asked to sit down outside the experimental area, heading away from it to de-focus from the trial. (3) Participants were asked to fill out a questionnaire between the trials, introducing a cognitively different task. With this, we expect the learning effects to be less biased and more consistent with operant conditioning [132], in which rewards for random actions lead to the intrinsic repetition of those actions. Considering a safe crossing and reaching the target as a reward, our results could be linked to this effect. Our results indicate that the effect is more pronounced for PID, exclusively among those who performed a gesture. This is noteworthy, as conventional active learning tasks often present high cognitive barriers for PID [133,134,135]. This suggests that the learning effect associated with intrinsic gestures is subconscious.

In conclusion, regarding RQ 1 and 2, we highlight that not only cultural differences but also situational and individual differences could affect the use of gestures, influencing the performance of future detection systems. Regarding interindividual differences, the existing literature has not directly examined how they influence the performance of specific communicative gestures, such as hand signals. However, several individual factors, such as age [136,137], gender [136,138]. Personality traits (particularly neuroticism, conscientiousness, and agreeableness [138]) have been shown to influence pedestrian behavior in traffic, with personality traits exhibiting the strongest effects on emotional expressions, and age affecting safety-related behaviors. Regarding the influence of situational factors, pedestrian gesture performance in traffic was shown to be influenced by vehicle approach speed [139], physical infrastructure characteristics (traffic signal presence, street width, crossing type) [120], environmental conditions (weather, scene, complexity, visibility), and cultural and geographical norms [91]. These factors operate interdependently to shape both the frequency and expressiveness of gestures used to communicate with drivers and vehicles. These interindividual and situational factors were not part of the present study, yet they may exert equal or even stronger influence on gesture performance than the variables examined.

5.1. Implications for Development

While the experiment was conducted in VR, we can still assume some implications for future development in the real world.

• Importance of High-Prevalence Gestures:

We consistently observed three main gestures (Greet, Barrier, Stop) across both groups. These findings should be validated in real-world scenarios and other cultural contexts to enable the development of future AVs that can reliably recognize and respond to these simple, high-prevalence gestures, ensuring robust recognition of a small, shared set [140,141].

2.

Inclusive Communication Strategies:

Even the descriptive (non-significant) tendency toward less gesturing among PID, and the fact that roughly a third of PID trials involved no gesture at all, suggests that gesture-based human-machine interfaces (HMIs) alone may not ensure inclusive interaction. AVs should thus add gesture recognition to the set of visual and acoustic modalities [39] to ensure accessibility across cognitive abilities. This complies with the suggested combination of multiple modalities, as proposed in related work [59,142].

3.

Design for Operant Conditioning Effects:

The observed learning effects, particularly among PID, imply that simple, consistent feedback from AVs (e.g., vehicle slowing down, visual confirmation signals) can reinforce communication behavior over time. HMIs designs should provide clear, immediate, and consistent responses to user signals, promoting safe learned interaction patterns.

5.2. Directions for Future Research

While our findings suggest that the gestures Greet, Barrier, and Stop account for the most prevalent interaction attempts, it remains unclear whether additional gestures emerge across different cultural settings, traffic scenarios, or real-world environments. VR studies are known to be efficient for studies substituting real-world scenarios [143]. Nevertheless, future research should validate our results in a real-world scenario and extend the scope of gesture analysis beyond the three major gestures identified in this study. Future studies should also include measures for CL, as traffic density [144], intellectual disability [145], and the usage of gestures [116] are known influencing factors.

Another important research direction lies in the comparative study of diverse user groups. Our work focused on PID and PnDD, but further research should include pedestrians with other cognitive, sensory, or motor disabilities. Such investigations are crucial for advancing universal design principles and ensuring that AV communication strategies do not unintentionally exclude vulnerable populations.

Finally, the observed short-term learning effects raise new questions about long-term habit formation. Future studies should investigate whether repeated exposure to AV feedback promotes stable, safe communication behaviors over time and whether these learning processes differ across user groups. In particular, it is important to assess how AVs can consistently reinforce positive behaviors through clear and immediate feedback.

Multimodal integration remains an open research challenge. While gestures, visual cues, and acoustic signals each have strengths and limitations, the optimal combination for different pedestrian groups remains unclear. Systematic studies should investigate how multimodal HMIs can balance inclusivity and clarity without creating cognitive overload. To validate these findings, subsequent work ideally will move beyond controlled VR settings and test interactions in real-world traffic environments, where environmental complexity and noise may strongly influence pedestrian behavior.

5.3. Limitations

First, as we applied the adjustments from Haimerl and Riener [103] to our questionnaires by converting scales to smiling faces to foster affection for answers and provide gradual support while filling out the questionnaires, enhanced positivity [146] and interviewer bias [147] might be reflected in the subjective data.

Second, the implemented gesture detection system was not working reliably due to a predominance of false positives, likely caused by overfitting of our gesture recognition model [148], which is a common limitation of pedestrian gesture recognition [149]. Therefore, we had performed an additional Wizard of Oz gesture detection, where the experimenter manually entered the moment a gesture was detected and took notes and scribbles of the performed gesture for subsequent classification based on the dataset by Brand and Schmitz, which makes the setup more prone to human error, while still more reliable than automated detection. The classification of gestures was partially ambiguous: The gesture “Greet”, with a hand held slightly higher, could be interpreted as “Raise Hand”, or, when held lower, as “Stop”, requiring the experimenter to weigh the classifications. Furthermore, we only classified static gestures: for example, the gestures “Wave to greet” and “Thank you!” from [47] were both classified as “Greet” both from the “Attention” class. Finally, we want to mention that this study was conducted in Germany, and therefore, participants were familiar with regional gesturing behavior. As our participants were familiar with regional gesturing behavior, our results may not be generalizable to other populations or cultural contexts.

Given the complementary nature of our interviews and the limited data available, we want to highlight that our findings should be interpreted as exploratory insights rather than a comprehensive account of participants’ experiences.

Furthermore, relying on caregivers or close relatives to complete the EBI introduces the potential for proxy reporting bias. While this proxy approach ensured standardized scoring of participants’ daily support needs, we acknowledge that third-party assessments can sometimes differ from a participant’s self-perception. This may, on the one hand, introduce bias in the supporting persons’ perception of capabilities. On the other hand, self-assessment of capabilities is prone to introspection illusion [150], which can be enhanced by cognitive conditions [151]. This could mean that a caregiver’s assessment yields more objective results. While the EBI provides a simple classification of cognitive functioning, this methodological choice inherently limits participant autonomy in this specific aspect of the data collection process.

It is important to acknowledge that VR studies have limited transferability to the real world [143,152], which is highly dependent on the fidelity of the simulation and the presence of the participant [93,153]. For our study, we followed the approach described in Haimerl et al. by using high-resolution assets and a low-latency standalone VR device to maximize participant presence [42]. With the good presence benchmark of the TUI-IMM (see Section 4.4), we expect our results to be transferable to the real world.

6. Conclusions

This study investigated the prevalence of intrinsic gestures by exploring differences in behavior and use of gestures from PID and PnDD when interacting with AVs in a crossing scenario simulated in VR. Descriptively, PID gestured less often toward AVs than PnDD, though this difference was not statistically significant; the assumption that people would still gesture even when no driver is present [42] was therefore neither confirmed nor refuted. Across both groups, we identified three main gestures (“Greet”, “Barrier”, and “Stop”) and concluded that many factors, including individual differences, situational factors, and cultural background likely influence gesture use. With no statistical effect, spoken communication played only a small role, highlighting the importance of non-verbal communication. We also found that gestures did not affect where or when people crossed the street. However, repeating the task showed signs of learning, especially for PID, suggesting that people may adapt their behavior over time in response to feedback.

Overall, this work extends the scientific body of inclusive automated traffic and supports the successful introduction of AVs. Future systems should combine multiple communication methods and better adapt to different user needs. If future AVs support gesture detection, communication could become more intuitive and easier to learn for VRUs. Instead of relying on complex or exclusive actions, such as reading instructions or pressing buttons, pedestrians might use natural gestures, like raising a hand, to safely cross the street.