Immersive Interaction for Inclusive Virtual Reality Navigation: Enhancing Accessibility for Socially Underprivileged Users

Abstract

Existing virtual reality (VR) street view and 360-degree road view applications often rely on complex controllers or touch interfaces, which can hinder user immersion and accessibility. These challenges are particularly pronounced for under-represented populations, such as older adults and individuals with limited familiarity with digital devices. Such groups frequently face physical or environmental constraints that restrict their ability to engage in outdoor activities, highlighting the need for alternative methods of experiencing the world through virtual environments. To address this issue, we propose a VR street view application featuring an intuitive, gesture-based interface designed to simplify user interaction and enhance accessibility for socially disadvantaged individuals. Our approach seeks to optimize digital accessibility by reducing barriers to entry, increasing user immersion, and facilitating a more inclusive virtual exploration experience. Through usability testing and iterative design, this study evaluates the effectiveness of gesture-based interactions in improving accessibility and engagement. The findings emphasize the importance of user-centered design in fostering an inclusive VR environment that accommodates diverse needs and abilities.

1. Introduction

Virtual reality (VR) is a technology that allows users to experience various contents in an immersive environment [1]. Its applications include fields such as entertainment, education [2], medical care [3], and tourism [4]. VR simulates reality, allowing users to indirectly experience environments that they cannot experience directly in a virtual space. These characteristics can provide new opportunities, especially for users with limited physical activity.

Individuals belonging to socially disadvantaged groups, particularly the elderly and those with limited digital proficiency, may face restrictions on their outdoor activities due to various factors, including physical limitations, mobility concerns, and challenges in utilizing digital technology due to aging. Consequently, there is an ongoing focus on research aimed at facilitating these individuals’ indirect exposure to the outside world through the use of VR technology [5]. Research has demonstrated that VR content designed for the elderly can provide significant benefits, including reducing the risk of falls by improving balance and mobility [6], as well as enhancing overall postural control and stability [7]. Furthermore, there has been a growing increase in the availability of VR content tailored for the elderly population [8,9].

For individuals with social vulnerabilities, 360-degree street view content, a subcategory of virtual reality (VR), can serve as a valuable instrument. This technology enables users to explore the world virtually without the need for physical travel, visit various locations, and engage in social interactions with other users through multi-party features. For individuals with physical and environmental limitations, this experience can serve as a crucial alternative to external activities [10]. Beyond mere navigation, the potential for 360-degree views to facilitate learning extends to various educational domains. A systematic literature review analyzing 64 studies highlighted the educational applications, advantages, and interaction characteristics of 360-degree VR videos and real VR [11]. The findings suggest that 360-degree VR can enhance learners’ performance, motivation, and knowledge retention while also fostering immersion.

However, the current commercially available VR street view applications are not designed to be easily used by the socially disadvantaged. A notable illustration of this challenge is the predominance of VR-based road view applications, including Google Earth, which necessitate the use of a controller. This requirement poses a substantial obstacle for users who are not accustomed to the digital environment due to the need for both hands. Furthermore, many existing applications demand a sophisticated interface and operation method, impeding their accessibility to the elderly and digital novices. To address these challenges, this study aims to explore the usability and cognitive impact of a gesture-based VR navigation system. Specifically, this study investigates the following research questions:

• RQ1: How does a gesture-based navigation system affect user experience (UX) and usability compared to a traditional controller-based system?
• RQ2: Does a gesture-based interface reduce cognitive load compared to a controller-based interface?
• RQ3: How effective is a gesture-based navigation system for older adults and individuals with limited digital proficiency?

To answer these research questions, this study focuses not only on the development of a VR street view application but also on evaluating how an intuitive, gesture-based interface can improve accessibility and usability for socially disadvantaged users. By eliminating the need for a controller, the proposed system allows users to navigate and interact with the environment using natural hand movements, increasing immersion and accessibility. The usability of this system is assessed through the SUS questionnaire, while the cognitive load differences between gesture-based and controller-based navigation are measured using NASA-TLX. The findings of this study provide insights into whether a controller-free VR navigation approach can serve as a viable alternative for older adults and individuals with limited digital literacy.

The experimental results support the effectiveness of gesture-based navigation in addressing the research questions. For RQ1, the System Usability Scale (SUS) evaluation demonstrated that the gesture-based system achieved significantly higher usability scores compared to the controller-based navigation system, confirming its effectiveness in improving user experience. For RQ2, NASA-TLX assessments revealed that the gesture-based system significantly reduced mental demand, physical effort, and frustration, indicating that it imposes a lower cognitive workload than controller-based navigation. Lastly, for RQ3, user feedback highlighted that older adults and individuals with limited digital proficiency found gesture-based navigation more intuitive and easier to use, supporting its potential as a viable alternative to traditional controllers for digitally disadvantaged users.

2. Related Work

2.1. Google Earth VR

Google Earth VR offers a highly immersive virtual experience, allowing users to explore the world in an interactive and intuitive way (Figure 1). By utilizing Google Earth’s extensive satellite imagery and 3D terrain data, the application provides a seamless, lifelike representation of global locations. Users can freely navigate through cities, natural landscapes, and famous landmarks, experiencing them as if they were physically present. One of its core features is 360-degree 3D exploration, which enables users to traverse the Earth with a complete panoramic view. This feature allows for the detailed exploration of various locations, from towering skyscrapers in metropolitan areas to remote natural wonders such as mountains, forests, and oceans. The integration of Google Street View further enhances this experience, offering realistic street-level perspectives that allow users to virtually “walk” through real-world locations.

For movement within the virtual world, Google Earth VR offers two primary navigation methods: teleportation and free movement. The teleportation feature enables users to instantly jump between locations, making it easy to visit distant places without disorientation. Meanwhile, the free movement mode allows users to “fly” smoothly through the environment, giving them the sensation of soaring above landscapes and cities. To enhance navigation control, Google Earth VR employs handheld VR controllers, such as those used with the HTC Vive, Oculus Rift, and Valve Index. These controllers allow users to manipulate the environment by pointing, selecting, and moving (Figure 2). However, this control method requires the use of both hands, which may pose challenges for users unfamiliar with VR controllers or those with limited mobility. Another key feature of Google Earth VR is the ability to adjust the scale and perspective of the view. Users can zoom out for a broad, globe-like overview of Earth or zoom in for a close-up, street-level exploration. This flexibility allows users to engage with the environment in different ways, whether they wish to explore vast geographical areas or inspect specific locations in detail. Additionally, Google Earth VR includes predefined virtual tours, guiding users through famous landmarks, historical sites, and natural wonders. These tours provide an educational and engaging way to discover new places, as they highlight important locations with built-in navigation prompts.

2.2. Wander

Wander VR is a virtual reality software that facilitates multiplayer functionality, enabling users to navigate virtual spaces collaboratively with friends and family (Figure 3). It enables real-time location sharing, thereby fostering a sense of collective exploration. This functionality allows users who are physically separated to partake in a shared virtual journey, which is anticipated to have a favorable impact on emotional stability, particularly for individuals experiencing social vulnerability. The incorporation of voice recognition capabilities in Wander VR empowers users to search for specific locations or execute commands by simply uttering the name of a city or landmark. This hands-free interaction method enhances accessibility, making it easier for users to explore without relying on manual input. Additionally, Wander VR supports various movement modes, including teleportation and smooth movement, allowing users to navigate different environments based on their comfort preferences. The teleportation feature minimizes the risk of motion sickness, while the free movement mode provides a more fluid and continuous travel experience.

Beyond solo exploration, Wander VR enhances social interactions through multiplayer voice chat, which enables users to communicate in real time while traveling together. This collaborative exploration feature makes the experience more engaging and immersive, allowing users to share discoveries and discuss locations as they explore. By combining multiplayer exploration, real-time voice chat, and flexible navigation modes, Wander VR provides a versatile and social virtual travel experience, making it more inclusive and practical for a wide range of users.

2.3. Wooorld

Wooorld is a virtual reality (VR) application designed for Meta Quest(Meta Platforms, Inc., Menlo Park, CA, USA) devices, including Quest 2, Quest Pro, and Quest 3, which allows users to explore the world in an interactive and immersive way (Figure 4). The application leverages Google Earth’s 3D data, enabling users to visit over 2500 cities in stunning detail. Unlike traditional VR travel experiences, Wooorld offers multiplayer functionality, allowing users to explore locations together with friends and family in real time, fostering a shared virtual travel experience.

One of the standout features of Wooorld is its hand-tracking support, which enables users to interact with the environment naturally without the need for a VR controller. This system allows for intuitive hand gestures, making navigation and interaction more seamless and accessible. Users can perform actions such as hand-ray pointing and pinching to interact with the interface. The hand-ray function acts as a virtual pointer, enabling users to select locations or interact with elements from a distance, while the pinch gesture allows users to make precise selections and zoom in and out of the 3D map. Additionally, Wooorld features a tabletop mode, allowing users to view and manipulate a 3D map as if it was placed on a table, providing a different perspective for exploring global locations. The inclusion of a mini-map feature further enhances navigation by helping users orient themselves within the environment, making exploration smoother and more intuitive.

2.4. Additional VR Exploration Solutions

Beyond the previously discussed VR applications, several other immersive VR exploration experiences offer unique approaches to virtual travel. These applications provide diverse methods for interacting with virtual environments, each with its own strengths and limitations. We examine four notable solutions—Blueplanet VR, BRINK Traveler, Titans of Space, and Realities.io—comparing their main features, navigation methods, level of immersion, accessibility, and control schemes. By analyzing these applications, we gain deeper insights into how current VR travel solutions address usability challenges and where improvements can be made to enhance accessibility for a wider range of users.

2.4.1. Blueplanet VR

Blueplanet VR offers a collection of over 40 curated virtual locations featuring high-quality photogrammetric volumetric scans of natural wonders and cultural heritage sites. Unlike traditional 3D models, these scans capture intricate details and depth, providing users with a realistic representation of various landscapes and historical landmarks. Users explore these environments primarily through a teleportation-based navigation system. At each location, multiple viewpoints are available, allowing users to move between predefined vantage points and examine different aspects of the scene. The application focuses on passive exploration, meaning that while users can view the environment from various perspectives, they have limited ability to interact with the surroundings beyond movement. Although photorealistic scans significantly enhance immersion, the reliance on standard VR controllers for navigation may introduce usability barriers, particularly for elderly users or those unfamiliar with VR technology. The teleportation mechanism requires precise pointing and clicking, which can be unintuitive for individuals with limited hand–eye coordination or motor control. Moreover, Blueplanet VR lacks alternative interaction methods such as voice commands or gesture-based controls, making it less accessible to users who may struggle with traditional VR interfaces.

2.4.2. BRINK Traveler

BRINK Traveler is designed to provide a highly interactive virtual tourism experience, using photogrammetry-based 3D reconstructions of real-world landmarks. Unlike many VR travel applications that focus solely on visual fidelity, BRINK Traveler integrates dynamic elements to create an engaging and educational experience. A defining feature of BRINK Traveler is its AI-powered virtual travel assistant, which offers contextual information and interactive guidance. Users can ask questions about different locations and receive real-time responses, enhancing their understanding of the environment. This functionality transforms passive exploration into an interactive learning experience, making it particularly valuable for educational applications. In terms of movement, users navigate through teleportation and can interact with specific elements within the scene. The application supports a guided experience, meaning that users can follow a predetermined path while listening to narrated explanations. However, the primary mode of interaction remains controller-based, requiring users to select points of interest, activate features, and move through the environment manually. Although BRINK Traveler introduces several enhancements over traditional VR travel applications, it still relies on conventional input methods, which may present challenges for users who are not comfortable with VR controllers.

2.4.3. Titans of Space

Titans of Space is an educational VR application designed to provide an immersive tour of the Solar System. Unlike terrestrial VR travel applications, which focus on real-world locations, Titans of Space presents a guided journey through space, allowing users to observe celestial bodies in a detailed and visually engaging manner. One of the most notable aspects of Titans of Space is its hands-free mode.Unlike other applications that require users to manually navigate through an environment, Titans of Space offers an automated tour experience where users can sit back and observe as the system takes them through different parts of the Solar System. This feature significantly lowers the barrier to entry for individuals who may struggle with complex controls or navigation systems. The experience is structured, meaning that users follow a predetermined path rather than freely exploring an environment. The application provides detailed explanations and interactive visualizations, making it a valuable tool for astronomy education. The ability to experience scale comparisons between planets and other celestial bodies enhances understanding and engagement. By removing the need for active navigation, Titans of Space makes VR exploration more accessible to users who may find traditional locomotion systems difficult to use.

2.4.4. Realities.io

Realities.io is a VR platform that specializes in photorealistic 3D-scanned locations, allowing users to visit historical sites, natural landscapes, and abandoned structures. Unlike applications that rely on synthetic 3D modeling, Realities.io employs advanced scanning technology to create highly detailed digital replicas of real-world environments. The platform provides an immersive experience by allowing users to move within these scanned environments using teleportation. Each scene is designed to capture the atmosphere and details of the original location, giving users a sense of presence. However, the primary mode of interaction remains teleportation-based locomotion and handheld controller input. This approach presents accessibility challenges for users who may not be familiar with VR systems or who struggle with precise controller interactions. Realities.io focuses primarily on passive observation rather than interactive engagement. Unlike BRINK Traveler, which includes guided narration, Realities.io relies on the visual fidelity of its environments to create immersion. The absence of interactive elements or voice-assisted exploration means that users must manually navigate through each environment, which may not be intuitive for all user groups. Although Realities.io excels in realism, its reliance on traditional control schemes limits its accessibility for individuals who are new to VR or who have physical limitations.

2.4.5. Comparison and Implications

Each of these VR exploration applications highlights both the strengths and limitations of current virtual travel solutions. Although they successfully deliver high levels of immersion and realism, most of them continue to rely on controller-based interaction, which remains a significant barrier for certain user groups. Titans of Space stands out as an accessible experience due to its guided, hands-free navigation, eliminating the need for manual locomotion and complex input methods. This approach aligns closely with efforts to improve accessibility by reducing reliance on traditional controllers. In contrast, BRINK Traveler enhances usability through its AI-powered assistant, which provides real-time information and voice interaction. However, despite these advancements, BRINK Traveler and Realities.io still require users to engage with VR controllers for movement and selection, which may limit accessibility for individuals unfamiliar with gaming interfaces. Blueplanet VR and Realities.io excel in providing high-fidelity visual environments but fall short in offering alternative interaction methods. These applications demonstrate that high-quality virtual exploration is possible, but their design choices often prioritize realism over accessibility.

These findings reinforce the necessity of developing alternative navigation systems that eliminate the need for handheld controllers while maintaining immersion and interactivity. A gesture-based navigation system has the potential to bridge the gap by providing a more intuitive interaction method that is accessible to a wider range of users. By incorporating gesture-based controls, future VR applications can ensure that socially disadvantaged users, elderly individuals, and those unfamiliar with gaming interfaces can fully participate in immersive virtual experiences without facing usability barriers.

2.5. Limitation for Socially Underprivileged Users

Despite the advanced features provided by Google Earth VR, Wander VR, and Wooorld (

Table 1. Comparison of Google Earth VR, Wander VR, Wooorld, and our system.

Feature	Google Earth VR	Wander VR	Wooorld	Our System
Controller Required	Yes (Both hands)	Yes (Both hands)	Yes (Optional)	No
Gesture-Based Interaction	No	No	Yes (Hand Tracking: Pinch & Touch)	Yes
Voice Recognition	Yes (Requires Controller)	Yes (Requires Controller)	Yes	Yes (Gesture-Based Activation)
Multiplayer Support	No	Yes (Real-Time Location Sharing)	Yes (VR Social Travel)	Yes (Real-Time Location Sharing)
Navigation Mode	Teleportation Free Movement Voice Commands	Teleportation Free Movement Voice Commands	Hand Ray+PinchFree ExplorationVoice Commands	Gesture-Based Voice Commands
Street View Integration	Yes	Yes	Yes	Yes

), these applications still present significant challenges for digitally disadvantaged users, such as the elderly or individuals unfamiliar with digital devices. One of the primary obstacles is operational complexity, which makes navigation difficult for users who lack experience with VR technology. Google Earth VR and Wander VR require the use of VR controllers, which involve button mapping, directional controls, and precise hand movements. For digitally disadvantaged users, these interactions can be unintuitive and difficult to master, creating a steep learning curve that prevents easy access to virtual exploration [12,13]. Prior research has highlighted that VR controllers pose a significant usability barrier for elderly users and individuals with motor impairments. Gluck et al. [12] found that complex controller-based interactions negatively impacted engagement, leading to frustration and reduced willingness to use VR applications. Similarly, Kruse et al. [13] emphasized that physically demanding interactions, such as pressing small buttons or coordinating joystick movements, impose cognitive and motor burdens that hinder accessibility.

While many applications offer voice input, activating this feature typically requires using a controller to cast a selection ray, an action that is not straightforward for users with limited digital proficiency. Similarly, although text-based search functions exist, entering text via a controller is challenging, and even general text input methods may prove difficult for those unfamiliar with digital devices. Wander VR’s voice recognition feature could improve accessibility, but since it still requires manual activation via a VR controller, the benefit is reduced for users who struggle with controller-based interactions [14]. Furthermore, traditional VR navigation methods often require fine motor skills, making them difficult for users with limited mobility or dexterity challenges. Wooorld, which offers hand-tracking capabilities as an alternative to controllers, still presents challenges for digitally disadvantaged users. Although it supports pinch and touch gestures, the hand-ray-based pinch gesture, which is used for selecting and zooming, can be difficult for users unfamiliar with precise hand-tracking interactions [15]. Unlike physical controllers with tangible buttons, hand-tracking requires precise hand positioning and controlled finger movements, which can be hard to execute for users who are not accustomed to such interactions.

Research on gesture-based interactions suggests that natural hand movements can offer a more intuitive and accessible alternative to traditional controllers, particularly for elderly users. Gerling et al. [15] demonstrated that older adults adapted quickly to in-air gesture interactions and found them more engaging than traditional input methods. Similarly, Del Rio Guerra et al. [14] explored gesture-based interactions for users with cognitive and motor impairments, concluding that intuitive hand gestures improved task performance and reduced frustration. These findings indicate that replacing complex controllers with simpler gesture-based interfaces could enhance usability and accessibility for socially disadvantaged groups.

Previous studies have shown that gesture-based interactions enhance usability for elderly individuals and those with limited experience in digital environments. For instance, Chen (2013) explored how gesture-based applications could improve accessibility for elderly users [16], while Carreira et al. (2017) evaluated the effectiveness of in-air hand gestures for older populations [17]. Studies comparing gesture-based and controller-based interactions have also highlighted the benefits of gestures for accessibility. Zhao et al. (2023) examined the differences between the two interaction methods and found that gestures provide a more natural and engaging experience for external users [18]. Similarly, Dharavath et al. (2024) demonstrated that integrating gesture and voice control in virtual environments improved usability for elderly users [19]. Furthermore, Premaratne et al. (2010) investigated hand-gesture-based human–computer interactions, emphasizing their advantages in enhancing user experience compared to traditional controllers [20].

Beyond improving accessibility for elderly users, gesture-based interactions also hold significant potential for individuals with disabilities. Siddique et al. [21] conducted a comprehensive survey on gesture control techniques for smart object interactions in disability support. Their findings highlight how gesture recognition systems can empower individuals with physical impairments by enabling intuitive control over digital and physical environments. By reducing the reliance on traditional input devices, gesture-based systems can foster greater independence and improve the overall quality of life for people with disabilities. This research further underscores the importance of developing inclusive interaction methods that cater to users with varying levels of motor ability, making virtual and augmented reality environments more accessible and user-friendly.

While Google Earth VR, Wander VR, and Wooorld provide sophisticated virtual exploration experiences, they lack intuitive, accessible interaction methods for digitally disadvantaged users. The reliance on controller-based navigation, complex gestures, and physically demanding interactions presents significant barriers that prevent these users from fully engaging with VR technology. To ensure a more inclusive and accessible virtual experience, there is a need for simpler, more intuitive interaction methods that minimize physical effort and reduce technological complexity for socially and digitally disadvantaged individuals. Gesture-based navigation, as explored in prior research, offers a promising alternative that warrants further investigation.

3. Our System

3.1. Features of the Proposed System

Existing VR-based road view applications, including Google Earth VR and Wander VR, offer advanced navigation capabilities. However, these applications present significant challenges for people with limited digital literacy skills due to their intricate operation and limited accessibility. Specifically, reliance on physical controllers can create barriers for users unfamiliar with VR hardware, while rapid viewpoint movement may induce motion sickness, reducing overall usability. To address these challenges, this study has developed a gesture-driven road view application, as depicted in (Figure 5). The proposed application is designed to facilitate the intuitive navigation of virtual spaces by introducing gesture-based interactions, eliminating the need for a physical controller. Hand movements are utilized to display and manipulate the map, allowing users to navigate back and forth, adjust their position, and interact with the environment seamlessly. A distinctive attribute of the application is the incorporation of a translucent map interface, which seamlessly integrates with the surrounding environment, thereby enhancing user immersion. Additionally, touch-sound feedback is provided upon operation, offering an intuitive experience and reinforcing user confidence. Unlike traditional VR navigation applications, which rely on controller-based inputs, the proposed system integrates multiple interaction methods to accommodate digitally disadvantaged users. The main functionalities of the system are summarized in

Table 2. Summary of key features in the proposed system.

Feature	Description
Gesture-Based Map Control	Open, close, and manipulate the map using hand gestures, eliminating the need for a controller.
Voice-Based Search	Activate voice search via a gesture-based trigger and navigate using speech commands instead of text input.
Random Landmark Selection	Enables users to randomly select and explore famous landmarks.
Direct Touch-Based UI Interaction	Allows users to interact with UI elements via touch gestures, replacing complex controller-based navigation.
Multi-User View Sharing	Supports multiple users sharing their virtual view in real-time, enabling collaborative exploration.
Voice Communication	Provides real-time voice chat functionality for seamless interaction between users.

.

3.2. Design Consideration

This study was specifically designed to compare the usability and cognitive impact of controller-based and gesture-based VR navigation systems. To facilitate this comparison, Google Earth VR was selected as the controller-based navigation system, as it represents a widely used VR street view solution that relies on handheld controllers for movement and interaction. The gesture-based navigation model, developed in this study, replaced the use of controllers with hand-tracking gestures and voice commands, allowing users to interact with the virtual environment in a more natural and accessible manner.

This study does not focus on developing a commercial application or service but rather investigates how alternative interaction methods can enhance accessibility for digitally disadvantaged users. As part of this effort, external APIs have been integrated into the system to support key functionalities, including Google Street View API for 360-degree panoramic views and location search, and OpenAI’s Whisper API for voice recognition. These APIs provide a robust technical foundation, allowing us to explore interaction design without being constrained by back-end data management, while the Google Street View API requires a paid subscription for extended usage, there are no legal constraints in using these APIs under the given terms. Additionally, since the research aims to explore interaction methods rather than develop a standalone product, API dependencies are evaluated in the context of user experience rather than long-term service sustainability.

Previous research has identified keyboard input in VR as a significant challenge for elderly users and those unfamiliar with digital environments. Dudley et al. [22], in Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality, note that text entry via virtual keyboards remains suboptimal, with issues related to efficiency and ease of use. Similarly, Nicolau and Jorge [23] highlight in their study on elderly text entry performance that older adults experience considerable difficulty typing on touchscreens due to the absence of physical feedback and increased cognitive load. These findings emphasize the necessity of replacing conventional text input methods with more accessible and intuitive interaction techniques, such as gesture-based navigation and voice commands.

3.3. Implementation

The Google Street View API was utilized to obtain a panoramic image of a specific location by requesting multiple image tiles using the panorama ID and subsequently merging them into a single, complete panorama (Figure 6). The process initiates with the retrieval of the panorama ID from the Google Street View API, based on the coordinates (latitude and longitude) provided by the client. After the retrieval of the panorama ID, 16 × 8 image tiles at zoom level 4 are downloaded in parallel to ensure efficient processing. These tiles are then merged to construct the entire panoramic image. To enhance performance and provide faster responses to repeated requests, the downloaded images are cached locally. An API request is handled using the Flask framework in Python 3.12.4, while image processing is performed using the Pillow library [24]. To further optimize the process, multithreading is applied, allowing parallel downloads to be executed efficiently. After the culmination of these steps, the final panoramic image is generated, ensuring a seamless experience for the user.

The downloaded panoramic image was applied as a skybox to configure the appropriate environment in Unity. To do so, first, the TextureShape property of Unity was set to Cube to convert the panoramic image to a cubemap format that wraps around 360 degrees, as shown in Figure 7. This allows users to enjoy an immersive virtual environment, and the entire space is rendered more realistically using Unity’s Skybox rendering system. The application of the skybox entailed rendering the cubemap-formated image using the skybox shader in Unity’s material settings, thereby enabling users to naturally rotate their heads and explore their surroundings in the VR environment. Additionally, the field of view (FOV) was adjusted to prevent image distortion, and the camera position was optimized to provide a more natural and stable search environment.

3.4. Gesture Interaction

Gesture-based interactions have been recognized as a more accessible alternative to controller-based interactions, particularly for socially disadvantaged users, such as individuals with physical disabilities, the elderly, and digital novices. Unlike controller-based interactions, which require precise manipulation of physical devices, gesture-based interactions enable natural and intuitive control through body movements, thereby reducing physical constraints and improving accessibility. The rationale behind employing gesture-based interactions instead of controller-based interactions is twofold: first, to mitigate the risk of unintended gestures being erroneously recognized, and second, to enhance recognition accuracy by requiring distinct, intentional actions. Additionally, gesture-based interactions facilitate intuitive operations, allowing users to engage with technology without the need for external input devices, which is particularly beneficial for users with physical impairments.

In this study, two distinct gestures were designed for interactions: the map display gesture and the voice recognition activation gesture. The map display gesture is executed through a “squeeze” forward motion, mimicking the unfolding of a physical map (Figure 8). This design choice enhances immersion and intuitiveness, allowing users to interact with the map in a direct and engaging manner. By extending the hands forward with palms outward, users naturally simulate spreading out a map, ensuring ease of use. Holding the hands stationary for one second triggers the map display, accompanied by auditory feedback, which reinforces user confidence in the interaction. The voice recognition activation gesture is based on a prayer-like movement, where both hands come together in a symbolic posture before initiating speech input (Figure 9). This gesture enhances immersion by associating voice recognition with a deliberate action, making it a natural method for activating speech input. Users must maintain this gesture while speaking, ensuring a seamless transition between physical action and speech recognition. Furthermore, since this gesture is distinct from everyday hand movements, it significantly reduces the likelihood of accidental activation, thereby improving the reliability and precision of the system. By designing gesture-based interactions grounded in existing research, this study aims to provide an accessible and intuitive interface, reducing learning barriers for socially disadvantaged users. The principles outlined by Saffer (2008) in Designing Gestural Interfaces informed the development of these gestures, ensuring they align with usability standards and maximize user engagement [25].

3.5. Speech Interaction

The development of the speech recognition function was implemented using the high-performance Whisper API, which has been demonstrated to offer high recognition accuracy [26] and is optimized for the conversion of user speech into text. When a user issues a voice command, the Whisper API converts it into text, and the converted text is then transmitted to the Google Search API to identify the most relevant location. Subsequently, a function was implemented that automatically directs the user to the top-ranked search result based on the retrieved search data.

Although multiple locations may be associated with a given search term, the system was designed to teleport the user to the most relevant location—determined by the first result in the search ranking—rather than presenting a list of choices. This approach was adopted to minimize additional user effort, as requiring manual selection from multiple results could introduce unnecessary complexity and disrupt the fluidity of the VR experience. However, this design choice also introduces a limitation, as it removes user autonomy in selecting among potential search results. In cases where the top-ranked location does not match the user’s intended destination, they would need to restart the search process. We acknowledge that this limitation could impact user control, particularly for individuals who prefer to verify search results before navigation. Future improvements could explore incorporating an optional confirmation step or an alternative selection mechanism to balance automation with user control. Nonetheless, for this study, prioritizing seamless navigation through direct redirection was considered the most efficient approach for enhancing accessibility and ease of use.

4. Experiment

4.1. Experimental Design

A total of 15 participants were included in the experiment, all of whom had no prior experience with virtual reality. The participants were divided into two groups: seven men and eight women, ranging in age from 50 to 60. This demographic was chosen to represent digitally disadvantaged users, who may face challenges when interacting with complex VR systems.

Participants were instructed to use both Google Earth VR and the developed gesture-based navigation system to perform the following tasks:

• Visit and explore their home and school within the virtual environment.
• Navigate through the downtown area, experiencing the street view of the city.
• Explore a tourist destination of their choice, simulating a virtual travel experience.

Each participant performed the same set of tasks in both systems, ensuring a direct comparison of interaction methods. The participants were first introduced to the VR interface and given time to freely explore and become familiar with the system before beginning the assigned tasks. After completing the tasks, participants were asked to complete the NASA Task Load Index (NASA-TLX) [27] and System Usability Scale (SUS) [28] questionnaires to assess cognitive load and usability.

By ensuring identical task conditions across both systems, this study aimed to isolate the effects of interaction method (controller-based vs. gesture-based) on user experience and workload. The main objective of the experiment was to evaluate and compare the workload and usability of controller-based and gesture-based VR navigation systems. By analyzing both cognitive demand and usability, this study provides insights into whether a gesture-based VR navigation system can serve as a viable alternative to traditional controller-based interaction methods for digitally disadvantaged users.

4.2. Study Procedure

To systematically assess the impact of each interaction method, this study was conducted in three key phases:

• Baseline Comparison: Google Earth VR was used as the baseline system, while the gesture-based navigation model was developed within a similar VR environment to enable a direct comparison.
• User Study: Participants navigated and interacted with virtual locations using both systems. Identical navigation tasks were assigned to compare their experiences and performance.
• Usability and Cognitive Load Evaluation: The usability and cognitive load of each method were quantitatively assessed using SUS and NASA-TLX metrics. These evaluations provided objective insights into whether the gesture-based interface improved usability and reduced cognitive strain compared to the controller-based interface.

By directly comparing Google Earth VR with a gesture-based alternative, this study evaluated the feasibility of controller-free VR navigation as a viable option for individuals with limited digital literacy. The findings provide insights into how intuitive, controller-free interaction methods can enhance accessibility and improve user experience for socially disadvantaged groups.

4.3. Evaluation Methods and Statistical Analysis

In this study, NASA-TLX (Task Load Index) (

Table 3. NASA-TLX questionnaire.

NASA–TLX (Task Load Index) Questionnaire
1. Mental Demand	How much mental effort was required to complete the task?
2. Physical Demand	How much physical effort was required to complete the task?
3. Temporal Demand	How much pressure did you feel due to the time constraints in completing the task?
4. Performance	How successful do you think you were in accomplishing the task?
5. Effort	How much effort did you have to put into completing the task?
6. Frustration	How much frustration or annoyance did you experience while performing the task?

) and SUS (System Usability Scale) (

Table 4. SUS (System Usability Scale) questionnaire.

SUS (System Usability Scale) Questionnaire
1.	I think I would like to use this system frequently.
2.	I found the system unnecessarily complex.
3.	I thought the system was easy to use.
4.	I think I would need the support of a technical person to use this system.
5.	I found the various functions in this system were well integrated.
6.	I thought there was too much inconsistency in this system.
7.	I would imagine that most people would learn to use this system very quickly.
8.	I found the system very cumbersome to use.
9.	I felt very confident using the system.
10.	I needed to learn a lot of things before I could get going with this system.

) evaluations were conducted to compare Google Earth VR with the system developed in this study. The purpose of these evaluations was to determine whether a gesture-based interaction system could reduce cognitive workload compared to controller-based navigation and to assess whether system usability remains acceptable even after replacing traditional controllers with gesture-based controls. These evaluations were designed to directly address the research questions. To ensure the validity of the comparison, an independent samples’ t-test was applied to analyze the statistical significance of the differences between the two systems, with a p-value of <0.05 considered as the threshold for significance. The findings from these evaluations provide empirical evidence to answer the research questions by measuring usability and cognitive workload in both interaction models.

To systematically evaluate usability and cognitive load, this study employed two well-established assessment tools: the System Usability Scale (SUS) and the NASA Task Load Index (NASA-TLX). NASA-TLX, introduced by Hart and Staveland [27], is a subjective workload assessment tool designed to evaluate cognitive load based on six key dimensions: mental demand, physical demand, temporal demand, performance, effort, and frustration. SUS, developed by Brooke [28], is a widely used metric for measuring system usability across various domains. It provides a quick and reliable assessment of perceived ease of use and user satisfaction.

The NASA-TLX (Task Load Index) is a widely used tool for assessing user workload when interacting with a system. It evaluates six factors that collectively represent cognitive and physical workload:

• Mental demand: The cognitive effort required to perform the task.
• Physical demand: The physical effort needed to interact with the system.
• Temporal demand: The perceived time pressure experienced by the user.
• Performance: The user’s self-assessed success in accomplishing the task.
• Effort: The amount of work exerted to complete the interaction.
• Frustration: The level of stress or annoyance experienced during task execution.

Previous studies have validated the effectiveness of these tools in evaluating usability and cognitive load in virtual environments. For instance, NASA-TLX has been applied to assess cognitive workload in human–computer interaction studies, confirming its utility in identifying workload differences between interaction methods [29,30]. Participants were instructed to assign a score on a scale from 0 to 100 for each factor, typically utilizing a Likert scale in 10-point increments. The NASA-TLX was selected to evaluate RQ2, which investigates whether gesture-based interactions reduce cognitive workload compared to controller-based navigation. By analyzing NASA-TLX scores, we assessed whether users experienced lower mental demand, physical effort, and frustration while using the gesture-based system. A comparative statistical analysis was conducted using an independent t-test, where mean workload scores were compared, and the results were interpreted based on statistical significance and effect size.

The System Usability Scale (SUS) was selected as a complementary evaluation method to assess the usability of the system. The SUS questionnaire consists of ten items that measure usability aspects using a five-point Likert scale (1: Strongly Disagree to 5: Strongly Agree). The key usability dimensions evaluated through SUS include the following:

• System acceptance: The likelihood of continued use.
• Complexity: Whether the system is unnecessarily difficult.
• Ease of use: The perceived simplicity of interaction.
• Required assistance: The need for external help to operate the system.
• System integration: How well the system’s features work together.
• Interface consistency: The clarity and predictability of interactions.
• Learnability: The ease of acquiring proficiency in using the system.
• User burden: The perceived difficulty in operation.
• User confidence: The level of comfort and competence using the system.
• Learning curve: The perceived effort required to become familiar with the system.

SUS has been extensively used to measure the usability of VR applications, demonstrating its reliability in assessing user experience [31,32]. The SUS evaluation was directly linked to RQ1, which examines whether gesture-based navigation enhances usability compared to a traditional controller-based system. By analyzing SUS scores, we assessed whether users found the gesture-based system easier to use, more intuitive, and better integrated into their interaction experience. To analyze usability differences between the two systems, the SUS scores were statistically evaluated using an independent t-test to determine whether the gesture-based system provided a significant improvement over the controller-based system. Additionally, this evaluation also contributes to answering RQ3, which focuses on the effectiveness of gesture-based navigation for older adults and individuals with limited digital proficiency. By analyzing usability scores and user feedback from participants aged 50–60 with no prior VR experience, we assessed whether gesture-based navigation provided a viable alternative to controller-based navigation for digitally disadvantaged users.

4.4. Statistical Analysis Methods

To ensure statistical reliability, the analysis was conducted using standard methods in human–computer interaction (HCI) research:

• Independent samples t-test: Used to compare the mean NASA-TLX workload scores and SUS usability scores between the gesture-based navigation system and Google Earth VR (controller-based system).
• Significance threshold: A p-value of <0.05 was used to assess statistical significance.

All statistical analyses were conducted using SPSS version 28.0 and SciPy 1.12.0. The results of these analyses provided empirical evidence to determine whether gesture-based navigation can serve as an effective alternative to controller-based VR navigation, particularly for older adults and digitally disadvantaged users.

5. Results and Discussion

The analysis revealed that the mean SUS score for Google Earth VR was 43.83, with a standard deviation of 6.04, indicating significant variability in the usability experience among users. In contrast, the average SUS score for our system was 59.67, which surpassed that of Google Earth VR. The standard deviation of 3.88 indicates that the user experience was relatively consistent (see

Table 5. SUS questionnaire results.

System	Mean Score	Min Score	Max Score	Standard Deviation
Google Earth VR	43.83	32.5	55	6.04
Our System	59.67	52.5	65	3.88

and Figure 10).

The System Usability Scale (SUS) score was calculated using the standard formula:

$$SU{S}_{\mathrm{score}}=\left(\sum _{i=1}^{10}(s_i-{\mathrm{offset}}_i)\right)\times 2.5$$

(1)

where $s_i$ represents the score for each of the 10 SUS questions (rated from 1 to 5), and the offset value (${\mathrm{offset}}_i$) is adjusted based on whether the statement is positive or negative:

• For odd-numbered questions (positive statements): ${\mathrm{offset}}_i=1$ (i.e., $s_i-1$)
• For even-numbered questions (negative statements): ${\mathrm{offset}}_i=5$ (i.e., $5-s_i$)

The final SUS score ranges from 0 to 100, where higher scores indicate better usability. To verify whether the difference in usability scores between the two systems was statistically significant, an independent samples’ t-test was conducted. The results of the analysis are as follows.

As demonstrated in

Table 6. SUS t-test results.

Measure	Statistic	p-Value
SUS Score	−8.54	0.000101

, the p-value is astonishingly less than 0.001, suggesting that the discrepancy between the two systems is statistically significant. Through meticulous analysis, it has been determined that our system provides superior usability for users in comparison to Google Earth VR.

User responses further support this finding:

User 1: “Interacting through gestures felt intuitive. In particular, the gesture for displaying the map closely resembled the real-world action of unfolding a physical map.”

User 2: “Compared to simple controller-based interactions, gesture-based controls provided a higher sense of immersion.”

Based on user responses and SUS evaluations, our proposed gestures not only enhance usability but also significantly contribute to a greater sense of immersion.

As demonstrated in Figure 11 and

Table 7. NASA-TLX t-test results.

Category	t-Statistic	p-Value
Mental Demand	−6.199	0.000002
Physical Demand	−3.309	0.002645
Temporal Demand	1.146	0.261678
Performance	0.264	0.794028
Effort	−3.020	0.005353
Frustration	3.315	0.002719

, the system under investigation exhibited a comparable level of usability to Google Earth VR, while significantly reducing cognitive and physical workload. Specifically, our system yielded lower mental demand ($p<0.001$) and physical demand ($p<0.01$) scores, indicating that users experienced reduced cognitive and physical strain when performing tasks. Furthermore, a statistically significant difference was observed in effort ($p<0.01$) and frustration ($p<0.01$) categories, suggesting that our system requires less effort to complete tasks and induces lower levels of frustration compared to Google Earth VR. These findings highlight that our system not only maintains usability on par with Google Earth VR but also enhances user experience by offering a more accessible and less burdensome interaction. Conversely, no statistically significant differences were found in temporal demand ($p=0.26$) and performance ($p=0.79$), indicating that both systems offer similar levels of efficiency in task execution and user-perceived performance. This suggests that users can complete tasks within a comparable timeframe and achieve similar performance outcomes. Thus, while maintaining an equivalent level of usability and task efficiency as Google Earth VR, our system further minimizes workload and frustration, enhancing overall user experience. The results from the independent t-test analysis reinforce that our system provides notable advantages over Google Earth VR by reducing mental and physical workload while preserving high usability. This demonstrates that users can perform tasks in a more natural and less fatiguing environment, making interactions both intuitive and effortless. Furthermore, unlike traditional controller-based applications, our system incorporates gesture and speech interactions, allowing for a more intuitive and immersive experience.

A key strength of our system lies in its gesture-based interactions, which significantly enhance accessibility and ease of use, particularly for socially disadvantaged users who may struggle with traditional controllers. Unlike controller-based methods that require fine motor skills and familiarity with complex button mappings, gesture-based interactions leverage natural hand and body movements, thereby reducing cognitive load and physical effort. This approach not only lowers the entry barrier for new users but also ensures that individuals with limited VR experience can navigate seamlessly without difficulty. Additionally, the gesture interactions in our system further amplify immersion, aligning with natural human movement to create a more engaging and intuitive user experience. By eliminating the need for complex button inputs, users can explore virtual environments with minimal disruption to their sense of presence. This is particularly advantageous for individuals who find traditional controller-based mechanics cumbersome, as our system enables effortless and fluid interactions with the virtual space.

5.1. Discussion on Research Questions

RQ1: How does a gesture-based navigation system affect user experience (UX) and usability compared to a traditional controller-based system?

Our study demonstrates that a gesture-based navigation system can provide an intuitive and accessible user experience compared to traditional controller-based systems. The SUS results indicate that users found the gesture-based system easier to use and more immersive, aligning with previous studies on the benefits of gesture interfaces for accessibility [16,17,18]. Unlike traditional controllers, which require precise hand coordination and button interactions, gestures enable a more natural interaction, reducing cognitive effort. This suggests that integrating gesture-based navigation into VR applications can enhance usability and engagement, particularly for users with limited digital proficiency.

RQ2: Does a gesture-based interface reduce cognitive load compared to a controller-based interface?

Our findings confirm that the gesture-based interface significantly reduces the cognitive load. The NASA-TLX results showed lower mental demand, physical demand, and frustration scores for the gesture-based system compared to Google Earth VR. Prior research [19,20] has also indicated that gesture-based interactions reduce the cognitive load by leveraging natural movement instead of complex button mapping. These results highlight that removing the need for controllers and simplifying interactions can lower cognitive strain, making VR navigation more intuitive and efficient. This is particularly beneficial for socially disadvantaged users, who may struggle with learning and operating traditional controller-based systems.

RQ3: How effective is a gesture-based navigation system for older adults and individuals with limited digital proficiency?

The results of this study strongly support the effectiveness of gesture-based navigation for older adults and digitally disadvantaged users. Our usability analysis showed that participants aged 50–60 found the gesture-based system easier to use and more engaging than controller-based alternatives. These findings align with prior research [14,15], which suggests that elderly users adapt more quickly to gesture-based interfaces than to controllers. Furthermore, the reduction in cognitive load and frustration indicates that this approach provides a viable alternative for users with limited digital literacy. This reinforces the importance of designing VR systems with accessibility in mind, ensuring that they can accommodate users with varying levels of technical proficiency.

5.2. Comparison with Existing VR Navigation Systems

Prior research has highlighted the usability barriers that traditional VR navigation systems pose for socially disadvantaged users [12,13]. Applications such as Google Earth VR, Wander VR, and Wooorld rely primarily on controller-based interactions, which require precise hand coordination, button mapping, and joystick control. These input methods present significant challenges for elderly users and individuals with motor impairments, leading to frustration and reduced engagement [14,15]. Furthermore, even systems that offer limited hand-tracking support, such as Wooorld, still require fine motor control, making them difficult to use for individuals with disabilities [15].

In contrast, our study demonstrates that a gesture-based interaction model can serve as a more intuitive and accessible alternative for VR navigation. Our experimental results indicate that users with limited digital proficiency found our system significantly easier to use than traditional controller-based applications. This aligns with the findings from previous studies that have shown gesture-based interfaces to be particularly beneficial for elderly individuals and those with physical impairments [16,17,18]. Moreover, although prior research has explored gesture-based VR applications in other contexts, such as rehabilitation and gaming [19,20], our study is one of the first to apply this approach specifically to VR street-view navigation. By eliminating the need for controllers and leveraging intuitive gestures, our system reduces cognitive and physical workloads, making VR navigation more accessible for socially disadvantaged groups.

6. Conclusions

This study developed a 360-degree road view application using virtual reality (VR) to improve accessibility for individuals with limited digital literacy. Existing VR street view applications, such as Google Earth VR and Wander VR, rely on complex controllers, making them difficult to use for older adults and digital novices. To address this issue, we introduced a gesture-based interface that simplifies navigation and enhances usability for socially disadvantaged users. Experimental results confirmed that the proposed system significantly reduces both mental and physical workload compared to controller-based systems, as measured by NASA-TLX and SUS evaluations. Gesture-based interactions provided a more intuitive and engaging experience, reducing cognitive strain and enhancing accessibility. By eliminating the need for controllers, the system allows users to navigate virtual environments more naturally, making VR technology more inclusive.

Our findings demonstrate that gesture-based interactions significantly enhance usability for digitally disadvantaged users, particularly older adults and individuals unfamiliar with traditional controllers. Unlike conventional VR navigation, which requires precise hand–eye coordination and fine motor skills, gesture-based navigation leverages natural human movements, lowering the entry barrier for first-time VR users. Additionally, gesture-driven navigation minimizes cognitive and physical workloads by simplifying interactions, allowing users to focus more on immersion rather than struggling with control mechanics. Although our study focused on individuals with limited digital proficiency, future research could explore how gesture-based navigation impacts other demographics, such as users with visual impairments or those with motor disabilities. A hybrid interaction model integrating gesture-based and controller-based navigation could provide greater flexibility, allowing users to choose interaction modes based on their needs.

Despite its advantages, the proposed system has certain limitations. It does not fully support complex functions such as text input, which may be necessary for tasks requiring detailed searches. Enhancing the system with integrated speech recognition and improved gesture-based text entry methods could address this limitation. Additionally, the current implementation requires two-handed interactions, which may pose challenges for users with limited mobility. Future research should explore alternative interaction models, such as adaptive gesture recognition or one-handed navigation options, to ensure broader accessibility. Beyond street-view navigation, the principles outlined in this study can be applied to various VR applications, including education, remote tourism, and virtual social interactions. Exploring how gesture-based navigation can be adapted for these contexts will further validate its effectiveness in enhancing accessibility and usability across diverse user groups.