Can Nonliterates Interact as Easily as Literates with a Virtual Reality System? A Usability Evaluation of VR Interaction Modalities

Abstract

The aim of the study is twofold: to assess the usability of a virtuality (VR) interaction designed for nonliterate users in accordance with ISO-Standard 9241-11 and to compare the feasibility of two interaction modalities (motion controllers and real hands) considering the impact of VR sickness. To accomplish these goals, two levels were designed for a VR prototype application. The system usability scale (SUS) was used for self-reported satisfaction, while effectiveness and efficiency were measured based on observations and logged data. These measures were then analyzed using exploratory factor analysis, and the ones with high factor loading were selected. For this purpose, two studies were conducted. The first study investigated the effects of three independent variables on the interaction performance of a VR system, i.e., “User Type,” “Interaction Modality,” and “Use of New Technology.” The SUS results suggest that all the participants were satisfied with the application. The results of one-way ANOVA tests showed that there were no significant differences in the use of the VR application among the three selected user types. However, some measures, such as task completion time in level one, showed significant differences between user types, suggesting that nonliterate users had difficulty with the grab-and-move interaction. The results of the multivariate analysis using statistically significant variables from both ANOVA tests were also reported to verify the effect of modern technology on interactivity. The second study evaluated the interaction performance of nonliterate adults in a VR application using two independent variables: “Interaction Modality” and “Years of Technological Experience.” The results of the study showed a high level of satisfaction with the VR application, with an average satisfaction score of 90.75. The one sample T-tests indicated that the nonliterate users had difficulty using their hands as the interaction modality. The study also revealed that nonliterates may struggle with the poses and gestures required for hand interaction. The results suggest that until advancements in hand-tracking technology are made, controllers may be easier for nonliterate adults to use compared to using their hands. The results underline the importance of designing VR applications that are usable and accessible for nonliterate adults and can be used as guidelines for creating VR learning experiences for nonliterate adults.

1. Introduction

The ability to read and write is vital in today’s world, yet a global literacy crisis is affecting many nations, with 771 million adults lacking basic literacy skills [1]. Illiteracy is the cause of various problems like poverty and non-sustainable economic growth [2]. Thus, the elimination of illiteracy is considered a key goal in the United Nations Sustainable Development Goals 2030. There are various approaches to enhancing adult literacy, including traditional instructor-led literacy programs that use direct instructional strategies [3,4] and the increasingly popular use of Information and Communication Technologies (ICT) that follow a learner-centered, active learning approach [5,6,7]. The curriculum utilized should be appropriate for instructing adults, as they possess a matured cognitive ability and their motivation to learn is shaped by their life experiences, which direct their method of acquiring knowledge [8]. There are multiple examples of research that suggests different techniques, such as using environmental print material to tutor the nonliterate adult population [6].

According to research, nonliterate adults tend to struggle with the cognitive processing of spoken language [9]. They have weaker abilities in retaining both verbal and visual information, and they possess lower visual-spatial abilities [10]. Katre et al. [11] discovered that these limitations stem from differences in cognitive development. Therefore, ICT should be designed according to the requirements of nonliterate adults. It is crucial to determine if the design principles that have been successful in conventional ICT can also be applied to create usable VR applications for nonliterates, considering the various interactivity and audio/visual differences in VR compared to traditional ICT. Furthermore, the design should be evaluated as per design standards such as ISO-9241-11 [12].

The interaction experience in traditional ICT applications includes hardware devices such as a mouse or a keyboard that provide an indirect manipulation experience to the users. Both devices provide a unique experience of interaction that is often used to do the same task in applications. So, these are the two communication channels or modalities that are engaged to interact with traditional ICT applications as defined by Bartneck et al. [13]. Multiple studies indicate that novice users face initial difficulties while using the devices [14,15,16,17,18]. One of the major difficulties faced by the user is the investment of extra time in learning input via keyboard or moving a mouse to click interfaces. Current VR systems, e.g., Oculus Quest 21 and HTC Vive2 use head-mounted displays (HMD) for the perception of vision and audio, and motion controllers or real hands are used as modalities for reality-based interaction. Therefore, it provides unique experiences of interaction, a sense of presence, and immersion in three-dimensional (3D) virtual environments. Despite several studies aimed at assessing the interaction experiences of the general population with VR [19,20], there is a lack of research specifically focused on nonliterate users. This means there are no established guidelines for evaluating the VR interaction experiences of the nonliterate population.

This study aimed to evaluate and compare the usability of various interaction modalities of VR systems in the context of nonliterate users following ISO Standard 9241-11. The study also aimed to compare the modalities among different user groups, such as tech-literate, non-tech-literate, and nonliterate, through a designed VR educational application. Therefore, the targeted research questions are as follows:

• RQ 1: Is the designed educational application usable for the nonliterate population?
• H₁:The designed educational application will be usable by the nonliterate population.
• RQ 2: If yes, then how easy is the application for nonliterate users as compared to the two other groups?
• H₂:The designed VR application will be as easy to use for nonliterate users as it is for literate users.
• RQ 3: Which interaction modality is more usable by nonliterate users?
• H₃:Nonliterate people will find hands to be more usable due to their intuitive and reality-based interaction styles as compared to controllers.

The rest of the paper is organized as follows: Section 2 discusses literature in the context of the objectives of this research. Section 3 explains the design of the VR prototype, and Section 4 explains the measures used in the research. Section 5 is about study 1 and the analysis of the data and the discussion of the results. Section 6 covers the analysis of results and discussions for study 2. Section 7 summarizes the results, highlights some of the limitations of the research, and provides suggestions and recommendations. Finally, Section 8 concludes the research.

2. Literature Review

In the context of the aims and objectives of this research, the following sub-sections of the literature review will discuss the modalities of VR, the interactivity of VR, VR-induced sickness problems, and finally the characteristics of the nonliterate adult population that must be taken into consideration when designing VR experiences.

2.1. Virtual Reality

Virtual reality refers to a computer-generated simulation of a 3D environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a headset with sensors [21]. This technology creates an immersive experience for the user, in which they can feel as though they are present in and a part of the artificial environment. With the availability of affordable consumer VR systems and rapid research and development in terms of vividness and interactivity, there is a growing trend toward its adaptation in various fields of education and training.

The psychological aspects of VR encompass how the human mind processes and perceives the experience in a simulated 3D environment. These aspects include presence and immersion as the most important concepts that make the VR medium stand apart from others [22]. Presence refers to the experience of being physically present within the virtual environment rather than simply viewing it on a display. Immersion, on the other hand, is the sensation of being completely enveloped in the virtual world, with one’s attention fully devoted to it. These two concepts work together to create the perception of a convincing virtual environment through the interplay of sensory input and the brain’s processing of these stimuli [23,24]. Perception is an active system that uses inputs from our sensory system and information from our cognition. For example, if you see a hurdle blocking your path, your sensory system provides data related to the hurdle, and your cognition provides information about the type of hurdle and how to overcome it. This provides enough cues to our perceptual system to infer a model of our surroundings so we can act accordingly [25]. The same phenomenon occurs in VR. In VR, immersion experience is defined by its capability to support natural sensorimotor possibilities; this stimulates our perceptual system to generate an illusion of being there, called presence [26]. Figure 1 shows that flow is the ultimate objective of any VR experience.

Figure 1. Implementation elements influencing VR Characteristics.

Flow is an autotelic experience, which refers to a self-contained activity that is done for a present reward without the anticipation of any future benefits [27]. The flow alienates one from reality to ecstasy without ruminating on the after-effects. A person in the state of flow has a boosted sense of self and concentration due to greater control, involvement, and enjoyment [28]. Awed by this experience, time is transformed, and the person is disconnected from his surroundings [29]. Consequently, this deep focus and enjoyment affect performance and its quality [30]. For a user to be in the state of flow, he must experience a sense of presence and immersion [31,32], which are directly affected by the quality of engagement, vividness, and interactivity [21].

For an engaging flow, content is always the most important commodity [33]. The implication of content is so diverse that we cannot decern its definition. In VR, most of the development resources are allocated to the creation, management, and marketing of content [34]. Content is divided into audio/visual components such as environments, objects, visual effects, sound effects, and music. Simply displaying the content is not enough to provide an optimal flow experience in VR. A player’s interaction with the content and its response are equally important. Therefore, for the optimal flow experience to happen, one must be engaged in an activity that is aptly challenging for his skill level [35]. These activities should be designed to easily achieve the optimal flow experience by providing appropriate equipment and an appropriate environment, devising rules that require learning skills, setting achievable goals, providing consistent feedback, and ensuring control [36]. Moreover, virtual reality sickness is a characteristic that can negatively affect the VR experience [37]. The focus of this work is to measure the interaction usability of the VR system for the nonliterate population; therefore, the topics being discussed in the sub-sections are interactivity [21], VR sickness [37], and the characteristics of the nonliterate adults that must be considered in the interaction design of the VR applications.

2.2. Interactivity

Interactivity in VR refers to the ability for a user to actively engage with and control elements within a virtual environment. This can include actions such as moving objects, selecting items, and communicating with other users or virtual characters. Interactivity is a key aspect of VR, as it allows users to experience a sense of agency within the virtual world and helps to increase their sense of presence and immersion [38]. Interactivity is stimulated by the technological aspects of the VR system [21]. In consumer VR systems, interactivity is achieved through the HMD, handheld motion controllers, motion tracking of the HMD and controllers, and hand tracking. These can also be called interaction modalities of VR systems. The presence of the head and hands is enabled through motion tracking of the HMD and controllers/hands within a specified room-scale play area. There are two types of motion tracking: outside-in and inside-out. Outside-in external motion sensors are used, while these sensors are integrated inside the HMD for inside-out tracking [39]. The most common interactions enabled by this technology are a simulation of hand presence, recognition of head and hand gestures, manipulation of 3D content, and facilitation of physical and artificial locomotion.

Current VR devices provide hand presence and interaction with the virtual world using wireless motion controllers and/or hand tracking. The handheld controllers have varied interactivity based on their designs. Figure 2 illustrates controllers for popular VR systems. These controllers are ergonomically designed to enable users to realistically interact with the VR environment. VR controllers have traditional action buttons and analog triggers, as found in many gaming controllers. Thumb sticks are provided for movement/locomotion, and internal motors are used for haptic feedback. These controllers track their position and send the data to the HMD to detect user hand movements and interactions. They can be programmed for several types of interactions, such as grab, pinch, poke, etc. These controllers provide a high degree of interaction fidelity and have a minimal temporal offset between real and virtual actions [40]. Hand tracking is also enabled in the current state-of-the-art VR systems. Instead of motion controllers, the user’s real hands are tracked by the cameras/sensors attached to the HMD and rendered in the virtual world. Users can use natural hand movements and gestures for interaction in VR. Regarding the controller, designing and implementing real hand interactions and gestures may be challenging. Errors in position tracking and gestures can also occur when hands are not visible or angled awkwardly to the HMD’s camera/sensors [41]. A high temporal offset between real and virtual actions can be experienced because the hand-tracking data is first processed by the HMD [42]. Several locomotion techniques are also possible in the current generation of VR systems and are divided into two categories: artificial locomotion and physical locomotion. Physical locomotion in VR is controlled by the user’s movements in the real world using the motion tracking of the HMD. Motion tracking of the HMD and controllers controls artificial locomotion. Examples are teleportation, walking in place, and world-pulling.

Figure 2. Left HTC Vive, Middle Oculus Rift, Left Oculus Quest 2.

There can be unimaginable possibilities for interactions in VR, but it is necessary to evaluate the contextual interactivity of a VR experience [43]. In research [44], the interactivity of handheld controllers is measured, and interaction design guidelines are proposed. However, these guidelines are general and are not specific to a certain type of VR experience.

2.3. VR Sickness

VR sickness, also known as cybersickness or simulator sickness, is a phenomenon where users of virtual reality (VR) systems experience discomfort or symptoms similar to motion sickness [37]. This is caused by a mismatch between what the user sees and what their body feels, leading to feelings of nausea, dizziness, and headaches. Factors contributing to VR sickness include high levels of motion, rapid changes in visual information, and a lack of stability in the virtual environment [45]. A person experiences motion sickness when there is a sensory conflict between visual stimuli and the vestibular system [46]. In VR, visual stimuli are the major sensory input and, therefore, can induce motion sickness and adversely affect the user experience. Eye movements and the vestibular system are predicted to be the major contributors to VR sickness, and it was advised that reducing the eye movements and incorporating motion simulation synchronized with visual stimuli may reduce it [37,45]. Since VR sickness (nausea and discomfort) may reduce the user experience, it is essential to measure it. The most used subjective method is the simulator sickness questionnaire (SSQ) [47]. Research has shown that instead of using the complete items of the SSQ, we can use the most common question from the SSQ to get some basic output from the user [48]. Similarly, for objective measurement, postural sway can be observed [37].

2.4. Characteristics of Nonliterate Adults

Designing VR applications for nonliterate people requires considering certain specific characteristics of this population. For example, they may have limited or no experience with technology, which means that interfaces and interactions must be intuitive and simple to understand. They may also have difficulty with visual information, so non-verbal cues such as audio, haptic feedback, and gesture-based controls may be more effective. Additionally, it is important to consider cultural factors, such as whether the language used in the application is appropriate, and to keep in mind that literacy levels can vary widely even within a single population. Overall, the design of VR applications for illiterate people requires careful consideration of their specific needs and abilities in order to create a successful and accessible experience. Therefore, it is recommended in multiple studies that ethnographic characteristics must also be considered when designing the content for ICTs [49,50,51,52]. These characteristics include life experiences, sociocultural factors, gender disparity, etc. Based on these observations and recommendations, several well-known general guidelines were proposed. These guidelines focus on audio/visual and task elements, for example, using hand-drawn images with visual cues, short and explicit audio cues for complex concepts, and showing consecutive steps during a task [50,53]. These guidelines might influence the implementation of engagement and vividness elements, but not the interactivity aspect of VR systems. Whereas, the use of modern technology, for example, smartphones or computers, can be considered a factor of interactivity.

The design and development of the VR applications as per the users’ characteristics demand the investigation of the interaction behavior of the users with the applications [49,50,51]. Such findings are elaborated as user-centered design (UCD). The UCD process is always conducted when underlying technology changes, or if the technology remains the same but a group of users changes. For example, Rasmussen et al. [54] elaborated that if an interface shape changes, it should be evaluated on a group of users every time.

3. VR Prototype Design

The aim of the study was two-fold. Firstly, to evaluate how well a certain interaction implemented in VR suits the nonliterate population in terms of effectiveness, efficiency, and satisfaction as per ISO-Standard 9241-11. Secondly, to evaluate the usability of two interaction modalities, i.e., motion controllers and real hands, along with the negative impact of VR sickness. To achieve the aims, two VR application levels were designed. As the focused user groups are adult, nonliterate people; therefore, the content for the VR application was designed considering the users’ characteristics as described above. The following Section 3.1 explains the design of the VR application levels and the content, and Section 3.2 provides information about the tools and technologies used.

3.1. VR Environment and Level Design

The artifacts of engagement and vividness were designed and organized in such a way that correlates with the cognitive differences and characteristics of nonliterate adult users. Elements of game-based learning were also incorporated, as it is considered an effective approach to transferring knowledge [55]. Wade et al. [56] found that prior knowledge-induced curiosity leads to higher learning. Therefore, we designed the game objects and environments following this concept while also considering the life experiences and sociocultural norms of nonliterate adults. Cognitive load is a major factor in increased error rates during a task based on gesture input [57]. Therefore, considering the lower visual-spatial skills of nonliterates, confined environments are designed for reduced cognitive load. Contemplating the lower language comprehension of nonliterates, easily comprehendible language was used to compose the audio instructions. Complements are considered positive emotion-laden word types [58], and positive emotions facilitate learning [59]; therefore, complementary remarks were also added. To invigorate an all-encompassing feeling, relaxing background music along with atmospheric sounds and effects were added. Language learning techniques were selected that can be effectively implemented in VR. Interaction schemes were programmed for both motion controllers and hand tracking. Only physical locomotion was used within a specified room-scale boundary to ponder VR sickness. Unintended accidents may occur while wearing the HMD, therefore, the real-world view was displayed on HMD screens using pass-through cameras upon crossing the virtual boundary. The prototype was thoroughly used and tested by expert VR users to find any bugs or exceptions. The final iteration was installed on the Oculus Quest 2 VR System. Figure 3 illustrates the level design process.

Figure 3. Level Design Process.

Two levels were designed to evaluate the interaction possibilities of the VR system. Level one was used to test and evaluate the usability and interactivity of different kinds of basic interactions using motion controllers and hand tracking. These basic interactions include grabbing, pinching, and poking. These basic interactions were then used to create complex interactions, as shown in Table 1.

Table 1. Implementation scheme of Interaction modalities and Interaction types.

Interaction Modalities	Interaction Types
	Grab	Pinch	Poke	Move	Haptics
Motion Controllers	Press the Grip button.	Press the trigger button.	Press the grip button and point the index finger.	The controllers track translation and rotation.	Feedback on interaction
Hand Tracking	Make a grip.	Join the index finger and thumb.	Make a grip and point the index finger.	Translation and rotation are tracked by HMD.	None

Level two introduced some more complex interactions and presented the user with a learning experience with the Urdu alphabet and their learning resources. The learning processes of writing, memorizing, and recognizing the alphabet were implemented through the three practice modes.

In the first level, the user was presented with an environment consisting of some basic game objects like small and large cupboards, a table, and a TV stand. Some electronics items, such as a TV and music system, were also placed. Everyday items were placed in such a way that the user required minimal locomotion to interact with them. Most of these items were grabbable, and some were pressable or usable. Some unrealistic objects based on real-world concepts were also placed in the environment, such as a floating TV remote with big buttons. The virtual environment and the interaction modes and possibilities are shown in Figure 4. In VR, spatial audio plays a vital role in affecting the sense of presence [60]. Therefore, several audio sources were placed in the environment to play music, ambiance sounds, interaction sounds, and instructions for the user.

Figure 4. Gameplay Images of Level One, designed to test and evaluate several possible interactions using motion controllers and hand tracking. Images (a–c) show the user interacting with the objects using motion controllers, while (d–f) show interaction with hands.

Furthermore, this level was comprised of six tasks. The user must complete all tasks to progress forward. Each task was designed to evaluate a specific type of interaction, as shown in Table 2. Periodic audio instructions were played to guide the user toward task completion. After the successful completion of the task, complementary audio was played to motivate the user.

Table 2. Level One Tasks and Interactions Mapping.

#	Task	Interactions
1	Grab the objects, 2 are required for the next task.	Grab and/or pinch
2	Grab the blue object on the map and move it to the appropriate province.	Grab and move (translate)
3	Press the buttons on the left of the table to change the music and/or on the right to change the TV channels.	Poke
4	Open the black drawer of the small cupboard and put the bracelet inside.	Grab and move (translate), grab and place
5	Grab the sword on the pedestal and cut the hay sticks.	Grab, move (translate and rotate), and use.
6	Open the red box in the big cupboard and grab the pistol inside. Shoot the target.	Grab and move (rotate), Grab, move and use.

The user can progress to the next level by pressing the big red button that will only appear after each task in level one was completed. The main objective of this level was to evaluate the basic interaction possibilities while training the users for more complex interactions in the next level.

The second level presented the user with an Urdu alphabet board with interactable alphabet cards. A new interaction type, “distance grab” was introduced at this level. After grabbing the alphabet card, the user can press the button on that card to play an educational video associated with the alphabet. Three alphabet writing modes were also created at this level so that the user can practice the learned alphabet. The first mode presented a traditional blackboard and marker setup where the user could grab a marker and write on the board. The second mode was an Urdu keyboard with color-coded alphabetic keys. Color-coding alphabet families was proven effective in teaching nonliterates [61]. The third mode gives the user the ability to write in the air. This mode was used to practice writing. Itaguchi et al. [62] have demonstrated that this is an effective way to memorize language shapes and letters both consciously and unconsciously. Figure 5 illustrates the user’s interaction with the VR environment using different interaction modalities.

Figure 5. Gameplay images of Level Two, designed as the application of the interaction types in Level One, and a complex interaction type known as distance grab is also used. Images (a–c) show the user interacting with the objects using motion controllers, while (d–f) show interaction with hands.

Like level one, the user had to complete tasks; Table 3 shows the task interaction mapping of level two.

Table 3. Level Two Tasks and Interactions Mapping.

#	Task	Interactions
1	Grab the alphabet card from the alphabet board by pointing your palm toward the board and pinch or grab it.	Distance Grab and pinch
2	Press the button on the alphabet card to watch an informative video about the alphabet. (Task 1 must be completed for this to work).	Poke
3	Use all three practice modes of writing the alphabets. Air-Writing, Board writing, Using Urdu keyboard.	Pinch and Move, Grab and Move, Poke

3.2. Tools and Technologies

Unity 2021.3.83 was used as the game engine with Oculus Interaction SDK4 and C# was used as a scripting language. Open-source tools such as Blender5 for 3D modeling, Gimp6 for textures, and Audacity7 for sound recording were used. Moreover, free assets from Unity Asset Store8 and Quixel’s Megascans9 were also used. Informative videos on the Urdu alphabet and Pakistan were linked in the game from YouTube channels “MUSE Lessons—Education Cartoons for Kids,” “Urdu Reading,” and “Discover Pakistan.”

This study used the Oculus Quest 210 VR System, which includes an HMD and two motion controllers. Oculus Quest 2 is among the cheapest VR Systems that provide several excellent features. This system requires no external computer and sensors, as it has its own computer and inside-out tracking system that is managed by Android. Users can interact with the VR using motion controllers or their hands. The portability, simplicity, and feature-rich characteristics of this system make it an ideal choice for this research.

4. Measures

Two studies were conducted to evaluate the usability of VR systems for nonliterate adult users. In the first study, we compared the usability of VR applications amongst three user groups, i.e., tech literates, non-tech literates, and nonliterates, and analyzed the differences based on the usability of interaction modalities and the effect of using technology. The second study was conducted only on nonliterate people. In this study, we used the data from study 1 as hypothesized values for tests in study 2. Furthermore, we added variables for technological experience and analyzed their effect on usability.

The interaction outcomes were measured as per ISO Standard 9241-11. The standard dictates usability in terms of effectiveness, efficiency, and user satisfaction. Effectiveness is the capability of users to carry out tasks and the quality of the productivity of those tasks. Efficiency is the amount of resource consumption by the user in executing the tasks. Satisfaction is the personal response of the user to using the system. These three aspects of usability should be measured holistically to evaluate the interaction usability of the designed application. There are multiple examples of such ICT evaluation, such as designing ATM user interfaces [10], designing multimedia content [11], and developing a website for nonliterate people [50]. The personal reaction of satisfaction can be measured subjectively using SUS [63]; thus, a survey was created using 10 items of SUS and a 5-point Likert scale. The questions are shown in Table 4. Based on previous research, a SUS score ≥ 68 is considered above average and <68 below average. SUS score was calculated by summing the score contributions of items 1, 3, 5, 7, and 9, i.e., item value minus 1, and for 2, 4, 6, 8, and 10, the contribution is 5 minus the item value and then multiplying by 2.5. This gives the SUS score a range of 0 to 100. The objective measures of efficiency and effectiveness were measured by logged gameplay data, video recordings of gameplay, and external videos, and observations were documented for several variables listed in Table 5. VR Sickness was measured using a 2-item questionnaire selected from the SSQ [37]. All the measures for dependent variables were verified using exploratory factor analysis.

Table 4. SUS Questions.

Id	Questions
1	I think that I would like to use this system frequently. مجھے لگتا ہے کہ میں اس سسٹم کو کثرت سے استعمال کرنا چاہوں گا۔
2	I found the system unnecessarily complex. میں نے سسٹم کو غیر ضروری طور پر پیچیدہ پایا۔
3	I thought the system was easy to use. میں نے سوچا تھا کہ سسٹم استعمال کرنا آسان ہوگا۔
4	I think that I would need the support of a technical person to be able to use this system. میں سمجھتا ہوں کہ اس سسٹم کو استعمال کرنے کے لیے مجھے کسی تکنیکی شخص کی مدد کی ضرورت ہوگی۔
5	I found the various functions in this system were well integrated. میں نے پایا کہ اس سسٹم میں مختلف طریقے اچھی طرح سے ضم تھے۔
6	I thought there was too much inconsistency in this system. میں نے سوچا کہ اس سسٹم میں بہت زیادہ عدم مطابقت ہے۔
7	I would imagine that most people would learn to use this system very quickly. میں تصور کرتا ہوں کہ زیادہ تر لوگ اس نظام کو بہت جلد استعمال کرنا سیکھ لیں گے۔
8	I found the system very cumbersome to use. میں نے سسٹم کو استعمال کرنے میں اپنے آپ کو بہت بوجھل پایا۔
9	I felt very confident using the system. میں نے سسٹم کا استعمال کرتے ہوئے بہت پر اعتماد محسوس کیا۔
10	I needed to learn a lot of things before I could get going with this system. اس سسٹم استعمال کرنے سے پہلے مجھے بہت سی چیزیں سیکھنے کی ضرورت تھی۔

Table 5. Variables and their measures.

Variables	Level 1		Level 2
	Id	Measures	Id	Measures
Effectiveness	1	Total Interaction in Level 1	1	The user is confident while interacting in VR.
	2	The user was confident while interacting in VR.	2	The user required external help.
	3	The user required external help.	3	The user followed in-app instructions.
	4	The user followed in-app instructions.	4	The user tried varied poses for interaction
	5	The user tried varied poses for interaction.	5	The user tried to interact with every object.
	6	The user tried to interact with every object.	6	Total Interactions in Level 2
Efficiency	1	1st Task Completion Time	1	1st Task Completion Time
	2	2nd Task Completion Time	2	2nd Task Completion Time
	3	3rd Task Completion Time	3	3rd-a Task Completion Time
	4	4th Task Completion Time	4	3rd-b Task Completion Time
	5	5th Task Completion Time	5	3rd-c Task Completion Time
	6	6th Task Completion Time	6	Errors in Distance Grab Interaction
	7	Errors in Grab Interaction	7	Errors in Poke Interaction
	8	Errors in Grab and Move Interaction	8	Errors in Pinch and Move Interaction
	9	Errors in Grab and Use Sword	9	Errors in Grab, Move, and Use Interaction
	10	Errors in Grab and Use Pistol	10	Errors in Poke and Use Interaction
VR Sickness	1	Discomfort reported.	1	Discomfort reported.
	2	Fatigue reported.	2	Fatigue reported.
	3	Postural sway observed.	3	Postural sway observed.

Validity of Measures

The questions in the SUS scale are proven to have both internal and external validity and have been used in numerous other studies [64,65]. The validity of measures for effectiveness and efficiency was verified by exploratory factor analysis. VR Sickness measures were also verified as they can negatively affect other variables. The results in Table 6 and Table 7 show that the selected measures can be used as factors in this research. All the components that have factor loading > absolute value of ±0.5 are included as measures in this research.

Table 6. Results of Exploratory Factor Analysis—Level One.

Factors	Factor Analysis
	Components	Factor Loading	Communality	Eigen Value	Explained Variance %	Cumulative Variance %
Effectiveness	Effectiveness 1	0.530	0.331	2.994	37.422	37.422
	Effectiveness 2	−0.773	0.796	1.836	22.951	60.373
	Effectiveness 3	0.880	0.778	0.943	11.791	72.164
	Effectiveness 4	−0.691	0.651	0.770	9.622	81.786
	Effectiveness 5	0.634	0.408	0.600	7.495	89.282
	Effectiveness 6	−0.724	0.671	0.193	2.418	100.000
Efficiency	Efficiency 1	0.855	0.765	2.856	25.964	25.964
	Efficiency 2	0.707	0.681	2.274	20.673	46.637
	Efficiency 3	0.581	0.709	1.273	11.572	58.209
	Efficiency 4	0.797	0.671	1.091	9.918	68.127
	Efficiency 5	0.808	0.738	0.908	8.258	76.386
	Efficiency 6	0.758	0.678	0.671	6.104	82.490
	Efficiency 7	0.735	0.634	0.624	5.675	88.165
	Efficiency 8	0.760	0.685	0.426	3.870	92.036
	Efficiency 9	0.767	0.775	0.378	3.433	95.469
	Efficiency 10	0.823	0.733	0.305	2.773	98.242
VR Sickness	VR Sickness 1	0.926	0.877	2.517	27.964	27.964
	VR Sickness 2	0.848	0.770	1.883	20.924	48.888
	VR Sickness 3	0.644	0.641	1.224	13.604	62.492

Used Principal Component Analysis as an extraction method. Used Varimax as a rotation method, KMO = 0.610, p = 0.003.

Table 7. Results of Exploratory Factor Analysis—Level Two.

Factors	Factor Analysis
	Components	Factor Loading	Communality	Eigen Value	Explained Variance %	Cumulative Variance %
Effectiveness	Effectiveness 1	−0.749	0.812	2.880	35.997	35.997
	Effectiveness 2	0.881	0.788	1.821	22.761	58.758
	Effectiveness 3	0.697	0.766	1.075	13.440	72.199
	Effectiveness 4	−0.601	0.377	0.843	10.543	82.742
	Effectiveness 5	−0.671	0.701	0.268	3.348	97.574
	Effectiveness 6	0.947	0.908	0.194	2.426	100.000
Efficiency	Efficiency 1	0.820	0.766	3.614	36.139	36.139
	Efficiency 2	0.904	0.834	1.859	18.591	54.730
	Efficiency 3	0.985	0.992	1.527	15.268	69.997
	Efficiency 4	0.985	0.992	1.046	10.464	80.461
	Efficiency 5	0.985	0.992	0.896	8.958	89.419
	Efficiency 6	0.850	0.755	0.716	7.156	96.575
	Efficiency 7	0.512	0.387	0.199	1.992	98.567
	Efficiency 8	0.911	0.866	0.143	1.433	100.000
	Efficiency 9	0.908	0.854	1.480 × 10−6	1.480 × 10−5	100.000
	Efficiency 10	0.696	0.607	7.838 × 10−7	7.838 × 10−6	100.000
VR Sickness	VR Sickness 1	0.926	0.877	2.517	27.964	27.964
	VR Sickness 2	0.848	0.770	1.883	20.924	48.888
	VR Sickness 3	0.644	0.641	1.224	13.604	62.492

Used Principal Component Analysis as an extraction method. Used Varimax as a rotation method, KMO = 0.605, p = 0.003.

5. Study 1

The main aim of this study was to evaluate the usability of the VR system and its interaction modalities for nonliterate adult users. Unfortunately, we have not found any similar research to compare our results against. Therefore, we decided to test our prototype VR on adult users that belonged to the following user groups:

• Tech-Literate: This group encompasses literate individuals with a high level of expertise and proficiency in utilizing computer systems, software applications, and digital devices. Participants in this group were mainly from computer science and software engineering backgrounds.
• Non-tech-Literate: This group consists of literate individuals with a basic or limited familiarity with the use of technology. Participants in this group came from non-technical fields.
• Nonliterate: This group comprises individuals who are not literate, regardless of their level of technology literacy. These individuals may have difficulties using digital devices and computer systems and may need support or training to effectively utilize technology. Participants in this group came from a variety of fields that did not require education or technical experience.

This study was conducted to answer the following research questions:

• RQ 1: Is the designed educational application usable for the nonliterate population?
• H₁: The designed educational application will be usable by the nonliterate population.
• RQ 2: If yes, then how easy is the application for nonliterate users as compared to the two other groups?
• H₂: The designed VR application will be as easy to use for nonliterate users as it is for literate users.

5.1. Procedure

All participants in the study provided their consent, either by completing a form (for literate participants) or giving verbal approval (for nonliterate participants), which was recorded by the experimenter. Some of the female participants refused external video capture, so the experimenter recorded the observations on paper. The experiment was continued only after the consent of the participant. The procedure of the experiment was duly approved by the COMSATS University Islamabad, Research and Evaluation Committee (CUI-REC). The experiments were conducted from 7–11 November 2022. The experiment was based on a between-subjects design. The experiments were conducted at separate locations as per users’ suitability and availability.

The VR prototype was designed in such a way that an average user took about 6 to 8 min to complete it. VR gameplay data was logged, and internal gameplay and external videos were recorded for objective evaluation. After the gameplay, the literate participants completed a concise survey based on 10 items of SUS and two items of VR sickness for 3 to 5 min. The experimenter read the same questions in the local language to the nonliterate participants, and their responses were recorded. Before using the VR devices, each participant was asked to sanitize their hands. Remedial counteractions were prepared in case VR sickness was experienced by the user. Some of the remedial counteractions were a place for the user to lie down, the availability of drinking water, and a jar of lemon and orange sweets.

5.2. Participants

The participants for the study were all residents of Abbottabad, KPK, Pakistan. The participants were recruited from different departments at COMSATS University Islamabad Abbottabad Campus: tech literates from the Computer Science department, non-tech literates from the Management Science department, and nonliterates from the Establishment department. Some nonliterate participants were also sourced from outside the university. They all practiced Islam as their religion and were fluent in Urdu, the national language of Pakistan. Due to cultural sensitivity, a female experimenter was arranged, as the female participants felt uncomfortable performing the experiment in front of male individuals. The tech-literate user group was composed of software engineering students and faculty from the COMSATS University Islamabad Abbottabad Campus in Pakistan. The non-tech-literate user group included students and faculty from non-technical degree programs. Meanwhile, the nonliterate user group consisted of individuals from various backgrounds who lacked literacy skills. A total of 30 participants took part in the experiment, with ages ranging from 21 to 55 years old. Out of the 30 participants, 12 (11 males and one female) were from the tech-literate group, eight (five males and three females) were from the non-tech-literate group, and 10 (seven males and three females) were from the nonliterate group. Only one tech-literate participant had prior experience with VR. All the literate participants used modern ICT equipment, while some of the nonliterates had limited interaction with ICT, using only mobile phones (N = 3), or mostly using smartphones to watch multimedia content (N = 7). The nonliterate participants were aware of computers and smartphones but had never used VR before, yet they were eager to try it. Twelve participants used the VRLE prototype with motion controllers, and 18 participants used their hands. A table with more detailed information on the participants is provided in Table 8.

Table 8. Information about the participants.

Participant’s Demography	Literacy Statistics	Interaction Modalities Used
Area:     Abbottabad, KPK, Pakistan	No Education 7 Less than 2 years 3 Higher Education 20	12—used motion controllers.   3—Tech-Literate   4—Non-Tech-Literate   5—Nonliterate 18—used their hands.   9—Tech-Literates   4—Non-Tech-Literate   5—Nonliterate
Numbers:   30—23 Male, 7 Female
Occupation:  17—Teaching/Training
11—Tech-Literate
6—Non-Tech-Literate
4—Labour/Worker
All Nonliterate
3—Technical/Operational
1—Tech-Literate
2—Nonliterate
3—Supervisory/Managerial
1—Non-Tech-Literate
2—Nonliterate
3—Unemployed
All Nonliterate
Technology Use: 19—Smartphone/computer
12—Tech-Literate
7—Non-Tech-Literate
8—Only smart phone
1—Non-Tech-Literate
7—Nonliterate
3—Only regular phone
All nonliterate
Age Range:  21–55

5.3. Analysis of Results

Three independent variables were used in this experiment. The comparison between the interaction of distinct types of users with the VR system was evaluated using “User Type” with three classes (tech-literate, non-tech-literate, and nonliterate). Interaction performance between controllers and hands was evaluated using the variable “Interaction Modality” with two classes (controllers and hands). The effect of technology use on interactivity was measured using “Use of New Technology” with two classes (Yes, No). SPSS 20 was used to analyze the acquired data. Two one-way ANOVA tests were conducted. In the first test, “User Type” was used as a factor for the dependent variables listed in Table 5, and in the second test, “Interaction Modality” was used as a factor.

In our survey, the satisfaction level reported by the participants is between 78 to 100 inclusive (nonliterate participants: 78 to 90 and literate participants: 83 to 100), which indicates that the users are satisfied with the designed application. Two one-way ANOVA tests were conducted to measure effectiveness and efficiency. In the first test, “User Type” was used as the independent variable, and in the second test, “Interaction Modality” was used as the independent variable. The descriptive and ANOVA results are attached as Appendix A, Appendix B, Appendix C and Appendix D and Table A1, Table A2, Table A3 and Table A4. All the statistically significant results along with their descriptive values are shown in Table 9 and Table 10. Out of 30 measures (seven for effectiveness, 20 for efficiency, and three for VR sickness), only eight variables were found statistically significant for “User Type” as the independent variable and only four for “Interaction Modality.” The results suggest that there were no significant differences in the use of the designed VR application for any type of user among the selected types (tech-literate, non-tech-literate, and nonliterate).

Table 9. ANOVA Analysis Results—user type as the independent variable.

Measure	p-Value	Descriptive Values
		Class	Number	Mean	Std. Deviation
Level 1 2nd Task Completion Time	0.021	Tech-Literate	12	25.58	27.158
		Non-Tech-Literate	8	54.25	36.850
		Nonliterate	10	58.80	20.741
Level 1 4th Task Completion Time	0.018	Tech-Literate	12	21.50	11.302
		Non-Tech-Literate	8	23.38	11.057
		Nonliterate	10	38.60	17.977
Level 1 6th Task Completion Time	0.022	Tech-Literate	12	53.83	21.294
		Non-Tech-Literate	8	32.25	10.553
		Nonliterate	10	53.10	16.231
Level 2 2nd Task Completion Time	0.019	Tech-Literate	12	39.22	22.222
		Non-Tech-Literate	8	36.22	13.764
		Nonliterate	10	69.21	37.228
Errors in distance grab	0.018	Tech-Literate	12	1.00	1.954
		Non-Tech-Literate	8	1.75	2.435
		Nonliterate	10	4.00	2.667
The user required external help	0.000	Tech-Literate	12	1.33	0.778
		Non-Tech-Literate	8	1.13	0.354
		Nonliterate	10	2.30	0.483
The user tried to interact with every object	0.004	Tech-Literate	12	3.08	1.165
		Non-Tech-Literate	8	2.88	1.458
		Nonliterate	10	1.40	0.699
The user followed in-app instructions	0.011	Tech-Literate	12	3.58	0.900
		Non-Tech-Literate	8	4.00	1.195
		Nonliterate	10	4.80	0.422

Table 10. ANOVA Analysis Results—interaction modality as the independent variable.

Measure	p-Value	Descriptive Values
		Class	Number	Mean	Std. Deviation
Level 1 6th Task Completion Time	0.003	Controllers	12	35.75	10.931
		Hands	18	55.89	19.638
Errors in grab and use sword	0.021	Controllers	12	2.25	1.603
		Hands	18	5.28	4.056
Errors in grab and use pistol	0.001	Controllers	12	2.42	1.165
		Hands	18	8.72	5.959
Errors in grab, move, and use a marker	0.037	Controllers	12	1.17	1.403
		Hands	18	2.78	2.264

The variables for which significant differences were found between the user types and interaction modalities are discussed in detail in the following sections. Furthermore, multivariate analysis using statistically significant variables from both ANOVA tests was conducted to verify the effect of the use of modern technology on interactivity. The related data is attached as Appendix E and Table A5. All the measures having significant mean differences are displayed in Table 11. The results are elaborated on in later subsections.

Table 11. Multivariate Analysis of Statistically Significant Variables.

Dependent Variable	Use of New Technology	User Type	Interaction Modality	Mean
Level 1 2nd Task Completion Time (Grab & Move Interaction)	No	Nonliterate	Controllers	78.000
			Hands	87.500
	Yes	Nonliterate	Controllers	48.000
			Hands	47.667
		Non-Tech-Literate	Controllers	23.500
			Hands	85.000
		Tech-Literate	Controllers	20.667
			Hands	27.222
Level 2 2nd Task Completion Time (Distance Grab, Pinch & Poke Interaction)	No	Nonliterate	Controllers	78.600
			Hands	117.800
	Yes	Nonliterate	Controllers	52.350
			Hands	56.167
		Non-Tech-Literate	Controllers	33.100
			Hands	39.350
		Tech-Literate	Controllers	39.700
			Hands	39.056
Errors in Distance Grab Interaction	No	Nonliterate	Controllers	5.000
			Hands	7.500
	Yes	Nonliterate	Controllers	1.750
			Hands	4.333
		Non-Tech-Literate	Controllers	2.250
			Hands	1.250
		Tech-Literate	Controllers	1.000
			Hands	1.000

5.3.1. Analysis of Results with “User Type” as Predictor

There were three user classes: tech-literate, non-tech-literate, and nonliterate. The dependent variables listed in Table 9 and Table 8 were found to be statistically significant with respect to the user type. The first significant variable was the “2nd task completion time,” which was logged during level one gameplay. This task was created to evaluate the grab-and-move interaction. In the interaction, the user must grab a blue pebble placed on the map of Pakistan and move it to the KPK province. The pebble movement was constrained in x and y coordinates; therefore, the user will not be able to grab and move it like other objects. It was observed that tech literates learned this interaction in three to four tries, yet this task proved difficult for the non-tech literates and nonliterates. This is also evident in Table 9, where the mean of the task completion time for non-tech-literate and nonliterate users is 54.25 and 58.80 s, respectively, while tech literates completed the task, on average, in 25.58 s. The results indicated that constrained movement in three-dimensional space was not comprehended properly by the non-tech-literate and nonliterate groups.

The next significant measure, i.e., “4th Task completion time” was also logged while playing level one. Grab, move, and place interactions were tested in this task. The user must open a drawer in the small cupboard and put a gold bracelet inside. Like in the 2nd task, the drawer was also constrained to be moved in only z coordinate, but this time, the difference was not due to the constrained movement. It was observed that nonliterate users were not grabbing the drawer by the handle; instead, they tried to open it from the sides. That may have been the reason nonliterates lagged as compared to the other groups.

Another variable from level one, i.e., “6th Task completion time” was also found to be significant. This task introduced interaction with usable objects. In this task, the user must open a red box, grab the pistol from inside, and shoot the target. This task was among the most complex ones because the user had to interact in multiple ways to complete it. The sequence of interactions started with opening the box lid; the user must grab the lid handle and move it upwards to open it. The next interaction in the sequence was to grab the pistol and use it to destroy the target. A pistol must be grabbed with grip interaction and then shot with the index finger. Interestingly, the tech-literate and nonliterate users completed this task later than the non-tech-literate ones. Some of the earlier experiments were conducted in a lecture room where the brightness levels were not adequate for hand tracking by the HMD and all the users were tech literate. Therefore, it can be assumed that this disparity resulted from the factor highlighted above. It was observed that all the above significant measures involved grab-and-move interaction, which proved difficult for nonliterate users. Therefore, we have decided not to implement this interaction in the later levels.

The next significant measure was the second task of level two, where the user must press a small button on the alphabet card to watch an informative video on TV. To complete this task, the user either must watch the whole video (30 to 45 s) or press the button again to stop the video. Despite knowing that the video can be stopped, most of the nonliterate users watched the video while the others stopped it, which was also reflected in the above result that nonliterate users completed this task in double time, but this also indicates their curiosity to learn.

The measure of “Errors in distance grab” was also related to level two. The observations revealed that nonliterate users tried to grab more than one card at a time. Another odd behavior by the nonliterates was uncovered during the observations, i.e., it seemed that they wanted to collect the alphabet cards. Therefore, after the experiment, they were asked about this phenomenon. Their responses disclosed that the displayed reticle while trying to grab a card was confusing. The observation was noted for a future iteration of VRLE.

The next three statistically significant measures were related to behavioral observations. The first one, “User required external help,” means that the user sometimes relied on outside help to complete the task. For example, in one of the experiments, it was observed that a nonliterate user was not moving while wearing the VR. The experimenter intervened and helped him move physically to get him started. Later investigation revealed that the user was waiting for some event to happen. Moreover, the user said that he had never seen such content and was confused. However, if given another chance, he would do better. The second observed behavior was that the “user tried to interact with every object.” This means that the user was not just picking the required object but also trying to grab other static or irrelevant objects, which were deliberately placed in the level to cater to the curiosity element. This observation was more prominent in level one. From the results, it was evident that most of the nonliterate users interacted with only the objects required for the task completion, and therefore they have the highest mean in the next measure, i.e., “User follows in-app instructions.” It was also apparent from the results that prior knowledge-induced curiosity was commonly observed in tech-literate users.

5.3.2. Analysis of Results with “Interaction Modality” as a Predictor

This analysis was made to compare the differences in interaction performance between controllers and hands as interaction modalities. We have found four statistically significant measures. The first measure was the completion time of the sixth task in level one, which also relates to the measure of errors in grabbing and using a pistol. As discussed in the above section, in this task, the user must use a pistol to shoot a target. It was observed that shooting using the controllers was much quicker than using actual hands, and that was because the controller had physical buttons for grip and trigger. Users could grab the gun using the grip button and shoot using the trigger button. The results in Table 10 revealed that grab-and-use interaction was more performant using the controllers.

The results also indicated that the temporal offset between real and simulated actions while using real hands as a modality could affect fast-paced interactions in VR. Users’ hands were tracked by the HMD, and if the real-world environment was not properly lit, then there could be problems with hand tracking due to a greater temporal offset, whereas controllers were self-tracked and were not plagued by it. This could be the only logical explanation for the observed significance. Some unrecorded experiments were conducted with different lighting conditions to confirm the above phenomenon. A reduction in errors was observed when an infrared illuminator was used, but more data is needed to confirm it.

5.3.3. Analysis of Results with the “Use of the Technology” as a Predictor

A multivariate analysis was conducted to analyze the effect of the use of modern technology on the interactivity of VR systems by nonliterate users. Therefore, an independent variable called “Use of New Technology” was introduced along with the other independent variables used in the above sections. This analysis was made using only the statistically significant measures identified in ANOVA tests. The variable “Use of New Technology” has two classes (“Yes” and “No”). The use of smart phones, computers, laptops, or any other state-of-the-art technology was categorized as “Yes.” Moreover, only the significant measures shown in Table 11 that have notable mean differences were considered in this analysis. It was observed that the use of modern technology did not affect the behavior of the user, yet it affected some interactions. It was observed that for both interaction modalities, nonliterate users who did not use modern technology had a tough time completing task 2 of level one, which involved constrained grab and move interaction. The other two measures were related to task two of level 2. In this level, the user had to distance grab an alphabet card and press a button on it. The interaction types associated with this task were distance grab, pinch, and poke. Overall, if the user uses modern technology, that may affect some of the complex two-handed interactions.

5.4. Discussion of Results

Research question 1 hypothesizes that the designed educational application will be usable for nonliterate users. The results of the experiment showed that the satisfaction level reported by participants was between 78 to 100, indicating that users were satisfied with the application. The results of the two one-way ANOVA tests showed that there were no significant differences in the use of the VR application among the three user types (tech literate, non-tech literate, and nonliterate). However, a few variables were found to be statistically significant.

Research question 2 hypothesis states that the educational application will be easier for nonliterate users to use, which is partially supported by the results of the experiment. The satisfaction level reported by the participants indicates that all types of users are satisfied with the designed application. However, the results of the one-way ANOVA tests suggest that there are significant differences in task completion times between the user types, with non-tech-literate and nonliterate users taking longer to complete certain tasks compared to tech-literate users. The results also indicate that nonliterate users had difficulty with some of the interactions, such as the grab-and-move interaction and interactions with usable objects. The results suggest that the educational application is generally easier for nonliterate users, but the application may need to be improved to better cater to the needs of nonliterate users in certain interactions.

6. Study 2

In our previous study, we evaluated the usability performance of a VR application among three user groups: tech-literate, non-tech-literate, and nonliterate. Our findings indicated that the tech-literate and non-tech-literate groups performed relatively better compared to the nonliterate group. As a result, we took the average values of the dependent variables obtained from the tech-literate and non-tech-literate user groups for each interaction modality and used the values to establish test values. The main objective of this study was to use these test values shown in Table 12 to analyze the differences in efficiency and effectiveness between the two modalities. Another predictor was added to the study to analyze the effect of years of technological experience on VR usability. This study was conducted to answer the following research question:

Table 12. Test values.

Variables	Measures	Average Values (Controllers)	Average Values (Hands)
Effectiveness	Total Interaction in Level 1.	24.14	30.00
	Total Interactions in Level 2.	9.00	8.85
	The user was confident while interacting in VR.	5.00	4.69
	The user required external help.	1.14	1.31
	The user followed in-app instructions.	4.29	3.46
	The user tried varied poses for interaction.	1.57	2.46
	The user tried to interact with every object.	2.71	3.15
Efficiency	Level 1 1st Task Completion Time	40.00	48.00
	Level 1 2nd Task Completion Time	22.29	45.00
	Level 1 3rd Task Completion Time	27.14	31.38
	Level 1 4th Task Completion Time	16.86	25.15
	Level 1 5th Task Completion Time	20.00	22.31
	Level 1 6th Task Completion Time	29.86	53.46
	Leval 2 1st Task Completion Time	38.54	37.40
	Leval 2 2nd Task Completion Time	35.93	39.15
	Level 2 3rd-a Task Completion Time	72.81	61.89
	Level 2 3rd-b Task Completion Time	91.01	77.36
	Level 2 3rd-c Task Completion Time	109.19	92.83
	Level 1 Errors in Grab Interaction	1.29	3.00
	Level 1 Errors in Grab and Move Interaction	4.71	9.77
	Level 1 Errors in Poke Interaction	4.00	7.85
	Level 1 Errors in Grab and Use Sword	1.71	5.08
	Level 1 Errors in Grab and Use Pistol	2.14	9.54
	Level 2 Errors in Distance Grab Interaction	1.71	1.08
	Level 2 Errors in Poke Interaction	0.86	1.31
	Level 2 Errors in Pinch and Move Interaction	0.71	1.92
	Level 2 Errors in Grab, Move, and Use Interaction	0.14	2.46
	Level 2 Errors in Poke and Use Interaction	1.86	1.92
VR Sickness	Discomfort reported.	1.29	1.15
	Fatigue reported.	1.29	1.15
	Postural sway observed.	1.00	1.08

• RQ 3: Which interaction modality is more usable by nonliterate users?
• H₃: Nonliterate will find hands to be more usable due to their intuitive and reality-based interaction style as compared to controllers.

6.1. Procedure

The participants in the study voluntarily agreed to participate in the experiment after being fully informed by the experimenter. Some female participants declined to have video recordings taken, so their observations were recorded by the experimenter through written documentation. The study was approved by the CUI-REC at COMSATS University Islamabad and only carried out after obtaining the participants’ consent. The experiments took place from 5–10 January 2023 and employed a between-subjects design, conducted at various locations depending on the participants’ schedules and accessibility.

The VR prototype was designed to take approximately 6 to 8 min to complete, and data was recorded during the gameplay, including internal gameplay and external videos for evaluation. After completing the VR session, the questions for SUS were read to the participants and the responses recorded by the experimenter. Before using the VR device, each participant was required to sanitize their hands, and remedial measures such as a place to lie down, water, and lemon/orange sweets were available in case of VR sickness.

6.2. Participants

The participants in the experiment were all from Abbottabad, KPK, Pakistan, and were recruited through the establishment department of COMSATS University Islamabad, Abbottabad Campus. Additionally, some nonliterates were also sourced from external locations. All the participants followed Islam as their religion. Urdu is the national language of Pakistan; hence, all the participants understand the language. Female participants were hesitant to perform the experiment in front males; therefore, the female experimenter was arranged. A sample of 10 nonliterate adults (split evenly between males and females), ranging in age from 19 to 50, participated in the experiment. All the participants had smartphones except for one, and three of them also used computers. The participants who had access to modern technology, such as smartphones or computers, used these devices primarily for consuming multimedia content. During the experiment, half of the participants used the prototype VRLE with motion controllers, while the other half used their hands. A full list of participant information can be found in Table 13.

Table 13. Information about the participants.

Participant’s Demography	Literacy Statistics	Interaction Modalities Used
Area:     Abbottabad, KPK, Pakistan	No Education 8	5 participants used motion controllers.
Numbers:   10—5 Males, 5 Females	Less than 2 years 2	5 participants used their hands.
Occupation:  5—Labour/Worker
1—Technical/Operational
1—Supervisory/Managerial
3—Unemployed
Technology Use: 5—Smartphone/computer
4—Only smart phone
1—Only regular phone
Technological Experience:    4—More than 5 years
4—1 to 5 years
2—Less than 1 year
Age Range:    19–50

6.3. Analysis of Results

The study is focused on nonliterate users and aims to compare the interaction performance between controllers and hands when using a VR application. Additionally, the effect of technological experience on interactivity was also measured. In the experiment, two independent variables were employed. The “Interaction Modality” variable, which had two classes (Controllers and Hands), was used to assess the interaction performance between controllers and hands. The “Years of Technological Experience” variable, which had three classes (less than 1 year, 1 to 5 years, and over 5 years), was used to evaluate the impact of technological experience on interactivity. The analysis was conducted using SPSS 20. One-sample T-tests were performed on the measures of effectiveness and efficiency, using the hypothesized mean values from Table 12. The tests were conducted for both interaction modalities, and the results are shown in Appendix F, Table A6 and Appendix G, Table A7. The statistically significant results and their descriptive statistics are presented in Table 14 and Table 15.

Table 14. Significant measures of T-tests with Controller as an interaction modality.

#	Variables	Statistics
1	Level 1 2nd Task (Map—Grab & Move Interaction) Completion Time	Test Value = 22.29
		t	df	Sig. (2-tailed)	Mean Difference
		4.536	4	0.011	21.110
2	Level 1 4th Task (Put Jewellery—Grab & Move + Grab & Place Interaction) Completion Time	Test Value = 16.86
		t	df	Sig. (2-tailed)	Mean Difference
		3.611	4	0.023	14.740
3	Level 1 5th Task (Sword & Cut—Grab & Use Interaction) Completion Time	Test Value = 20
		t	df	Sig. (2-tailed)	Mean Difference
		3.162	4	0.034	4.000
4	Level 2 1st Task (Alphabet Cards—Distance Grab + Pinch Interaction) Completion Time	Test Value = 38.54
		t	df	Sig. (2-tailed)	Mean Difference
		−7.207	4	0.002	−16.340
5	Gameplay User tries to interact with every object.	Test Value = 2.71
		t	df	Sig. (2-tailed)	Mean Difference
		−5.348	4	0.006	−1.310
6	Level 1 Errors in Grab and Use Sword	Test Value = 1.71
		t	df	Sig. (2-tailed)	Mean Difference
		−5.348	4	0.006	−1.310

Table 15. Significant measures of T-tests with hands as an interaction modality.

#	Variables	Statistics
1	Level 1 4th Task (Put jewellery—Grab & Move + Grab & Place Interaction) Completion Time	Test Value = 25.15
		t	df	Sig. (2-tailed)	Mean Difference
		6.744	4	0.003	13.250
2	Level 1 5th Task (Sword & Cut—Grab & Use Interaction) Completion Time	Test Value = 22.31
		t	df	Sig. (2-tailed)	Mean Difference
		3.686	4	0.021	12.090
3	Level 2 3rd Task Air Writing (Pinch + Move Interaction)	Test Value = 61.89
		t	df	Sig. (2-tailed)	Mean Difference
		4.860	4	0.008	18.870
4	Level 2 3rd Task Board Writing (Grab or Pinch + Move interaction)	Test Value = 77.36
		t	df	Sig. (2-tailed)	Mean Difference
		4.839	4	0.008	23.560
5	Level 2 3rd Task Typewriter (Poke Interaction)	Test Value = 92.83
		t	df	Sig. (2-tailed)	Mean Difference
		4.849	4	0.008	28.290
6	Gameplay User tries to interact with every object.	Test Value = 3.15
		t	df	Sig. (2-tailed)	Mean Difference
		−3.637	4	0.022	−1.150
7	Gameplay User is following the in-app instructions.	Test Value = 3.46
		t	df	Sig. (2-tailed)	Mean Difference
		6.700	4	0.003	1.340
8	Gameplay User tried varied poses to interact with the objects.	Test Value = 2.46
		t	df	Sig. (2-tailed)	Mean Difference
		−3.511	4	0.025	−0.860
9	Level 1 Errors in Grab and Move Interaction	Test Value = 9.77
		t	df	Sig. (2-tailed)	Mean Difference
		−4.548	4	0.010	−4.770
10	Level 1 Errors is Poke Interaction	Test Value = 7.85
		t	df	Sig. (2-tailed)	Mean Difference
		−16.169	4	0.000	−6.050
11	Level 1 Errors in Grab and Use Sword	Test Value = 5.08
		t	df	Sig. (2-tailed)	Mean Difference
		−10.941	4	0.000	−2.680
12	Level 2 Errors in Distance Grab Interaction	Test Value = 1.08
		t	df	Sig. (2-tailed)	Mean Difference
		4.704	4	0.009	3.120

The participants reported a satisfaction level ranging from 70 to 100 with an average of 90.75, indicating a high level of satisfaction with the VR application. Out of 31 measures, only 6 (comprising 1 measure of effectiveness and 5 measures of efficiency) were statistically significant when controllers were employed as the interaction modality. However, when hands were used as the interaction modality, the number of statistically significant measures increased to 12 (with 3 measures of effectiveness and 9 measures of efficiency) out of 31. The findings suggest that nonliterate adults may have struggled when using their hands for interaction and that controllers may be a preferable option until hand-tracking technology improves. Detailed discussions are presented in the later subsection.

6.3.1. Analysis of Results with Controllers as an Interaction Modality

Controllers were found to be easier to use for nonliterate adults compared to using hands as an interaction modality. The controllers were equipped with triggers and buttons that the users used to perform the interactions. Our prototype implemented three basic interactions: grab, pinch, and poke. Before the start of the experiment, the users were informed about the controllers and how they worked. A 3D model of the controllers was also provided in VR for reference.

The first significant measure was “Level 1 2nd Task,” in this task, the user had to move a blue pebble on the map while grabbing it to a specific region. This pebble can only be positioned along the x and y-axis, and the user had to continuously press the grip button to drag it along the map. On average, nonliterate users completed this task 21.110 sec later than the test value. The second measure was “Level 1 4th Task,” in this task, the user had to open a drawer and place jewelry in it. This was a complex task that required multiple interactions, but it was observed that the nonliterates struggled to find the drawer handle that was intentionally colored the same as the cupboard. That was the main reason that this variable was significant in both interaction modalities. Moreover, the mean difference for both interaction modalities was also similar, i.e., 14.74 and 13.25. The third and sixth measures were correlated and needed to be evaluated together. Both interaction modalities resulted in significant findings for these measures, where the goal was to grab a sword and use it to cut hay sticks. The results showed significance in both interaction modalities; however, the values and causes of this significance varied. The mean difference from the test value was 4.0 s and 12.09 s for the interaction modalities of controllers and hands, respectively. The results showed that when using controllers, nonliterate users were quicker to grab the sword with the grip button but had difficulties holding onto it when trying to cut the hay sticks, resulting in more mistakes and longer completion times. On the other hand, the slower completion time while using hands was due to a higher temporal offset. The next measure was “Level 2 1st Task,” in which the user is required to grab an alphabet card using a distance grab interaction. The reasons for this significance were aligned with the ones observed in study 1 and discussed in Section 5.3.2. The next significant measure that occurred in both analyses was related to the user’s curiosity. The results indicated that nonliterates prioritized completing the task at hand and did not spend much time interacting with unmentioned objects, as evident from the lower mean difference.

6.3.2. Analysis of Results with Hands as Interaction Modality

It is evident from the results of the one-sample t-tests shown in Appendix F, Table A6 and Appendix G, Table A7 and Table 15 that using real hands as an interaction modality proved to be challenging for nonliterate users. Although the same interactions were implemented for both interaction modalities, to interact with objects using hands, the user had to form the required pose for the interaction. It was observed that users created varied poses for specified interactions. For example, to grab an object, the user had to make a fist, but most of the nonliterates tried to grab the object according to their perception of the real world. This proved to be the major cause of increased errors and completion time delays. However, on the other hand, the reported satisfaction level was higher when hands were used as an interaction modality, yet the observed effectiveness and efficiency were better with controllers.

We have already discussed measures 1, 2, 6, and 11 in the section above. The next three significant measures in the study were related to three different modes for practicing alphabet writing and understanding at level two. The first mode involved writing letters in the air, where the user was required to make a specific pinch pose to write. Many participants struggled to do so because of the variations in poses created. The second mode involved writing with a marker on a board. It was noted that nonliterate users may not know how to hold a marker properly, leading to difficulties with creating the necessary pose to grab it. The third mode was a typewriter-style interface for identifying alphabets. Yet again, it was observed that the users struggled to comprehend the correct pose, or it may be due to the numerous buttons in close proximity. The next significant measure was related to human behavior in terms of compliance; it was observed that nonliterate users are more compliant with instructions than the non-tech-literate ones. The next significant measure is related to the poses and gestures required for interaction. In the above discussion, we have mentioned several times that the creation of varied poses was the primary reason for the difficulties in interaction when using hands as the interaction modality. The remaining significant measures dealt with errors in complex interactions. It was noted that tasks that required two-handed interaction had the highest error rates, such as grabbing an alphabet card with one hand and pressing the button on it with the other hand. Nonliterate users were also found to struggle in unexpected situations. For instance, in level one, the user may have to adjust the speed of their sword swing to cut the hay stick due to a temporal offset, but nonliterate users were unable to do so.

6.3.3. Analysis of Results for Years of Technological Experience on Interaction

A Kruskal–Wallis H-test was also performed to evaluate the impact of technological experience on interaction. Initially, it was assumed that this factor could affect the effectiveness and efficiency of interactions, but the results shown in Appendix H, Table A8 showed that there was no significant impact of the number of years of experience with technology on the usability of VR applications.

6.4. Discussions of Results

The results of the experiment suggest that controllers are more usable by nonliterates compared to hands as an interaction modality, which goes against our initial assumption. The results indicated that the nonliterate users had difficulty using their hands as the interaction modality. Nonliterates found controllers easier to use as they were equipped with triggers and buttons for performing interactions. In contrast, using hands as an interaction modality proved to be difficult for nonliterates, as they had trouble creating the correct pose for the specified interactions. Despite the higher reported satisfaction level with hands, the observed effectiveness and efficiency were better with controllers.

7. Discussions

In this study, the usability of the VR prototype was analyzed by evaluating its effectiveness and efficiency parameters. The users’ satisfaction with the experience was gauged, and it was found that all participants were generally pleased. Task completion was considered an indicator of effectiveness, and all users were able to successfully complete each task in the VR prototype. The effectiveness aspect of productivity and the usability parameter of efficiency were further analyzed using statistical techniques such as ANOVA (as presented in Appendix A, Appendix B, Appendix C and Appendix D) and one-sample T-tests (in Appendix F and Appendix G).

7.1. Summary of Results

The following summary of results is related to Study 1. The results of the study showed that for non-tech-literate and nonliterate users, moving in 3D proved to be challenging, as seen in the slow completion time of task 2. Nonliterate users faced difficulties when trying to open the drawer in task 4, attempting to do so from the sides instead of the handle. Task 6, which was the most complex, took longer for both non-tech-literate and nonliterate users to complete, which may have been due to the low lighting in the room where the experiment was conducted. The results also showed that nonliterate users were slower at completing task 2 of level 2, but their curiosity to learn was demonstrated by watching the entire video. Nonliterate users struggled with some tasks requiring hands as an interaction method, such as opening a drawer, which required finding a handle that blended in with the cupboard. The results indicated that nonliterate users sometimes sought external help and were confused by the reticle while trying to grab an object. They only interacted with the required objects and followed the in-app instructions, while non-tech-literate users showed a higher level of curiosity and tried to interact with all objects. The study found that controllers were a better option for tasks requiring fast actions, such as shooting a target with a pistol, as they led to faster completion times and fewer errors. On the other hand, hand tracking in VR resulted in a delay between real and simulated actions, affecting fast interactions, whereas controllers were self-tracked and not impacted. The multivariate analysis showed that the use of modern technology had a significant impact on some VR interactions for nonliterate users, making task 2 of level 1 more challenging for those who did not use modern technology. However, it also influenced some complex two-handed interactions.

The following summarizes the findings of Study 2. The results showed that using controllers as an interaction mode was easier for nonliterate adults than using their hands. The controllers were equipped with triggers and buttons, making it easier for users to perform interactions. Nonliterate users were able to complete certain tasks faster and with fewer errors when using controllers, such as shooting a target. However, nonliterate users had difficulties with some tasks that required multiple interactions using controllers, such as cutting hay sticks. The study suggests that controllers were a more effective and efficient interaction mode for nonliterate users compared to using hands. The variations in hand poses caused difficulties in interaction when using hands as the interaction mode. Nonliterate users were found to be more compliant with instructions but struggled in unexpected situations. Tasks requiring two-handed interaction had the highest error rates, and nonliterate users struggled with understanding and performing the correct poses. The Kruskal–Wallis H-test performed in the study did not find a significant impact of the number of years of experience with technology on the usability of VR applications, suggesting that more experience with technology does not necessarily lead to better VR interaction performance.

7.2. Limitations

The study had some limitations that should be considered. One of the limitations was the small sample size, which might have limited the generalizability of the findings. Further research with a larger sample size is needed to make more robust comparisons. Another limitation was the variation in lighting conditions between different experimental locations, which caused difficulties with hand tracking due to the temporal offset. To mitigate this issue, an infrared illuminator was used, and a decrease in errors was seen, but more data is needed to confirm this effect.

7.3. Suggestions and Recommendations

Based on the results of Study 1 and Study 2, the following suggestions and recommendations are proposed to direct future research and development of VR experiences for nonliterate users:

• Consider using controllers as the interaction mode instead of hands: Results from both studies indicate that nonliterate users find controllers easier to use and perform interactions faster and with fewer errors compared to using their hands.
• Enhance the design of the VR educational application: Based on the results of Study 1, it is recommended that the design of the application be modified to cater to the specific needs of nonliterate users, taking into consideration their behavior patterns and the difficulties they face.
• Improve lighting conditions: Results from Study 1 indicate that low lighting levels can negatively impact the completion time of complex tasks. Hence, it is recommended to ensure adequate brightness levels in the room where the VR experience is conducted.
• Provide clear instructions and a reticle: Results from Study 1 show that nonliterate users sometimes sought external help and were confused by the reticle while trying to grab an object. Hence, it is recommended to provide clear instructions and a visible reticle to help users with their interactions.
• Consider alternative interactions: Results from Study 1 suggest that gestures could be a suitable alternative interaction mode for nonliterate users and warrant further investigation.
• Consider user training and familiarization: Results from Study 1 indicate that user training and familiarization could impact the performance of nonliterate users in VR systems. Hence, it is recommended to explore the effect of training on VR interaction performance for nonliterate users.
• Consider the impact of technology experience: Results from Study 2 suggest that more experience with technology does not necessarily lead to better VR interaction performance. Hence, it is recommended to examine the impact of technology experience on specific VR interaction tasks to better understand how experience affects performance.
• Consider the impact of technology use: Results from Study 1 show that the use of modern technology can have a significant impact on VR interactions for nonliterate users. Hence, it is recommended to explore the impact of using different types of technology (e.g., smartphones, laptops, and computers) on VR interactivity.
• Expand the study to include a wider range of tasks and interactions: Results from Study 1 indicate that nonliterate users struggled with some VR tasks and that the study could be expanded to include a wider range of tasks and interactions to gain a more comprehensive understanding of the abilities of nonliterate users in VR systems.

8. Conclusions

Our study found that the educational VR application is effective and efficient for nonliterate individuals due to its simple interface and use of visual and auditory aids for learning. The results showed high levels of satisfaction among all types of users, with 78–100% reporting being pleased with the application. The data showed no significant differences in usage of the VR app among tech-literate, non-tech-literate, and nonliterate individuals. The results also showed that controllers were more usable for nonliterate individuals than hands as an interaction modality and that experience with technology and familiarity with modern technology had little impact on the usability of VR applications.

The first research question stated that the educational VR application will be usable for nonliterate individuals. The results of the study showed high levels of satisfaction among all types of users, with 78–100% reporting being pleased with the application. The data from the two one-way ANOVA tests showed no significant differences in usage of the VR app among tech-literate, non-tech-literate, and nonliterate individuals. However, a few variables were found to have a significant impact.

The second research question speculated that the educational VR app would be easier for nonliterate individuals to use, and this was partially supported by the results. All user types reported a high level of satisfaction with the app. However, the one-way ANOVA tests indicated that there were significant differences in task completion times among the user types, with non-tech-literate and nonliterate individuals taking longer to finish certain tasks than tech-literate users. The results also revealed that nonliterate individuals had difficulty with certain interactions, such as grabbing and moving objects and interacting with usable objects. This suggests that the educational VR app may need to be improved to better cater to the needs of nonliterate users in certain areas.

The third research question focused on the interaction performance between controllers and hands for nonliterate individuals. The results showed that controllers were more usable for nonliterate individuals than hands as an interaction modality, which negates our hypothesis. 19.4% of measures were found to be statistically significant when using controllers, compared to only 38.7% with hands. Nonliterate individuals found controllers easier to use because of the triggers and buttons for performing interactions. On the other hand, using hands as an interaction modality was challenging for nonliterate individuals as they struggled to create the correct pose for interactions. Despite a higher reported satisfaction level with hands, the observed effectiveness and efficiency were better with controllers.

The results also indicated that experience with technology and familiarity with modern technology had little impact on the usability of VR applications.