Tactile Feedback in Hierarchical Menu Interaction Within Peripersonal Space: A Comparison Between Virtual and Real Environments

Abstract

Virtual reality (VR) interfaces increasingly rely on interaction within peripersonal space. However, the conditions under which interaction performance in virtual environments can approximate those of comparable real-world tasks remain underexplored, particularly for hierarchical menus requiring precise sequential input. This study investigated how the presence or absence of tactile feedback influences movement time and selection accuracy during hierarchical menu interaction in peripersonal space across different task difficulty levels. Twelve participants performed a three-level hierarchical selection task on a 4 × 3 menu in two controlled experiments with a stereoscopic 3D TV. Two interaction conditions were tested: a surface-based condition, with the menu attached to the physical screen, and a mid-air condition, with the menu positioned 35 cm and 45 cm in front of participants. Selections were confirmed using a handheld remote. Results showed no statistically significant difference in movement time and selection accuracy between the virtual and real environments when screen-surface targets provided tactile feedback, but performance declined for mid-air targets without tactile references, particularly under higher task difficulty levels. These findings suggest that tactile feedback, coordinated visual target placement, and users’ familiarity with touchscreen-like interaction jointly act as key factors for designing effective, immersive, and user-friendly VR menu systems in peripersonal space.

1. Introduction

Recent advances in virtual reality (VR) systems have substantially improved visual fidelity, tracking accuracy, and interaction responsiveness, enabling users to perform increasingly complex interactive tasks in immersive environments [1,2,3,4]. These developments have supported the broader adoption of VR beyond visualization and exploration, extending its use to domains that require reliable and precise user input, such as healthcare, education, professional training, and simulation [5,6,7,8,9]. As VR systems are increasingly adopted not only for interaction but also for performance evaluation, an important question in human–computer interaction (HCI) is to what extent interaction performance in virtual environments can approach that achieved in comparable real-world settings.

A large proportion of VR interaction occurs within the user’s peripersonal space, the immediate area surrounding the body in which visual, tactile, and proprioceptive inputs are integrated to support reaching and manipulation. Within this spatial volume, motor execution is calibrated through multisensory feedback loops, with tactile cues playing an important role in confirming target acquisition and refining spatial boundaries. Research in cognitive neuroscience and VR has shown that peripersonal space supports multisensory facilitation for stimuli near the body and can be systematically probed and reshaped using virtual environments [10,11]. Prior work has highlighted that interaction performance in VR within this reachable space is sensitive to the availability of direct physical contact and tactile feedback, with several studies reporting slower or less accurate performance when these cues are absent [12,13]. It remains unclear, however, whether performance can be maintained when these near-field tasks involve structured interface operations rather than isolated reaching movements. This makes peripersonal-space interaction a critical context for evaluating the validity and limits of VR interaction performance.

In practice, interaction performance in peripersonal space is often evaluated using functional, repetitive interface tasks, with menu interaction among the most common examples. Menus are fundamental interface components that support navigation and command selection in interactive systems [14,15]. In many VR applications, menus are organized as hierarchical structures, requiring users to perform a sequence of selections across multiple levels to reach a desired function [16]. Prior studies have shown that hierarchical menus with two to three levels are widely used and relatively efficient in balancing the number of items, cognitive load, and interaction efficiency [15,17,18]. Hierarchical menus nevertheless remain comparatively underexplored as a distinct problem in VR interaction research, despite their practical relevance. Their study is methodologically more demanding than that of simplified target-selection tasks because hierarchical menu interaction involves dependent sequential selections as well as additional structural and interaction demands that are not fully captured by simplified tapping or pointing paradigms [18,19,20,21,22]. Hierarchical menus also introduce design challenges related to submenu placement, overlap, focus-context management, and transitions across levels. These features complicate implementation and hinder direct comparison across studies. Prior research has therefore more often focused on simpler pointing tasks or on the optimization of specific menu layouts and interaction techniques, whereas controlled comparisons of hierarchical menu interaction remain limited.

Hierarchical menu interaction differs from single-target selection tasks because it involves a sequence of dependent selections in which each step must be completed before proceeding to the next level [18,23]. The motor efficiency of these tasks is often discussed in terms of Fitts’ law, which defines the relationship between movement time (MT) and the Index of Difficulty (ID). However, the standard Fitts’ model assumes the availability of a physical surface to support movement termination. Mid-air VR environments lack a tactile reference at the target location, which may increase endpoint uncertainty and require greater reliance on slower visual closed-loop corrections for target verification. This condition may disrupt the usual speed-accuracy relation, thereby increasing MT variability and reducing selection accuracy relative to surface-based interaction. To address these limitations, prior work has explored the use of visual guides, input-device-based confirmation, and various forms of haptic or passive tactile feedback [22,24].

Among these feedback modalities, tactile feedback plays a particularly important role in supporting reliable near-field interaction. Physical contact with an interaction surface provides users with clear confirmation that a selection has been successfully completed, which helps stabilize target selection and reduce uncertainty during interaction [25,26]. Direct touch has been shown to improve both efficiency and accuracy in interaction tasks within reach [17,27]. However, existing haptics research in VR has mostly examined whether one VR technique outperforms another, rather than whether tactile alignment helps virtual interaction approach the performance of comparable real-world interaction. This distinction is important because a technique that performs relatively well within VR is not necessarily one that reduces the gap between virtual and real task execution. The gap is especially relevant for hierarchical menu interaction, where tactile support may influence not only single selections but also the stability of an entire multi-step sequence.

The objective of this study is to investigate whether hierarchical menu interaction in VR can approximate performance in a comparable real-world task within peripersonal space, with particular attention to the role of tactile alignment at the target location. We evaluated interactions with a 4 × 3 hierarchical menu located within peripersonal space (<100 cm) in two controlled experiments. In Experiment 1, interaction was performed under a screen-based condition in which visual targets were aligned with the point of contact on a physical screen. In Experiment 2, the menu was presented in mid-air to examine performance when the virtual targets were displaced from the screen, and no tactile reference was available at the target location. Interaction performance was assessed using movement time and accuracy. This study uses a comparable real-world baseline to examine the conditions under which hierarchical menu performance in VR is maintained or degraded, rather than limiting the comparison to alternative VR techniques in isolation. The contribution of the study, therefore, lies in providing a more explicit comparison between controlled VR interaction conditions and their real-world counterpart for sequential menu selection within peripersonal space.

Based on these objectives, we propose two hypotheses: (H1) hierarchical menu interaction shows no statistically significant difference in movement time and selection accuracy in virtual environments when tactile feedback is spatially aligned with the visual target; and (H2) hierarchical menu interaction performance in virtual environments degrades when spatial alignment between the visual target and tactile feedback is disrupted, regardless of depth variations within the peripersonal space.

2. Materials and Methods

2.1. Study Design Overview

The study employed a within-subject design comprising two experiments that used the same apparatus and hierarchical menu selection task. In Experiment 1, a 2 × 3 within-subject factorial design was used with environment (real vs. virtual) and task difficulty (easy, medium, hard). The menu was displayed on a physical screen at a fixed viewing distance of 50 cm from the participant’s eyes, and participants selected targets by directly touching the corresponding menu button and then confirming the selection with a wireless remote. Thus, in Experiment 1, both environments provided physical contact with an interaction surface at the same depth within the participant’s peripersonal space.

In Experiment 2, a 2 × 2 × 3 within-subjects factorial design was used, with environment (real vs. virtual), depth, and task difficulty as factors. In the real-world condition, the menu was displayed on the same physical screen, which was repositioned so that the screen surface was located at viewing distances of 35 cm and 45 cm from the participant’s eyes. Participants again selected targets by touching the corresponding menu button on the screen and confirming the selection with the remote. In the virtual condition, the menu was visually presented at the same depths within the participant’s peripersonal space, appearing to float in front of the screen. Participants selected targets by moving their hand to the location of the virtual button in mid-air and confirming the selection with the remote, without making physical contact with the screen or any other surface.

In both experiments, participants performed the same hierarchical menu selection task, and performance was assessed using movement time and selection accuracy. The two experiments were analyzed separately due to their differing factorial structures and the presence versus absence of physical contact during selection. Experiment 1 established a baseline screen-based condition in which the visual target and the point of physical contact were spatially aligned, allowing performance in the virtual and real environments to be compared when tactile support was available at the target location. Experiment 2 examined performance under a mid-air condition in which this alignment was disrupted in the virtual environment, with depth introduced as an additional factor. Separating the experiments allowed these two interaction conditions to be examined more clearly without collapsing aligned and disrupted tactile-reference conditions into a single inferential structure. Accordingly, the two experiments were analyzed separately because they differed in both factorial structure and the presence or absence of physical contact during selection.

2.2. Participants

Twelve graduate students from National Taiwan University of Science and Technology voluntarily participated in the study (M = 25.92 years, SD = 2.50). All participants completed both experiments (Experiment 1 and Experiment 2) in a within-subject design. The sample size was planned a priori using G*Power 3.1.9.7 for a repeated-measures within-subject design. In the absence of pilot data directly matching the present hierarchical menu task, a conventional medium effect-size assumption (f = 0.25) was used as an initial planning reference. To further contextualize the achieved sample size, a supplementary sensitivity analysis was conducted with α = 0.05, power = 0.80, one within-subject group, an assumed correlation of 0.30 among repeated measures, and a non-sphericity correction of 1.0. Under the 12-condition repeated-measures structure of Experiment 2, the achieved sample size of 12 participants was sufficient to detect effects of approximately f = 0.30 or larger. Under the 6-condition repeated-measures structure of Experiment 1, the corresponding detectable effect size was approximately f = 0.37. Accordingly, the achieved sample size appears adequate to detect effects of at least moderate magnitude under the present assumptions, but does not guarantee sensitivity to all possible interaction effects.

This sample size is also broadly consistent with prior within-subject studies of interaction performance in virtual environments [28,29,30]. The participant group was relatively homogeneous, consisting of graduate students from a single institution with a narrow age range, which should be taken into account when assessing the external validity of the findings. All participants were right-hand dominant. In addition, prior VR experience was generally limited, and several participants had little or no experience with VR systems. All participants reported normal or corrected-to-normal vision and no history of musculoskeletal disorders affecting the hands, arms, or shoulders.

2.3. Hierarchical Menu Structure

Participants performed hierarchical menu selection tasks using a matrix-based menu with a consistent structure across all experimental conditions. The menu consisted of three hierarchical levels, such that each selection task required participants to complete a sequence of selections from the first to the final level. Three-level hierarchical menus were selected based on prior findings indicating their effectiveness in balancing the number of items, cognitive load, and interaction efficiency [15,17,18,31].

The menu was arranged in a 4 × 3 matrix layout. On each level, one item in the active column was designated as the target and highlighted in green, while all other items were displayed in gray and served as non-targets. Upon successful selection of the green target button in the current column, the next hierarchical level (i.e., the next column) was immediately presented with an identical layout until all three levels were completed. Task difficulty (easy, medium, hard) was controlled by varying the movement distance and target width in the selection task, as detailed in Section 2.5.

The layout and visual appearance of the menu were kept constant across environments, depths, and task difficulty levels, so that any observed performance differences could be attributed primarily to differences in pointing and selection behavior rather than to variations in interface design. To minimize the influence of semantic processing and higher-level cognitive factors, the menu items were rendered as simple rectangular buttons without icons or textual labels.

2.4. Apparatus and Stimuli

The hierarchical menu described in Section 2.3 was implemented using Unity 3D (2017.4.1f1) and presented under controlled stereoscopic viewing conditions. The menu was displayed within a blue rectangular frame corresponding to a 22-inch area (52 × 31 cm) on a 55-inch Sony Bravia 3D TV (Model: KDL-55W800B, Sony EMCS, Bandar Baru Bangi, Malaysia), as illustrated in Figure 1. Stereoscopic presentation was achieved using an active-shutter system to ensure consistent depth cues across experimental conditions.

Figure 1. Experimental apparatus and display setup. Participants were seated in front of a 55-inch stereoscopic 3D TV and performed hierarchical menu selections using the dominant index finger and a wireless remote to confirm selections. Hand movements were tracked using an optical marker attached to the fingertip.

Participants were seated in an adjustable chair positioned in front of the 3D TV throughout the experiment. The study was conducted in a 3 m × 3 m × 2.5 m room enclosed with black curtains to minimize external visual distractions and maintain stable stereoscopic viewing conditions. During virtual conditions, participants wore Sony TDG-BT500A (Sony Group Corporation, Tokyo, Japan) active-shutter 3D glasses synchronized with the television via an NVIDIA 3D IR emitter (NVIDIA Corporation, Shenzhen, China), enabling clear visualization of menu items rendered in gray and green. The glasses were selected for their lightweight design and minimal setup requirements to reduce discomfort and visual fatigue during prolonged interaction [32]. The stereoscopic display settings were adjusted individually to match each participant’s interpupillary distance (IPD) within the system’s supported range (≤6.5 cm) in order to minimize vergence-accommodation conflict and visual fatigue during stereoscopic viewing [33].

Hand movements were recorded by attaching a small optical marker to the tip of the participant’s dominant index finger. The marker was tracked at 50 Hz using an OptiTrack (Natural- Point, Inc., Corvallis, OR, USA) motion capture system and did not obstruct direct finger contact with the screen in touch-based conditions. Finger position data were streamed to Unity and logged to obtain movement trajectories and timestamps associated with target selections. Target selections were confirmed using a Logitech Spotlight Presenter (Logitech, Shanghai, China) remote held in the dominant hand, with the dominant index finger kept free for pointing at the target contact.

The handheld remote was used to standardize the confirmation step across all experimental conditions. This two-step interaction method, consisting of finger pointing followed by button-based confirmation, was adopted to reduce variability associated with system-dependent gesture-recognition methods, such as air-tap detection or dwell-based activation, and to ensure that selection commitment was operationalized consistently in both real and virtual environments. This design choice was intended to minimize differences in confirmation mechanisms across conditions so that observed performance differences could be attributed more directly to tactile alignment and target presentation, rather than to variation in the selection-confirmation method. This procedure also strengthened internal comparability across conditions, although it may limit direct generalization to VR interaction paradigms in which confirmation is integrated with selection. The additional button press may also introduce some coordination demands beyond target acquisition itself; however, this requirement was kept constant across all experimental conditions. Button-press events were recorded on a desktop computer (Intel^® Core™ i7-6700 CPU, 3.40 GHz) and time-synchronized with the motion-capture data to ensure consistent timing measurements across all conditions.

2.5. Experimental Design

2.5.1. Independent Variables

The study examined three independent variables: interaction environment, menu depth, and task difficulty.

The interaction environment had two levels: real and virtual. In both experiments, the real environment involved direct touch interaction with menus presented on the display surface (Figure 2a and Figure 3a). In the virtual environment, menu presentation depended on the experimental condition. In Experiment 1, virtual menus were presented on the same plane as the display surface (Figure 2b).

Figure 2. Experimental conditions in Experiment 1 at a viewing distance of 50 cm: (a) real screen-based interaction, in which participants directly touched the display surface, and (b) virtual screen-aligned interaction, in which the stereoscopic menu was presented at the screen plane.

Figure 3. The experimental conditions in Experiment 2 at viewing distances of 35 cm and 45 cm: (a) real screen-based interaction, in which participants directly touched the display surface, and (b) virtual mid-air interaction, in which the menu was presented stereoscopically in front of the screen without tactile contact at the target location.

In Experiment 2, virtual menus were presented stereoscopically in front of the display surface, resulting in mid-air interaction without tactile feedback at the visual target location (Figure 3b).

Menu depth was defined as the perceived distance between the participant and the menu. A preliminary study was conducted to determine appropriate depth parameters, for which the minimum observed arm length was 53 cm (M = 74 cm, SD = 9.6 cm). Based on this study, a depth of 50 cm was used in Experiment 1, while depths of 45 cm and 35 cm were used in Experiment 2. Prior work and ergonomic guidelines suggest that distances of approximately 45 cm are appropriate for direct hand interaction, and that approximately 30 cm is a recommended minimum adult viewing distance [34,35]. Depth was not treated as a primary hypothesis factor in the present study. Instead, it was included in Experiment 2 as a secondary factor to test whether the effect of disrupted tactile alignment in the virtual mid-air condition remained consistent across two target positions within peripersonal space. For this reason, depth was retained in the factorial design of Experiment 2 and analyzed together with interaction environment and task difficulty, even though the main hypotheses were not centered on depth itself.

Task difficulty was modeled using the Index of Difficulty (ID) derived from Fitts’ law and treated as a categorical factor with three levels: easy, medium, and hard. Difficulty levels were defined a priori using specific combinations of target width (W) and movement distance (D), as summarized in Table 1 and Figure 4. ID values were computed using Equation (1) [36]:

$$\mathrm{ID} = {\log }_2\left({\displaystyle \frac{\mathrm{D}}{\mathrm{W}}}+1\right)$$

(1)

where D represents the average movement distance from the starting position to the target (cm), and W represents the target width measured along the direction of movement (cm). For reporting purposes, task difficulty levels were associated with nominal ID ranges (easy: ID ≤ 3.0; medium: 3.01 ≤ ID ≤ 5.0; hard: ID ≥ 5.1). ID served as a descriptive metric to classify task difficulty level and was not entered as a separate factor in the statistical analyses. In the present study, ID was used to define task-demand conditions a priori and to structure the repeated-measures design into three interpretable levels of difficulty rather than to estimate a continuous Fitts’ law function. This categorization was adopted to support comparison of hierarchical menu performance across interaction environment, depth, and controlled levels of task demand within a factorial design. The use of broader ID-based task difficulty categories is also consistent with prior published studies that have classified task precision into low, medium, and high levels following ISO 9241-9 [37]. Accordingly, ID was treated as a design-based manipulation of task difficulty, and the statistical analyses focused on comparisons among the predefined easy, medium, and hard conditions.

Table 1. Target width (W), movement distance (D), index of difficulty (ID), and predefined task difficulty levels used in Experiments 1 and 2.

W (cm)	D (cm)	ID (bits)	Task Difficulty Level
10	18.02	1.49	Easy
	19.18	1.55	Easy
	22.32	1.69	Easy
	26.73	1.88	Easy
1	18.02	4.17	Medium
	19.18	4.26	Medium
	22.32	4.48	Medium
	26.73	4.74	Medium
0.5	22.15	5.50	Hard
	23.90	5.61	Hard
	28.53	5.86	Hard
	34.89	6.15	Hard

Figure 4. Menu layouts used to manipulate task difficulty. Three target widths were tested: W = 10 cm in (a), W = 1 cm in (b), and W = 0.5 cm in (c). Within each layout, four movement distances (D1–D4) were used to define the task difficulty conditions. The target item is shown in green and the remaining non-target items in gray.

2.5.2. Dependent Variables

The dependent variables in this study were movement time and accuracy. Movement time (MT) was defined as the duration required for a participant to move the fingertip from the starting position to the target menu item and confirm the selection using a wireless remote. Higher MT values indicate lower interaction efficiency. MT data were automatically recorded by the Unity system in a timestamp format.

Selection accuracy was measured as a continuous spatial accuracy index representing the proximity between the detected fingertip position and the actual target position at the moment the selection was confirmed. This normalized formulation has been used in related stereoscopic and virtual-environment studies to quantify how closely an estimated or indicated position corresponds to its reference value [38,39]. Unlike a conventional binary error-rate measure, the present metric retains the magnitude of endpoint deviation and was therefore considered appropriate for the present hierarchical menu task, in which selections could still be completed successfully while varying in spatial precision. This measure should therefore be interpreted as complementary to conventional binary error-rate measures rather than as a direct replacement. Accuracy was calculated using Equation (2) [38]:

$$\mathrm{Accuracy} = 1-{\displaystyle \frac{‖{\mathrm{D}}_{\mathrm{p}}-{\mathrm{D}}_{\mathrm{a}}‖}{‖{\mathrm{D}}_{\mathrm{a}}‖}}$$

(2)

where $‖{\mathrm{D}}_{\mathrm{p}}-{\mathrm{D}}_{\mathrm{a}}‖$ represents the Euclidean distance between the detected fingertip position and the actual target position, and $‖{\mathrm{D}}_{\mathrm{a}}‖$ denotes the Euclidean norm of the actual target menu position. Because the same predefined menu layout and target positions were maintained across experimental conditions, any variation in $‖{\mathrm{D}}_{\mathrm{a}}‖$ was systematic and consistent across the main comparisons. The normalization by $‖{\mathrm{D}}_{\mathrm{a}}‖$ was used to express endpoint deviation relative to the target position, thereby providing a dimensionless measure that supports comparison across the predefined target locations used in the task. Condition-level accuracy was computed by averaging across all predefined target positions within the same fixed menu layout, thereby reducing the influence of any single target location on the final accuracy estimate for that condition. Accuracy values ranged from 0 (completely inaccurate) to 1 (perfect accuracy), with higher values indicating closer agreement between the fingertip position and the target location. Mean accuracy for each experimental condition was obtained by averaging across all trials within that condition for each experiment separately.

2.5.3. Task and Procedure

The study followed a within-subject design consisting of one training session and two main experiments. Each participant completed 19 conditions, including 1 (one) training condition, 6 (six) conditions in Experiment 1, and 12 (twelve) conditions in Experiment 2. Participants were informed of the study’s purpose, procedures, and potential risks, and provided written informed consent before the experimental sessions began. Basic demographic information, including age and visual status, was collected at the beginning of the experiment. Participants were seated in front of a 3D TV, used the dominant index finger to point at and select menu targets, and held a wireless remote in the dominant hand to confirm selections throughout all sessions (Figure 5). The use of active-shutter 3D glasses depended on the interaction environment being tested.

Figure 5. Illustration of the menu-selection procedure, in which the participant touched the target location with the dominant index finger and confirmed the selection using the handheld remote.

The training session was designed to familiarize participants with the hierarchical menu structure and interaction method, and to reduce initial unfamiliarity before the formal experiment. During training, participants were seated in a chair, wore 3D glasses, and interacted with a mid-air virtual menu positioned 35 cm in front of them, ensuring clear target visibility. The training session consisted of three practice trials for each task difficulty level, for a total of nine per participant. Training performance was not formally analyzed, and readiness was determined by completion of the predefined practice session rather than by a performance-stabilization criterion. This standardized familiarization procedure was applied equally to all participants. The 35 cm mid-air virtual condition was selected for training because it was considered the least familiar and most demanding interaction condition, requiring target localization without tactile support at the point of selection. Training under this condition was intended to help participants adapt to the core task structure and the selection-confirmation sequence before the formal experiments began.

The study employed a two-step interaction method in which participants first positioned the fingertip at the intended target location and then confirmed the selection using the wireless remote. The remote was held in the dominant hand while the dominant index finger remained free for pointing and target contact. This procedure was applied consistently across all experimental conditions in order to maintain the same selection-confirmation sequence throughout the study.

Following the training session, all participants completed the formal experiment in a fixed sequence: Experiment 1, followed by Experiment 2. This order was selected to maintain a consistent progression from the baseline screen-based condition to the more demanding mid-air condition. In Experiment 1, interaction was performed at a depth of 50 cm, and the menu was displayed on a 3D TV. In the real environment, participants interacted directly with the menu without 3D glasses, touching the buttons on the display surface and confirming selections with the remote. In the virtual environment, participants wore active-shutter 3D glasses; the menu was rendered stereoscopically at the same depth and plane as the physical screen, and participants again touched the screen at the button locations and confirmed selections with the remote.

In Experiment 2, interaction was performed at depths of 45 cm and 35 cm. In the real environment, menus were presented on the same physical screen, which was repositioned so that the screen surface was located at the corresponding viewing distances, and interaction involved direct touch. In the virtual environment, the menus were presented stereoscopically at the same nominal depths within peripersonal space, appearing to float in front of the screen. Participants selected targets by moving their fingertip to the visually indicated button location in mid-air and confirming the selection with the remote, without making physical contact with the screen or any other surface.

In both the training and experimental sessions, participants performed hierarchical menu-selection tasks by selecting a designated target item highlighted in green. At each menu level, one button was predefined as the target and visually highlighted in green, whereas all remaining buttons functioned as non-target items. At the first menu level, the target button was positioned in the first column of the menu (Figure 6a). Upon successful selection of the first-level target, achieved by touching the button with the tip of the index finger and confirming the selection via a wireless remote, the subsequent menu level was displayed immediately, as shown in Figure 6b. This procedure was iteratively repeated until the final menu level was reached, thereby constituting a complete hierarchical menu selection sequence (Figure 6c). All participants completed all conditions within each experiment, but the order of those conditions was varied across participants. In Experiment 1, the order of interaction environment and task difficulty conditions differed across participants. In Experiment 2, the order of interaction environment, depth, and task difficulty conditions also differed across participants. Participants were instructed to complete each menu selection sequence as quickly and as accurately as possible.

Figure 6. Sequence of the three-level hierarchical menu-selection task. The target item is shown in green and the remaining non-target items in gray. Participants selected the highlighted target at the first level (a), after which the second level appeared (b). The same selection procedure continued until the third and final level was completed (c).

Each three-level hierarchical menu sequence yielded three target selections and two movement time measurements (between successive selections). Accordingly, each participant contributed 18 target selections and 12 MT records in Experiment 1, and 36 target selections and 24 MT records in Experiment 2. Across both experiments, each participant contributed 54 target selections and 36 MT records. No observations were excluded as outliers, and all recorded data were retained for analysis. The formal experimental procedure required approximately 45 min per participant on average. A scheduled 15 min rest period was provided during the formal experiment to reduce fatigue. Including the training session and the scheduled rest period, the total procedure required approximately 60 min per participant. Although the training session, the varied order of conditions within each experiment, and the scheduled rest period were intended to mitigate learning-related, fatigue-related, and within-experiment order effects, the order of the two experiments was not counterbalanced; therefore, possible carryover effects related to practice or fatigue across experiments cannot be fully excluded.

3. Results

The two experiments were analyzed separately according to their respective analytical objectives. Experiment 1 employed a 2 × 3 within-subjects design with interaction environment (real vs. virtual) and task difficulty (easy, medium, hard) as factors, while depth was held constant at 50 cm. Experiment 2 employed a 2 × 2 × 3 within-subjects design with interaction environment (real vs. virtual), depth (35 cm vs. 45 cm), and task difficulty (easy, medium, hard) as factors.

Shapiro–Wilk tests were initially applied to the raw dependent-variable scores as a check of distributional normality before inferential analyses were conducted. The results indicated that the assumption of normality was not met in either experiment. Accordingly, statistical analyses were performed using the Aligned Rank Transformation (ART), which provides a nonparametric framework for factorial analysis while preserving the interpretability of main and interaction effects [40]. In Experiment 1, the ART model included interaction environment, task difficulty, and their interaction, whereas in Experiment 2, it included interaction environment, depth, task difficulty, all two-way interactions, and the three-way interaction. The ART-transformed responses were subsequently restructured by participant and analyzed using separate full-factorial repeated-measures analyses of variance, with participant specified as the subject in the repeated-measures structure. Only the effect corresponding to the alignment of each ART-transformed response was interpreted. The significance level was set at α = 0.05. The sphericity assumption was tested using Mauchly’s test, and the Greenhouse–Geisser correction was applied when this assumption was violated. Significant main effects were followed by Bonferroni-corrected post hoc tests, whereas significant interactions were examined using simple-effects analyses. Effect sizes were reported as partial eta-squared (${\eta }_p^2$) for omnibus ANOVA effects and Cohen’s d for the post hoc pairwise comparisons reported in the text. Accordingly, comparisons between Experiment 1 and Experiment 2 in the following figures and text are descriptive rather than inferential.

3.1. Movement Time (MT)

In Experiment 1, there was no statistically significant difference in movement time (MT) between the real (M = 1.382 s, SD = 0.341) and virtual environments (M = 1.566 s, SD = 0.513). However, MT was significantly lower in the real environment (M = 1.337 s, SD = 0.329) than in the virtual environment (M = 1.583 s, SD = 0.465) in Experiment 2, with a large effect size, ${\eta }_p^2=0.687$; Cohen’s d = 0.998. Figure 7a summarizes the MT values for the real and virtual environments in each experiment; any comparison between the two experiments in this figure should be interpreted descriptively, as the experiments were analyzed separately.

Figure 7. Movement time (MT) results for (a) interaction environment and (b) task difficulty level in Experiment 1 and Experiment 2. Panel (a) shows MT in the real and virtual environments for each experiment. Panel (b) shows MT across the easy, medium, and hard task difficulty levels for each experiment. Box plots represent participant distributions, and points represent individual observations. Brackets indicate significant Bonferroni-adjusted pairwise comparisons (* indicates p < 0.05, ** indicates p < 0.01, and *** indicates p < 0.001).

Task difficulty had a significant effect on MT in both experiments. In Experiment 1, MT increased from easy (M = 1.251 s, SD = 0.328) to medium task (M = 1.481 s, SD = 0.424) and hard tasks (M = 1.690 s, SD = 0.465), with a large main effect of task difficulty (${\eta }_p^2=0.634$). Bonferroni-corrected pairwise comparisons showed significant differences among all three task difficulty levels, with large effect sizes (easy vs. medium: Cohen’s d = 0.853; easy vs. hard: Cohen’s d = 1.517; medium vs. hard: Cohen’s d = 1.165). A similar trend was observed in Experiment 2, where MT increased from easy task (M = 1.298 s, SD = 0.405) to medium task (M = 1.414 s, SD = 0.395) and hard tasks (M = 1.667 s, SD = 0.379), with a large main effect of task difficulty (${\eta }_p^2=0.851$). All pairwise comparisons were significant, particularly those involving the hard task (easy vs. medium: Cohen’s d = 1.098; easy vs. hard: Cohen’s d = 2.791; medium vs. hard: Cohen’s d = 2.879), as shown in Figure 7b.

In Experiment 2, depth did not show a statistically significant main effect on MT (p = 0.229). Likewise, no statistically significant two-way or three-way interactions involving interaction environment, task difficulty, and depth were observed. Taken together, these findings indicate that MT was influenced primarily by interaction environment and task difficulty under the present conditions, whereas the tested depth variation within peripersonal space did not show a detectable effect on temporal performance. Full ANOVA results, including effect sizes, are presented in Table 2.

Table 2. Results of repeated-measures ANOVA on ART-transformed data for movement time and accuracy in Experiment 1 and Experiment 2.

Experiment and Factors	df	Movement Time (MT)			Accuracy
		F-Value	p-Value	${\mathit{\eta }}_{\mathit{p}}^2$	F-Value	p-Value	${\mathit{\eta }}_{\mathit{p}}^2$
A. Experiment 1
Environment	1	3.830	0.076	0.258	3.786	0.078	0.255
Task Difficulty	2	19.079	<0.001	0.634	65.859	<0.001	0.857
Environment × Task Difficulty	2	0.087	0.917	0.008	0.0025	0.976	0.002
B. Experiment 2
Environment	1	24.175	<0.001	0.687	33.635	<0.001	0.754
Task Difficulty	2	62.912	<0.001	0.851	23.139	<0.01	0.678
Depth	1	1.621	0.229	0.128	4.286	0.063	0.280
Environment × Task Difficulty	2	1.482	0.249	0.119	6.692	0.005	0.378
Environment × Depth	1	4.056	0.069	0.269	40.201	<0.001	0.785
Task Difficulty × Depth	2	0.419	0.663	0.037	2.734	0.087	0.199
Environment × Task Difficulty × Depth	2	0.286	0.754	0.025	3.026	0.069	0.216

3.2. Accuracy

In Experiment 1, no statistically significant difference in accuracy was observed between the real (M = 0.970, SD = 0.032) and virtual environments (M = 0.965, SD = 0.035) when tactile feedback was available, and the menu targets were presented on the screen (p = 0.078). In Experiment 2, however, accuracy was significantly higher in the real environment (M = 0.910, SD = 0.064) than in the virtual environment (M = 0.785, SD = 0.098), with a large effect size (${\eta }_p^2=0.754$; Cohen’s d = 1.671). This result indicates that the absence of tactile feedback at the visual target location in the virtual mid-air condition substantially reduced selection accuracy. Figure 8a summarizes the accuracy values for the real and virtual environments in each experiment. Although the separation between the two environments appears greater in Experiment 2 than in Experiment 1, this cross-experiment contrast should be interpreted descriptively rather than inferentially because the two experiments were analyzed separately.

Figure 8. Accuracy results for (a) interaction environment and (b) task difficulty level in Experiment 1 and Experiment 2. Panel (a) shows accuracy in the real and virtual environments for each experiment. Panel (b) shows accuracy across the easy, medium, and hard task difficulty levels for each experiment. Box plots represent participant distributions, and points represent individual observations. Brackets indicate significant pairwise comparisons, and *** indicates p < 0.001.

Task difficulty also had a significant effect on accuracy in both experiments. In Experiment 1, the highest accuracy was observed in the easy task (M = 0.993, SD = 0.011), followed by the medium task (M = 0.950, SD = 0.034) and the hard task (M = 0.958, SD = 0.032). Post hoc comparisons showed that accuracy in the easy task was significantly higher than in both the medium and hard tasks (Cohen’s d = 1.744 and 1.869, respectively), whereas no statistically significant difference was observed between the medium and hard tasks. A similar pattern was observed in Experiment 2. Accuracy declined from easy (M = 0.887, SD = 0.092) to medium (M = 0.823, SD = 0.111) and hard tasks (M = 0.832, SD = 0.097), with significant differences between the easy task and the two higher difficulty levels (easy vs. medium: Cohen’s d = 2.341; easy vs. hard: Cohen’s d = 1.461), whereas no statistically significant difference was observed between the medium and hard tasks. The accuracy differences across experiments at each task difficulty level are illustrated in Figure 8b.

Accuracy did not show a statistically significant main effect of depth in Experiment 2 (p = 0.063). However, significant interactions were found between interaction environment and depth (p < 0.05, ${\eta }_p^2=0.785$) and between interaction environment and task difficulty (p < 0.05, ${\eta }_p^2=0.378$). Bonferroni-adjusted simple-effects comparisons for the significant interaction between interaction environment and depth showed that accuracy differed significantly between the real and virtual environments at both tested depths (both adjusted p ≤ 0.030). As illustrated in Figure 9a, accuracy remained higher in the real environment than in the virtual environment at both depths, and the descriptive pattern suggests that the reduction in the virtual condition was more pronounced at 35 cm than at 45 cm. For the interaction between interaction environment and task difficulty, Bonferroni-adjusted simple pairwise comparisons did not isolate statistically significant differences between the two environments at any individual task difficulty level, nor significant pairwise differences among task difficulty levels within either environment. Accordingly, Figure 9b is interpreted as illustrating the overall interaction pattern rather than a single localized pairwise contrast. No statistically significant three-way interaction among interaction environment, depth, and task difficulty was observed. Full ANOVA results, including effect sizes, are presented in Table 2.

Figure 9. Accuracy interaction plots for Experiment 2. Panel (a) shows the interaction between interaction environment and depth. Panel (b) shows the interaction between interaction environment and task difficulty. Points represent participant-level observations, lines indicate condition means, and error bars show within-subject standard errors.

4. Discussion

This study investigated hierarchical menu interaction performance in virtual and real environments within peripersonal space, with a particular focus on the role of physically aligned versus non-aligned tactile reference during hierarchical menu selection. Two primary interaction conditions were tested: (1) direct touch selection on the screen surface (Experiment 1) and (2) mid-air selection without physical contact (Experiment 2).

Experiment 1 supported H1, as no statistically significant difference in movement time or accuracy was observed between the real and virtual environments when visual targets were aligned with the point of physical contact on the screen. Experiment 2 supported H2, as performance in the virtual environment declined relative to the real environment when selections were performed in mid-air without tactile reference at the target location. These findings are consistent with prior studies indicating that passive-haptic support can reduce performance differences in simple pointing tasks and that mid-air interaction can increase spatial uncertainty, particularly within peripersonal space [28,41,42,43]. The experimental design, however, did not manipulate visual target alignment and tactile feedback independently. In the screen-based condition, aligned visual targets and tactile feedback were available simultaneously, whereas in the mid-air condition, displaced visual targets and the absence of tactile feedback co-occurred. The present findings, therefore, do not permit a clear separation of the respective contributions of visual target alignment and tactile feedback. Accordingly, the following sections discuss three plausible contributors to the descriptive differences observed between experiments: (1) target placement and visual alignment on a physical surface, (2) the availability of tactile support during target acquisition, and (3) task familiarity and VR experience.

4.1. The Effect of Visual Target Position and Tactile Feedback on Hierarchical Menu Performance

4.1.1. Visual Target Location

In the condition where targets were presented on the 3D TV screen (zero parallax), the visual target locations corresponded directly to the physical screen. Presenting the targets on the screen allowed participants to align gaze and finger movements towards a single spatial position and to terminate movements upon contact with the screen, resulting in a clear and stable endpoint. Positioning the menu on a physical surface is expected to reduce vergence-accommodation conflict (VAC) and stabilize depth perception in stereoscopic displays, thereby supporting more consistent selection performance [33,44,45,46]. Although VAC was not directly measured in the present study, this interpretation is consistent with previous studies reporting that lower visual conflict is associated with shorter movement times and higher accuracy, particularly when targets are located in the central display region [30,47].

In contrast, when the menu was rendered as a floating stimulus in front of the screen at depths of 35–45 cm (negative parallax), the visual system had to converge to the virtual target depth, while accommodation remained at the screen plane [46]. Participants therefore had to move the finger toward a target perceived as protruding from the display while maintaining visual focus on a surface behind that target. Such a configuration is likely to reduce depth perception stability and increase the need for small corrective movements prior to confirmation, particularly in the virtual environment where the targets were presented in mid-air. One plausible explanation is that this condition imposed greater VAC-related visual demands, although VAC and related visual strain were not directly measured in the present study. Prior work nevertheless indicates that vergence-accommodation conflict and increased parallax can reduce the accuracy of depth perception, impair stereoscopic selection performance, and increase visual load [28,48,49,50].

The significant interaction between interaction environment and depth suggests that the influence of target distance depended on how selection was physically supported. In the real condition, direct contact with the screen appears to have stabilized performance across the tested distances, whereas in the virtual mid-air condition, the shorter-distance setting was associated with a more pronounced reduction in accuracy. This finding suggests that bringing the virtual target closer did not simply make the task easier. Instead, it may have increased the demand to localize and control the fingertip within a visually protruding space while accommodation remained tied to the screen plane. Under these conditions, small endpoint deviations may have become more consequential, particularly because no tactile confirmation was available at the target location. The reduced accuracy observed in this condition should therefore not be attributed solely to visual target displacement or stereoscopic depth demands, but should be interpreted more cautiously as arising from the combined effect of visual and tactile differences between the compared conditions. The findings of this study are consistent with the role of multisensory integration in peripersonal space, where stable performance depends on the coordination of visual, proprioceptive, and tactile information during near-body actions. When tactile confirmation is absent, motor control must rely more heavily on visual and proprioceptive estimation alone, which may increase endpoint uncertainty during sequential menu interaction. These findings suggest that visual factors related to target position and stereoscopic presentation contributed to the reduced accuracy observed in the virtual condition.

These interpretations should nevertheless be generalized with caution to contemporary HMD-based VR systems. Unlike the stereoscopic 3D TV used in the present study, HMDs typically provide a more immersive and head-centered visual presentation, which may alter peripersonal-space perception, depth judgment, and the role of aligned visual references during target acquisition. As a result, the magnitude and practical relevance of the present findings may not extend directly to HMD-based interaction contexts.

4.1.2. Tactile Feedback on Menu Selection Performance

In both experiments, participants completed the same two-stage interaction sequence: placing the finger on the target, followed by pressing a handheld remote button to confirm the selection. The critical difference lay in the availability and quality of physical contact during target acquisition. When the menu was attached to the screen surface, movement termination occurred through direct fingertip–screen contact, providing a discrete tactile signal that the intended target position had been reached. Such tactile events can act as a physical stop that stabilizes the endpoint, reduces error incidence, and decreases selection time variability, particularly for small targets and dense layouts [36]. Similar benefits have been reported in studies of passive haptics in VR, where physical surfaces aligned with virtual objects increase throughput and make performance more comparable to touchscreen interaction by providing tactile guidance and hand-position references aligned with the target location [51]. Tactile feedback has further been shown to support action confirmation, enhance spatial awareness, and increase user confidence, thereby promoting more stable and natural interaction behavior [9,52,53]. However, in the present experiments, tactile feedback was not manipulated independently of visual target location. In the screen-based condition, tactile contact was available together with spatially aligned visual targets, whereas in the mid-air condition, the absence of tactile contact co-occurred with displaced visual targets. The results, therefore, support tactile feedback as a plausible contributing mechanism, but they do not isolate its effect independently of the visual configuration.

A different pattern emerged when the menu was visualized in mid-air. In the absence of a physical surface at the depth of the virtual targets, participants had to rely almost entirely on visual information and estimated hand position to determine when to confirm a selection. The lack of a tactile reference increases uncertainty about when the finger is “close enough” to the target, induces additional corrective sub-movements, and elevates the risk of selection errors during target acquisition [54]. Under this heightened spatial uncertainty, the handheld remote effectively serves as a final commitment mechanism, consistent with the increased movement-time variability and reduced accuracy observed in mid-air VR interaction. Prior studies on pseudo-haptics and mid-air button controls further demonstrate that adding physical tactile cues or artificial visuo-haptic illusions (e.g., visual impact effects, deformations, or pseudo-haptic buttons) can improve perceived contact, confirmation reliability, and overall experience for mid-air button interaction [54,55,56].

The significant interaction between interaction environment and task difficulty further suggests that the influence of task demand differed between environments. The follow-up pairwise comparisons, however, did not isolate statistically significant differences at any individual task difficulty level. The interaction is therefore more appropriately interpreted as an overall divergence in performance patterns across environments. Increasing task difficulty appears to have imposed a greater performance burden in the virtual condition, in which hierarchical selections were completed without physical contact at the target location. Participants in this condition were required to maintain an internal estimate of hand depth along the anterior–posterior axis throughout the selection process, consistent with previous reports [57,58]. Small uncertainty in endpoint control may therefore have accumulated across successive selections, making accurate confirmation more difficult in the VR mid-air condition than in the real condition. This interpretation also has implications for motor-control theory, as it suggests that the absence of a physical endpoint may increase reliance on visually guided closed-loop correction during sequential action, which may in turn increase the performance cost associated with higher precision demands across menu levels.

Taken together, these findings suggest that the absence of a tactile reference during mid-air interaction contributed to the performance degradation observed in the mid-air VR condition. The interaction pattern is consistent with the interpretation that tactile support may assume greater importance when hierarchical menu selection is performed under higher precision demands, although the follow-up pairwise contrasts did not isolate a single dominant task difficulty level after correction.

The findings further support the use of surface-based interaction, or other forms of stable tactile support, when hierarchical menus require precise sequential input. Designers should reduce precision demands where possible when mid-air interaction is required and should consider additional support mechanisms, such as stronger visual guidance, clearer confirmation cues, or passive- and pseudo-haptic techniques, to improve selection stability across levels. Tactile feedback and visual target location nevertheless remained confounded in the present design, and their relative contributions, therefore, cannot be determined conclusively. The design implications of this study should accordingly be interpreted in relation to the combined interaction condition rather than as a direct estimate of tactile-feedback effects alone.

4.2. Task Familiarity and VR Experience

The screen-touch menu-selection task closely reflects routine interaction with touchscreen devices, such as selecting options and pressing buttons on smartphones and monitors. Participants, therefore, are expected to have established sensorimotor mappings for coordinating visual attention and finger movement on a stable planar surface. This interpretation is supported by the training protocol: before the experiment, participants practiced only with a floating virtual menu at 35 cm, yet performance remained more stable in the screen-touch condition. This condition was reflected in more consistent terminal finger placement, faster confirmation decisions, and lower variability in movement time and error than in the mid-air condition. These observations are consistent with prior studies suggesting that experience with similar task types and interaction patterns facilitates adaptation and improves the quality of the user experience in VR [59,60].

However, the mid-air selection condition has less direct similarity to daily interaction. In this setting, users had to control hand position without physical contact references while pressing a button on a separate device to confirm selections, so strategies effective for touchscreen-like tasks could not be transferred directly and instead required additional adaptation. This requirement may explain the higher variability in movement time and the lower selection accuracy in the mid-air VR condition. However, task familiarity and prior VR experience were not measured explicitly in the current design. Therefore, their contribution to performance differences between screen-touch and mid-air conditions cannot be quantified directly.

Previous work also demonstrated that changes in input modality (e.g., hand-based versus controller-based input, or mid-air versus device-assisted interaction) can alter control demands, performance, and user perception, implying that strategies effective in one mode do not automatically transfer without adaptation and practice [61,62]. Furthermore, this study extends earlier work by employing a menu-based interaction paradigm that more closely reflects sequential menu use in practical VR applications than the standard multidirectional tapping task with circular virtual targets specified in ISO 9241-9. This task choice is important both methodologically and theoretically, as it situates multisensory control demands within a sequential interface task that more closely approximates real menu use in such environments.

4.3. Limitation

The findings of this study should be interpreted in light of several limitations. First, the two experiments were completed in a fixed sequence, with Experiment 1 always preceding Experiment 2. Although this order was selected to maintain a consistent progression from the baseline screen-based condition to the more demanding mid-air condition, possible carryover effects related to practice or fatigue cannot be fully excluded. Future studies should either counterbalance the order of experiments or separate them across sessions to isolate such effects more clearly.

Second, the participant sample was relatively small and homogeneous within a narrow age range. Accordingly, the external validity of the findings is limited, and the results may not be extended directly to broader populations with different educational backgrounds, age groups, levels of VR familiarity, or professional experience. Future studies should therefore include more diverse participant samples to examine the robustness of the present findings across wider user groups.

Third, the experimental platform relied on a stereoscopic 3D TV with active-shutter glasses rather than a head-mounted display (HMD). This limitation is important because contemporary VR is predominantly experienced through HMD-based systems, which provide a different visual and interaction context from stereoscopic screen-based setups. In particular, HMD-based VR has been associated with different levels of immersion and presence, different depth-perception characteristics, and HMD-specific visual constraints such as vergence-accommodation conflict, all of which can influence 3D selection and direct interaction performance [63,64]. The present findings should therefore be generalized to modern HMD-based VR systems with particular caution.

Fourth, the present design did not independently manipulate aligned visual target location and tactile feedback. In the surface-based condition, the visual target and the physical point of contact were spatially co-located, so aligned visual information and direct tactile contact were available simultaneously. Accordingly, the findings do not permit a strict causal separation of the respective contributions of visual alignment and tactile feedback. Future studies should manipulate these two factors independently to determine their relative contributions more directly.

Fifth, the study relied exclusively on objective performance measures and used only a simplified rectangular grid menu without icons or text. Although these choices supported strict experimental control, they limit both interpretive breadth and ecological generalizability. In particular, the absence of subjective assessments such as workload, usability, presence, preference, or visual comfort makes it difficult to determine whether the observed performance differences were accompanied by differences in perceived effort or discomfort, while the simplified menu does not capture the range of menu configurations found in real-world VR applications. Future studies should therefore combine objective and subjective measures and examine a broader range of menu archetypes.

5. Conclusions

This study evaluated selection performance for a 4 × 3 hierarchical menu in virtual reality (VR) and in a real environment within peripersonal space, comparing two menu presentation conditions: (1) the menu presented on a physical screen surface and (2) the menu rendered as a floating element in front of the screen. When targets were displayed on the screen and aligned with the fingertip contact point, no statistically significant differences in movement time or accuracy were observed between the virtual and real environments. This arrangement, therefore, provides a reference condition for hierarchical menu interaction in peripersonal space. However, when the menu was presented in mid-air, performance in the virtual environment declined relative to the real environment, particularly in terms of selection accuracy and under higher task demands.

Taken together, the findings indicate that aligned visual targets and tactile support are important for maintaining stable hierarchical menu interaction in VR. The results further suggest that the performance cost of mid-air interaction is shaped jointly by spatial presentation and task demand, rather than by either factor in isolation. Although the present study does not permit a strict causal separation between visual target alignment and tactile feedback, it provides evidence that the combined absence of tactile reference and mid-air target presentation imposes greater control demands within peripersonal space. These findings offer practical guidance for the design of VR menu interfaces by highlighting the value of surface-based interaction and tactile support, particularly for tasks that require precise sequential input.