The Trail Making Test in Virtual Reality (TMT-VR): Examination of the Ecological Validity, Usability, Acceptability, and User Experience in Adults with ADHD

Abstract

Background: Virtual Reality (VR) is transforming neuropsychological assessment by providing immersive environments that closely replicate real-world conditions. This study presents the Trail Making Test in VR (TMT-VR), a novel adaptation of the traditional TMT, aimed at evaluating cognitive functions in adults with Attention Deficit Hyperactivity Disorder (ADHD). We examined the ecological validity, convergent validity, usability, and user experience of the TMT-VR compared to the traditional version. Methods: Fifty-three adults (18–40 years old), including 25 with ADHD and 28 neurotypical controls, completed both the traditional and VR versions of the TMT. The participants also provided feedback on the VR experience via standardized questionnaires. Results: The statistical analyses demonstrated a significant positive correlation between TMT-VR scores and both the traditional TMT and ADHD symptomatology as measured by the Adult Self-Report Scale, confirming the TMT-VR’s ecological and convergent validity. High usability and positive user experience ratings indicated strong acceptability of the VR platform. Notably, the ADHD group exhibited greater performance differences in the VR environment, suggesting that VR may more effectively capture real-world cognitive challenges. Conclusions: These findings suggest that TMT-VR is a valid, engaging, and ecologically valid tool for cognitive assessment in ADHD and other clinical populations, offering enhanced insights over traditional methods.

1. Introduction

Traditional neuropsychological tests have long been the dominant method for assessing cognitive functions in relation to everyday functioning. However, these tests are significantly limited by their lack of ecological validity. Ecological validity refers to the extent to which an assessment’s tasks resemble real-life activities (verisimilitude) and correlate with real-world performance (veridicality; [1,2]). The primary issue with traditional measures is that they typically take place in laboratory settings, where the simplicity and static nature of tasks fail to represent the complexities of daily life [3,4,5]. This lack of ecological validity can result in inconsistent findings when evaluating cognitive functions relevant to everyday activities [6].

1.1. Ecological Validity of Virtual Reality Assessments

Traditional neuropsychological measures often lack ecological validity because the simple and static nature of tasks does not accurately represent the complexities of everyday life [3,5,7]. This lack of ecological validity is a significant limitation because it impairs the ability to relate task performance scores to real-life functioning, even in neurologically intact individuals [2,6]. Furthermore, non-ecologically valid measures can yield inconsistent results when assessing cognitive functions relevant to daily activities [5].

Efforts to replicate everyday tasks in naturalistic settings have attempted to address these limitations, but they face significant challenges, including the inability to standardize for clinical use, the tendency to overlook specific groups (e.g., mental health patients), and limited control over extraneous variables [5]. These challenges have driven the development of alternative methods for neuropsychological assessment, with immersive virtual reality (IVR) emerging as a highly promising substitute for traditional testing methods [4,8,9].

IVR encompasses a suite of technologies that significantly enhance ecological validity in neuropsychological assessments by closely mimicking real-world experiences [3,5]. Key components of IVR systems include head-mounted displays (HMDs), motion controllers, motion sensors, haptic feedback, and 3D sound environments, all of which contribute to a more immersive experience [3,5]. These elements collectively foster a heightened sense of presence, or the feeling of ‘being there’, which is crucial for realistic simulation and active engagement. Additionally, IVR’s interactive features enable users to manipulate objects and navigate environments in ways that mirror real-life interactions, making assessments more reflective of daily activities [3,5]. Through these advancements, IVR technology offers a robust platform for replicating complex cognitive tasks in controlled yet lifelike settings that are adaptable to the specific needs of diverse populations, including those often underserved by traditional assessment methods [3,5].

IVR technologies have been shown to enhance ecological validity by creating more adaptable and realistic environments that better reflect real-world scenarios. For example, VR-based assessments like the Virtual Reality Everyday Assessment Lab (VR-EAL) have demonstrated high ecological validity compared to traditional tests, particularly in assessing executive functions (e.g., mental flexibility, attentional set-shifting, and processing speed) within activities of daily living (ADL) [5,10]. Systematic reviews have confirmed that VR technology is highly ecologically valid as a neuropsychological tool for various cognitive functions, including attention and memory [3,4,9,11]. Such assessments have proven valuable in predicting overall performance in everyday life [3,5,12].

Virtual reality assessments also allow clinicians and researchers to extend studies to specific populations and cognitive domains in alternative settings, providing more accurate evaluations of everyday functionality by examining the impact of environmental stimuli (e.g., sensory distractions) on cognitive performance [13]. While IVR simulations of everyday activities have been extensively studied and shown to possess high ecological validity in terms of verisimilitude [2,5,11], there is a notable lack of literature exploring their veridicality.

IVR technologies as neuropsychological instruments are designed to be user-friendly, requiring no prior experience or specific age group restrictions [5,14]. Research indicates that factors such as gaming ability, age, and educational background do not significantly impact performance in VR environments, suggesting broad applicability [5]. Interestingly, individuals with no prior gaming experience often perform better in VR settings because their interactions closely resemble real-life scenarios [3,7,15]. Moreover, both younger and older adults show greater motivation and engagement with VR-based cognitive tasks compared to traditional paper-and-pencil methods, while both age groups have demonstrated high levels of acceptability, adaptability, and familiarity with VR technologies [9,10,16].

One challenge frequently encountered with IVR technology is VR-induced symptoms and effects (VRISE), commonly known as cybersickness. In the past, VR assessments often led to adverse physical symptoms, such as nausea, disorientation, fatigue, and vertigo [17,18]. These symptoms could severely undermine the effectiveness of VR neuropsychological assessments [19,20,21]. However, recent advancements in VR technology have significantly mitigated these issues by incorporating systems designed to minimize adverse effects, resulting in participants experiencing little to no VRISE symptoms [10,14,21,22].

1.2. Trail Making Test in Virtual Reality

The Trail Making Test (TMT) is a widely recognized and extensively utilized tool in neuropsychology for evaluating cognitive functions, particularly attention, processing speed, cognitive flexibility, and task-switching [23,24]. The TMT can be employed either as a standalone screening instrument to assess neurological impairments or as part of a broader battery of neuropsychological tests [24]. Performance on the TMT is a reliable indicator of cognitive dysfunction and is frequently used to assess various cognitive abilities across different populations.

Given the TMT’s significance, there has been increasing interest in developing alternative versions of the test to enhance its usability and application. Numerous studies have consistently demonstrated the TMT’s effectiveness as a neuropsychological assessment tool. However, despite the test’s broad applicability, research exploring a virtual reality (VR) adaptation of the TMT has been limited.

In one study, researchers developed an immersive VR version of a TMT alternative, specifically the Color Trail Test, and reported high ecological and convergent validity [25]. However, this study faced several limitations: it employed complex methodologies and required expensive equipment, making the system inaccessible to many clinicians and difficult for researchers with limited funding to replicate. Moreover, the study appeared to focus more on the technological aspects rather than clinical applications, thus limiting its practical use in everyday clinical settings. Notably, while this was the first known adaptation of a TMT alternative into a VR context, the original Trail Making Test has yet to be fully explored and utilized in VR environments.

The TMT has been employed in various neuropsychological contexts to evaluate cognitive functioning, particularly in populations with attention deficits. Numerous studies on adults with ADHD have utilized both parts of the TMT—Part A and Part B—as supplementary assessment tools [26,27,28,29]. Part B, which assesses cognitive flexibility and task-switching, has been particularly effective in identifying set-shifting issues among adults with ADHD [26].

The current study represents the first adaptation of the traditional TMT into a virtual reality format, the TMT-VR, with the goal of confirming its validity and effectiveness as a clinical tool. In a preliminary study, three different interaction modes—eye-tracking, head movement, and controller—were tested to compare performance among 71 neurotypical young adults [30]. The results indicated that both eye-tracking and head movement were significantly more efficient than the controller in terms of task performance, including accuracy and task completion time. Specifically, eye-tracking was found to be more accurate, while head movement facilitated faster task completion. Despite these differences, no significant overall performance differences were observed between eye-tracking and head movement, suggesting that both modes could be viable alternatives. However, further analysis revealed that eye-tracking demonstrated superior accuracy among the non-gamer participants, indicating that it may be more universally applicable and better suited for clinical settings, especially in studies involving individuals with ADHD.

1.3. Neuropsychological Assessment in Attention Deficit Hyperactivity Disorder

Attention Deficit Hyperactivity Disorder (ADHD) is a neurodevelopmental disorder characterized by persistent patterns of inattention, hyperactivity, and impulsivity that can vary in severity among individuals [31,32]. ADHD is among the most prevalent neurodevelopmental disorders, affecting approximately 5% of children and adolescents globally [33,34]. Traditionally, ADHD has been perceived as a childhood disorder, with the expectation that symptoms would diminish as individuals age. However, recent longitudinal studies have provided compelling evidence that ADHD often persists into adulthood, challenging this outdated notion [34,35].

Meta-analytic research indicates that about 65% of individuals diagnosed with ADHD in childhood continue to exhibit symptoms into adulthood, highlighting the chronic nature of the disorder [36,37]. Despite this persistence, the prevalence of ADHD in adults is estimated at around 2.5%, which may be underreported due to the complexities and subtleties of diagnosing ADHD in adult populations, where symptoms may manifest differently than in children [36,38].

ADHD can have a profound impact on everyday functioning, affecting multiple cognitive domains essential for daily tasks. Core cognitive deficits in individuals with ADHD include impairments in sustained attention, behavioral inhibition, working memory, and perceptual processing speed [6,39]. These deficits can lead to significant difficulties in academic achievement, occupational performance, and social interactions [40,41]. For example, sustained attention, crucial for tasks requiring prolonged mental effort, is often compromised in individuals with ADHD, resulting in difficulties completing tasks, following instructions, and maintaining focus in environments such as classrooms or workplaces [39].

Behavioral inhibition, which involves the ability to control impulses and delay gratification, is another critical cognitive function often impaired in ADHD [39]. Deficits in this area can lead to impulsive decision-making, challenges in resisting distractions, and difficulties adhering to social norms or expectations [42]. These behaviors can significantly impact relationships, academic success, and career advancement [43].

Research has shown that while symptoms of hyperactivity and impulsivity tend to decrease with age, inattention often persists across the lifespan, continuing to impact daily life and cognitive performance [35,44]. Persistent inattention can contribute to ongoing challenges in organization, time management, and completion of tasks that require sustained mental effort, making it difficult for adults with ADHD to manage responsibilities such as finances, deadlines, and household organization [40].

Given the significant and lasting impact of ADHD on cognitive and functional abilities, accurate neuropsychological assessments are crucial for effective diagnosis and treatment planning. Traditional neuropsychological tests have been widely used to evaluate the cognitive deficits associated with ADHD. However, these assessments often lack ecological validity, which refers to the extent to which test results reflect an individual’s performance in real-life situations [1,11]. This lack of ecological validity is a significant limitation, particularly for adults with ADHD, who may face different cognitive challenges than children [40].

The need for ecologically valid assessment tools is particularly pressing for adults with ADHD, as their cognitive difficulties are often context-dependent and influenced by the complexities of daily life [40]. For instance, while children with ADHD may struggle primarily with hyperactivity in structured environments, adults may face greater challenges related to inattention and executive dysfunction in unstructured settings [45].

In this context, the development of more ecologically valid assessment tools, such as virtual reality-based tests, is of paramount importance. These tools offer a more realistic and interactive testing environment that closely mimics the demands of everyday life, providing more accurate and meaningful insights into the cognitive abilities and challenges of individuals with ADHD [3,11]. By capturing the dynamic and multifaceted nature of cognitive functioning in real-world contexts, VR-based assessments can inform more effective treatment strategies and improve the overall quality of life for individuals with ADHD [4].

1.4. Trail Making Test in ADHD

The TMT is widely used in neuropsychological assessments to measure cognitive functions such as attentional set-shifting, mental flexibility, and processing speed. These functions are often impaired in individuals with ADHD [32,46]. The TMT is typically divided into two parts: Part A, which involves connecting numbered circles in sequence, and Part B, which requires alternating between numbers and letters, thereby adding a cognitive load that tests mental flexibility and other executive functions, such as inhibitory control and working memory operations. Notably, both parts of the Trail Making Test assess psychomotor speed, visual attention, and visuomotor coordination. However, studies have shown that insufficient alphabet automatization, as seen in individuals with dyslexia or those unaccustomed to the Latin alphabet, can adversely affect performance on the TMT-B, with alphabet support mitigating these effects without compromising the test’s validity as a measure of executive function [47].

In the context of ADHD, the literature suggests that TMT is an integral part of a broader assessment battery aimed at evaluating cognitive deficits related to attention and executive functions [46]. However, the existing body of research on the application of TMT in ADHD populations presents some inconsistencies. While many studies report that individuals with ADHD tend to perform worse on both parts of the TMT, particularly Part B, due to its higher demands on cognitive flexibility and task-switching abilities [32,46,48], other research has not found significant differences between ADHD and control groups, highlighting the variability in how ADHD affects cognitive functioning [49]. For instance, one study found that although exercise interventions benefit cognitive performance overall, tasks that revolved around set-shifting or mental flexibility, as measured by the TMT-B were not found to benefit performance in individuals with ADHD [50]. These discrepancies may be due to variations in study design, sample characteristics, or the specific cognitive demands of the TMT in different contexts.

Nevertheless, a substantial body of evidence supports that individuals with ADHD, particularly adults, exhibit greater impairments on the TMT Part B than on Part A, reflecting their broader challenges in tasks requiring attention, inhibition, executive functioning, and set-shifting [28,51]. This pattern is indicative of deficits in mental flexibility, a core cognitive function often disrupted in ADHD [26,51].

These findings suggest that the TMT, particularly Part B, is a valuable tool for identifying cognitive deficits associated with ADHD, specifically those related to mental flexibility and executive functioning. However, the variability in results across studies underscores the need for further research to clarify the conditions under which the TMT is most effective in differentiating between ADHD and neurotypical individuals. Such research is crucial for refining the use of the TMT in clinical practice, particularly in developing more tailored and accurate assessments for individuals with ADHD.

1.5. Virtual Reality and ADHD in Clinical and Non-Clinical Populations

The overall lack of ecological validity in traditional neuropsychological assessments significantly impacts the accuracy of clinical evaluations for ADHD [44]. Common neuropsychological tests used to assess inattention and hyperactivity often have limited diagnostic utility and can yield inaccurate results [44]. As a potential solution, a growing literature suggests that VR technologies could offer more reliable assessments by integrating interactive environments that closely resemble real-life scenarios, thereby enhancing ecological validity [11,44]. Also, VR headsets such as the Meta Quest series constitute accessible VR technology, offering affordable and user-friendly headsets that require minimal training for healthcare professionals, thereby addressing both cost and usability concerns in clinical settings [3,5,52].

Individuals with ADHD frequently experience cognitive deficits, including difficulties in maintaining attention during monotonous or repetitive tasks, and challenges in task monitoring, preparatory processing, and response inhibition [53,54]. VR technologies are designed to allow sensory information (e.g., visual) to interact directly with the virtual environment while simultaneously monitoring the participant’s movements and positions during assessments [53]. Furthermore, VR environments can be manipulated to create realistic yet highly interactive settings, in contrast to the monotony often associated with traditional assessment measures [53].

Although research on VR interventions in ADHD populations remains limited, the findings are promising. The integration of VR not only enhances assessment methods for individuals with ADHD but also improves performance in cognitive functions such as sustained vigilance [53]. However, most studies have focused on children and adolescents, leaving adult ADHD relatively underexplored. This focus on younger populations likely stems from the higher prevalence of ADHD in children and adolescents, but there remains a critical need to investigate ADHD in adults. For example, symptomatic adult ADHD—where symptoms first present during adulthood—is a condition often lacking a clinical diagnosis due to the absence of noticeable symptoms during childhood [34].

Studies on children and adolescents have provided valuable insights into the effectiveness of VR for ADHD. A meta-analytic study suggested that VR technologies could be used as a treatment method for attention and hyperactivity deficits and improve memory performance and overall cognitive function in children with ADHD [55]. Interestingly, another study found that VR interventions were as effective in treating ADHD symptoms in children as traditional drug treatments and could potentially serve as a replacement in some cases [56]. Another relative study found increased motivation to learn when children with ADHD were administered VR training compared to individual training [57]. The VR training in that case reinforced active participation and direct involvement of the children in regards to the learning experience [57]. The present study aims to extend this literature by integrating the Trail Making Test (TMT) into a VR setting, with a specific focus on adults with ADHD, a population that has been underexplored.

The development of tools such as the Adult ADHD Self-Report Scale (ASRS) has been crucial for assessing ADHD symptoms, including inattention, hyperactivity, and impulsivity, in both clinically diagnosed individuals and undiagnosed adults. While the ASRS cannot independently provide a diagnosis, it is considered a highly accurate measure for identifying prevalent ADHD symptoms in non-clinical populations [58]. By combining these validated tools with innovative VR technology, this study seeks to advance the assessment and treatment of ADHD across different age groups and diagnostic statuses.

1.6. Usability, User Experience, and Acceptability

The perception of neuropsychological assessment instruments by those being assessed has been extensively studied, with recent research indicating a strong preference for technology-based tools. Instruments that incorporate technology, such as virtual reality (VR), are generally rated highly in terms of usability and acceptability by users [59,60,61,62]. For instance, one study demonstrated the high technical acceptance of a TV-based platform for cognitive evaluations, highlighting its high usability and acceptability [59]. Additionally, technology-based neuropsychological tools are promising in terms of resource efficiency, speed, and cost, factors that are often positively perceived by users and contribute to higher ratings [61]. Furthermore, user experience reports indicate very positive attitudes towards these technologies across various age groups, including older adults [5].

Although there are relatively few studies focused specifically on validating VR instruments as neuropsychological assessment tools, the available research shows promising results. For example, one study validated VR training software for adults with autism spectrum disorder, reporting high system usability and positive user experiences, suggesting minimal effort was required from users and that the software was easy to handle [12]. High usability scores indicated that the participants were able to inhibit automatic responses and ignore distractions effectively, engaging positively with the software [12]. Similarly, high acceptability scores suggest that the VR software was well received as a neuropsychological assessment tool [12]. However, while these findings offer insights into the overall technical acceptance of VR technology, research in neuropsychology remains limited, and further expansion is needed to establish VR as a valid neuropsychological assessment method.

1.7. Present Study

The development of the TMT-VR marks a significant effort to integrate virtual reality technologies into neuropsychological assessments. As the first VR adaptation of the traditional Trail Making Test, the TMT-VR aims to provide a comprehensive tool for evaluating various cognitive functions. This study was designed to validate the TMT-VR by assessing its ecological validity and convergent validity. Additionally, the study evaluated the usability, user experience, and acceptability of the TMT-VR based on feedback from participants.

The study focused on a clinical population, particularly young adults with ADHD, with the goal of establishing the TMT-VR as a reliable instrument for neuropsychological assessments in clinical practice. The study compared performance on both the traditional TMT and the TMT-VR between neurotypical and neurodivergent populations. To assess the ecological validity of the TMT-VR, the Adult ADHD Self-Report Scale (ASRS) was utilized in both diagnosed and undiagnosed participants.

The study hypothesized the following:

H1. 

The TMT-VR will demonstrate high ecological validity, particularly in terms of veridicality.

H2. 

TMT-VR scores will show a significant positive correlation with traditional TMT scores, indicating high convergent validity.

H3. 

The TMT-VR will exhibit high usability, user experience, and acceptability among individuals with ADHD.

2. Materials and Methods

2.1. Participants/Sample

The study received approval from the Ad-hoc Ethics Committee of the Psychology Department, which abides to the Declaration of Helsinki. The individuals with ADHD presented to the experimenter a medical document from the national health system proving their diagnosis. None of the ADHD participants was under treatment with medication. The inclusion criteria were absence of psychiatric disorders, neurological disorders, neurodevelopmental disorders other than ADHD (for the ADHD group), learning difficulties, use of stimulants, and abuse of alcohol or other drugs. A total of 53 Greek-speaking young adults participated in the study. Of these, 47.17% had been diagnosed with ADHD (N = 25), and 52.83% were neurotypical individuals (N = 28). In the sample, 46.9% were male (N = 15), 50.0% were female (N = 16), and 3.1% (N = 1) did not disclose their gender. The participants’ ages ranged from 18 to 40 years (M = 23.87, SD = 3.82), and their total years of education ranged from 12 to 22 years (M = 15.98, SD = 3.26).

2.2. Materials

2.2.1. Immersive VR Setup

This study exclusively utilized exclusively fully immersive VR. In accordance with hardware recommendations aimed at significantly reducing the likelihood of cybersickness [52], the VR setup in this study was centered around a high-performance PC, complemented by a Varjo Aero headset (Varjo Technologies Oy, Helsinki, Finland), HTC Vive Lighthouse stations (HTC Corporation, New Taipei City, Taiwan), SteamVR software (Valve Corporation, Bellevue, WA, USA), noise-canceling headphones (Sony Corporation, Tokyo, Japan), and HTC Vive controllers (HTC Corporation, New Taipei City, Taiwan). The PC featured an Intel64 Family 6 Model 167 Stepping 1 Genuine Intel processor with a speed of approximately 3504 MHz and 32 GB of RAM, ensuring a smooth and responsive VR performance.

The Varjo Aero headset was selected for its superior visual quality, boasting dual mini-LED displays with a resolution of 2880 × 2720 pixels per eye and a 115-degree horizontal field of view. Operating at a refresh rate of 90 Hz and equipped with custom lenses to reduce distortion, the headset delivered a highly immersive and high-fidelity visual experience. Positional tracking was managed by Vive Lighthouse stations, strategically placed diagonally to provide a large tracking area and precise tracking accuracy.

SteamVR served as the main platform for managing the VR experiences, offering robust room-scale tracking capabilities. To enhance auditory immersion, noise-canceling headphones were used, providing high-quality audio with active noise cancellation to minimize external distractions. Interaction within the virtual environment was facilitated by Vive controllers, designed ergonomically with haptic feedback, grip buttons, trackpads or joysticks, and trigger buttons, ensuring a responsive and tactile user experience. The integration of these components resulted in a comprehensive and immersive VR setup, enabling detailed and precise virtual reality experiences for the study.

Additionally, the participants underwent a familiarization phase to ensure comfort with VR interactions before testing. The task instructions were provided in both visual and auditory formats, and real-time tracking of head and hand movements was utilized to capture response accuracy and engagement. These methodological considerations were designed to optimize attentional focus while maintaining an ecologically valid testing environment.

2.2.2. Demographics

A custom demographics questionnaire was administered that collected information on the participant’s gender, age, years of education, and whether participants had received an ADHD diagnosis.

2.2.3. Greek Version of the Adult ADHD Self-Report Scale (ASRS)

The Adult ADHD Self-Report Scale (ASRS) is a brief self-report screening tool for ADHD in adults [58]. It is a widely recognized tool for identifying ADHD symptoms in adults. The ASRS was developed through a collaboration between the World Health Organization (WHO) and researchers from New York University [58]. It consists of 18 items rated on a 5-point Likert Scale (0 = Never, 4 = Very Often). An example question is “How often do you feel restless or fidgety?” Scores are calculated by summing all responses. The Greek version of the ASRS was used [63]. The Greek version retains the strong psychometric properties of the original scale, with Cronbach’s alpha values typically ranging between 0.88 and 0.89, indicating excellent internal consistency, and it shows robust test-retest reliability around 0.79, suggesting reliable results over time [58,63].

2.2.4. Trail Making Test (TMT)

The Greek version of the traditional Trail Making Test was administered [64], including both its parts, Part A and Part B. The test takes approximately 5–10 min. The primary measure is the total completion time for both parts, with a maximum score of 300 s, which serves as the cut-off for discontinuing the test [24].

Part A

This section evaluates visual search and motor speed skills. Participants must draw a line connecting numbers in ascending order from 1 to 25. Performance is assessed based on the total time in seconds required to complete the task.

Part B

This section assesses higher-level cognitive skills, such as mental flexibility. Participants must draw a line alternating between numbers and letters in ascending order (e.g., 1 to A, 2 to B, 3 to C, etc.) until completion. Performance is evaluated based on the total time in seconds needed to complete the task.

2.2.5. Cybersickness in Virtual Reality Questionnaire (CSQ-VR)

The CSQ-VR is a brief tool for evaluating VRISE [21]. It consists of three categories, each with two items (Nausea A, B; Vestibular A, B; Oculomotor A, B) rated on a 7-point Likert Scale (1 = Absent Feeling, 7 = Extreme Feeling). An example question is “Do you experience disorientation (e.g., spatial confusion or vertigo)?” Participants answer based on their current feelings. Each category is scored by summing Score A and Score B. The total CSQ-VR Score is the sum of all category scores (Nausea Score + Vestibular Score + Oculomotor Score). The CSQ-VR is administered twice during the assessment (before and after the VR session). The internal consistency of the CSQ-VR is particularly strong, with Cronbach’s alpha values exceeding 0.90, demonstrating excellent reliability. Additionally, its test–retest reliability is greater than 0.80, indicating that the CSQ-VR provides stable and reliable results across repeated measures. These metrics demonstrate that the CSQ-VR is a superior tool compared to other cybersickness questionnaires like the SSQ and VRSQ, especially in detecting performance declines due to cybersickness [20,21].

2.2.6. Trail Making Test VR (TMT-VR)

The Trail Making Test in Virtual Reality (TMT-VR) represents a significant advancement over the traditional paper-and-pencil version of the Trail Making Test [30]. This VR adaptation is designed to leverage the immersive qualities of virtual reality to create a more naturalistic and engaging environment for cognitive assessment, specifically targeting young adults with ADHD. In the TMT-VR, participants are fully immersed in a 360-degree virtual environment where numbered cubes are distributed across a three-dimensional space, including the Z-axis, introducing a depth component that mirrors real-world spatial challenges (see Figure 1).

A key feature of the TMT-VR is its naturalistic approach to target selection. Unlike the traditional TMT, which is limited to a flat, two-dimensional paper format, the TMT-VR allows participants to select targets using an intuitive, real-world interaction method—eye-tracking. This setup closely simulates how individuals naturally interact with their surroundings, providing a more accurate assessment of cognitive functions like visual scanning, spatial awareness, and motor coordination.

The administration of the TMT-VR is fully automated, including comprehensive tutorials that guide participants through the tasks. This automation ensures that all participants receive consistent instructions, eliminating the variability that can occur with human administrators. The scoring process is also automated, with precise timing of task completion automatically recorded, thereby eliminating the need for manual start and stop actions and providing a more accurate measure of cognitive processing speed.

Another innovative feature of the TMT-VR is the randomization of target placement within the virtual environment. Each time a participant undertakes the TMT-VR, the cubes are positioned in new, randomized locations, requiring participants to adopt a different spatial strategy each time. This randomization is essential for ensuring that the test maintains perfect test–retest reliability, as it prevents participants from simply memorizing target locations from previous attempts, which is particularly valuable for longitudinal studies or repeated assessments.

Participants interact with the virtual environment using eye-tracking technology, where the direction of the participant’s gaze is used to select objects. The task requires participants to select numbered cubes in ascending order. Each cube must be fixated upon for 1.5 s to confirm selection, ensuring deliberate and accurate interactions. This process minimizes errors due to inadvertent movements, which are common in both traditional and virtual environments. Participants navigate the 3D virtual space by orienting themselves toward numbered cubes, using their gaze to select each target in sequence. Upon locating a target, they must fixate on it for 1.5 s to confirm selection, which triggers the visual and auditory feedback to confirm correct or incorrect choices. This sequence—identifying, fixating, receiving feedback, and moving to the next target—continues until the task is complete, providing a controlled and immersive experience that closely resembles natural spatial interactions.

Visual and auditory feedback are provided to guide participants throughout the tasks. Correct selections are highlighted in yellow, while incorrect ones are marked in red and accompanied by auditory cues. This immediate feedback allows participants to quickly correct mistakes, reducing frustration and helping maintain engagement. Previously selected cubes remain highlighted, assisting participants in tracking their progress and avoiding the re-selection of cubes.

The TMT-VR comprises two tasks that correspond to the traditional TMT-A and TMT-B, as follows:

• TMT-VR Task A: Participants connect 25 numbered cubes in ascending numerical order (1, 2, 3, …, 25). This task measures visual scanning, attention, and processing speed, similar to the traditional TMT-A but within a more immersive environment.
• TMT-VR Task B: Participants connect 25 cubes that alternate between numbers and letters in ascending order (1, A, 2, B, 3, C, …, 13). This task assesses more complex cognitive functions, including task-switching ability and cognitive flexibility, reflecting the traditional TMT-B but with the added benefits of the VR setting.

Performance on both tasks is depicted in terms of accuracy, mistakes, and completion time. Accuracy is measured as the average distance from the center of the target cube during selection, providing a precise metric of participant performance. Additionally, the number of mistakes—instances where participants selected the incorrect target—is recorded, offering further insight into cognitive control and task accuracy. Task completion times are automatically recorded, ensuring objective and consistent measurements of cognitive performance.

In summary, the TMT-VR offers a fully immersive, naturalistic, and randomized testing environment that enhances the traditional TMT [30]. Its automated administration, precise scoring, and intuitive interaction method make it a powerful tool for cognitive assessment, offering greater ecological validity and reliability than traditional methods. The TMT-VR is particularly well-suited for diverse populations, including those with limited experience with technology, ensuring consistent and accurate cognitive evaluations across repeated assessments. A video presentation of each task of TMT-VR can be accessed using the following links: TMT-VR Task A: https://www.youtube.com/watch?v=npki7i4OnwY and TMT-VR Task B: https://www.youtube.com/watch?v=immvIkOyVuA (accessed on 1 September 2024).

2.2.7. System Usability Scale (SUS)

The System Usability Scale (SUS) is a tool for assessing system usability [65]. It consists of 10 items on a 5-point Likert Scale from “Strongly Agree” to “Strongly Disagree”. Responses are aggregated to create a total score that reflects the system’s usability, requiring specific calculations to convert raw scores into a usability score out of 100. An example question is “I thought this virtual reality system was easy to use”. The questionnaire includes five reverse-scored items. SUS demonstrates excellent psychometric properties, with Cronbach’s alpha ranging from 0.85 to 0.92, indicating high internal consistency, and test–retest reliability of 0.84, showing good stability over time [12,65].

2.2.8. Short Version of the User Experience Questionnaire (UEQ-S)

The Short Version of the User Experience Questionnaire (UEQ-S) evaluates users’ subjective opinions about their experience with a technological product [66]. It comprises 26 items rated on a 7-point Likert Scale, with pairs of terms having opposite meanings (e.g., annoying vs. innovative, unpredictable). Responses range from −3 (completely disagree with the negative term) to +3 (completely agree with the positive term). The total score, representing the overall user experience, is calculated by summing all responses. It shows strong psychometric properties, with Cronbach’s alpha values between 0.70 and 0.90 and test–retest reliability scores greater than 0.80, demonstrating both internal consistency and reliability [12,66].

2.2.9. Service User Technology Acceptability Questionnaire (SUTAQ)

An adapted version of the Service User Technology Acceptability Questionnaire (SUTAQ) was employed to assess the acceptability of the TMT-VR among the target population [67]. This version of the SUTAQ, tailored for public assessments, consists of 10 items on a 6-point Likert Scale (6 = Strongly Agree, 1 = Strongly Disagree). An example question is “I am satisfied with the neuropsychological assessment of cognitive functions in virtual reality”. The SUTAQ demonstrates good psychometric properties, with Cronbach’s alpha values ranging from 0.75 to 0.85, and test–retest reliability greater than 0.70, ensuring both consistency and reliability in its measurements [12,67].

2.3. Procedures

Two groups (healthy young adults/young adults with ADHD) were assessed, and participants from each group were exposed to all conditions. The experiment took place in the Psychology Network Lab of the American College of Greece, with all sections conducted in Greek. The participants were recruited through ads, posters, and email invitations distributed via the American College of Greece’s email list. Appointments for participation were scheduled through the Calendly app. To control for potential order effects, the participants were counterbalanced: half were administered the TMT-VR first, while the other half began with the traditional paper-and-pencil version of the TMT.

At the start of the experiment, the participants were seated in front of a computer or laptop screen and asked to follow written instructions. After completing the informed consent form, the participants were given the Greek version of the Adult ADHD Self-Report Scale to fill out. Once they finished the questionnaire, they were instructed to notify the experimenter and wait for further instructions.

The administration of the tests was counterbalanced for both groups. Half of the participants of each group completed the paper-and-pencil TMT first, while the other half started with the TMT-VR. Counterbalancing was achieved through participant codes that were assigned by the experimenter (the odd-numbered participants began with the TMT-VR, the even-numbered participants with the traditional TMT). Before each TMT-VR assessment, the participants completed the Cybersickness in Virtual Reality Questionnaire.

When the paper-and-pencil TMT was administered first, the experimenter provided the participant with a pencil and gave verbal instructions. The participant was then asked to complete Part A and Part B of the TMT, with each part timed separately using a timer on the experimenter’s mobile phone. The times were recorded in the Qualtrics form by the experimenter. After finishing the paper-and-pencil test, the participant moved to the designated VR area.

In the VR area, the experimenter helped the participant put on the VR headset and headphones. The participants were informed that the experimenter would assist with adjusting the VR headset and guide them to press a button to start the calibration. During the calibration phase, the participants focused on a dot on the screen and followed it with their gaze. After the calibration, the participants had a trial phase that lasted approximately 10–15 min in order to ensure familiarity and accuracy with the software and device. During that phase, the participants listened to instructions for TMT-VR Part A through the headphones, had a short trial before each task, and then proceeded with the actual task. The same procedure was followed for TMT-VR Part B. Throughout the VR tasks, the participants were advised they could move their heads to adjust the screen if necessary. After the trial phase, the participants were exposed to the actual assessment which only lasted approximately 5–7 min and contained the TMT-VR A and TMT-VR B parts without the instructions.

After completing the TMT-VR, the participants filled out the Cybersickness in Virtual Reality Questionnaire again. This process was followed for all participants, regardless of the order in which the tests were administered.

After completing the tests and the CSQ-VR, the participants were instructed to return to the same screen they used earlier to continue the study. They were then presented with the System Usability Scale, followed by the Short Version of the User Experience Questionnaire, and finally the Service User Technology Acceptability Questionnaire (SUTAQ), in that order. At the end, participants were given a debriefing statement.

2.4. Statistical Analyses

All statistical analyses were conducted using R language version 4.4.1 [68] within the RStudio software version 2024.9.1.394 [69]. The dataset was first examined for normality using the Shapiro–Wilk test, and variables that were not normally distributed were transformed where necessary using the bestNormalize package [70]. Descriptive statistics, including means, standard deviations (SD), and ranges, were calculated for all demographic variables, task performance metrics, and user feedback scores. All data visualizations were created using the ggplot2 package [71], which provided detailed graphical representations of the key findings, including group differences and correlations.

2.4.1. Usability, User Experience, and Acceptability

Frequencies were calculated to analyze responses on the usability (SUS), user experience (UEQ-S), and acceptability (SUTAQ) scales. Pearson’s correlations were conducted to explore relationships between performance on the TMT-VR tasks (task time, accuracy) and the self-reported scores on these scales. The results were visualized using scatterplots and correlation matrices created with ggplot2 [71] and corrplot [72].

2.4.2. Independent Samples t-Tests

Independent samples t-tests were conducted to compare performance metrics (task time, accuracy, and mistakes) between the individuals diagnosed with ADHD and the neurotypical individuals. Separate t-tests were performed for both TMT-VR tasks (A and B) as well as the traditional paper-and-pencil TMT. The t-tests also compared the ASRS scores between the two groups to evaluate group differences.

2.4.3. Repeated Measures ANOVA

A repeated measures ANOVA was conducted to assess potential differences in task time, accuracy, and mistakes between the TMT-VR and traditional TMT tasks. The group (ADHD vs. neurotypical) was included as a between-subjects factor. For significant effects, post hoc pairwise comparisons were conducted using the Bonferroni correction to account for multiple comparisons. The ANOVA was conducted using the afex package [73], and effect sizes are reported as partial eta-squared (η²p). Significance was determined as p < 0.05.

2.4.4. Correlation Analyses

Pearson’s correlation coefficients were calculated to explore relationships between demographic variables (e.g., age and education) and performance metrics (e.g., task time and accuracy) across both the TMT-VR and the paper-and-pencil versions. Additionally, correlations were analyzed between performance outcomes and self-reported scores from the ASRS and user feedback scores (SUS, UEQ-S, and SUTAQ). Correlation matrices were visualized using the corrplot package [72] to provide a clear summary of these relationships. Pearson correlations were interpreted according to Cohen’s (1992) guidelines: correlations of ±0.10 to ±0.29 were considered small, ±0.30 to ±0.49 moderate, and ±0.50 or above large [74].

2.4.5. Linear Regression Analysis

To further examine the ecological validity of the TMT-VR, linear regression analyses were performed to assess the predictive power of TMT-VR performance (accuracy and task time) on the ASRS scores. The models included accuracy and task time from both The MT-VR A and the TMT-VR B as predictors, and a hierarchical approach was used to determine the best predictors of the ASRS scores. The percentage of variance explained by the model (R²) and standardized beta coefficients (β) were reported for each predictor, with significance set at p < 0.05. The regression analysis was performed using the lm function in base R.

3. Results

3.1. Descriptive Statistics

The sample consisted of 53 Greek-speaking young adults, with 47.17% (N = 25) diagnosed with ADHD and 52.83% (N = 28) classified as neurotypical. The participants’ ages ranged from 18 to 40 years (M = 23.87, SD = 3.82), with total years of education ranging from 12 to 22 (M = 15.98, SD = 3.26).

Table 1. Descriptive statistics.

	Mean	SD	Minimum	Maximum
Age	23.868	3.8232	18	40
Education	15.981	3.2610	12	22
TMT Task A–Task Time	30.435	9.8485	15.080	59.730
TMT Task A–Mistakes	0.132	0.590	0	4
TMT Task B–Task Time	65.485	25.3947	28.200	148.500
TMT Task B–Mistakes	0.604	1.291	0	6
TMT-VR Task A–Accuracy	0.185	0.0572	0.152	0.318
TMT-VR Task A–Task Time	83.835	26.7971	38.174	142.443
TMT-VR Task A–Mistakes	1.151	2.3729	0	14
TMT-VR Task B–Accuracy	0.185	0.0568	0.151	0.313
TMT-VR Task B–Task Time	98.607	28.2016	43.681	171.150
TMT-VR Task B–Mistakes	1.585	2.4685	0	14
SUS	41.208	5.1004	29	49
UEQ-S	143.547	19.4971	101	174
SUTAQ	49.113	8.5522	20	60
ASRS	39.528	9.5507	15	56

TMT = Trail Making Test; TMT-VR = Trail Making Test in Virtual Reality; SUS = System Usability Scale; UEQ-S = Short Version of the User Experience Questionnaire; SUTAQ = Service User Technology Acceptability Questionnaire; ASRS = Adult ADHD Self-Report Scale.

portrays the descriptive statistics of the participants. Furthermore, the ADHD and neurotypical groups were matched in terms of sex, age, and education. Specifically, there were 12 female participants in each group, while the ADHD and neurotypical groups had 13 and 16 male participants, respectively. Also, the ages and education of the ADHD group ranged from 19 to 35 years (M = 23.80, SD = 3.75) and from 12 to 22 years (M = 16.70, SD = 2.49), respectively. Comparably, the age and education of the neurotypical group ranged from 18–40 years (M = 23.90, SD = 3.96) and from 12 to 20 years (M = 16.30, SD = 3.74), respectively. Notably, there were no significant differences between the ADHD and neurotypical groups in terms of age, t(51) = −0.05, p = 0.961, or education, t(51) = 1.17, p = 0.247.

3.2. Usability, User Experience, and Acceptability

The usability, user experience, and acceptability ratings for the TMT-VR were notably high. For usability, 64.15% of the participants responded in the highest quartile, with an additional 33.96% in the third quartile, indicating widespread positive ratings. For user experience, 56.60% of the responses were in the highest quartile, while 41.50% were in the third quartile. Acceptability ratings followed a similar pattern, with 66.03% of the participants reporting scores in the highest quartile and 28.30% in the third quartile.

The correlation analysis revealed that age did not correlate with usability (SUS), r(51) = −0.084, p = 0.548, user experience (UEX-S), r(51) = 0.170, p = 0.222, or acceptability (SUTAQ), r(51) = 0.181, p = 0.195. Similarly, education did not significantly correlate with usability (SUS), r(51) = 0.096, p = 0.493, user experience (UEX-S), r(51) = 0.228, p = 0.100, or acceptability (SUTAQ), r(51) = 0.109, p = 0.437. However, usability (SUS) and user experience (UEX-S) were significantly correlated, r(51) = 0.546, p < 0.001, as were usability (SUS) and acceptability (SUTAQ), r(51) = 0.330, p = 0.016, and user experience (UEX-S) and acceptability (SUTAQ), r(51) = 0.309, p = 0.024. These results indicate that while age and education are related, neither factor appears to influence usability, user experience, or acceptability, suggesting that VR technology usability is not significantly impacted by these demographic variables.

Significant negative correlations were found between task time on the TMT-VR and the usability (SUS) scores, with the TMT-VR A, r(51) = −0.312, p = 0.023, and the TMT-VR B, r(51) = −0.291, p = 0.034, indicating that the participants who completed the tasks more quickly rated the system higher in terms of usability. Similar negative correlations were found between the user experience (UEX-S) scores and task time, further suggesting that efficient task performance was linked to better overall user experience.

A significant negative correlation between the acceptability (SUTAQ) scores and the TMT-VR A task time, r(51) = −0.291, p = 0.035, revealed that the participants who completed the task more quickly also rated the software higher in terms of acceptability, reflecting a more favorable perception of the technology among those who performed well.

3.3. Performance on TMT-VR and TMT

For the TMT-VR A, the accuracy scores ranged from 0.152 to 0.318 (M = 0.185, SD = 0.057), task times ranged from 38.17 to 142.44 s (M = 83.84, SD = 26.80), and the number of mistakes ranged from 0 to 14 (M = 1.15, SD = 2.37). For the TMT-VR B, the accuracy scores ranged from 0.151 to 0.313 (M = 0.185, SD = 0.056), task times varied from 43.68 to 171.15 s (M = 98.61, SD = 28.20), and the number of mistakes ranged from 0 to 14 (M = 1.59, SD = 2.47).

The self-reported scores on the System Usability Scale (SUS) ranged from 29 to 49 (M = 41.21, SD = 5.10). The user experience scores, measured using the UEQ-S, ranged from 101 to 174 (M = 143.55, SD = 19.50), and acceptability (measured via SUTAQ) ranged from 20 to 60 (M = 49.11, SD = 8.55). ADHD symptomatology was assessed via the Adult ADHD Self-Report Scale (ASRS), with scores ranging from 33 to 74 (M = 57.53, SD = 9.55). Importantly, reported cybersickness symptoms were either absent or mild for all participants, suggesting that cybersickness did not interfere with task performance.

Group Comparisons: ADHD vs. Neurotypical

In

Table 2. Performance on the TMT and the TMT-VR per diagnostic Group: ADHD vs. neurotypical.

	Diagnosis	Mean (SD)	Min–Max	t-Statistic	p-Value	Cohen’s d
TMT Task A–Task Time	ADHD	29.38 (10.24)	16.05–59.73	−0.73	p = 0.467	−0.20
	Neurotypical	31.37 (9.57)	15.08–53.80
TMT Task A–Mistakes	ADHD	0.08 (0.27)	0–1	−0.60	p = 0.549	−0.17
	Neurotypical	0.17 (0.77)	0–4
TMT Task B–Task Time	ADHD	65.40 (27.92)	31.62–148.50	−0.02	p = 0.983	−0.01
	Neurotypical	65.55 (23.42)	28.20–136.81
TMT Task B–Mistakes	ADHD	0.52 (1.12)	0–5	−0.44	p = 0.660	−0.12
	Neurotypical	0.67 (1.44)	0–6
TMT-VR Task A–Accuracy	ADHD	0.21 (0.07)	0.15–0.31	4.89	p < 0.001 ***	1.34
	Neurotypical	0.15 (0.01)	0.15–0.17
TMT-VR Task A–Task Time	ADHD	80.75 (29.29)	38.17–142.44	−0.98	p = 0.835	−0.27
	Neurotypical	86.58 (24.56)	44.84–136.07
TMT-VR Task A–Mistakes	ADHD	1.88 (3.17)	0–14	2.38	p = 0.011 *	0.65
	Neurotypical	0.50 (0.96)	0–3
TMT-VR Task B–Accuracy	ADHD	0.21 (0.07)	0.15–0.31	3.69	p < 0.001 ***	1.02
	Neurotypical	0.15 (0.01)	0.15–0.17
TMT-VR Task B–Task Time	ADHD	94.47 (27.71)	43.68–152.68	−0.93	p = 0.822	−0.26
	Neurotypical	102.29 (28.61)	52.58–171.15
TMT-VR Task B–Mistakes	ADHD	2.28 (3.08)	0–14	2.29	p = 0.013 *	0.63
	Neurotypical	0.96 (1.57)	0–6
ASRS	ADHD	43.96 (6.81)	28–56	3.57	p < 0.001 ***	0.98
	Neurotypical	35.57 (9.99)	15–52

ADHD = Attention Deficit Hyperactivity Disorder; TMT = Trail Making Test; TMT-VR = Trail Making Test in Virtual Reality; ASRS = Adult ADHD Self-Report Scale. Task time measured in seconds; Accuracy in the TMT-VR reflects the distance in meters from the center of the target during selection; Min–Max = minimum to maximum; * for p < 0.5 and *** for p < 0.001.

, the performance metrics are presented for the two diagnostic groups (i.e., ADHD vs. neurotypical). Independent samples t-tests were conducted to compare performance between the ADHD and the neurotypical participants across the TMT and the TMT-VR tasks. No significant differences were observed between the groups in terms of task time on either the paper-and-pencil TMT or the TMT-VR (see Table 2). However, significant differences were found in accuracy, the number of mistakes, and cognitive control in both the TMT-VR A and the TMT-VR B (see Table 2 and Figure 2, Figure 3, Figure 4 and Figure 5).

For the TMT-VR A, the ADHD participants demonstrated significantly lower accuracy than the neurotypical individuals (see Table 2 and Figure 2) and made significantly more mistakes (see Table 2 and Figure 3). Similarly, in the TMT-VR B, the ADHD participants performed with lower accuracy (see Table 2 and Figure 4) and made more mistakes (see Table 2 and Figure 5). In contrast, no significant group differences were found in terms of mistakes on the paper-and-pencil TMT (see Table 2), highlighting the stronger ecological validity and discriminatory power of the TMT-VR for distinguishing attentional deficits.

To further examine attentional challenges, the ASRS scores were compared between the groups. As expected, the ADHD participants scored significantly higher on the ASRS (see Table 2), reflecting greater attentional difficulties and everyday impairments in the ADHD group.

3.4. Convergent Validity

Pearson’s correlations were performed to evaluate convergent validity by comparing performance on the TMT and the TMT-VR. A significant positive correlation was found between task times for the TMT-A and the TMT-VR A, r(51) = 0.360, p = 0.004, as well as between the TMT-B and the TMT-VR B, r(51) = 0.281, p = 0.021. These findings suggest strong convergent validity (see Figure 6), with performance on the VR adaptation aligning with performance on the traditional paper-and-pencil test. Additional significant correlations were observed between the TMT-A and the TMT-B (r(51) = 0.587, p < 0.001) and between the TMT-VR A and the TMT-VR B (r(51) = 0.391, p = 0.003), further confirming consistency across traditional and VR-based cognitive tasks.

3.5. Ecological Validity

Ecological validity was assessed through correlations between the TMT-VR performance and the ASRS scores. Significant positive correlations were observed between the ASRS scores and the TMT-VR A task time, r(51) = 0.358, p = 0.004, as well as between the ASRS scores and the TMT-VR B task time, r(51) = 0.411, p = 0.001. This indicates that the participants with higher ASRS scores—indicative of more severe attentional deficits—took longer to complete the TMT-VR tasks, suggesting that TMT-VR performance accurately reflects everyday attentional challenges (see Figure 7).

Significant positive correlations were found between the ASRS scores and TMT-VR accuracy and mistakes, further supporting the ecological validity of the TMT-VR. Specifically, the ASRS scores were significantly positively correlated with TMT-VR A accuracy r(51) = 0.331, p = 0.008, and TMT-VR B accuracy r(51) = 0.325, p = 0.009. Additionally, a positive correlation was found between the ASRS scores and TMT-VR A mistakes r(51) = 0.248, p = 0.036, highlighting the ability of the TMT-VR to capture attentional difficulties. This association with real-world impairments, as measured by the ASRS, further supports the ecological validity of the TMT-VR.

Interestingly, traditional TMT task performance showed a significant negative correlation with years of education, with TMT-A task time, r(51) = −0.263, p = 0.029, and TMT-B task time, r(51) = −0.278, p = 0.022, decreasing as years of education increased. In contrast, no significant correlation was found between education and TMT-VR performance, p > 0.05, suggesting that educational background influenced performance on the traditional TMT but not the TMT-VR. This further supports the ecological validity of the TMT-VR, as it appears to offer a more realistic measure of cognitive functioning unaffected by educational disparities. Demographic factors did not have a significant impact on the results, highlighting that the TMT-VR is a highly applicable instrument across diverse populations.

Regression Analyses for Predicting ASRS Scores

A hierarchical linear regression was performed to explore the ability of TMT-VR task performance (task time, accuracy, and mistakes), TMT task performance, user evaluation (UX, SUS, and AS), and demographic information (age, education) to predict attentional challenges as measured by ASRS scores. Crucially, demographic variables (age and years of education), as well as traditional TMT performance metrics, were not significant contributors to the final model. Only TMT-VR metrics were included, underscoring the superior predictive power of the VR version over the traditional task. The exclusion of demographic factors highlights the ability of the TMT-VR to provide an accurate reflection of attentional challenges across diverse populations, independent of age or educational background.

The model accounted for 38.1% of the variance in ASRS scores, F(3, 49) = 10.10, p < 0.001, indicating that TMT-VR performance metrics are robust predictors of everyday attentional difficulties (see Figure 8). The model did not suffer from multicollinearity, since the predictors TMT-VR A accuracy, TMT-VR A task time, and TMT-VR B task time presented low variance inflation factors (VIF) of 1.01, 1.49, and 1.50, respectively. Among the significant predictors, TMT-VR A accuracy emerged as the strongest predictor of ASRS scores (β = 0.439, p < 0.001), followed by TMT-VR A task time (β = 0.269, p = 0.012) and TMT-VR B task time (β = 0.341, p = 0.003). These results suggest that higher accuracy and faster task completion in the TMT-VR are associated with lower ASRS scores, indicating fewer attentional challenges. Accuracy in the TMT-VR A being the most significant predictor reflects the importance of precise cognitive control in real-world attentional functioning.

In summary, the fact that neither demographic variables nor traditional TMT scores were significant predictors further reinforces the ecological validity of the TMT-VR, which seems to better capture attentional deficits in a manner reflective of everyday functioning, unlike the traditional, lab-based assessments.

4. Discussion

This study is the first to validate the TMT-VR, a VR adaptation of the traditional TMT, for assessing cognitive functioning in adults with ADHD. Beyond psychometric evaluation, the study examined user perceptions of usability, acceptability, and overall experience. The results were promising: the TMT-VR demonstrated strong ecological and convergent validity, aligning well with real-world cognitive demands, an improvement over traditional assessments. Usability ratings were high, with the participants finding the system intuitive and easy to navigate and with no reports of VRISE symptoms (e.g., nausea and fatigue), highlighting the TMT-VR as both a practical and comfortable tool for clinical settings.

Interestingly, no significant differences were found in task completion times between the neurotypical participants and those with ADHD on the TMT-VR. This suggests that in an immersive virtual reality environment, individuals with ADHD may perform similarly to their neurotypical counterparts, potentially due to the increased engagement and real-life simulation that the VR environment provides. This result may indicate that the TMT-VR reduces cognitive load or increases participant motivation, offering a more representative measure of real-world performance, especially in tasks that require sustained attention and cognitive flexibility. Furthermore, the lack of significant differences in task completion times between the neurotypical and neurodivergent participants on the TMT-VR could be related to the characteristics of the virtual reality environment. The immersive nature of VR environments might limit external distractions that are commonly present in real-world settings.

Although task completion times showed no significant differences, the neurotypical participants outperformed those with ADHD on accuracy and error rates in the TMT-VR. This indicates the TMT-VR’s effectiveness in distinguishing between groups based on cognitive performance, particularly capturing attentional and executive deficits typical of ADHD—deficits that traditional TMT may overlook. Unlike the traditional TMT, the immersive nature of the TMT-VR enhances sensitivity to impairments in attention, task-switching, and cognitive flexibility by offering a dynamic, realistic environment. This makes the TMT-VR a valuable tool for clinicians assessing ADHD and similar conditions.

4.1. Usability, User Experience, and Acceptability

In line with previous findings on the usability, user experience, and acceptability of virtual reality-based neuropsychological measures, such as the VRESS [12], this study revealed that the TMT-VR received similarly high scores in these areas from young adult participants. The importance of evaluating participants’ perceptions of technological properties in VR assessments is often overlooked in neuropsychological research [61,62]. Understanding these dimensions is critical, as technological properties such as usability can directly impact user engagement and task performance [60].

The participants gave high usability ratings, suggesting that the TMT-VR is both intuitive and highly functional. Design features, such as the eye-tracking interaction mode, allowed the participants to navigate the tasks without using their hands or other body parts, simplifying the process and reducing cognitive and physical effort. Research indicates that minimizing physical and cognitive demands in VR can enhance the user experience, especially in populations that may face additional cognitive challenges [10]. These findings suggest that the TMT-VR is effective and easy to use, requiring little improvement in terms of interface or design, making it suitable for diverse settings, including clinical environments [5].

Similarly, the participants rated their experience in the highest quartiles. Positive user experiences are essential in VR-based assessments, as high immersion and engagement can improve task performance [9]. Immersive environments can also improve motivation and reduce fatigue during cognitive tasks, which is particularly important when assessing individuals with attentional disorders like ADHD [53]. The high ratings on these dimensions suggest that the TMT-VR effectively engages users, supporting its use as an ecologically valid neuropsychological tool.

Finally, the acceptability of the TMT-VR, as measured by the SUTAQ scale, was also rated highly, suggesting that the participants viewed the system as a suitable and reliable instrument for neuropsychological evaluation. Previous studies have shown that the acceptance of VR-based tools in clinical populations can be influenced by usability and engagement factors [3,11]. The TMT-VR’s combination of ease of use, immersive engagement, and minimal adverse effects (such as VRISE) makes it a promising tool for broader clinical applications, particularly in populations with cognitive or attentional deficits.

Virtual Reality in Neurodivergent Individuals

The findings of this study are consistent with the growing body of evidence supporting the use of VR-based tools for assessing cognitive performance in individuals with ADHD [54]. The TMT-VR demonstrated strong validity and was well-received by the participants, reflecting its potential as an effective tool for evaluating cognitive functions such as attention, memory, and executive function in a controlled, yet immersive environment. Previous studies have emphasized the importance of integrating eye-tracking and body movement into these assessments, as they can provide deeper insights into cognitive and behavioral functions in neurodivergent populations [75].

Also, the current study employed eye-tracking as the primary interaction mode for the TMT-VR. Research has shown that eye movement can serve as a reliable predictor of cognitive performance, offering a non-invasive and intuitive way to assess attention, task-switching, and other executive functions [75]. By capturing subtle variations in gaze and visual attention, eye-tracking enhances the ecological validity of VR-based assessments, making them more reflective of real-world cognitive challenges. This method also aligns with broader trends in neuropsychological research that emphasize the need for more immersive, interactive, and contextually relevant tools for assessing cognitive performance in neurodivergent individuals [5,11].

4.2. Convergent Validity

The TMT-VR as an immersive computerized neuropsychological assessment includes several advantages over the traditional paper-and-pencil version, including the elimination of human error during the test administration and result calculation, standardization of the test conditions, and millisecond accuracy in recording task performance [30], which are critical in understanding attention lapses and variations in cognitive performance, particularly in clinical populations, such as those with ADHD [76,77]. In this study, both TMT-VR tasks demonstrated significant correlations with their paper-and-pencil counterparts, indicating satisfactory levels of convergent validity and confirming the TMT-VR’s reliability in measuring the same cognitive constructs. This supports its use as an alternative to traditional methods [78]. While test format differences can affect convergence [78], prior research has shown that VR-based neuropsychological tools often align well with traditional assessments [5,8,12,79].

These findings suggest that the TMT-VR is a reliable, valid, and precise tool for cognitive assessment, consistent with previous validations of alternative TMT forms, such as computerized versions and the Color Trail Test [25,80]. Unlike studies that require expensive equipment [25], the TMT-VR developed in this study is practical and accessible, utilizing affordable VR technology for broader clinical use. As the first immersive VR variant of the TMT, it offers a dynamic, ecologically valid alternative for cognitive assessment that can be readily adopted in research and clinical settings [24].

4.3. Ecological Validity

Unlike traditional paper-and-pencil tests, which are often limited by their artificial and controlled settings, VR assessments can simulate complex, real-life scenarios, allowing for a more accurate evaluation of cognitive functions, such as attention, memory, and executive function [81,82]. Previous studies, such as those using tools like the Nesplora Aquarium and VR-EAL, have demonstrated strong psychometric properties and superior ecological validity compared to traditional methods [4,83], which render VR-based assessments particularly valuable for measuring real-world performance in clinical populations, including individuals with ADHD [53,55].

This study has built on these findings by examining both verisimilitude and veridicality—two key aspects of ecological validity. While previous research has often focused on verisimilitude or how closely a VR task resembles real-world activities, our study also assessed veridicality, which examines the ability of an instrument’s scores to predict everyday performance [2]. In this context, the TMT-VR’s scores were compared to those of the ASRS, a well-validated measure of attentional and everyday cognitive functioning, revealing a significant correlation between the two.

By demonstrating strong verisimilitude and veridicality, the TMT-VR not only effectively simulates real-world conditions but also accurately predicts everyday cognitive performance in individuals with ADHD. These findings highlight the TMT-VR’s potential as a more ecologically valid tool compared to traditional neuropsychological assessments, providing deeper insights into cognitive functioning within naturalistic settings.

4.4. Limitations and Future Studies

While this study offers valuable initial validation of the TMT-VR as a neuropsychological tool, further research is essential to fully understand its capabilities and limitations. Given the modest sample size, future studies with larger and more diverse populations would strengthen the generalizability of these findings. Additionally, since this study focused primarily on young adults, assessing the TMT-VR across different age groups—from adolescence to older adulthood—could determine its effectiveness in evaluating cognitive functions throughout various life stages.

The novelty effect of VR may influence usability and user experience. Specifically, while novelty could negatively impact usability (SUS) due to initial unfamiliarity with the system, it may enhance user experience (UEX-S) by increasing engagement and motivation. However, our study assessed ecological validity through correlations between VR performance, traditional paper–pencil tasks, and real-world cognitive functioning as measured by the ASRS. This allowed us to evaluate the extent to which VR-based assessments generalize to everyday functioning rather than being purely influenced by novelty.

Importantly, while VR is generally considered safe for research and clinical applications, its use in children under the age of six is often discouraged due to concerns regarding visual development and cybersickness [84]. Although VR technologies and VR-based cognitive assessments have been successfully implemented in school-aged children (e.g., [53,84,85]), additional research is needed to determine its suitability for extensive use. While this limitation is not relevant to the present study, it should be considered when evaluating the broader applicability of VR-based neuropsychological tools in children.

Expanding the TMT-VR’s application to other clinical groups, such as individuals with Mild Cognitive Impairment, dementia, or traumatic brain injury, would provide a more comprehensive picture of its utility. Validating the TMT-VR in these contexts could underscore its versatility as a reliable tool for identifying cognitive impairments across a range of conditions. Furthermore, establishing the tool’s long-term reliability, including test–retest consistency, would confirm its robustness for repeated use in clinical practice.

To enhance accessibility, future iterations of the TMT-VR could explore additional interaction modes, such as hand tracking or gesture control. These developments would allow broader applicability for individuals with varying physical or cognitive needs, further expanding its clinical potential. By continuing to refine and validate the TMT-VR across diverse populations and clinical settings, future research can ensure that this innovative tool meets the demands of neuropsychological assessment in dynamic, real-world conditions.

5. Conclusions

This study provides key insights into the TMT-VR’s validity and utility as an innovative tool for neuropsychological assessments, especially in clinical populations like adults with ADHD. The TMT-VR demonstrated strong ecological and convergent validity, showing its effectiveness in capturing real-world cognitive challenges, particularly in attentional deficits and executive dysfunctions. This immersion-based approach offers a clear advantage over traditional assessments, which often lack ecological validity. Its capacity for real-time feedback and continuous assessment holds promise for personalized clinical interventions. High ratings in usability, user experience, and acceptability confirm that the participants found the TMT-VR intuitive, engaging, and well-suited to clinical contexts. These positive user experiences suggest strong potential for the TMT-VR’s integration into neuropsychological practice, where engagement is critical for accurate and consistent assessments. Overall, these findings highlight the TMT-VR as a valuable step toward enhancing cognitive assessments through VR technology. As VR applications in neuropsychology continue to expand, tools like the TMT-VR could offer more accurate, immersive, and dynamic methods for evaluating cognitive functions across diverse populations. Future research should focus on further exploring VR technologies to ensure their widespread acceptance and adoption in clinical practice.