Validity of a Novel Head-Mounted Display Augmented Reality Device with a Body-Worn Accelerometer (HMD-WRIT) and Dynamic Executive Function Assessment Against Standardized Tests in Healthy Older Versus Young Adults

Abstract

Background: Neurologic pathologic conditions, including traumatic brain injury (TBI), and natural aging often present with concurrent physical and cognitive challenges. While functional physical capabilities may be accurately and reliably assessed, traditional cognitive measures rely on static testing methods that fail to incorporate functional movements. Objective: These two studies aimed to validate a novel portable version of the previously validated laboratory-based Walking Response and Inhibition Test (WRIT), utilizing a head-mounted device (HMD) and a body-worn position sensor for dynamic measure of Executive Function (EF) during functional mobility. Methods: Two observational studies were conducted across cohorts of healthy older and younger adults. To validate the portable device (HMD-WRIT), participant performance was benchmarked against two 3D Motion Analysis Systems. Additionally, a healthy elderly cohort study compared three standardized cognitive tests, while both studies utilized a functional mobility test. Results: Data from both observational studies demonstrated that the portable HMD-WRIT accurately captured postural responses to visual commands during walking. Performance metrics successfully correlated with motion-capture data and established static cognitive benchmarks. Conclusions: The HMD-WRIT is a viable, portable option for measuring EF during functional movement. This system bridges the gap between static cognitive testing and real-world mobility assessments in both young and aging populations.

1. Introduction

Persons with neurologic pathologies (e.g., Alzheimer’s, Parkinson’s, stroke), traumatic brain injuries (TBI) (e.g., sports injuries, falls, and auto accidents), and normal aging often present clinically with concomitant physical and cognitive impairments (Huang et al., 2017; Kudlicka et al., 2018). In older adults, falls are commonly attributed to physical limitations but may also be due to disfunctions of Executive Functions (EF) (inability to plan, focus, recall from memory, and multitask) (Eman Abdulle & Van Der Naalt, 2020). Recent findings suggest that EF plays a critical role in the regulation of gait in older adults, especially under complex and challenging conditions, and that EF deficits may, therefore, contribute to fall risk (Mirelman et al., 2012). Cognitive deficit measures for these groups traditionally are seated computerized tests, administered by licensed practitioners, and do not incorporate functional (e.g., walking) or dual-tasking activities, commonly encountered during daily activities (e.g., cooking, house cleaning, etc.). When symptoms manifest from central and/or peripheral vestibular dysfunctions along with cognitive and behavioral changes, objective measurement and intervention design challenges arise. These are relevant for determining fall-prevention strategies, intervention efficacy, safe return to sports participation, and safe basic Activities of Daily Living (B-ADLs) participation guidelines. While vestibular systems testing and interventions are effective in reducing symptoms and improving functionality in these populations, little research-supported evidence exists of how EF during dynamic activities impacts functionality during recovery (Hall et al., 2016).

Components of EF are the mental processes that enable planning, focusing attention, remembering instructions, and handling multiple tasks. ADLs require filtering of distractions, task prioritization, setting and achieving goals, and impulse control. Studies have shown that interventions that concurrently address both physical limitations and EFs improve cognition, regardless of whether complementary cognitive rehabilitation methods are employed (Alsalaheen et al., 2016; Gottshall & Hoffer, 2010). Broad measures of EF are commonly the only diagnostics used to monitor these multidimensional approaches and are administered in controlled, static clinical or laboratory settings, often with the client seated. Therefore, novel testing methods involving recognition of visual cues and resultant decision making, demonstrated by reactive functional movements, are needed to better assess the use of cognitive processes during B-ADLs.

The Walking Response Inhibition Test (WRIT) and the Stroop Walking Task (due to their dual tasking and possible cognitive overload) assess cognitive influences on physical tasking (Leyva et al., 2017; Perrochon et al., 2015). Both tests employ walking and reacting to visual stimuli that incorporate basic EF components. They require a controlled, laboratory-based large space setting, large, cumbersome equipment, and specialized measurement devices, making them impractical for common clinical use. The original WRIT was validated as an EF measure during dynamic activity (Leyva et al., 2017); therefore, a portable version usable in any environment is merited. Head-mounted display-based virtual augmented reality (AR) devices, such as the Microsoft HoloLens^® (Redmond, WA, USA), along with wearable technology like inertial measurement unit (IMU) devices for tracking spatial movements, may allow these tests to be administered in any clinical setting, needing only a hallway or walking space for the test. The use of AR or mixed reality (MR) devices is advantageous because they merge the real and virtual worlds to produce new environments, allowing physical and digital objects to co-exist and interact in real time. The use of VR simulators for rehabilitation has gained favor and shown efficacy in enhancing vestibular and balance functions (Sulway & Whitney, 2019; Dunlap et al., 2019). However, immersion into a virtual environment is not completely innocuous and may provoke motion intolerance symptoms, known as “simulator sickness” (Malinska et al., 2014; Aldaba et al., 2017). The subject’s adaptation to a simulator-generated environment may create a sensory conflict between visual stimuli and movement detected by the vestibular organs, eliciting nausea and motion intolerance (Kiryu & So, 2007). Maladaptation to these immersive stimuli can also result in Mal de Debarquement syndrome, a vestibular system disorder resulting in phantom perception of self-motion, having prolonged debilitating effects (Tal et al., 2014). Additionally, immersion in the VR environment prevents the performance of dynamic functional activities, as the surrounding environment is blocked. Unlike VR, AR devices allow simultaneous environment viewing and HMD displayed visual cues, avoiding simulator sickness and motion intolerance, while allowing safe dynamic activities performance (Baus & Bouchard, 2014). Use of wearable sensor technologies, such as IMUs with accelerometers, is useful during rehabilitative and sports-related applications, and can be interfaced with AR visual simulators.

Leyva et al. (2017) demonstrated that the original WRIT was better reflective of EF in the everyday environment than traditional computerized analyses, as it more specifically measures mobility-linked inhibitory control/functional Executive control during walking/moving (Leyva et al., 2017). Adaptation to a portable AR environment using HMD and ‘wearable technologies’ (e.g., inertial measurement unit (IMU) (HMD-WRIT)) makes its administration more viable in common clinical settings with limited resources (Buskirk, 2021). This novel, updated device primarily taxes the inhibitory control, particularly the response inhibition portion of EF while embedded within a walking/postural-control context. By contrast, the traditionally employed computerized EF tasks measure related but not identical constructs under low motor demands. The List-Sorting Memory Test was chosen for comparison as ‘working memory’ (primarily measured by the test) is required for performing a postural response to previously learned visual ‘commands’ during the HMD-WRIT. The Flanker and Stroop tests were chosen for comparison as they both primarily measure the ‘inhibitory control’ portion of EF. Therefore, these two studies in different locations were conducted, first on healthy older adults and subsequently on a younger adult cohort, to objectively measure the EF of ‘inhibitory control’ during a walking activity. The purpose of these two studies was to evaluate a new portable motion analysis and Executive Function (EF) assessment tool by comparing its accuracy, real-world relevance, and age-related performance against established standards.

Specifically, the research aimed to achieve the following:

• Validate the HMD-WRIT device: Test the accuracy of this portable tool against two different “gold-standard” 3D Motion Analysis Systems.
• Assess cognitive testing accuracy: Compare the HMD-WRIT’s Executive Function assessments against standard computerized EF tests in older adults.
• Link cognitive data to daily function: Confirm if the EF results correlate with real-world mobility by comparing them to the Timed Up-and-Go (TUG) test.
• Analyze age differences: Compare the performance and assessment results between the two distinct age groups.

To our knowledge, this is the first study to compare EF results of an older adult cohort against three standardly administered computerized cognitive tests and the Timed Up-and-Go functional mobility test (TUG) utilizing a portable device. The purpose of the second study (younger adults) was to compare results to the older adult cohort.

The HMD-WRIT device responses were hypothesized to positively correlate to both the 3D Motion Analysis Systems—BTS Bioengineering (BTS) and Qualisys Track Manager (QTM) results, the Montreal Cognition Assessment (MoCA), and TUG tests—and the young adults’ results would differ from the older adults with shorter latencies of responses (LAT), higher accuracy of correct responses (ACC), and lower scores variability.

2. Methods

2.1. Recruitment

Recruitment for the initial older adult project occurred at the University of Miami from August through December 2019, and from March through June 2022, at Rosalind Franklin University for the younger adult project. Both studies were approved by the respective Universities’ Institutional Review Boards (IRB) for the Use and Protection of Human Subjects (Approval numbers 20200818 and CHP22-325) and conducted according to the World Medical Association Declaration of Helsinki. Approved recruitment flyers were posted throughout the campuses and communities. The majority of the older adult study subjects were recruited via phone and email lists (maintained by the Lab) who had participated in prior studies. Younger adults were also recruited utilizing email notifications. Sample sizes were determined via Power Analyses for both studies (Power: 80%, Alpha: 0.05, Effect Size: Cohen’s d 0.05 (medium)), and both cohorts exceeded the targeted minimum 31 subjects required for the study to have a high probability of detecting a statistically significant effect.

2.2. Inclusion

Forty-five healthy adult male (n = 22) and female (n = 23) subjects, communicative in English, aged 50–86 (mean = 69.51, SD ± 9.98 years), able to walk 300 feet unaided, with normal color vision, and living independently in the community participated in the older adult study. For the subsequent younger adult study, forty-four adult male (n = 21) and female (n = 23) subjects aged 18–49 (mean = 26.25, SD ± 6.47 years) meeting the same inclusion criteria participated.

2.3. Exclusion Criteria

Individuals were excluded with a history of joint replacement, any psychiatric disorder, history of diagnosed balance or neurologic problems, history of traumatic brain injury (TBI) within the last 6 months, color blindness, or non-correctable vision limitations. Adults unable to consent, aged outside 50 to 86 years, pregnant women, and prisoners were also excluded.

2.4. Procedures

A portable version of the original WRIT technology was created (HMD-WRIT) using a head-mounted augmented reality (AR) device (HMD) with heads-up display (HUD) and integrated inertial measurement unit (IMU) and was compared to a gold-standard 3-D Motion Analysis System (BTS) (Buskirk, 2021). The BTS and QTM systems are primary software used to control, collect, and process data from motion-capture systems. They employ automatic marker tracking, a real-time skeleton solver/analysis, and built-in plotting to analyze biomechanics, gait, and 3D kinematics. The forty-five healthy older adults performed nine randomized trials (three trials of Congruent (indicating one direction-changing command) and six trials of Incongruent (indicating combinations of two or three direction-changing commands)). Measurements from both devices, including Accuracy% (ACC) (correctness of response) and Latency (LAT) (time from displayed command symbols in the HMD to postural responses), were recorded. An EF score was tabulated by (ACC * LAT). Calculations utilized for these responses were consistent with those used in the original study (Leyva et al., 2017). Subjects completed two NIH Toolbox computerized cognition tests (the Eriksen Flanker Test, a response inhibition measure, and the List-Sorting Memory Test, a working memory test) and a Stroop Color Word Test, which tests the ability to inhibit cognitive interference measures, well-known as the ‘Stroop Effect’. Finally, subjects performed the TUG test to assess functional mobility.

The subsequent healthy young adult study utilized the same portable HMD-WRIT device and protocol. For consistency of measures, the same 3-D Motion Analysis Systems (BTS) and analysis software were used at both testing locations for the two studies. Comparison of Accuracy% (ACC), Latency (LAT), and EF was made to a Qualisys 3-D Motion Analysis System (QTM). Subjects performed the Montreal Cognitive Assessment (MoCA), validated as a highly sensitive tool for early detection of mild cognitive impairment (MCI). It was used also in the older adult study (with score ≥23 for inclusion criteria) and included the TUG test.

Upon arrival, a screening protocol (temperature and COVID-19 questionnaire) was administered, and masks and gloves were provided. A maximum of three researchers wearing surgical gowns or laboratory coats, masks, plastic face shields, and disposable gloves performed all tests. University-approved COVID-19 precautions were utilized and all equipment properly sanitized. Subjects completed a written Informed Consent form, a CDC-approved COVID-19 screening questionnaire, a Physical Activity Readiness Questionnaire (PAR-Q) form, a health questionnaire, and a photography release form. The use of photographs and videotaping were limited to subjects performing the HMD-WRIT wearing the AR HMD; therefore, their faces were obscured. Subjects wore comfortable clothing, allowing secure IMU placement onto a Velcro closure belt at the umbilicus level with four reflective markers (3D motion analysis-required) at the anterior and posterior superior iliac spines (ASIS and PSIS). Subjects wore their own comfortable shoes during the HMD-WRIT and TUG tests.

Subjects’ heights and weights were recorded. Subjects completed the Ishihara Plate color blindness test (as the commands shown on the HMD were different colors) and the MoCA. They were instructed on the set-up and procedures for the HMD-WRIT with IMU and moved to the testing area, where they were shown the projected HMD command signals (Figure 1). The HMD displayed combinations of cues: two round circles at the top (one red and/or one blue) and one of two green arrows at the bottom, pointing right or left. A green left-pointing arrow indicated a left body turn and a green right-pointing arrow indicated a right body turn. A blue circle above an arrow indicated to turn opposite to the pointing arrow. A red circle, whether presented alone or in combination, indicated to stop.

Figure 1. Representation of the WRIT/HMD test commands.

Prior to testing, subjects demonstrated a thorough understanding of the commands and correct postural responses. They were then fitted with the HMD and completed the nine test trials. For the older adult study only, after removal of the HMD and Velcro belt with IMU, subjects were seated and completed the three computerized cognition tests. Finally, subjects completed 3 TUG trials. Figure 2 shows the flow of subjects through the study. Each subject was allotted sixty minutes for testing completion.

Figure 2. A total of 41 of 45 underwent Final Analyses, as 4 subjects’ data were incomplete (raw data not recorded to the phone during testing trials) and therefore not included for analyses. Only participants with complete recorded data for all trials were included in Final Analyses.

2.5. HMD-WRIT

The HMD-WRIT utilized a portable HoloLens^® version 2 HMD (Microsoft Corp., Irving, TX, USA) with a custom heads-up display (Charles River Analytics, Baltimore, MD, USA) allowing subjects to view and identify displayed cues’ shapes and colors, eliminating the need for a cumbersome screen for commands display. An IMU (Life Performance Research, Tokyo, Japan) interfaced through telemetry to the HMD determined the subjects’ postural responses to displayed cues. This allowed evaluation of the subjects’ abilities to stop or change directions by providing data for Z (direction of walking), X (lateral left/right), and Y (vertical) planes. Data were stored on an Android phone application (Google, LLC, Mountain View, CA, USA), then uploaded to an Excel spreadsheet (Microsoft Office, Microsoft Corporation, Redmond, WA, USA) and compared to the 3D systems timeline data for each trial. Subjects initially rose onto their toes and returned to feet-flat position prior to immediately initiating the gait cycle, allowing timeline synchronization for the two devices.

Recorded time stamp measurements quantified the displayed randomized command times and the subjects’ earliest postural responses (change in movement direction or stop), and were recorded as Latency (reaction time) (LAT). The Accuracy % score (ACC) was determined as a percent of correct reactions to total commands. The minimum threshold for gait speed was 1 m·s⁻¹.

Figure 3 is an illustration of the HMD-WRIT testing environment. Course floor space was 9.4 m long by 5 m wide. A center yellow line provided a visual reference for assessing subjects’ direction change.

Figure 3. Representation of the original WRIT course (from Leyva et al., 2017) with commands shown on a wall-mounted screen (replaced in both studies by the portable Hololens 2 device).

Sobolewski et al. (2017) reported longer mean reaction times for an older adult group (1.40 ± 0.126 s) compared to younger groups during the Functional Reactive Agility Test (FRAT), a task similar to the HMD-WRIT (Sobolewski et al., 2017). In the absence of other published works of similar test procedures, this test data (1.53 s, mean + 1 SD) was used as the threshold value to determine the “purposeful” postural response from a “drop” step or “plant” step (where the subject “plants” a foot firmly on the ground (either behind or to one side) prior to making the “purposeful” whole-body response) or a corrective (“double”) whole-body response for the older adult study, and was applied to the sixty-eight identified trials. When subjects stopped or paused for >1.53 s prior to making a body response, the ACC was deemed incorrect, and the LAT value was marked at the response initiation. When the stop or pause lasted <1.53 s the response made immediately after was considered the “purposeful” movement. The LAT was marked at the second response initiation and responses determined ACC scores. Calculated EF scores were utilized for analyses.

The younger adult study utilized the same processes for determining response LAT and ACC. There is an absence of published works with similar test procedures to determine a LAT threshold value for healthy, young adults. Therefore, a mean + 1 SD of the sixty-one obtained responses was calculated (0.72 s) and used to determine LAT and ACC of “purposeful” responses.

During testing trials utilizing the HMD-WRIT device and during standardized testing procedures, subjects were asked and encouraged to report to the examiner any adverse symptoms or effects of the testing trials. No adverse effects were reported.

Computerized cognitive tests (Flanker, List Sorting Memory, and Stroop Color Word). Older adult subjects completed three computerized tests of cognition: two (Eriksen Flanker and List Sorting Memory) from the NIH Toolbox Cognitive battery and the Stroop Color Word Test. All subjects utilized the same test order. A five-minute rest period between tests minimized physical or cognitive fatigue. The NIH tests utilized an iPad, while the Stroop used an adapted gaming computer. Subjects received test protocol performance instructions and demonstrated successful understanding of all test battery components using the iPad, keyboard, and mouse responses.

The Eriksen Flanker incorporates “flanker” stimuli surrounding a central target and assesses the ability to suppress inappropriate responses in a particular context (Eriksen & Eriksen, 1974). A directional response (left or right) was assigned to a central target stimulus. Three test components were utilized, where “flanker” images were presented to the left and right of a central target as either congruent (same direction as the central target), incongruent (opposite direction from the central target), or neutral (different symbols than the central target). These responses are believed to be controlled at the anterior cingulate cortex, a frontal brain structure responsible for a wide variety of autonomic and EFs, particularly when processing incongruent stimuli (Davelaar, 2013).

Subjects responded by pressing a specific keyboard key or, if using a mouse, by selecting a “left” or “right” box indicating the central image directionality. Correct scores in percentage and response times were recorded. The classic Flanker effect shows faster reaction times and higher accuracy percentage for congruent vs. incongruent or neutral “flanker” symbols.

Scoring for the Flanker test typically uses the sum of Accuracy and Reaction times for a composite score (CS).

Accuracy scores were computed as 0.125 * Number of Correct Responses (there are 40 possible Accuracy points).

Reaction time scores were computed as:

Reaction Time (RT) Score = 5 − 5 × [(log(RT) − log(500))/(log(3000) − log(500))]

(1)

The List-Sorting Working Memory Test assessed working memory (Tulsky et al., 2014). A computerized version presented two image groups (food and animals) that subjects ranked from smallest to largest size order. The image sequence was initially presented as a single category, followed by a more difficult second task, containing both. The number of images increased with each test trial and ended when the subject incorrectly identified the size order of two successive sequences. Score calculations used the same formulae as the Flanker test.

Not included in the NIH Toolbox, the Stroop Test was used to measure selective attention capacity, cognitive flexibility, response inhibition, working memory, and processing speed (Stroop, 1935). The test (Milisecond Software Version 5, Seattle, WA, USA) involved processing of a displayed stimulus (word) where one feature (identification of a word vs. the print color) may affect the simultaneous processing of a new stimulus (Stroop, 1935; Scarpina & Tagini, 2017). There were two subtasks: (1) an incongruent trial, where the color name differed from the word print color and the subject was to choose the print color rather than the color the word indicated, and (2) a congruent trial, where the print color matched the meaning of the word (Howieson et al., 2004; Strauss et al., 2006). These demonstrated the subjects’ abilities to inhibit cognitive interference, a domain of EF.

The subjects were presented with four color options and responded by pressing the F, G, J and K keyboard keys for red, blue, green or black colors. Response LAT and ACC results were determined. One short duration practice trial (20 symbols) was allowed.

2.6. Physical Performance Testing—Timed Up-and-Go Test (TUG)

An accepted test of functional mobility and balance (TUG) was used to measure subjects’ abilities to change position, balance, and walk (Podsiadlo & Richardson, 1991). Subjects sat in a standard armless chair (seat height 43 cm), feet flat on the floor. When instructed, in the shortest time possible, they stood and walked to a taped line three meters (9.8 ft) away, then returned and sat. Time was recorded from the starting vocal cue until the subject returned to the chair. Each subject performed three trials and a mean time calculated.

3. Statistical Analyses

After applying the newly formed criteria for “purposeful” HMD-WRIT postural responses, data were analyzed by Pearson correlation analyses, comparing the HMD-WRIT to the two 3D Motion Analysis Systems (BTS and QTM) outputs for LAT, ACC, and calculated EF scores. Bland–Altman plots provided further relationships analyses. The standard error of measurement (SEm) was computed (as a measure of the error in the scores not due to true changes) using the following formula:

SEm = σM/√N

where σM = the standard deviation of the HMD-WRIT scores; √N = the square root of the population mean.

Pearson correlation analyses compared the three computer-based cognition tests’ ACC and LAT data to the HMD-WRIT.

Finally, Pearson correlation analyses tested the HMD-WRIT LAT scores to the mean of the three timed TUG scores.

All analyses utilized SPSS version 26 software (SPSS, Inc., IBM, Chicago, IL, USA). Significance was set a priori at alpha <0.05.

4. Results

Table 1 shows subjects’ demographics data. Ethnicity and racial composition were not acquired. All produced MoCA scores above the recommended normal cut-off value (23/30) (Podsiadlo & Richardson, 1991). Both cohorts showed a significant statistical difference between genders for height and weight. Data were analyzed by gender for the younger adult cohort and in aggregate for the older adults.

Table 1. Subject demographics.

Variable	Men	Women	Sample
Older adult cohort
N	22	23	45
Age	69.5 ± 10.4	69.6 ± 9.8	69.5 ± 10.0
Height (in)	70.47 ± 3.15 *	64.17 ± 3.54	67.32 ± 4.72
Weight (lb)	198.2 ± 29.98 *	143.52 ± 29.1	170.42 ± 40.12
Younger adult cohort
N	21	23	44
Age	26 ± 6.74	26.47 ± 6.36	26.25 ± 6.47
Height (in)	70.76 ± 2.3 *	64 ± 3.11	67.23 ± 4.37
Weight (lb)	182.95 ± 30.32 *	145.87 ± 28.79	163.57 ± 34.68

All values are mean ± SD; * significantly greater than women, p < 0.001.

4.1. Older Adults

A significant strong correlation (r = 0.94, p < 0.0001) existed between the HMD-WRIT EF and BTS 3D Motion Analysis System, confirmed by Bland–Altman analyses (Figure 4 and Figure 5 and Table 2 and Table 3). Trials were omitted if valid data were not adequately recorded by either of the two recording devices or if the subjects did not walk at the minimal required speed (1 m/s), which eliminated four subjects from this analysis.

Figure 4. Regression analysis comparing HMD-WRIT and BTS values for Executive Function (EF).

Figure 5. Bland-Altman plot comparing HMD-WRIT and BTS EF values.

Table 2. Correlation analyses between HMD-WRIT to BTS.

Variable	N	HMD-WRIT	BTS	r	p
ACC (%)	41	81.5 ± 19.1	81.5 ± 20.20	0.994 *	<0.0001
LAT (s)	41	1.29 ± 0.24	1.29 ± 0.20	0.916 *	<0.0001
EF	41	103.66 ± 24.80	103.05 ± 26.52	0.936 *	<0.0001

Values are mean ± SD. Results presented represent 343 trials. * Significant at p < 0.01 (2-tailed). HMD-WRIT = Walking Response Inhibition Test on Head-Mounted Augmented Reality Display; BTS = SMART-DX 7000, BTS Bioengineering; EF = Executive Function; LAT = Latency; ACC = Accuracy; r = Coefficient of correlation.

Table 3. Results for bias and LOA for Accuracy, Latency and Executive Function between BTS and HMD-WRIT trials (n = 41).

Variable	Bias ± SD	Bias 95% CI	LOA	Lower LOA 95% CI	Upper LOA 95% CI
Accuracy (%)	0.00 ± 2.48	−0.78 to 0.78	−4.87 to 4.87	−6.23 to −3.51	3.51 to 6.23
Latency (s)	0.00 ± 0.96	−0.03 to 0.03	−0.03 to 0.03	−0.24 to −0.13	0.14 to 0.24
EF	−0.51 ± 0.90	−0.98 to −0.05	−2.27 to 1.25	−3.08 to −1.47	0.44 to 2.051

HMD-WRIT = Head-mounted device-based Walking Response Inhibition Test; BTS = BTS 3D Movement Analysis System; LOA = limits of agreement; SD = standard deviation; CI = confidence interval.

A small, not significant positive correlation (r = 0.146, p = 0.368, r = 0.076, p = 0.64, and r = 0.236, p = 0.142 respectively) existed between the HMD-WRIT EF and the three computerized cognition tests.

Table 4 displays the Pearson correlation analyses of the HMD-WRIT ACC, LAT and EF to these variables for the Flanker, Stroop Color Word, and List-Sorting Memory Tests.

Table 4. Correlation analyses of HMD-WRIT and 3 computerized cognition tests.

Variable	N	HMD-WRIT	Flanker	r	p
ACC (%)	41	81.5 ± 19.1	99.98 ± 0.16	0.310 †	0.052
LAT (s)	41	1.29 ± 0.21	0.90 ± 0.18	0.166	0.306
EF	41	103.37 ± 25.05	89.78 ± 17.48	0.146	0.368
Variable	N	HMD-WRIT	List Sort	r	p
ACC (%)	41	81.5 ± 19.1	63.40 ± 10.65	0.076	0.640
Variable	N	HMD-WRIT	Stroop	r	p
ACC (%)	41	81.5 ± 19.1	94.90 ± 9.39	0.053	0.748
LAT (s)	41	1.29 ± 0.21	1.49 ± 0.43	0.170	0.295
EF	41	103.37 ± 25.05	139.88 ± 34.82	0.236	0.142

Values are mean ± SD. † Trend toward significant at p < 0.05 (2-tailed). HMD-WRIT = Walking Response Inhibition Test with Head-Mounted Augmented Reality Display; Flanker = Flanker test; List Sort = List Sort test; Stroop = Stroop test; EF = Executive Function; LAT = Latency; ACC = Accuracy; r = Coefficient of correlation.

Correlation of the Flanker ACC with the HMD-WRIT approached significance r(40) = 0.310, p = 0.052. No significant correlations were found between HMD-WRIT and the Flanker for LAT or EF. There were no significant correlations between the List Sort Memory and HMD-WRIT for ACC, nor between the Stroop and HMD-WRIT for ACC, LAT and EF scores.

A Pearson correlation analysis examined the HMD- WRIT EF and mean TUG scores. The mean HMD-WRIT EF was 103.66 and the forty-five TUG times (averaged) was 6.23 s, showing a significant, moderate correlation (r = 0.32, p = 0.04).

4.2. Clinical Significance

Effect Size

A Cohen’s d (d = 0.024) for the difference between WRIT-HMD and BTS EF scores revealed no effect.

Standard Error of the Measurement

To further determine the clinical significance of the differences between the HMD-WRIT EF scores and BTS EF scores the SEm was determined using the following formula:

SEm = SD × √ (1 − r)

where

SD = the average standard deviation between BTS and HMD-WRIT; r = the intraclass correlation (r).

Therefore:

SEm = ((24.80 + 26.52)/2) × √ (1 − 0.936) = 6.5, indicative of a small SEm.

4.3. Younger Adults

A significant, strong correlation existed between the HMD-WRIT and the QTM 3D Motion Analysis System (r = 0.98, p < 0.001), confirmed by Bland–Altman analyses, showing no significant statistical difference between the two devices (t(43) = 1.66, p = 0.104) (Figure 6 and Table 5).

Figure 6. Bland-Altman plot comparing HMD-WRIT and QTM EF values.

Table 5. Correlation analyses between HMD-WRIT to QTM for EF.

Variable	N	HMD-WRIT	QTM	r	p
EF	43	94.22 ± 39.37	96.59 ± 37.30	0.977 *	<0.001

Values are mean ± SD. Results presented represent 381 trials. * Significant at p < 0.01 (2-tailed). HMD-WRIT = Walking Response Inhibition Test on Head-Mounted Augmented Reality Display; QTM = Qualsys 3D Motion Analysis System; EF = Executive Function; r = Coefficient of correlation.

Statistically significant height and weight differences existed in the younger adults; therefore, results were interpreted by gender, rather than in aggregate (Table 6). Males had a significantly shorter mean LAT (0.76 s) compared to females (0.92 s, p = 0.003), and males had significantly higher ACC (85.89%) to females (71.00%, p = 0.008).

Table 6. ANOVA of HMD-WRIT results by gender.

Variable	N	Males (n = 21)	Females (n = 22)	F	p
ACC (%)	43	85.89 ± 12.57	71 ± 21.28	7.799 *	0.008
LAT (s)	43	0.76 ± 0.11	0.92 ± 0.19	9.946 *	0.003
EF	43	64.67 ± 9.19	61.77 ± 12.62

Values are mean ± SD. Results presented represent 381 trials. * Significant at p < 0.01 (2-tailed).

A significant, moderate positive correlation existed for MoCA to group ACC (r = 0.398, p = 0.007) and no correlation existed between TUG to group LAT (r = 0.071, p = 0.647). A significant positive correlation of MoCA to group EF (r = 0.331, p = 0.028) existed, and a negative correlation of TUG to group EF (r = −0.202, p = 0.189) was shown (Table 7).

Table 7. Correlations of TUG to LAT, TUG to group EF, MoCA to ACC%, and MoCA to group EF.

Variable	N	Mean	r (TUG)	r (MoCA)	p
ACC (%)	43	78.27 ± 17.03		0.398 *	0.007
LAT (s)	43	0.84 ± 0.15	0.071		0.647
EF	43	63.19 ± 10.94	−0.202	0.331 *	0.189, 0.028
TUG (s)	43	5.097 ± 1.01
MoCA	43	28.47 ± 1.25

Values are mean ± SD. Results presented represent 381 trials. r = Pearson correlation; * significant at p < 0.05 (2-tailed).

Lastly, Table 8 shows a comparison between the two age groups:

Table 8. Comparison of the two age groups.

Variable	Younger Males	Younger Females	Older Adults
Latency (LAT) sec	0.76 ± 0.11	0.92 ± 0.19	1.29 ± 0.24
Accuracy (ACC) %	85.89 ± 12.57	71.00 ± 21.28	81.50 ± 19.10
EF	64.67 ± 9.19	61.77 ± 12.62	103.66 ± 26.52
Younger Adults		Older Adults
EF Variance	122.99	703.31
TUG	5.09 ± 1.00	6.23 ± 1.46
MoCA	28.47 ± 1.25	28.07 ± 1.84

All values are mean ± SD.

Healthy young adults had shorter LAT values than the older group (0.84 s ± 0.18 s compared to 1.29 s ± 0.21 s).

Young males had higher mean ACC (85.89% ± SD12.57% to 81.5% ± SD19.1%), but young females did not (71% ± SD21.28%).

Due to shorter LAT values, the young adults’ calculated mean EF scores were lower (63.15 ± SD11.09 to 103.66 ± SD24.8), where EF = LAT (sec) × ACC (%).

The young adults had less variable EF scores (122.87 ± 11.08 to 615.04 ± 24.8), where EF = LAT (sec) × ACC (%).

5. Discussion

Many medical maladies present clinically with concomitant physical and cognitive decline (especially in advanced age groups) and are increasing worldwide (Moriwaka et al., 1996). Previous studies have shown age-related differences in functional brain activity during cognitive tasking (Cabeza et al., 1997).

The primary aims of these two projects were to examine the validity of the portable HMD-WRIT device compared to two previously validated “gold standard” 3D motion analysis tools, to compare the HMD-WRIT to currently utilized computerized EF assessment tests (in the older adult cohort), to confirm that measured EF results are closely related to daily function when compared to a previously validated functional movement test (TUG), and finally to compare the two age groups.

Findings showed significant, strong correlations between the HMD-WRIT and the two 3D Motion Analysis Systems, confirmed by Bland–Altman analyses, validating the portable device for anywhere measure of the primarily ‘inhibitory control’ portion of EF. The SEm was 6.5, which was only 4% of the range of mean scores between HMD-WRIT and BTS in the older adult study.

No correlation was found between the HMD-WRIT and the three computerized cognition tests in the older adult cohort and to the MoCA in the younger adults, similarly to the results from the original Leyva project (Leyva et al., 2017). This lack of significant correlations was not unexpected due to their procedural and test environment differences. The HMD-WRIT incorporated functional movements as the subjects made whole-body postural responses to visual cues by either changing directions or stopping while walking. In contrast, the computerized tests were administered with the subjects seated and responding to stimuli on a computer and iPad. The computerized tests employed only small muscle groups and utilized fine motor movements of the wrist and hand, whereas the HMD-WRIT engaged large muscle groups and required whole-body postural responses. Further, the HMD-WRIT utilized a full-field visual environment (subjects could view their normal surroundings as the test cues were displayed onto the HMD). While this study is the first to compare an AR environment to traditional computerized EF test measures, Leyva et al. (2017) also demonstrated lack of correlation between the tests utilizing a monitor rather than an HMD for displaying commands. Juan et al. (2014) demonstrated that an AR environment utilizing the Augmented Reality Spatial Memory (ARSM) task was superior to the Dot Matrix subtest of the traditional computerized AWMA-2 test battery, commonly used for visuospatial short-term memory assessment in healthy children. The AR environment was found to be more acceptable to the subjects and better demonstrated an association with memory skills in real-life situations. Perrochon et al. (2015), using the “Stroop Walking Task” (which is similar to deciding whether to cross a street based on a pedestrian traffic light), demonstrated the use of dual tasking inspired by an everyday event and seemed to better detect aging subjects’ cognitive impairments than traditional psychometric tests. Therefore, it may be that the use of the HMD-WRIT differs from traditional computerized tests since it measures EF during functional situations rather than using the small muscles and limited visual field associated with computerized testing. Also, the lack of correlation between the tests may be due to the HMD-WRIT primarily measuring the ‘inhibitory control’ portion of EF whereas the static tests may provide a ‘broader’ measure of EF. Another possible explanation for the lack of test correlation is the mismatch of task difficulty. A task-difficulty mismatch occurs when the complexity of an assigned task does not align with an individual’s current skill level, cognitive capacity, or baseline knowledge. No pre-testing measures were obtained to measure any differences in capacities of the tested subjects. Lastly, though none was reported, potential confounds related to HMD use (e.g., adaptation, discomfort) may have contributed to the lack of correlation results. When examining the relationship between the HMD-WRIT and TUG, a functional mobility assessment highly associated with EF, there was a statistically significant positive relationship (Gothe et al., 2014; Mitchell & Miller, 2008). This is not surprising as a significant relationship between the WRIT and TUG was previously reported by Leyva et al. (2017). This relationship has considerable functional implications. Improvements in functional capacity and movement effectiveness are specific to practiced movements and limbs utilized (De Weerd et al., 2003). Walking control systems, located within the brainstem, facilitate both visual–motor coordination and accurate foot placement (Studenski et al., 2011) and are important during TUG performance, and both are measured by the HMD-WRIT. Since there was a significant statistical relationship between the HMD-WRIT and TUG, the two tests may be utilized in combination, or the HMD-WRIT may replace TUG as a functional EF assessment.

It was somewhat surprising that performance differences between genders were shown in the younger cohort but not the older (so the older cohort data were analyzed in aggregate). Young males had higher ACC than the older group, but young females did not. The EF scores of the young group were lower due to their shorter LAT values. The young group had less variable EF scores in contrast to healthy older adults. This was consistent with data from Sobolewski et al. (2017), where longer mean reaction times were reported for an older adult group (1.40 ± 0.126 s) compared with younger groups during the Functional Reactive Agility Test (FRAT), a task similar to the HMD-WRIT (but did not employ an HMD). This result confirmed prior study results showing that older adults demonstrate wider variability in responses than younger adults, and may impact their accuracy and function (Hultsch et al., 2002).

Limitations

There were several limitations to the two studies. First, the two studies took place at different locations and utilized distinct methodologies. They evaluated two separate cohorts—healthy older adults and healthy younger adults—without using identical comparative measures. Specifically, the older adult group was assessed using the BTS, Eriksen Flanker, List-Sorting Memory, Stroop, and TUG tests, whereas the younger adult group was evaluated only with the QTM and TUG tests. Despite being manufactured by different vendors, the BTS and QTM systems utilize the same foundational tools (e.g., motion-capture cameras, analysis software, and computers) to measure identical outcomes. Consequently, because the statistical analysis demonstrated that the HMD-WRIT was validated by both systems, this cross-device consistency strengthens the overall validation process. Conversely, the methodologies regarding traditional static cognitive testing differed because the specific tests used for the older cohort were unavailable in the same format at the younger cohort’s testing location. As a result, the younger adult study did not replicate the full cognitive comparison process, completing only the QTM validation and the TUG comparison. This data asymmetry ultimately limits the basis for broader comparisons or a definitive synthesis of the findings across both age groups.

The two studies were both conducted during the COVID-19 pandemic, which limited the number of participants despite enforced strict safety guidelines (as mandated by both Universities). This may have influenced the subject recruitment processes, especially since the initial study involved an older population (over age 50). The majority of participants in the older adult study were recruited from a phone-and-email subject list maintained by the Max Orovitz Laboratory, where the study was conducted. These subjects had participated in prior studies; therefore, their familiarity with the laboratory and its personnel may have motivated their participation to provide greater effort than the general population. Also, although not required for either study, all subjects had been vaccinated and boosted with the FDA- and CDC-approved vaccines prior to study participation. Since only those subjects who had received the vaccine opted to participate, this may have also biased the sampling processes, since those who had received the vaccine may have returned to a more active lifestyle and more normal functional activities, whereas the unvaccinated population may have continued to remain more homebound and sedentary. All participants were required to wear personal protective equipment (PPE), including masks and face coverings, which may have negatively influenced their physical performance during the functional tests.

For both studies, there was a paucity of published literature on normative postural reaction times related to the studies’ designs. For the older adults, only one prior study by Sobolewski et al. (2017) references full-body or postural response times to visual and auditory cues, relevant to this study (Sobolewski et al., 2017). No prior publications referenced postural responses to visual or auditory commands, similar to this study in a younger adult cohort. Therefore, results from this study have limited generalizable application.

A methodological limitation involves the inconsistent use of external validation criteria across the research. While the first study compared the HMD-WRIT against the BTS 3D Motion Analysis System, the second study utilized the QTM system. Although both function as 3D Motion Analysis Systems, they represent distinct reference standards, which prevents the overall validation framework from being fully uniform.

Other studies have shown large response-time variability in the elderly population as compared to younger subjects utilizing various stimuli and responses (Hultsch et al., 2002; Gorus et al., 2006; Grand et al., 2016; Bielak et al., 2010; Kail & Salthouse, 1994). Engel et al. (1972) demonstrated that auditory stimuli elicit faster responses than visual stimuli and adding simultaneous auditory stimuli to a visual stimulus reduces response times. Both stimuli were given simultaneously in both these studies so as to optimally minimize response latencies. No prior studies were found that reference a second or corrective reaction after an initial response, and no studies referenced “plant” or “drop” steps for response latencies. Therefore, for the older adult study, the mean and SD values from the Sobolewski study (Sobolewski et al., 2017) were utilized, and applying those accepted values to this study’s small sample was a limitation. For the young adult study, the mean and SD values were calculated from the cohort that made double or corrective postural responses. The mean + 1 SD was used as the cut-off threshold for determining the LAT and ACC of the trial. Application of this small cohort data as the threshold was a limitation.

For the HMD-WRIT data interpretation methodology, the raw data were converted into graphical representations for analyses via Excel software. Manual response interpretations were determined utilizing all three planes (X, Y, and Z planes) of motion. Choosing the exact timeline points selected manually for HMD-WRIT and 3D systems responses were not exacting and may have led to exaggerated data point differences or similarities. The 3D systems data, however, were considered more precise, since visual evaluation of the actual responses were measured by watching the movement in 3D graphic representation, whereas the HMD-WRIT data were acceleration data only from the IMU, making interpretation precision of the exact responses more difficult. Data from the two studies were analyzed by Statisticians at two different institutions, and their methodologies differed (e.g., SEm calculations) with inconsistent applications, thus raising an important issue of scientific coherence. The methodology of calculating EF was also a study limitation, as EF is defined as ACC multiplied by LAT (as was utilized in the original Leyva et al. (2017) study), where a higher latency represents poorer performance. This leads to a composite score where a lower EF implies better performance, an unintuitive and unlabeled directionality that lacks validation. Possibly a better calculation method for EF could have been ACC/LAT, but for consistency in validating the device, standardization of variables was maintained.

6. Conclusions

These studies, extensions of previous work by Leyva et al. (2017), demonstrated validity of the portable HMD-WRIT with an IMU device using two of the same previously validated 3D Motion Analysis Systems. This supports use of the HMD-WRIT as a viable option for measuring primarily the ‘inhibitory control’ portion of EF in healthy younger and older adults during functional movement, either in conjunction with or in place of traditional static computerized or paper–pencil-type testing and the TUG functional movement test.

These results were expected and confirmed the lack of construct validity, specifically ‘convergent validity’ between the tests, as Leyva et al. (2017) previously demonstrated the original WRIT as a measure of EF and its stronger relationship than computerized tests to functional performance (also assessed by comparison with the TUG test). Other systems have measured EF during functional activity (Perrochon et al., 2015) but to our knowledge, this study was the first to utilize an HMD AR device and IMU to provide a novel portable system that measured EF during functional movements. Additionally, once combined with a dedicated analysis program (e.g., AI technologies), the device may allow immediate measurement and display of ACC, LAT and EF in any clinical setting.

These two studies were designed with a central goal of improved ecological validity, mainly via ‘naturalistic observation’. Further research utilizing this method is merited for investigating findings related to other age groups and specific pathologies, resulting in valid and reliable alternative objective measures of EF (specifically ‘inhibitory control’) during functional activities. Measured deficits may direct Clinicians to implement targeted, optimized cognitive rehabilitation techniques, thus demonstrating incremental validity (e.g., reduced fall risk, improved mobility). In addition, the complete absence of any adverse effects (e.g., simulator sickness) with this testing method also makes it a safe option for clinical use.