Meta-Analysis on the Effectiveness of Virtual Reality Cognitive Training (VRCT) and Computer-Based Cognitive Training (CBCT) for Individuals with Mild Cognitive Impairment (MCI)

: This meta-analysis aims to assess the effectiveness of virtual reality cognitive training (VRCT) and conventional computer-based cognitive training (CBCT) in five specific cognitive domains (i.e., global cognitive function (GCF), memory (Mem), executive function (EF), language (Lang) and visuospatial skills (VS)) of individuals with mild cognitive impairment. A total of 320 studies were yielded from five electronic databases. Eighteen randomized controlled trials met the PRISMA criteria, with 10 related to VRCT and 8 related to CBCT. A random-effect model was used in determining the main effect of cognitive training in five specific cognitive domains. VRCT provided the largest effect size on VS and Lang while the smallest on EF. CBCT provided the largest effect size on Mem and Lang while the smallest on EF. VRCT and CBCT generate an opposite effect on VS. VRCT outweighs CBCT in treatment effectiveness of GCF, EF, Lang and VS. More immersive and interactive experiences in VRCT may help individuals with MCI better engage in real-life experiences, which supports skill generalization and reduces external distractions. CBCT tends to improve Mem but no definite conclusions can be made. Further investigation with more stringent research design and specific protocol are required to reach consensus about the optimum intervention regime.


Introduction
According to the World Health Organization, approximately 50 million people are affected by dementia globally, and the number is projected to triple to 131 million in 2050 [1,2]. Mild Cognitive Impairment (MCI) is characterized by slight but noticeable deficits in attention, learning and memory, executive function, processing speed, and semantic language [3,4]. If cognitive intervention is absent, individuals with MCI may not maintain current cognitive abilities and spiral down into full-fledged demented conditions. Therefore, it is important to identify effective cognitive trainings [5]. Technological advancement and enhanced education level of aging population demand more effective and evidence-based cognitive trainings. Besides conventional computer-based cognitive training (CBCT), immersive virtual reality cognitive training (VRCT) has been gaining popularity over the last decade [6]. Previous studies have suggested that implementation of CBCT and VRCT among individuals with MCI may lower the risk for developing Alzheimer's disease and related dementia [5,7]. VRCT and CBCT utilize distinct but similar approaches to provide immediate performance feedback, which is necessary for effective learning and rehabilitation [8].
VRCT differs from CBCT in terms of enhanced ecological validity, as high similarity between training environment and real world helps predict everyday functioning [8]. Moreover, VR is an advanced human-computer interaction and provides a realistic 3D environment to elicit individuals' perceptions and reactions [7,9]. VRCT also offers a cost-effective, accessible, and flexible intervention for individuals suffering from disability, distance, and transportation problems [7,8].
While some reviews indicated positive effects of VRCT and CBCT on cognitive domains such as global cognitive function (GCF), memory (Mem), executive function (EF), language (Lang) and visuospatial skills (VS) [5,[7][8][9][10], some had opposite conclusions and questioned their effectiveness on EF [11,12]. Moreover, there is limited information about effective intervention regime due to various training institutes, aims and objectives in different cognitive domains, of which their respective definitions are provided below.
GCF refers to mental processes in the acquisition of information and constitutes domains such as attention and decision making [13]; Mem involves the encoding, retaining and retrieval of information [14]. Deficits in Mem could lead to difficulties in learning and utilizing information in everyday tasks [14,15]; higher order EF (e.g., reasoning, problem solving) requires simultaneous use of multiple basic EFs (e.g., attention, inhibitory control), which are essential for goal-directed behaviors [16]. EF can be further categorized into five sub-domains, including attention, mental flexibility, inhibitory control, working Mem [17]; individuals with MCI are characterized by fluency and naming confrontation disorders as Lang is related to orthographic or phonological retrieval of semantic Mem and verbal production processes [18]; vs. refers to cognitive processes which identify, integrate, and analyze visual form, details, and spatial relations in multi-dimensions, and are essential for movement, depth perception, and spatial orientation [19].
The overall aim of this study is to assess the effectiveness on VRCT and CBCT in five cognitive domains, namely GCF, Mem, EF, vs. and Lang, for individuals with MCI. The specific objectives were to: (a) identify advantages and limitations of VRCT and CBCT; (b) observe and report the trend of cognitive trainings; and (c) suggest future research directions.

Search Strategy
A systematic search of relevant studies was conducted in April 2020 of published studies available on EMBASE, Virtual Health Library Search Portal databases (BVS), PsycInfo (Proquest), PubMed and MEDLINE (via EbscoHost). The search strategy combined keywords for three main concepts: virtual reality, cognitive training, and MCI.

Inclusion Criteria
Studies were selected from the initial search if they met the following criteria: (1) a VRCT or CBCT was evaluated; (2) individuals aged 65 or above having MCI or in a mixed data in which MCI individuals were available separately; and (3) study design was a randomized control trial (RCT); (4) published in a peer-reviewed English journal; (5) the result of at least one cognitive outcome measure is provided.

Exclusion Criteria
Studies were excluded if: (1) the effect of intervention on MCI individuals could not be extracted from effects among healthy or dementia individuals, (2) the publication was not in English, and (3) the study was published before 2009.

Appraisal of Study Quality
The review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Supplementary Materials S2). The use of PRISMA checklists improved the reporting quality and provided substantial transparency in the article selection process. Physiotherapy Evidence Database (PEDro) scale was used to assess methodological quality within studies (Supplementary Materials S3). The final score was settled when all authors reached an agreement after repeated review and analysis.

Analytical Approach
Relevant data from methodology and results of the selected studies were extracted to calculate effect sizes, statistical power and clinical relevance. Important characteristics such as cohorts, intervention type, intervention regimes, and cognitive outcome measures were also analyzed. Based on the five cognitive domains, Cohen's d was calculated by pooling standard deviations across pretest and post-test measurements for intervention and control groups. Thus, Cohen's d was obtained with 95% confidence intervals using the Practical Meta-Analysis Effect Size Calculator (retrieved from https://campbellcollaboration.org/research-resources/effect-size-calculator.html). Post-hoc power calculations were calculated with GPower Analysis Version 3.1 [20]. A random-effect model was used in determining the main effect of cognitive training in five specific cognitive domains. Forest plots of subgroup analysis were thus generated using Meta-Essentials Version 1.5 [21].

Search Results
The initial search strategy yielded 320 studies (including duplicates) from five databases. By using PRISMA guidelines, eighteen studies met the criteria, with ten studies related to VRCT and eight studies related to CBCT. A summary of the decision pathway for the final inclusion of eligible studies into the review was presented in Supplementary Materials S4.

Data Extraction and Analysis
The eighteen studies were analyzed with cohort and individual characteristics (year of publication, country of origin, MCI sample size, MCI criteria, screening assessment, age and gender) (Supplementary Materials S5), intervention characteristics (technology-used: VRCT or CBCT, type of interaction technique, intervention regime) (Supplementary Materials S6), and effect sizes (Supplementary Materials S7a, S7b), respectively.
Major features of cohort and individual characteristics, compliance and attrition factors, and intervention outcomes are reported below.

Age and Gender
The mean age of eighteen studies was 75.63 years old and age range was from 43 to 91 years old. Specifically, from the study done by Flak et al. (2019), age range was from 43 to 88 years old, with a mean age of 66 years old. Since our study interest is on individuals with MCI aged 65 or above, the study by Flak et al. (2019) was included in this review as the mean age was above the required age [22]. However, particular attention should be paid when analyzing this study.
The total number of the individuals with MCI included in this review is 733, including 300 males and 433 females. However, two studies reported inequality in gender ratios between treatment and control groups at baseline [23,24].

Intervention Outcomes
This meta-analysis assesses the effectiveness of five specific cognitive domains (i.e., GCF, Mem, EF, Lang and VS) for individuals with MCI with application of VRCT and CBCT. These five cognitive domains, which are also areas that occupational therapists focus on, were highlighted as they are commonly targeted during cognitive training for functional enhancement. Cognitive measurements used and effect sizes of each cognitive domain were reported in this section.

Correlation between the Five Cognitive Domains
The five cognitive domains are inter-correlated as evidenced by different research. GCF is most commonly assessed to provide an overview of cognitive abilities, while the Mem component is also heavily focused on as it brings about prominent inconvenience in daily living [36]. Mem largely correlates with EF through improving attention and memorizing stimuli more effectively, as the generalization of training effect often occurs due to neuroplasticity [28,37,38]. Moreover, Lang correlates with Mem to a large extent that Mem retrieval is closely linked with word finding [39,40]. vs. is closely linked to EF as working Mem and resistance to interference are used in maintaining visuospatial information [41].
The effectiveness of CBCT and VRCT on these cognitive domains would be evaluated below.

Cognitive Measurements Used
The eighteen studies reviewed adopted a wide variety of cognitive measurements in different cognitive domains. Cognitive measurements used in each of the cognitive domains are listed in Table  1.

Intervention Results
For CBCT, a total of three studies assessed GCF, seven studies assessed Mem, five studies assessed EF, three studies assessed Lang and three studies assessed VS. For VRCT, a total of six studies assessed GCF, seven studies assessed Mem, nine studies assessed EF, three studies assessed Lang and and studies assessed VS.
The results of the five specific cognitive domains are shown in Table 2. Weak effect of the treatment on visuoconstructive abilities was observed.

Overall Analysis
Heterogeneity has been identified in studies, so a random-effect model was applied. Subgroup analysis for the five cognitive domains was performed per intervention types and cognitive outcomes. This meta-analysis determined the magnitude of effects using standardized mean difference. Relative effect sizes (Cohen's d) were calculated by pooling standard deviations across pre-test and post-test for intervention and control groups. Thus, Cohen's d was obtained with 95% confidence intervals with forest plots generated (Figures 1-5).

Effect Sizes According to Different Cognitive Domains
For CBCT, the effect sizes of Mem (d = 0.39, CI = 0.05 to 0.73) and Lang (d = 0.39, CI = −0.24 to 1.02) are the largest while the effect size of EF (d = 0.08, CI = −0.14 to 0.30) is the smallest. This metaanalysis also found no significant effect on EF, which is in line with previous studies [11]. CBCT and VRCT on vs. generate an extreme and opposite effect (CBCT: d = −0.70, CI = −1.64 to 0.24; VRCT: d = 0.60, CI = 0.40 to 0.80). VRCT provides a medium-to-large effect size on vs., while CBCT provides an effect that favors the control group. However, the overall effect size of CBCT on the vs. calculated was incongruent with results reported in individual studies, which may be due to the heterogeneity in study populations and methodological differences between studies. Thus, careful interpretation of data is required.
Overall, compared with CBCT, VRCT generates a larger effect size in most cognitive domains. A more immersive environment in VRCT may indicate better impact in improving different cognitive domains, while CBCT tends to be more beneficial in improving Mem (CBCT: d = 0.39, CI = 0.05 to 0.73; VRCT: d = 0.35, CI = 0.03 to 0.66). No definite conclusions can be made from the small differences between Cohen's d.

Discussions
The overall aim of this study was to assess the effectiveness of the five cognitive domains (i.e., GCF, Mem, EF, Lang and VS) for individuals with MCI with application of VRCT and CBCT. Factors affecting the effectiveness of VRCT and CBCT will be discussed. As usage of VR in clinical fields has been a new growth area, the trend of cognitive trainings will be reported. Both advantages and limitations of CBCT and VRCT will be discussed and future research directions that may influence the intervention effectiveness will be suggested.

Effectiveness of CBCT and VRCT
Both VRCT and CBCT showed promising results in improving different cognitive domains for individuals with MCI. According to the forest plots, VRCT appears to be more beneficial in improving the GCF, EF, Lang and VS, while CBCT tends to be more beneficial in improving Mem. However, it may not be possible to illustrate with certainty which cognitive domain benefited the most due to the high variability in intervention nature, different study aims and objectives, intervention regimes, and cognitive outcome measures used.
For Mem (Figure 2), CBCT has a slightly larger effect size than VRCT. The effect sizes ranged considerably due to the different sample sizes and cognitive measurements used, and it may not be possible to state with certainty which types of Mem benefited the most. While effect sizes show a trend that CBCT is slightly more effective, it should be noted that the improvement claimed by Barnes et al. (2009) is not statistically significant (p = 0.07). However, its effect sizes consistently favor the intervention group especially for those scored below mean at baseline [31]. Further studies comparing the effectiveness of VRCT and CBCT on Mem and a research design with closer scrutinization is advocated.
For EF (Figure 3), both CBCT and VRCT show the smallest effect. Due to the complexity and variety of EF, it takes considerable time to train [16,42]. The existing intervention regime may not be enough to improve EF. Optale et al. (2010) revealed that the score of DTP remained stable after initial VRCT but increased after the booster session. This suggests that EF training requires more time and learning consolidation is needed.
The small effect size on EF may be explained by its broad definitions and correlation with other cognitive domains. Firstly, the definition of EF includes subtypes such as mental flexibility, inhibition, speed of processing, attention, etc. [16]. Different studies may focus on solely one or more subtypes. personalized scenes (e.g., home-setting, grocery shopping and taking public transport). Moreover, VR promotes cognition by enabling real-life experiences, new learning opportunities, and engagement [30,35]. VR helps to reenact past activities (e.g., visiting museum, going to firework party) in which individuals with MCI within aged care may not be able to join physically [30,35]. As enjoyment is critical for reaping cognitive benefits, the appealing elements in VRCT may foster participation and boost EF [42].
Some studies adopted a more immersive treatment, which supplemented VR learning with realtime visual and auditory feedback in triggering and improving EF [32][33][34]. Specifically, Mrakic-Sposta et al. (2018) revealed different visual and auditory supports (e.g., shopping list, guiding voice and hints) in the supermarket scenario were judged clear and useful by the individuals, which better improved their EF (e.g., planning and reasoning). Therefore, VRCT with an engaging virtual, dynamic and multisensory environment may be more effective in training EF in individuals with MCI, compared to CBCT.
For vs. (Figure 5), the results of CBCT and VRCT generate an extreme and opposite effect. VRCT provides a medium-to-large effect size while CBCT provides an effect that favors the control group. The difference may be explained by the effect of 3D stimuli and a more immersive environment in VRCT. Its immediate and internalized feedback improve learning efficiency, while higher similarity with the real world further engages individuals [8,29]. For instance, it resembles situations such as 'riding a bike in the park', 'crossing roads, avoiding cars', 'grocery shopping in a supermarket', 'tasks in an in-home setting scenario' and 'visiting the virtual museum', which further facilitates the transfer of training effect to real-life activities [28,29,35].

Trend of Rehabilitation
As technology advances, demand for more cost-effective and readily accessible training programs rises. Apart from CBCT, the use of VR in clinical fields has been a relatively new growth area [44]. VRCT addresses the previous concerns about CBCT by allowing more naturalistic and immersive experiences [9]. Simulating real-life situations, VRCT offers great potential in terms of generalization in functional areas, thus enhancing its transferability.
More than two-third (70%) of the studies in this systematic review on VRCT were published from 2017 to 2020, suggesting the growth in importance of VRCT. As this review has previously discussed, VRCT generates a greater effect size in most of the cognitive domains, which may indicate that VRCT is superior to CBCT in certain aspects of training. This may explain the recent shift in research interests from CBCT to VRCT in which an immersive and realistic training modality might prove to be a better alternative [9].

Advantages of CBCT
The training mode of CBCT is convenient and has lots of possibilities. CBCT enables tailor-made and standardized interventions, real-time monitoring of individual's cognitive performance, adjustment of intervention levels, and reduced implementation costs, which makes CBCT simple and easy to operate [45,46]. The training mode is flexible to operate either in self-administered form or with therapists' support. Meanwhile, it enables different training media, such as tablets or computers, providing high portability and flexibility for delivering various tasks.

Limitations of CBCT
CBCT shows limitations in higher levels of generalization. According to Han et al. (2017), the Ubiquitous Spaced Retrievalbased Memory Advancement and Rehabilitation Training (USMART) program could not improve individuals' SMCQ and MMSE scores, indicating that efficacy cannot be transferred to general cognitive function, which is the highest level in the hierarchy of generalization. Achieving the lowest level of generalization indicates training efficacy can only be transferred to other non-trained tasks in the same cognitive domain [15,45]. In terms of set-up, regular updates of software packages and touchscreens are required depending on the system, which may increase the possible maintenance fee [47].

Advantages of VRCT
VRCT provides internalized real-time feedback that improves individual's motivation, thus increasing adherence to cognitive trainings [27,[32][33][34]. Improved motivation is essential in continuing cognitive trainings, which helps maintain and boost training effects in the long run. This concurs with previous studies that suggest that VR was effective in arousing interest and satisfaction in individuals with MCI and is less anxiety-provoking than paper-based tasks [8].
VRCT allows for adjustment in learning pace that reinforces learning efficiency [23,27,34]. Man et al. (2012) revealed that the VR group subjectively perceived better use of Mem strategies due to a self-pacing learning mode and non-threatening training medium [23]. Similarly, Hwang and Lee (2017) revealed that self-selection of training programs may further stimulate the inner drive. Selfpaced learning is closely related to self-efficacy as individuals perceive a sense of autonomy in decision making, while a higher self-confidence may positively impact their performance [27,34]. Frequent verbal reinforcement also raises individuals' satisfaction [23]. This implies that self-paced training with therapist reinforcement may prevent learning pressure and increase engagement, which in turn heightens the training effect and satisfaction.

Limitations of VRCT
There are possibilities of undesirable effects, such as anxiety in older adults when they first encounter VRCT, which largely depends on the design of the VR system. Tarnanas et al. (2014) suggested that older adults without prior experience with technologies may feel anxious at the beginning, implying that user-friendly technology design and clear instructions may engage individuals and maximize training effects. It is essential to provide stand-by technical support to reduce the likelihood of anxiety and confusion [7,35].
VRCT may bring about side effects though they are uncommon. Park et al. (2020) reported that few individuals experienced dizziness and fatigue after intervention. Previous studies also revealed safety risks for older adults with reduced vision or other sensory impairment during VRCT [7]. Thus, safety measures such as staff supervision, positioning and regular rest breaks need to be given during intervention [7,30].
Some VR programs may involve a 3D-graphic interface set-up and multiple rooms were needed for participants' navigation to search certain targets [9]. Hence, multiple rooms may be needed for participants to navigate through to search for certain targets. VRCT may require a higher space demand and may not be suitable for crowded settings.

Comparison between CBCT and VRCT
Advantages of VRCT over CBCT VRCT brings better training effects in several cognitive domains compared to CBCT due to distinct characteristics of VR. This study further explains how VRCT improved various cognitive domains through repeated presentation of real-world, dynamic, multisensory, and interactive environments.
In terms of skill generalization, VRCT may have an advantage over CBCT [7,9]. As discussed above, CBCT shows limitations in higher levels of generalization while real-life and tailored-made scenarios in VRCT facilitate the transfer of training effect on daily activities [33]. Simulation of realworld situations (e.g., home environment, supermarket, park, etc.) could familiarize individuals with the transfer of training settings [24,28,29]. Moreover, the immersive and interactive experience in VR reduces external distractions, which improves the training efficiency and further facilitates skill generalization.
VRCT provides multisensory stimulation. With the usage of multiple sensory modalities (e.g., supplemented with audio-visual stimuli), VR may benefit the encoding of episodic information, which influences the later retrieval of Mem as multisensory contexts provide an implicit identification to experienced events, thus facilitating subsequent recognition [23,28].

Common Limitations of CBCT and VRCT
The effectiveness of both CBCT and VRCT is largely affected by the intervention regime, sequencing of training, task difficulty, and types of combination.
For CBCT, a less intensive intervention regime may lead to a lack of generalization since it only stimulates temporary processes (i.e., neurotransmitters upregulation) instead of long term changes (i.e., neurogenesis and formulation of extensive neural networks) [15,31]. The authors also suggested the importance of training sequence in which non-Mem domains should be trained before proceeding to complex Mem training [15].
Similarly, the effectiveness of VRCT is largely determined by intervention regime and types of combination [30]. Park et al. (2020) stressed that insignificant improvement in cognitive domains may be due to a relatively short regime (i.e., 12 weeks). However, VRCT targeting both cognitive and physical training may be more effective [30,32]. This indicates that the benefits of VRCT might be enhanced with a longer training duration and combination with exercise training [30,32]. Further analysis will be discussed in Supplementary Materials S8.

Compliance and Attrition Factor
The overall attrition rate of the eighteen studies was satisfactory (all less than 20%), except for the study done by Finn and McDonald (2012), which recorded a 33% attrition rate. Regarding the high attrition rate, authors explained primary reasons for drop-out included unrelated medical or personal issues and individuals using their own computers and Internet access at home for CBCT, while training by other studies were conducted in hospital or community settings with the therapist's support. This is in line with the findings by Lampit et al. (2014) stating that institutionalized CBCT is more effective than CBCT at home [11]. Thus, differences in training institute and availability of therapists' support may correlate with a higher attrition rate. Further evaluation of the compliance and attrition factors is needed.

Limitations of the Study
Since current interventions are diverse and variable, further studies should compare the effectiveness of both VRCT and CBCT with less variability of study design to reach definite conclusions. Presently, there is no consensus on an optimal intervention regime for CBCT and VRCT. Further investigation with more stringent research design and specific protocol is required to develop and implement safe, effective cognitive training. Secondly, some studies did not produce a significant effect size due to small sample sizes and the inclusion of pilot tests. Careful elaboration in this review should be advocated and further studies should adopt RCTs with greater sample sizes.

Conclusions
This meta-analysis integrated studies about VRCT and CBCT for individuals with MCI. By effect size calculation and comparison, VRCT outweighs CBCT in the treatment effectiveness of GCF, EF, Lang and VS. This can be explained by the fact that more immersive and interactive experiences in VRCT help individuals with MCI to better engage in real-life experiences, which supports skill generalization and reduces external distractions. CBCT tends to be more beneficial in improving Mem but no definite conclusions can be made from small Cohen's d differences. Although VRCT may be more effective for most cognitive domains, further investigation with more stringent research design and specific protocol is required to reach a consensus on the optimal intervention regime. However, it is certain that the trend for VRCT will grow continuously as it gains more momentum in the clinical field and continues in delivering promising results.