A Study of the Effectiveness Verification of Computer ‐ Based Dementia Assessment Contents (Co ‐ Wis): Non ‐ Randomized Study

Featured Application: Computer ‐ based dementia assessment content (Co ‐ Wis) has been developed for the purpose of efficient and accurate dementia prevention and screening. In this study, we verified the reliability and validity of the developed Co ‐ Wis which we anticipate will play an important role in future big data ‐ based dementia management systems


Introduction
Dementia is most commonly diagnosed in the elderly. In the Republic of Korea, more patients are being diagnosed with dementia due to the rapid aging of the population [1]. In 2017, it was estimated that the prevalence of dementia among those aged 65 or above was 9.94% and that approximately 700,000 individuals had dementia [2]. Along with the advance of pharmacological treatments or interventions, modifiable dementia risk factors need to be addressed to prevent this disease [3].
Dementia is a degenerative disease, its early diagnosis is critical, and considerable effort has been invested in preventing and curing dementia [4]. To receive a diagnosis of dementia, diverse medical assessment encompassing interpretation of neuropsychological tests, imaging or pathology, medical history, and assessment of activities of daily living are needed [5]. Neuropsychological assessments are both used to identify signs of cognitive impairment that are consistent with dementia or with neurological disorders [6] and include traditional paper-based and computer-based assessment methods [7].
Computer-based neuropsychological assessments have many advantages over traditional assessments including conservation of both costs and time, consistency of results, accurate recording of responses, and the ability to automatically store and compare a person's performance between assessment sessions [8]. However, most computer-based neuropsychological assessments are limited to those that use a keyboard and mouse as the primary inputs and besides there are limited data on the validity and reliability of them [9][10][11]. Although previous studies on patients with brain injuries evaluated the usefulness of computerized neurocognitive tests [12,13], further research is needed to overcome the limitations reported in the ethics, number of subjects, and heterogeneity of research methods. Thus, the present study aimed to validate the effectiveness of computer-based dementia assessment content (Co-Wis).

Clinical Trial Ethics (Institutional Review Board Approval)
The study procedures were performed in accordance with the Declaration of Helsinki and were approved by the Institutional Review Board of Yeungnam University (IRB No.: YUMC 2019-04-058-001). Coded identifiers were assigned to study subjects who visited the testing agency (medical institution) and provided written informed consent for study participation. The descriptive writing included the general content of the study, and the participants participated after fully understanding the study (Figure 1).

Clinical Trial Design
The purpose of this clinical trial was to confirm the effectiveness of the Co-Wis for the early diagnosis and prevention of dementia symptoms. Subjects were recruited from the Division of Clinical Psychology of the Department of Psychiatry at Yeungnam University Hospital in Daegu, Korea, from June 2019 to December 2019. All individuals with a subjective cognitive decline who attended the unit to undergo neurocognitive assessment were screened and among them, 113 subjects with an objective cognitive decline were included in the study. Participants were evaluated using standard and objective dementia cognitive test tools such as the Korean version of the Mini-Mental State Examination (K-MMSE), the Clinical Dementia Rating Scale (CDR), and the Standardized Seoul Neuropsychological Screening Battery-II (SNSB-II). Additionally, participants underwent a comprehensive clinical evaluation including a detailed medical history, neurological examinations, and a neuropsychological evaluation. The overall clinical trial progression is depicted in Figure 2. Subjects were included in the present study if aged between 45 and 90, having an expressed informed consent, a diagnosis of Neurocognitive Disorder according to DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition; APA, 2013) criteria, a total score of 10-26 on the K-MMSE completed within one year before screening or after screening, a score of 0.5-2 at the CDR, a completed SNSB-II within one year before screening or after screening, and not having alterations in the sensory perception and motricity which could interfere with the trial. Subjects were excluded if they needed hospitalization or outpatient care due to severe medical conditions, showed persistent behavioral problems requiring frequent hospitalizations or inpatient cares, medical re-access was needed due to rapid dementia symptoms presentation, having other medical illnesses, using drugs affecting cognitive function, participated in clinical trials, or other studies that are related to this study within 3 months before screening.
This was a non-randomized control single-group clinical trial. Only a single-blind method was applied hiding the purpose of the study from the subjects since the researcher's blindness was difficult to assure due to the neuropsychological test behavioral characteristics [14]. A total of 11 researchers participated in the clinical trial. The SNSB-II was administered at the screening stage and the Co-Wis at Visit 1 and 2. The test-retest was done in a time interval ranging between two weeks and three months. All tests were conducted in a quiet and focused environment to accurately examine the subject's cognitive function [15][16][17]. Two tablet PCs with a built-in Co-Wis application were prepared. Tests were conducted in a pre-specified order and, if they were discontinued, they were re-administered from the point of interruption. Clinical trials with the same procedure will be revisited within two weeks or three months after the first trial with Co-Wis to avoid learning biases. Contents and procedures implemented in this study were standardized and conducted per the maximum number of participants.

Computer-Based Dementia Assessment Contents (Co-Wis)
Co-Wis evaluated attention, memory, language, visuospatial, and executive function domains [18] (Table 1). To verify the effectiveness of the contents, the concurrent validity, test-retest reliability, construct validity, and signal detection analysis were used ( Figure 3). The Co-Wis was designed as a computer-based tool for dementia prevention with the aim to make its contents accessible for psychology and rehabilitation clinics worldwide. Furthermore, we improved the system to support multiple dementia prevention sessions simultaneously and patients used the application created with the Unity game engine. At the initial stage of the setup, the user was prompted to enter the identifier provided by the therapist. The identifier was used by the application to find the patient within the assessment database. Each therapist had access to his or her Co-Wis account and was able to accept new patients and track the historical session data. Co-Wis was designed to predict dementia disease by applying machine learning based on big data collected through content execution (Figures 4 and 5).

Data Analysis
Data were analyzed using IBM Statistical Package for Social Sciences (SPSS) version 25.0. To examine the concurrent validity of the Co-Wis, Pearson correlation coefficients were calculated for Co-Wis and the SNSB-II. A value of p < 0.05 was considered statistically significant [19,20]. The testretest reliability of Co-Wis was evaluated. Pearson correlation coefficients were used to test for testretest reliability within the clinical trial process [21,22]. Factor analyses (exploratory and confirmatory) were conducted for each cognitive domain subtest to investigate the Co-Wis composition feasibility [23,24]. To assess model fit, we examined the CMIN/df, Tucker-Lewis Index (TLI), the Comparative Fit Index (CFI), and the Root Mean Error of Approximation (RMSEA), which are less sensitive to sample sizes and reflect the simplicity of the model [25]. To evaluate the diagnostic utility, we examined the Co-Wis sensitivity and specificity using a receiver operating characteristic (ROC) curve [26].

Descriptive Statistics
Demographic information and screening data of the participants are presented in Table 2. The average age of the sample was 70.68 years (SD = 9.31), and the average years of education was 9.05 years (SD = 4.67). The mean K-MMSE score was 23.11 (SD = 3.62) and the mean CDR score was 0.80 (SD = 0.44) ( Table 2).

Concurrent Validity
To verify the public validity of Co-Wis construction, the study subjects also completed the SNSB-II within a year from the first test administration to compare results. Pearson's correlation coefficient between the raw scores of the Co-Wis subtests and the corresponding SNSB-II subtests scores were calculated. All the Co-Wis and the SNSB-II subsets were significantly correlated, with correlation coefficients that ranged from 0.409 to 0.798. Upon examination of convergent validity, we determined that the Co-Wis and SNSB-II results were highly correlated (Table 3).

Test-Retest Reliability
In this study, the correlation coefficient between Visit 1 and 2 trials was calculated to verify Co-Wis test-retest reliability. The test-retest was done in a time interval ranging between two weeks and three months. Most items showed significant test-retest reliability, with scores ranging from 0.529 to 0.872. Trail making (A type)'s test-retest low reliability results 0.251 were shown (Table 4).

Construct Validity
In this study, we explored the factor structure of the Co-Wis subtests. Factor analyses were conducted to see if the Co-Wis subtests had a factor structure that corresponded to the presumptive cognitive domain. We extracted Co-Wis factors using the principal component formula and SPSS version 25.0. Through the Kaiser-Meyer-Olkin (KMO) and Bartlett tests, the adequacy of the selection of the subtests and the suitability of the factorial analysis model were verified, as were results of the commonality (Table 5). Table 5. Appropriateness selection of Co-Wis subtest and suitability of factor analysis model.
The Kaiser-Meyer-Olkin (KMO) result was 0.928, meaning that the number of variables and cases used in the exploratory factor analysis was appropriate. Bartlett's test results suggested that factor analysis was appropriate (p < 0.05). The five criteria in the cognitive domain had a commonality of 0.3 for all the Co-Wis subtests, indicating a high commonality among the items (Table 6). Subsequently, confirmatory factor analysis was performed to verify the five sub-factors assumed in this study. The result of confirmatory factor analysis and goodness-of-fit are shown in Table 7  If the CMIN/df was less than or equal to three, it was considered acceptable. TLI and CFI were judged to have a good agreement when they were greater than 0.90. The RMSEA was considered good when below 0.06 and with a good agreement when below 0.08.

Signal detection analysis
All the Co-Wis subtest were significantly value, with area under the curves that ranged from .694-.860 (Figure 7, Table 8). We also analyzed and presented the sensitivity and specificity of each subtest (Table 9).   Table 9. Sensitivity and specificity of Co-Wis.

Discussion
Cognitive assessment could favor early detection of neurological disorders in comorbidity with psychiatry disorders and the necessity of computer-based neuropsychological tests rather than traditional paper-based ones is increasing [27]. The purpose of this clinical trial was to verify the validity and effectiveness of the Co-Wis compared with those of the SNSB-II, similar to a previous study on elderly participants comparing the prototype of a machine cognitive test with a traditional paper-based psychometric tool [28,29]. To confirm the consistency of the test results, we also checked the test-retest reliability between Visit 1 and 2 in a time interval between two weeks and three months to avoid fatigue and carryover effects linked to repeated measurements [30][31][32]. Exploratory and confirmatory factor analysis was conducted to investigate the factor structure of the Co-Wis subtests. We found that the Co-Wis was significantly correlated with the SNSB-II. Co-Wis has been tested and retested in two trials to ensure the stability of the test results. Our findings showed that Co-Wis results were significantly correlated with the SNSB-II scores. Test-retest reliability was assessed to ensure the consistency of the Co-Wis test results. The trail making test A type showed a test-retest reliability value of 0.251, which is considered as low. Given that the trail making test measured the time until test completion, this result seems to reflect the presence of a practice effect bias of the participants [32,33]. Therefore, it is reported that the participants showed a big difference in trail making test A between the first and repeated trials. Before using exploratory factor analysis, the KMO measure and Bartlett test were used to examine the suitability of the data for factor analysis. The KMO measure 0.928 and Bartlett test was 0.000. KMO measures have values greater than 0.90 (marvelous), 0.80-0.89 (meritorious), 0.70-0.79 (middling), and 0.60-0.69 (mediocre). A value between 0.50 and 0.59 was classified as poor and unacceptable if below 0.5. Bartlett's test of sphericity tests the null hypothesis and the alternative hypothesis for the existence of commonality and small values (less than 0.05, rejecting the null hypothesis) of the significance level indicate that factor analysis may be useful for data analysis. Bartlett's test of sphericity tests the null hypothesis and the alternative hypothesis for the existence of commonality and small values (less than 0.05, rejecting the null hypothesis) of the significance level indicate that factor analysis may be useful for data analysis [23]. Using the exploratory factor analysis results of the Co-Wis subtest, we confirmed a commonness of 0.3 over for all subtest items [23]. The cognitive domain was derived from the five components of attention, memory, language, visuospatial, and executive function. Confirmatory factor analysis showed that the five factors of the Co-Wis were acceptable. Considering the standardization coefficients of each factor, all factor coefficients showed statistically significant (p < 0.001), thus ensuring the stability of the structural model [24,34,35]. In the signal detection analysis, the overall scale showed good sensitivity and specificity, so that the Co-Wis contents could be identified as well as existing dementia assessment tools. Our results are in line with previous studies of Park   and patients with mild cognitive impairment or stroke. Additionally, Co-Wis showed high reliability and validity compared with the ones of paper-based assessments. This study confirmed the effectiveness of the Co-Wis and results will be of great value in the field, as the existing studies were conducted on small samples, without IRB approval and formal procedures [36][37][38]. The Co-Wis contents will direct guidelines for big data-based dementia care system development to solve challenging medical demands in dementia medical health experts. However, further research and data collection are needed. First, Co-Wis still requires a standardization score and big data collection in various environments. Secondly, to develop a computerized neurocognitive test that is effective in predicting and detecting dementia and multi-national data, the development of an English version and the verification of its validity and reliability are issues that need to be approached in the future [39]. Thirdly, in the future, it would be necessary to further verify the role of Co-Wis in dementia prevention and diagnosis by comparing patients with healthy controls (RCT). Co-Wis could play an important role in advancing dementia prevention, screening, and care. Thus, using big data collection and machine learning analysis, we may be able to reduce the social impact of dementia while also mitigating the symptoms of this disease.