Research Trends and Usability Challenges in Behavioral Data-Based Cognitive Function Assessment

Abstract

The prevalence of dementia, a condition associated with high social costs, is rising alongside the aging population. Early diagnosis of mild cognitive impairment (MCI), a precursor to dementia, is essential for effective intervention. Recent research has focused on diagnosing cognitive function in the elderly by analyzing behavioral data, such as gait and hand movements. Compared to traditional neuropsychological assessment methods, behavioral data-based assessments offer advantages, including reduced fatigue for patients and examiners, faster testing procedures, and more objective evaluation of results. This study reviews 15 research projects from the past five years (2018–2023) that have utilized behavioral data to assess cognitive function. It examines the specific gait and hand movement variables used, the technologies implemented, and user experiences reported in these studies. As these types of assessments require new technologies or environments, we analyzed usability issues that should be considered for accurate cognitive assessment. Based on this analysis, the paper proposes future directions for research in the field of behavioral data-based cognitive function assessment.

1. Introduction

Due to rapid population aging, the prevalence of functional cognitive disorders, such as mild cognitive impairment (MCI) and dementia, has increased [1,2]. In particular, South Korea has recently been projected to have the highest rate of population aging among the member countries of the Organization for Economic Cooperation and Development [3]. Hence, the increase in the number of dementia patients is also expected to increase social and economic costs. As of 2023, 10.34% of individuals aged 65 and over (i.e., more than 10 out of 100 people in this age bracket) in South Korea are diagnosed with dementia. The number of dementia patients is 976,923; 22.8% of these patients have MCI [4,5]. Moreover, by 2060, the estimated number of dementia patients aged 65 and over is projected to be 3,325,602, which is approximately four-fold the current number.

As a result, the national dementia management cost is anticipated to increase to approximately KRW 56.9 trillion [6]. Dementia is a severe health problem that transcends individual and family levels and must be resolved at the national level. The intermediate stage between the state of normal cognitive function and dementia is MCI. The long-term consequences of MCI are categorized as Alzheimer’s dementia (AD) and non-AD and vascular dementia [7,8]. Particularly, MCI corresponds to a middle phase in the cognitive function continuum connecting normality and dementia. It indicates “the status in which the cognitive function decline is beyond what is anticipated but not to the extent that dementia can be diagnosed” [9]. Given that MCI occurs prior to the onset of dementia, it must be differentiated from the decline in cognitive function due to normal aging. It is the phase that requires prompt early screening and preventive care before it progresses into dementia. Such screening and care are recognized as core management for reducing the incidence of dementia. Presently, the Mini-Mental State Examination is employed for the clinical diagnosis of MCI.

However, as of January 2021, neuropsychological tools, such as the Cognitive Impairment Screening Test, have been implemented in South Korea [10]. To assess cognitive functions (including MCI) with greater ease and speed compared with conventional diagnosis tools, various studies have been conducted. In particular, several studies have found that pathological changes in the brain are not invariably associated with an increased risk of AD [11]. Diagnostic markers of non-AD, which account for up to 50% of all dementia cases, remain lacking. Moreover, effective, affordable, and non-invasive methods for diagnosing initial dementia are necessary [12].

Based on this consensus, various studies centered on proactive diagnostic methods leveraging information and communication technology for cognitive disorders are ongoing. Particularly, these studies focus on employing behavioral data, such as hand movements and gait, to diagnose cognitive health problems in the elderly. Moreover, gait disorders as a predictor of dementia onset among the elderly have drawn attention [13]. Moreover, hand movement disorders, including limb apraxia, which is prevalent among patients with cognitive disorders (such as AD [14]), are regarded as representative clinical symptoms of cognitive decline.

Cognitive function assessment utilizing behavioral data, such as gait and hand movements, offers significant advantages over conventional neuropsychological assessment tools. These advantages include reduced fatigue for both patients and examiners, expedited testing procedures, and more objective evaluation of results. Additionally, the potential for automation through motion assessment sensors and artificial intelligence further enhances these benefits by mitigating the psychological strain associated with human-administered tests and minimizing errors stemming from subjective evaluations and inaccuracies. For examinees, this approach lowers the psychological barriers to cognitive screening tests, facilitating early diagnosis and potentially delaying the onset of dementia.

However, behavioral data-based cognitive assessments remain in an exploratory phase. In this study, we reviewed the specific behavioral actions examined by researchers, the variables utilized, and the measurement techniques employed over the past five years. Our analysis reveals several opportunities for further research, including the standardization of variables and methods, the exploration of behavioral indicators and diverse populations in previous studies, and the investigation of user experience factors to further reduce psychological barriers and improve the acceptance and usability of these assessment tools.

This study aims to derive the research trend as well as the technical and user experience (UX) problems encountered in studies assessing cognitive functions based on human body movements. These movements, specifically “gait” and “hand movements”, are essential for biometric recognition [15] and the maintenance of daily independence among the elderly [16]. For this purpose, research projects carried out in the last five years (2018–2023) that used behavioral data to assess cognitive functions were identified and analyzed.

2. Findings

Table 1. Literature regarding behavioral data assessed for cognitive impairment.

Assessed Category	First Author (Year)	Diagnostic Targets	Behavioral Data Assessment Items	Variables (1)	Assessment Devices	Remark
Gait	Lowe (2020) [17]	MCI	- Dual-task gait - Gait speed measured during natural daily walking; five-letter words vocally spelled while walking	Gait speed	Stopwatch	N/A
	Zhong (2021) [18]	SCD, MCI	- Natural walking	Gait speed, knee peak extension angle, knee angle at heel strike on right side	Force plate (AMTI BP400600, DC, USA, sampled at 1000 Hz), 3D Motion capture system (Vicon Nexus 2.8, Oxford Metrics, Oxford, United Kingdom)	N/A
	Zhang (2021) [19]	Dementia	- Single-task gait (stride) - Dual-task gait (walking while counting numbers)	- Mean speed, standard deviation of speed, mean step length, mean gait cycle time, mean acceleration	Kinect 2.0	N/A
	Lindh-Rengifo (2022) [20]	MCI	- Natural walking	Step velocity variability, mean step length, mean step time, swing and stance time asymmetry	GAITRITE (Platinum, CIR Systems Inc., NJ, USA)	N/A
	Borda (2022) [21]	SCD	- Natural walking	Gait speed	Stopwatch	TUG test
	Skillbäck (2022) [22]	SCD, AD	- Natural walking	Gait speed	N/A	Longitudinal study
	Guimarães (2023) [23]	MCI	- Single-task gait (preferred stride) - Dual-task gait (counting backwards in sevens from a randomly given number between 200 and 250 while walking)	Stride time, gait speed, Foot fat ratio, Pushing ratio, Liftoff angle, Minimum toe clearance, Heel 3D path length, Heel 3D path length variability, Toe 3D path length variability, Loading ratio asymmetry	Foot-worn inertial sensor-based gait analysis solution (Equipped with 3-axis gyroscope and 3-axis accelerometer (Bosch BMI160))	N/A
Hand movements (gesture)	Negina (2018) [14]	AD	- 29 gestures (14 dynamic and 15 static gestures)	Accuracy	Kinect v2	N/A
	Baumard (2020) [24]	Normal aging	- 45 meaningless gestures	Scoring System (+ Completion Time for Imitation; 0–2 additional points awarded for faster completion times in case of 2 points): - 0 point: Did not imitate or incorrectly imitated the presented hand gesture within the time limit - 1 point: Imitated hand gesture within the time limit; however, some fingers differ or hand direction is incorrect - 2 points: Perfect imitation within the time limit	N/A	Evaluated by an examiner
	Li (2022) [25]	Lewy body dementia, AD	- 4 meaningless gestures	Scoring System - 0 point: Any mistake in direction, finger, or intersection within 10 s - 1 point: Otherwise * Maximum score in gesture imitation test = 4 points	N/A	Evaluated by an examiner
Hand movements (for performing daily activities or specific tasks)	Curreri (2018) [26]	Dementia, MCI	- Putting on a shirt as quickly as possible and correctly buttoning it up	Completion time	N/A	Evaluated by an examiner
	Chua (2019) [27]	MCI	- Performing seven tasks (1) Opening a door using the correct key and password (2) Making a call by dialing eight predefined digits (3) Identifying a (a) celebrity, (b) grocery commercial, (c) four-digit lottery number in a newspaper (4) Classifying household items by category (5) Selecting an attire appropriate for a specific event (6) Withdrawing money from an automated teller machine (7) Grocery shopping at a market	Scoring system	N/A	Application of VR: - VR and leap motion are used in environment setup and input devices - They are evaluated by an examiner
	Liang (2019) [28]	Dementia, MCI	- Performing British Sign Language (BSL)	Hand movement trajectories and speed	Real-time web camera	Collected clinically meaningful facial expression data with hand movement data
	Umemura (2020) [29]	Dementia	- Hand movements while playing the tangram game; status of using both hands	Movement ratios of both hands, dominant hand, and non-dominant hand	Leap motion (3D hand recognition device)	Tangram: A shape - making game using seven pieces cut from a large square, including right isosceles triangles and squares
	Park (2022) [30]	MCI	- Performing six tasks using a kiosk (1) Deciding whether to eat at a store or get takeout food (2) Choosing a preferred hamburger (3) Choosing a preferred side (4) Choosing a preferred drink (5) Choosing a payment method (cash or credit card) (6) Entering the credit card password	Hand movement speed	Hand controller for VR (virtual reality)	Application of VR

⁽¹⁾ Dual-task effect (DTE) (%) = 100 × (dual-task score − single-task score)/(single-task score), including variables corresponding to a single task or dual task.

summarizes the 15 research projects utilizing behavioral data (such as hand movements and gait) to evaluate and diagnose the cognitive functions of the elderly. Seven of the projects are gait-related studies, and eight are hand movement-related studies (three gesture-related studies and five studies on hand movements for performing daily activities or specific tasks). The diagnostic targets, which encompass normal aging, subjective cognitive decline (SCD), MCI, and dementia, are listed in the table. Additionally, the table comprises behavioral data assessment items, variables, and devices employed for evaluating behavioral data.

3. Analysis on Behavioral Data Assessment Experimental Settings

3.1. Method for the Selection of Articles for Review

3.1.1. Eligibility Criteria

Original observational or interventional studies published in peer-reviewed international journals were considered eligible if they met all inclusion criteria and none of the exclusion criteria. Diagnostic subjects include all stages of cognitive health: normal aging, MCI (amnestic and non-amnestic types), and various types of dementia (e.g., AD, vascular dementia, and Lewy body dementia). Movement data used in cognitive function assessment were broadly categorized into gait (involving full-body movements) and hand movements (such as finger movements or gestures).

The selected projects excluded studies that included additional assessment factors beyond behavioral data, those focusing on treatment, the improvement of conditions, or the enhancement of the patient’s quality of life, and those that concentrated on verifying the effectiveness of behavioral data measurement techniques. Additionally, we focused on widely used diagnosis methods in existing neurocognitive tests (e.g., finger tapping and pegboard tests), as our focus was new types of behavioral data analysis that have not been used in traditional clinical trials.

3.1.2. Search Strategy

In this study, research projects performed in the last five years (2018–2023) that used behavioral data to assess cognitive functions were searched through the databases PubMed, Web of Science, Scopus, IEEE Xplore, and ACM Digital Library. The keywords used for the search were a combination of ‘dementia diagnosis’, ‘behavioral data’, ‘cognitive function assessment’, ‘gait analysis’, ‘hand movement’, and ‘gesture imitation’.

3.1.3. Selection Process

Screening for eligibility was executed by two of the authors. For that did not initially match, the authors discussed and reached a final decision. The selection process is explained in Figure 1. A total of 15 research projects were chosen for analysis.

3.1.4. Data Collection Process and Data Items

Following the screening of results, two of the authors extracted data including the author and publication year, diagnostic targets, behavioral data assessed, variables that were measured, assessment devices, the examiner or moderator involved in the examination, the population, and key results related to the research question. We subsequently grouped the results by outcome measure and relevancy.

3.2. Variables for Diagnosing Cognitive Functions

3.2.1. Variables Used in Gait Analysis

Gait indicates a series of lower limb and body movements to move the center of gravity forward. Reduced gait speed is a strong predictor of cognitive impairment and function decline and is regarded as an indicator of early diagnosis [31,32]. Accordingly, research on assessing cognitive function based on gait data is ongoing. Gait cycles are broadly classified into two stages (stance and swing phases) [33]. They are subdivided into eight stages: initial contact, which occurs in the gait cycle; loading response, which is the stage of moving the body weight forward; midstance and terminal stance, which belong to the single limb support stage; pre-swing, initial swing, midswing, and terminal swing, which belong to the swing limb advancement stage [34].

Seven studies focused on gait assessment commonly involved regular gait (i.e., “natural gait” was the analysis subject). Among these, three studies involved dual-task gait assessment, such as counting and calculating numbers and spelling presented words while walking [17,19,23].

Each study attempted to analyze the relationships between various time variables (e.g., stride time, gait cycle time, and step time) and space variables (e.g., stride length, step length, stride width, and heel 3D (three-dimensional) path length) calculated based on measured movement data and diminished cognitive function. For gait assessment, “gait speed” was mainly used (n = 6). It was followed by “variability in gait speed” and “mean step length”; n = 2 for both variables. Moreover, the kinematic characteristics of gait (such as “knee peak extension angle” and “knee angle at heel strike”) and spatiotemporal elements within the gait cycle (such as “asymmetry” and “variability”) were measured as variables for gait characteristic analysis.

3.2.2. Variables Used in Hand Movement Analysis

Hand movement research is implemented to assess cognitive functions. It considers gestures involving the hands and upper limbs, hand movements based on the ability to perform regular daily activities, instrumental daily activities, or specialized tasks (e.g., games). Hand movement studies utilize different types and dimensions of variables based on the properties of a task assigned to examinees.

Diagnostic measures leveraging gestures are primarily categorized into simple gestures (using the fingers of one hand) and complex gestures (using both hands or upper limbs). Conventionally, examinations imitating simple one-hand and complex two-hand movements [35,36] have been conducted. However, recent studies use both static and dynamic gestures requiring movements [14] or meaningless gestures to investigate the existence of a direct correlation between visual inputs and movement patterns [24]. Three studies on gesture commonly used the accuracy of imitation as a variable. Baumard (2020) [24] and Li (2022) [25] adopted a scoring system that assigned a score based on the level of imitation.

In addition to gesture imitation, an examinee’s hand movements in natural, everyday situations have been observed in previous research. A study in 2022 by Park (2022) [30] used kiosks in virtual reality as a task and an examinee’s hand movement pace during the given task as a variable. Curreri (2018) [26] used the completion time involving hand movements in putting on and buttoning up a shirt correctly as a variable. In a 2019 study, Chua (2019) [27] used an examinee’s performance based on correctly completing a given task within a set time. The number of trials involved and the rate of correctly performing a task were considered as variables. Additionally, the rates of two-hand movements and those of dominant and non-dominant hand movements when performing a simple game, such as Tangram, were used [29]. In a 2019 study, real-time hand movement trajectory [28] was used to examine cognitive functions based on hand movements for sign language. The study used the speed and trajectory of hand movements as variables for the cognitive function correlation analysis.

The most commonly used variable in hand movement research is either the accuracy of imitation or performing a given task (n = 4), which is typically converted to scores. Additionally, hand movement speed has been identified as the second most commonly used variable (n = 2).

3.3. Technology for Measuring Behavioral Data

3.3.1. Gait Assessment Technology

The five types of devices used to assess an examinee’s gait patterns are summarized in

Table 2. Comparison of gait data assessment devices.

Assessment Devices	Stopwatch	Wearable Inertial Sensor (3-Axis Digital Gyroscope + 3-Axis Accelerometer (Bosch BMI160))	Optical Motion Capture (Kinect v2)	Instrumented Walkways (GAITRite System)	3D Motion Capture System (Vicon)
Feature	- Simple usage - High portability and accessibility	- High portability, compact size - Capable of collecting detailed gait-related data via attachment cf. sensor developed in lab for pilot testing	- Markerless motion capture method - Capable of collecting full-body action data - Requires approximately 2 m of system installation space	- Capable of collecting detailed data specific to gait - Unable to assessment full-body gait data - Requires 4–6 m of system installation space	- Capable of collecting comprehensive full-body motion data, including gait - Requires the subject to wear markers - Requires a skilled examiner - Requires at least 6m of system installation space
Assessment Data	Temporal data related to gait	Acceleration and angular velocity data of gait motion	Relatively simple full-body gait data	Detailed gait specific data	Precise and comprehensive full-body gait data
Cost	Very affordable <——————————————————————————————————> Very expensive

. Because single gait data could be effectively collected without high-resolution capture or special equipment [15], relatively small affordable devices, including a stopwatch, Kinect, and wearable inertial sensors, were used. For a more complex and multi-dimensional gait assessment, professional analytic equipment, including instrumented walkways (e.g., GAITRite and 3D motion capture systems), was used. The range and dimension of collected data varied depending on the sensors and types of assessment devices.

Stopwatch

A stopwatch was used in two studies [17,24]. This device is frequently employed in various gait assessments due to its simple usage and high accessibility. Nevertheless, it has limitations in sensitively measuring motor impairments induced by the state of mild cognitive decline (e.g., SCD or MCI).

Foot-worn inertial sensor

Guimarães (2023) [23] devised an inertial sensor combining a three-axis gyroscope and a three-axis accelerometer (Bosch BMI160) to measure gait data. Because the inertial sensor is small and light, data can be collected while an examinee is wearing shoes; therefore, it is a relatively affordable solution. However, a widely commercialized solution is absent, necessitating the development of alternative equipment.

Optical motion capture device: Kinect v2

A research project in 2021, the “Study of Deep Learning-based Gait” [18], used Kinect v2, a markerless motion capture system, to compute the positions of body joints on a series of images. The capture resolution of this solution is 1080 p. Moreover, this approach can consistently detect the user’s state even when light sources are absent in the experimental space or when position change occurs. The absence of marker attachments reduces cost and the burden for both the examiner and examinee. Data precision and reliability are lower using this approach compared with those of a 3D motion capture system. However, the method can conveniently measure behavioral data.

Electronic pathway: GAITRite system

A study in 2022, the “Components of Gait in People” [20], employed the GAITRite system, a professional device for gait analysis. The study installed 4.88 m (length) × 0.69 m (width) GAITRITE (Platinum, CIR Systems Inc., NJ, USA) on the path. Examinees were provided a distance exceeding 1.5 m before and after the path for acceleration and deceleration, respectively. GAITRite is an electronic walking board equipped with 1-cm-diameter sensors arranged vertically every 1.27 cm along the board. This allows data measurement for temporal and spatial gait variables [37]. The GAITRite system has been proven to have high reliability and validity in measuring gait data [38]. However, its use is limited to environments with flat ground [39] and requires an installation area of 4–6 m. The other drawback of this system is its inability to measure kinematic components, such as the full-body gait state.

Three-dimensional motion capture system

Zhong (2021) [19] used the 3D motion capture system to collect gait data. The system collects and analyzes the reflection data from 12 infrared cameras after markers are attached throughout the body of an examinee; hence, precise full-body gait data are obtained. However, the system requires the attachment of markers to the entire body. Furthermore, setting its experimental environment, including optimizing locations and angles of multiple cameras to minimize the loss of marker data, is extremely intricate. Additionally, only trained examiners can operate the system, requiring at least 6 m of space [40]. Moreover, due to its high cost, the system has low accessibility compared with other devices.

3.3.2. Hand Movement Assessment Technology

Four of the eight studies that examined hand movement data utilized an examiner judgment-based evaluation method. In particular, two out of three studies that evaluated gesture imitation leveraged an examiner scoring method [24,25]. Because gestures have complex trajectories and relatively broad movement spans compared with simple and repetitive hand movements (e.g., finger tapping), the application of automated scoring methods is not easy. The remaining four studies on hand movement employ various devices depending on the experiment objective, method, and measured items and environments, as listed in

Table 3. Measurement device based on the experimental environment of hand movement data assessment.

	The Environment of Hand Movement Data
	Reality	Virtual Reality
Data measurement Device	Kinect v2 Leap motion, Real-time web camera	Hand controller

.

The measurement of hand movement data is categorized according to the experimental environment: actual physical space and VR. In particular, studies that sought to measure hand movements in relation to the ability of subjects to perform tasks or instrumental activities in daily living employed virtual experimental environments. Apparently, the studies attempted to mimic various everyday situations and increase the immersion of examinees in the experiments. Within the VR environment, hand movement data were collected using a hand movement controller. To provide a virtual scenario to an examinee, a study in 2022 [30] used a headset display (HTC VIVE Pro Eye, New Taipei City, Taiwan). In contrast, a research project in 2019 [27] presented experimental scenarios on a large display. A study in 2018, PRAXIS: Towards automatic cognitive assessment using gesture recognition [14], performed gesture imitation evaluation using Kinect v2, an optical movement capture system, to collect an examinee’s hand movement data. A study in 2020, “Significant Features of Hand Motion” [29], captured hand movement using leap motion while an examinee played a game.

3.4. Research Trends in Behavioral Data Analysis for Cognitive Function Assessment

3.4.1. Research Trends in Gait Analysis

Existing clinical tools for measuring natural gait functions include timed up and go (TUG), comfortable gait speed (CGS), and Tinetti tests. The TUG test is a gait assessment method commonly used in clinics to identify gait impairment. It is applied to cognitively impaired patients with diseases such as Parkinson’s disease and frail elderly individuals [41,42,43]. The CGS test is used to assess gait speed in a manner that is most comfortable and natural for an examinee; it involves walking a distance ranging from 2–10 m [43,44]. The Tinetti test for gait assessment utilizes a scoring approach predicated on the accumulation of scores for detailed gait components. These components include initial hesitancy in the gait phase, stride length, height, and continuity within a gait cycle, in addition to deviations from the path, the use of walking aids, or walking posture. The probability of falling among the elderly is determined by the total scores and the balancing test results. Current assessment methods solely consider the duration of time spent walking. Furthermore, gait analysis or diagnosis depends on an examiner’s subjective evaluation of the variability in stride speed or length. For evaluating cognitive functions, recent research on gait has used various sensors and devices to measure and evaluate diverse gait variables (including those imperceptible or those other than gait speed) precisely. Moreover, research has been extended to include the SCD group in addition to the MCI group (the primary subjective group considered in previous studies). This trend in the field of gait research is visually demonstrated in Figure 2.

Analysis of Kinematic Gait Data

Recent research has aimed to measure and examine kinetic and spatiotemporal variables, which have been the focus of analysis in previous studies [18,19,23]. Gait kinematics is the field of research on joint angles and the directionality of joint segments. Research on the impact of cognitive decline on gait parameters in kinematic aspects is scarce [18]. However, recent studies have endeavored to assess kinematic gait data using 3D behavioral data measurement technology (such as foot-worn inertial sensors and 3D motion capture systems) and investigate their relationships with the cognitive state of examinees.

Expansion in the Scope of Subjects toward SCD

Gait research has recently expanded its focus from MCI to SCD. The term SCD-I, which was introduced in 2014, referred to an individual’s experience of diminishing cognitive function in the absence of evidence of decline in objective cognitive functions in neuropsychological tests or daily functioning. An increasing number of studies indicate that SCD is highly related to the increased risk of objective cognitive functioning decline (including AD); hence, the significance of such research has increased [45,46]. Accordingly, recent studies on behavioral data have included the analysis of the SCD group. These studies have compared the TUG test results [21] or kinematic gait characteristics [18] of the SCD group with those of normal subjects. Longitudinal research on natural gait speed has also been performed [22].

3.4.2. Research Trends in Hand Movement Analysis

Recent studies on hand movements leverage various gestures that are challenging to interpret. These movements include dynamic gestures or relatively complicated hand motions involving tracing trajectories in 3D space (such as those in games or sign language). The accumulation of complex and multiple behavioral data indicates that the volume of information for analysis is increasing. Moreover, some studies have circumvented the physical constraints in experimental environments using VR and 3D avatars to maintain the consistency of experiments. The trend in hand movement research is depicted in Figure 3.

Increase in the Amount of Hand Movement Data for Analysis

Research on gesture imitation has examined static and dynamic gestures or gestures that are difficult to mimic. This is in contrast to the limited number of simple gestures considered in previous studies. Moreover, recent studies have devised and applied “meaningless gestures” to investigate the direct relationship between visual stimulus inputs and hand movement patterns. In other words, these studies endeavor to minimize the influence of an imitator’s semantic memory by complicating and increasing the forms of gestures presented to the examinee [14]. The hand movements of an examinee while playing games or performing sign language also correspond to dynamic behavioral data. A 2018 study on PRAXIS [14] used Kinect v2 (an optimal motion capture system) and a video-based gesture detection model to measure and classify considerable amounts of gesture data. The objective of the study was to increase the accuracy of gesture detection and classification compared with conventional methods, which relied on visual inspection to determine the success of imitation.

Overcoming Physical Constraints Using VR Environments and 3D Avatars

Recently, the assessment of cognitive functioning in the elderly has been based on certain activities. These are activities of daily living (ADL), such as putting on clothes (buttoning up or zipping), or instrumental ADL (IADL), such as shopping, making a phone call, or getting on a bus or a train. These activities indicate a higher degree of functioning than other basic daily activities [47]. Several studies consider such tasks because they demonstrate the daily living abilities that are considerably related to “hand movements”. Two MCI studies based on hand movements [27,30] employed a diagnosis approach using VR where tasks similar to daily activities can be performed. The VR-based cognitive functioning assessment can mimic various situations to diagnose an examinee’s cognitive state within a limited physical space. This approach offers the advantage of rapid diagnosis. Additionally, a 2018 study on PRAXIS [14] used a 3D avatar to present gestures instead of live demonstrations by the examiner. Consequently, all examinees were able to perform imitation experiments under the same conditions.

3.5. Limitations of Behavioral Data Analysis Technology and UX

3.5.1. Technical Limitations in Behavioral Data-Based Cognitive Assessment Research

Cognitive function assessment based on gait or hand movement data offers several benefits over existing neuropsychological examinations. It provides an easy, fast, and objective diagnosis of an examinee’s cognitive state. However, for the wide application of these resolutions to clinical settings and everyday environments, the reliability of behavioral data and diagnosis must be ensured. Additionally, setting up the diagnosis environment must be convenient and affordable. Based on these standards, the technical limitations of recent studies are summarized below.

Assessment and Analysis of Gait Data

Gait data-based cognitive function assessment is subject to technical constraints emanating from the intricate nature of collecting and analyzing behavioral data and physical space requirements. The incorporation of kinematic gait variables into the analysis has enabled the evaluation and analysis of more complex and high-dimensional data compared with those in the past. To achieve this objective, a 3D motion capture system allowing the collection of more sophisticated and comprehensive full-body gait data has been used. However, the system has drawbacks. For example, it requires a complicated setup of experimental devices and involves complex pre-processing as well as post-processing of vast volumes of data. Additionally, because the approach is costly, many studies using behavioral data have employed commercialized solutions, such as Kinect v2, or developed wearable inertial sensors. However, ascertaining whether gait analysis using the corresponding technology has adequate reliability and validity is necessary. Moreover, due to the gait task properties, securing a physical space where an examinee can naturally walk is required. The experimental section for gait speed measurement is classified into acceleration zone, walk-time section, and deceleration zone [20]. The use of acceleration and deceleration zones indicates that initial acceleration and final deceleration must be excluded from the measurement. Consequently, the path used in experiments in gait research ranges from as short as 4–5 m to as long as 20 m.

Assessment and Analysis of Hand Movements

To assess hand movement data, the availability of physical experimental space is not extremely necessary; even online experiments are feasible. This implies that spatiotemporal limitations are minimal compared with those of gait analysis. The time required to complete multiple experiments is brief, and the burden for both patients and examiners when undergoing multiple experiments is minor. Nevertheless, due to the necessity of imitating unfamiliar gestures and fully immersing oneself in a given task scenario, training and an additional practice phase may be required. Furthermore, the implementation of more intricate tasks necessitates the collection of vast amounts of data and numerous classification algorithms. Hand movement imitation tasks, such as gestures, require precise and objective analysis due to their predominant reliance on the subjective assessment and scoring of an examiner. This requirement arises from the possibility that an examiner’s demonstration is inaccurate or involves human error. Additionally, a precise analysis of the measured behavioral data is challenging, and the condition and skill level of the examiner may influence the qualitative scoring approach. To assess cognitive functions quantitatively through the analysis of hand movements, the use of sensors or devices and automated scoring technology seems necessary.

3.5.2. Limitations of Behavioral Data-Based Cognitive Assessment Research Considering UX

The limitations of gait or hand movement data-based cognitive assessment examinations relative to UX are as follows.

Gait Analysis

Research on gait-based behavioral data aids in minimizing the resistance or misunderstanding regarding experiments when the examinees are given familiar tasks. The measurement of “gait”, an ordinary and organic daily activity, obviates the necessity of separately training examinees for an experiment. However, in terms of dual-gait tasks, an additional phase is required to ensure that examinees fully comprehend experimental processes and methods. At that phase, experimental methods must be effectively delivered so that an elderly examinee can fully understand the tasks. Additionally, the risk of an examinee falling while walking cannot be dismissed. Due to these concerns, all gait analysis experiments exclusively involve examinees who can walk normally. Hence, diagnosing examinees who cannot walk or maintain stable walking is not feasible. Nevertheless, subjects who are unable to walk normally due to problems other than a decline in cognitive functioning (such as aging, musculoskeletal disorders, or vestibular function issues) must receive cognitive function diagnoses before they are selected as examinees.

Hand Movement Analysis

Constant examinations are required because the cognitive functioning of patients can occasionally vary with location and circumstance. However, the implementation of methods that can reduce the burden of examination and motivate an examiner has been attempted. In these attempts, a simple game or the automation of examination ensures that an examiner’s skills are leveraged. Moreover, errors in movements are prevented using 3D avatars [14]. Additionally, using VR to simulate varying everyday situations is another resolution to increase the immersion of an examinee in a given task [27,30]. However, the occurrence of physical and psychological discomfort or intimidation emanating from using unfamiliar devices (e.g., VR or 3D avatars) must be avoided. This prevents these factors from affecting the results of cognitive functioning diagnoses.

4. Discussion

A factor in behavioral data-based cognitive assessment research is UX. It affects not only an examinee’s understanding, convenience, or discomfort but also the examination result or its accuracy. This is because, during the examination, an examinee may misunderstand certain instructions or commit mistakes that can affect the cognitive health score. As depicted in Figure 4, careful consideration and attention to the UX aspect (including the examinee’s cognitive and physical limitations, impact of the examination environment, and situational backgrounds) in cognitive examinations are required. Most importantly, an examination of an elderly subject must be conducted with the presumption of cognitive and physical limitations. Compared with a young group of subjects, elderly examinees may not fully understand the experimental methods. Furthermore, the elderly are prone to commit mistakes during the examinations or fall during the gait tests. Thus, devices that prevent accidents and instructions for an experimental procedure that are simple to comprehend are required.

A study in 2022, “Gait Speed and Sleep Duration Is Associated” [48], investigated the relationship between the gait speed and sleeping hours of the elderly. In that study, the researchers accompanied the examinees as they walked on every path during the gait test to ensure safety. Second, the effects of everyday situations and varying experimental environments must also be considered in the UX aspect. In particular, unfamiliar experimental environments (e.g., VR or 3D motion measurement) can induce atypical or unnatural behavior among elderly participants. In terms of a diagnosis using VR, caution is necessary to prevent side effects (e.g., psychological burdens, inability to use the device, and dizziness in the VR environment). Third, the circumstances in cognitive examinations must be considered. An examinee’s resistance or psychological load during cognitive tests can also affect examination results. Continuous and repetitive data assessment is required for the accurate scoring and tracking of cognitive state. However, regular visits to a hospital, examination facility, or laboratory are challenging for the elderly. Strategies reducing the examinee’s physical and psychological barriers to improve examination accessibility are necessary. Lastly, an examiner’s inconsistencies in providing experiment instructions and scoring results, the effect of repetitive assessment on the examinee in terms of learning, the variability of the examinee’s ability to use systems and situational characteristics must all be considered in the UX aspect.

Considering the preceding discussion, this study presents three challenges that must be overcome to improve the accuracy of cognitive assessment. Moreover, overcoming these problems can alleviate the psychological and physical strains experienced by examinees during examinations.

First, the probability of certain conditions and subjectivity of examiners influencing data measurement and diagnosis must be minimized. This allows the continuous improvement of the accuracy of behavioral data measurement and diagnosis. Objective standards regarding instructions for the experiment method, movement demonstration, and evaluation are necessary. Furthermore, guidelines for aiding examinees to understand the experimental tests must be developed. When explaining the objectives and methodologies of experiments to elderly examinees, one potential approach is to employ commonly used or familiar language. In the evaluation, automated scoring may be implemented to assess behavioral data.

Additionally, mechanisms that can motivate examinees to undergo testing regularly and continuously are required. These mechanisms must also reduce the burden of examinees in undergoing testing and dispel their negative perspectives. For examinees to endure the demanding and strenuous process of “cognitive testing”, they must experience rewards or have self-efficacy in the state of their cognitive health.

Lastly, technology for assessing the cognitive functioning of the elderly must be designed so that it can be seamlessly integrated with the IADL. Cognitive functioning assessment based on collecting behavioral data can detect MCI in the elderly without the necessity of visiting a hospital or facility, thus providing an opportunity to maximize the benefits of treatment. For this reason, methods for identifying MCI patients based on IADL (including phone and transportation usage, taking medication, or financial management) continue to be employed. Examinations using IADL have benefits in assessing cognitive functioning by collecting the organic behaviors of the elderly without complicated instructions or training [47,49]. A research project by Rawtaer (2020) [50] attempted the early detection of MCI by installing sensors in households to monitor the daily behavioral patterns of the elderly. Daily behavioral data-based approaches offer advantages in reducing the burden of psychological barriers to examinations. Consequently, examinations can be integrated into daily life and regularly implemented. In particular, because MCI requires the tracking of examinations, the conduct of tests in everyday settings becomes even more meaningful.

5. Conclusions and Directions of Future Research

This study analyzes recent research projects using gait and hand movement data for the cognitive functioning assessment in aspects of data measurement technology, trends, and UX. The benefits of behavioral data-based cognitive functioning assessments include more intuitive examination as well as rapid and accurate diagnosis compared with conventional neuropsychological diagnosis tools. However, this approach has several constraints limiting its application to actual clinical situations. We believe that by refining these aspects for more accurate assessments, early diagnosis of cognitive impairments can be significantly improved, ultimately enhancing the quality of life for both patients and their caregivers.

Regarding gait data-based diagnoses, one drawback is the necessity of having physically large spaces. Solutions that are reasonable in terms of cost and environmental setup while allowing for accurate data collection are necessary. Moreover, the physical limitations of examinees must be considered, such that falling accidents do not occur during examinations.

Compared with gait-based assessments, evaluations based on hand movement data have fewer physical constraints. However, analysis shows that the latter requires more complex tasks involving experiments conducted in unfamiliar environments. Moreover, the collection and analysis of considerable amounts of hand gesture data are required. This necessitates the adoption of automated solutions for diagnosis. Compared with research based on movements (including gait analysis), the conduct of research on the relationship between cognitive and hand functioning is limited. Nevertheless, the capacity to perform all hand movements is critical to maintaining the independence of the elderly in daily life. Hence, continuous research in this area is essential [16].

This paper proposes the following future directions for cognitive functioning assessment studies based on behavioral data. First, effective solutions enabling the collection of reliable behavioral data are required. Currently, the assessment technology employed is highly reliable and accurate. However, the technology is costly and has low accessibility (e.g., expertise is required to operate devices). Second, automatic scoring methods must be implemented in diagnosis based on gesture imitation; to accomplish this, research is required. Algorithms capable of assessing cognitive functioning with high accuracy using relatively unfamiliar and intricate gesture data must be developed. Moreover, the data must be accurately collected and classified. Lastly, when simulating VR environments and interaction subjects, research on sophisticated designs is necessary. Such designs can reduce the physical and psychological burdens as well as improve the immersion of the elderly in corresponding environments and tasks.