1. Introduction
The world population is aging and with this, there is a concomitant increase in the number of dementia patients. Apathy, or the persistent lack of motivation to do things, is known to occur with aging, and recent evidence suggests that it may be a behavioral marker for more rapidly progressing forms of dementia. Therefore, tools to measure “motive” and methods to support positive emotions in older people are of great importance for human mental health. Clinically useful diagnostic and intervention tools for apathy must be more powerful, but must also be able to detect the early, preclinical condition. By designing home-use tools for this using information technology, there is the potential to promote novel cognition in ways that may be particularly important for targeting decreased motivation associated with aging in this population. Sensor controls used in digital art, such as depth image sensors with three dimensional (3D) reconstructing applications and software libraries like Processing [
1] (
https://processing.org/) and openFrameworks [
2] (
https://openFrameworks.cc/) have improved dramatically over the last ten years, becoming more available and easier to use. Using such tools, five students developed a digital art installation designed to diminish apathy in the elderly population in the creative design workshop at our university. This installation was then exhibited at our regional city illumination event, ‘Tokiwa fantasia’ (
https://www.tokiwapark.jp/event/tokiwa2018.html) for one month around the New Year’s holidays. Through this exhibition, we confirmed that the installation could successfully entertain visitors of varied ages, including older people, throughout a one-month period. In this paper, we describe the development of this innovative aging mental support tool and report on the results of a verification trial of this tool performed at a retirement home.
The elderly is often not comfortable with novel technology and the apathy associated with age-related neuronal deterioration can exacerbate this discomfort [
3]. Social interaction and communication between younger and older people can help alleviate the apathy associated with aging [
4]. The digital installation developed by the students highlights the benefits of psychological promotion and social facilitation for people of all ages, including the elderly. To evaluate this, we assessed the ability of the students’ digital art installation to ameliorate aging apathy in twenty elderly subjects living in a retirement group home. Our intention was to develop technology appropriate for use in the home, as opposed to the clinic. In order to accomplish this, our goal was to determine if elderly people would willingly participate and interact with the installation in a relaxed, stress-free environment, similar to their everyday life, as opposed to a more structured randomized clinical trial-type design.
The experiment consisted of having the subjects experience a series of seven animation scenes. Four of the scenes were human-interactive using a 3D depth camera sensor with reactive animation/sound controlled by Processing [
1] and included background healing music (
Figure 1, m1–m4). Another scene was also interactive, but utilized a microphone sensor controlled by openFrameworks [
2] (
Figure 1 (s1)). The final two animation scenes were presented passively to the subjects (
Figure 1 (p1, p2)). Each animation was designed to stimulate the subjects brain using sight, sound, motivation and motor function [
5]. An important unique feature of our hardware was the 3D effect we achieved by using multiple layered half-transparent screens to stimulate spatial cognitive function in the elderly. In order to enter this space, the subjects needed to shake the screen.
In addition to the presentation part of our aging intervention system, we included a second part designed to quantitatively analyze the psychological and learning effects of interacting with the system to provide feedback for future improvement. In the current study, we explored the possibility of meaningfully detecting joint motion by manually tracking video images every second during the most active 15 s period. If this works well, there is the future potential to automate this analysis with the use of wearable or remote sensors. Elderly with dementia may present with behavioral hypo- or hyper-responsiveness [
6,
7]. To assess this, motion of the upper body including the back, neck, both hands and shoulders were acquired with different installation scenes. After each testing day, subjects completed questionnaires of sensory pleasure and motivation. The detected motion patterns were analyzed taking the psychological questionnaire and cognitive impairment as measured by the Mini-Mental State Examination (MMSE) [
8] into account. We chose to use the MMSE for several reasons including its ease of administration, minimal exam and analysis time for both the subjects and the caregivers, and familiarity of the caregivers with this exam as it is the current standard examination used at the retirement home. The questionnaires of sensory pleasure and motivation were also carefully chosen for simplicity and to avoid any discomfort for the participants and caregivers. This preliminary study was designed to assess psychological aspects of the evaluation as opposed to deeper clinical questions. Specifically, we were interested in determining if older people would be willing to participate in the digital art entertainment and to understand whether this type of entertainment could play a role in normal elderly life.
2. Materials and Methods
2.1. Study Fundamental Protocols
The Institute Review Board of Yamaguchi University, and the Department of mechanical engineering reviewed and approved this study (2016004). All the study members have complied with the approved contents. The retirement home was randomly chosen and approached about participation in this study. Retirement home caregivers invited all residents of the retirement home to participate in this randomized trial, regardless of clinical background. Letters of intent to participate were signed by the participants or their responsible family members and were managed and controlled by the retirement home administration. Each participant underwent the same evaluation twice, Day 1 and Day 2, separated by one week. Evaluations were administered by the home caregivers at the participants’ retirement home in January, 2017. One to four subjects participated in the digital art entertainment installment at a time. Individual subjects selected who they would undergo the experience with, resulting in a random selection. During the entertainment, familiar caregivers interacted with participants with normal daily conversation to keep their minds relaxed. The students’ interactions with the participants were limited to a quick greeting, short responses to questions, or supporting the participant’s hand so they wouldn’t fall.
2.2. Participant Age, Sex, and MMSE Scores
The caregivers were also responsible for protecting the subjects’ personal information and assigning participant identification numbers which were then shared with the study personnel. Cognitive function for each subject was determined by MMSE scoring performed in the retirement home by caregivers within one month of the experiment. Participants were scored on MMSE from a maximum of 30 to a minimum of 9. Thirteen females (age: 83.8 y, 4.0 sd) (MMSE: 24.2, 6.2 sd) and seven males (age: 84.4 y, 5.6 sd) (MMSE: 25.0, 4.8 sd) participated in the first day of testing. A total of nine females (age: 85.4 y, 2.0 sd) (MMSE: 23.4, 6.9 sd) and four males (age: 86.9 y, 6.9 sd) (MMSE: 22.0, 4.2 sd) attended both the first and second test days. The average score per total and standard deviation (Ave/total, SD) of MMSE qualitative assessment were “orientation time” (3.1/5, 1.9), “orientation place” (3.8/5, 1.5), “registration” (2.8/3, 0.8), “attention and calculation” (2.8/5, 1.6), “recall” (1.8/3, 1.2), “language” (7.8/8, 0.6) and “copying” (0.92/1, 0.27). Subjects dropped from the study due to scheduling reasons, not because they found the experience unpleasant. No special steps were taken to introduce the intervention to the participants.
2.3. Digital Art Works Aapparatus and Presentation
Five university students developed the apparatus hardware including the 3D-effect screen space made by multiple parallel transparent plastic sheets (length 150, width 60, thick 0.03 mm), a Kinect sensor (Xbox360), PC, cameras and microphones (
Figure 1a). The same students assisted the caregivers in guiding participants into and out of the entertainment area, and remained in the area as assistants to the caregivers.
Seven scenes (
Figure 1b) were operated by either Processing [
1] (
Figure 1, m1–m4) or OpenFrameworks [
2] (
Figure 1, s1), or were prepared by animation maker applications (Adobe Creative Cloud) (
Figure 1, p1,p2). Processing was accomplished using the website source codes (Kinect4W 2.0 library for Processing [
9]) and were designed to respond to the participant’s motion using a Kinect 3D whole body sensor. OpenFrameworks referred to the website source codes (Particle system 20189713 OpenFrameworks [
10]) and used a microphone sensor to respond to the participant’s voice and handclap. Adobe Creative Cloud was used to create animations for passive presentation to the participants. The digital art default output consisted of a simple symbol mark (circles, lines, leaves, gates) in motion in each regular speed coefficient [
9,
10] on a two dimensional background. The interactive function (
Figure 1, m1–m4, s1) depended on sensor detection of the subjects’ motion and sound in the marks’ direction of movement and alteration in the two dimensional speed of output.
Each of the seven scenes took approximately three minutes, and were presented serially in random order for a total experience of approximately twenty-five minutes.
2.4. Questions of Sensory Pleasure and Motivation and Analysis
Immediately following their experience, a student asked each participant three types of Lickert five-point scale questions [
11] to determine their subjective impression of the digital installation; Q1. Visual pleasure: “Was it visually pleasant?”, Q2. Auditory pleasure: “Was it auditorily pleasant?” and Q3. Motive: “Do you currently feel motivated?”. Linear regression was used for correlation analysis between MMSE and sensory pleasure and motivation scores using Excel (Microsoft) and R software.
2.5. Data Acquisition and Analysis
Participant reactions during each of the seven scenes were recorded by video camera (Sony Handycam) without utilizing any wearable sensors to avoid causing any stress to the participants. To analyze each subjects’ representative motion per scene, observers manually focused on the most active fifteen seconds during the scene and obtained the video coordinates of the seven body joints per second by visual identification. This manual tracking method was considered appropriate given that in qualitative pre-observation of the overall video data, each scene was found to have uniform function over the duration. We first checked the validity of the motion data automatically acquired by the Kinect sensors against the data acquired manually. We found that the sitting elderly people move much less than other generation’s, one causing multiple acquisition failures by the Kinect sensors compared to the much more precise manually acquired data. Consequently, we adopted the manual method. To validate the reproducibility of the manual tracking over all the observers, we undertook preliminarily training with the same video data and feedback for each observer’s own difference, to acquire the same coordinate as well as time. The video data were converted into jpeg files and coordinates were measured using TMPGEnc and ImageJ (NIH) software. Next, each joint velocity per second was calculated (Excel, Microsoft). Several sets of video data were used to confirm acceptable inter-rater reliability for the manual measurements. These motion data were normalized by each participant’s sitting height measured from the same video data. Means and standard deviations were screened and data including MMSE and questionnaire scores (explained in
Section 2.4) were explored by single or multiple regression model correlation coefficients (R square value) [
12], principal component analysis (PCA) by correlation matrix [
13] and multivariate analysis of variance (MANOVA) and Wilks’s lambda [
14] in Excel (Microsoft) and R.
4. Discussion
The digital art installation was originally presented as an evening (17:00–21:00) illumination amusement in the regional play land around the New Year’s holiday. Many visitors of varied ages who had never experienced such an installation were able to spontaneously enjoy both the active and passive functions included. Given this success, we attempted to recreate the entertainment at the retirement home for the elderly who were not able to visit the original installment because of cold and dark winter nights. To adapt this installation to a potential in-home intervention to decrease apathy in the elderly, we designed the examination to be easy and stress free to perform for both the elderly participants and their caregivers. A comparison to clinical definitions for terms like “intervention” or “dementia” was undertaken. Then, all the older participants’ emotional responses were recorded in their own voice, showing that they were comfortable with video data. This was used to analyse the first reaction to the experience and a subsequent experience one week later to find comparative differences in behaviours between active and passive scenes in their usual home. The brief and simple interview was designed to enable each participant to immediately share their acute experiences after the unfamiliar experience. The participant’s visual and auditory impression of the installation revealed that it was a highly positive experience, but scores did not change between Day 1 and Day 2 (
Figure 2b,c). In contrast, the question scores indicated that the participant’s motivation was significantly elevated on Day 2 compared to Day 1 (
Figure 2d), comparing to visual or auditory pleasure (
Figure 2b,c), generally with lower correlation with MMSE (
Figure 2a).
Apathy, not clinically determined, but a simply decreased willingness to act, that was confirmed by the interview scores, is a serious negative characteristic observed in older people. Our students chose to target apathy in their next generation art work installation. In the elderly, spontaneous and active approaches to objects tend to decline due to motor, sensory, cognitive or emotional neuro-dysfunction. To prevent or delay age-associated apathy, we developed a task that the elderly could participate in, and from which they could gain a sense of accomplishment through motivation leading to self-directed participation. Originally, we assumed that the major impediment to motivation would be lack of familiarity with the art installation, and we learned that motivation did increase on Day 2 compared to Day 1. Furthermore, we found a certain learning-like quantitative effect in response to this novel prototype, shown in differences in head, neck and hip motion on Day 2 that were significantly greater compared to Day 1, especially in passive scenes (
Table 2). This difference was not dependent on cognitive condition which suggests that this motivation facilitating program would be valuable for elderly regardless of cognitive condition (
Table 1,
Figure 3 and
Figure 4). Based on our general observations at the winter festival, range of motion in the older people was qualitatively lower than that in younger people. This consistently low level of motion made it difficult to detect their motion in the dark experiment space. Therefore, we chose to track the participants’ motion manually in the video data. Because the stimulation contents of the installation were generally random and constant, it was difficult to distinguish what and when the participant reacted. Preliminary evaluation in our lab suggested a plan to detect the most representative behaviors. We found that the duration of the participants’ most representative behaviors could be determined by evaluating the most active 15 s during each 3 min scene.
Body motion analysis has been previously used to understand psycho-cognitive reactions in seated people in several contexts, including head movements in autistic children in social contexts [
15], adult body posture variation during Stroop test performance [
16], and use of a computerized tutor that simulates discourse patterns and pedagogical strategies of a typical human tutor [
17].
Subjects’ motion was activated more by passive than interactive scenes on Day 2 (
Table 2), possibly reflecting the additional time elderly may need to acclimate to unfamiliar situations particularly those requiring interaction. Reports indicate that several weeks of exposure to conventional art, such as sculpture [
18], confers benefits for dementia patients. The complexity, precision and balance of artwork created was significantly associated with their visuospatial function and severity of cognitive impairment [
19]. Our new approach, using one week of exposure to a digital art installation, was well accepted by older adults, though this approach did not allow for participants to create their own artwork. Our combined active and passive methods hold promise in that they may be varied to personalize the therapy for individual patients with different conditions. Based on studies using conventional art, further development of our virtual digital art methodology to allow patients to create their own 2D or 3D drawings and models and interact with the system over longer periods of time has therapeutic potential.
Regarding the similarity in response between days to the sound scenes, it could be because these did not require any body motion. In fact, we observed several participants singing their favorite songs during Scene s1, possibly enjoying a synchronized image reaction. These findings suggest other parameters that may be required to more fully quantify behavioral output, such as vocal discrimination, for full understanding of psychological expression.
In the multivariate analysis by PCA in
Figure 4, we found no less than three kinds of shifting directions from Day 1 to Day 2: 1. Head/Rhand, 2. VisPleas/AudPlease in the 1st–2nd components and 3. motive and VisPlease/AudPleas/Head/Rhand in the 1st–3rd components. The plots, variance ellipses and shift vectors allowed us to generalize responses by feature per scene. Our results suggest the importance of intervention technology being designed that utilizes diverse testing to elicit individual variation, as well as the importance of repeat testing to allow understanding of time and conditional shift directions [
15,
20,
21,
22,
23,
24,
25,
26].
This program was designed and implemented with the thought that older people would benefit psychologically from interacting with university students completing their engineering education. This is a group that does not normally communicate frequently with older people, at least not in Japan that has the largest population of older people in the world. The students specifically challenged themselves to apply their digital works in a novel way to the question of mental activation in older people. Before our evaluation, we were concerned that cultural differences between the students and older adults might cause discomfort for the older individuals and, as the worst possible case, accentuation of dementia symptoms. Contrary to this fear, the subjects had a pleasant response to the students’ digital works (
Figure 2). This might be interpreted as the ability to encourage crosstalk across generations. As the next stage in this research, basic functional responses to the digital work should be considered, such as the effect of a second exposure on brain neuronal activities as measured by electroencephalogram (EEG) [
27]. Increased interactions between individuals across the age spectrum is widely believed to be a key factor in improving neuropsychiatric disease outcomes, however, spontaneous communication between young and old is often restrained by differences in neuronal activities between the generations. It is possible that the technology we have developed could play the role of an intermediate in these interactions by attracting both younger and older generations. From our preliminary information, we learned that the participants appreciated daily communication with the younger generation, though this was seldom realized in their living environment. Our design, involving communicative interplay between the generations likely had a strong positive affect on subjects who reported sensory pleasure and motivation [
28]. We suggest that such inter-generational interactive programs show great potential for innovative technological approaches to improving apathy associated with the aging process [
29]. A limitation of this report is the lack of data on social interactions between the participants and the students or the participants and their caregivers. It is possible that these social interactions may have affected the participants behavior. It is important to note, however, that throughout the experiment the students attempted to communicate with the participants in a calm and consistent manner. Our study paradigm should be furthermore validated considering behavioral interactions between the participants and caregivers and students [
15,
20,
21,
22,
23,
24,
25,
26].
This study did not directly address the relevant brain locations activated during such a digital entertainment experience. It is likely that cerebral cortex regions including the inferior frontal gyrus, cingulate, Para hippocampal gyrus, parietal lobe, and thalamus, as well as cerebellum are involved [
30,
31], though further studies are required to clarify this point and to further the development of digital art therapy against aging-associated apathy. A technological limitation of the current study is the manual detection of body motion used. Preferably, an automatic detection method, such as a small wearable sensor like an accelerograph [
32,
33], would be used.
The individual sensor techniques used in this study were not novel, but taken together they suggest a novel design concept. The series of multimodal sensors used along with the 3D digital art projection output may induce users to interact both actively and/or passively. This has the potential to overcome apathy, one of the most difficult challenges associated with elderly care. The multivariate method we used to correlate changes in individual subjects’ responses between the two weeks of testing offers insight into the slow but progressive learning that the elderly achieved. This may represent learning plasticity and represents an area for potential intervention.
In this study, we chose to limit our neurocognitive and neuropsychiatric assessments to the MMSE and minimal questions to give priority to reducing the stress of the examination process for both the participants and caregivers. The next step in determining the clinical utility of this intervention is to perform more in-depth diagnostic assessments both before and after the intervention using tools such as the functional Clinical Dementia Rating (CDR), Assessment Staging Tool (FAST) [
34] and Apathy Evaluation Scale [
35] according to the international diagnostic criteria for a multidimensional construct [
36].