Touch Interactive System Design with Intelligent Vase of Psychotherapy for Alzheimer’s Disease

Abstract

This paper introduces the concept of an intelligent vase that combines the cultivation of plants with psychotherapy for the Alzheimer’s adjuvant therapy (from now on, referred to as AD) of patients. RECALL (The name of intelligent vase which defined by this paper), which includes hardware and software applications, as well as a unique hand gesture interaction method, is able to collect patients’ hand gestures and play photos, to remind patients of the plant’s growth status. To evaluate our conceptual system, we created a preliminary prototype. We conducted an experiment with leap motion. This project tries to explore a kind of interaction method by cultivating plants to simulate cognitive training activities, to stimulate memories through family pictures, to slow down the cognitive decline from AD, to incorporate non-drug intervention into the family life of AD patients, to reduce the expenditure, time, and energy costs, and to improve the quality of life of patients and their families.

1. Introduction

Alzheimer’s disease (from now on referred to as AD) is a disease with a large population and no effective therapeutic drugs but has non-drug interventions.

According to the data of the World Health Organization, the number of dementia patients worldwide continues to increase, especially in the elderly and those countries facing a demographic transition. In 2010, the total number of dementia patients in the world was about 35.6 million, which is expected to double every 20 years, reaching 65.7 million in 2030 and reaching 115.4 million in 2050. The number of new dementia patients is close to 7.7 million per year, which means that a new dementia patient appears every 4 s [1].

As there is no effective treatment for Alzheimer’s disease, the clinical use of these four drugs can only partially improve the patient’s condition [2]. Therefore, while new drugs are urgently needed, non-pharmacological interventions are playing an important part in individualized treatment options [3].

Caring for other living things can succeed where medications fail. Bill Thomas, MD, created a Garden of Eden, incorporating animals, plants, and children in the culture of a nursing home. Helping to care for animals gives residents a sense of purpose. It gives residents an opportunity to balance the care they receive with ways they can give care. This is an opportunity to “give back” to someone or something, fulfilling the human need to be needed. This plan shows a 71 percent reduction in the daily drug cost per resident and a 26 percent cut in the turnover of nurse aides [4].

Nostalgic therapy is a method of psychological treatment. It is one of the most commonly used methods in non-pharmaceutical interventions. It mainly reminds people of past events, emotions and thoughts through various tangible prompts (such as photos, music, scenes, videos, etc.,) to improve patients’ memory function [5]. On the basis of improving diet, moderate exercise, and safe nursing intervention, it helps patients to achieve happiness and improve quality of life. However, most of the psychotherapy interventions, such as nostalgic therapy, are carried out in hospitals, as they require professional guidance and planning that usually take eight weeks or more. Nostalgic therapy takes significant investments of human resources, material resources and time. Assuming that psychotherapy can be replaced by products with special interactions that are integrated into family interventions, it will greatly help improve the life quality of patients and their families.

Before proposing a touch interaction system to assist the cognitive rehabilitation training of Alzheimer’s disease, the authors of this paper have conducted comprehensive research on recent gesture recognition technology [6].

Depending on whether the body is touching the device, gesture recognition can be divided into contact recognition and vision-based hand gesture recognition. Compared with contact recognition, vision-based hand gesture recognition has the advantage of convenient interaction and has wide adaptability.

Sensors are necessary for collecting row gesture data; sensor technology can be divided into 3D-based approaches and 2D-based approaches, the latter showing some limitations in hand gesture interaction. There are three popular 3D depth sensors with 3D-based sensing technology: Kinect1.0, Leap Motion, and Time-of-flight sensors (ToF). Kinect1.0 is able to detect and track the human body’s skeleton joints quickly, but it focuses on the whole body, and the depth sensor of Kinect accuracy is not high at 4 mm. A ToF camera is a widely used and low-cost depth measurement device, which is more accurate than Kinect1.0, and has been used to generate the 3D point cloud and helped in recognizing complex hand trajectory gestures, where the accuracy of recognition can be up to 1 mm. However, Leap Motion, which was released in 2013, has the highest recognition accuracy at 0.01 mm, and is able to track hand and finger gesture accurately. Considering the importance of gesture recognition accuracy to the system proposed in this paper, the sensor adopted in the system design is Leap Motion.

This paper proposed RECALL (The name of intelligent vase which defined by this paper, Figure 1 shows an effect display of RECALL) and a touch interactive system based on intelligent vase of psychotherapy for Alzheimer’s Disease, hoping to help AD patients from three aspects: (i) Find alternatives to nostalgic therapy which can be handled easily. (ii) Integrate non-drug intervention of AD into family life, reducing the expenditure, time, energy cost of treatment as well as improving the life quality of patients and their families. (iii) Cultivate plants to enhance patients’ sense of responsibility and presence.

To test the practicability of this design, two practices were carried out in this paper (i) to simulate the functions of the hardware through Arduino and multiple sensors; (ii) to define interactive gestures assisted by leap motion.

Research results: (i) This paper proposes a kind of touch interaction system to assist the cognitive rehabilitation training of Alzheimer’s disease, which can provide a referential concept for the product design of this group. (ii) For the first time, this paper proposes to simplify the gesture in human–computer touch interaction according to the characteristics of cognitive decline in patients with Alzheimer’s disease, which can provide a reference for future gesture interaction in special groups.

2. Related Work

Faced with the large and expanding group of patients with Alzheimer’s disease, most devices or software techniques are concentrated on tracking location, the surveilling of patient’s status, helping with daily activities, and so on, but design of psychotherapy is still lagging. There are no devices to help people recall their families or friends.

Tracking location: For example, (1) GPS shoe: This consists of a tracking device in the patient’s right shoe. It is rechargeable and helps the caregiver track the patient easily within 30 min through the website [7], but the patient may remove their shoe on their way, due to an Alzheimer’s attack or may not wear that shoe as the patient does not realize what they are doing; (2) Project Saver, which tracks the patient’s location from sensors tied on the patient’s ankle, but the patient may throw/lose the device off their ankle or it may disturb their ankle and make them irritated. (3) Blue Water Security: a watch, which has a sensor fixed in that gives an alarm when the patient crosses a certain distance, and also has a panic button to be used in case of emergency. However, this device is very expensive and is not affordable for the everyday consumer. It has a month to month contract and no long-term commitments.

Surveillance of patient’s status: The Environment-Aware System developed at Fraunhofer Portugal, a remote monitoring system for patients suffering from Alzheimer’s disease, comprises a device able to monitor the environment temperature and humidity, the patient’s location using GPS, as well as the patient’s movements, including falls. The device sends this information to the caregiver, where it is stored in a database developed in this work [8]. However, patients may remove wearable electronic devices or may not wear that device.

Help with daily activities: (1) Jasvinder and Ashish proposed algorithmic Music Generation for the stimulation of Musical Memory in Alzheimer’s, playing music whenever a patient performs daily activities and can be replayed to assist in recollection of memories of that event. However, there are no supported tangible devices for this therapy [9]. (2) Tangible Virtual Kitchen: Thuong, Déborah, and Paul presented a tangible virtual kitchen system for the rehabilitation of Alzheimer’s patients; this system utilizes Sifteo cubes and supports intuitive and natural user interactions to improve motor skill rehabilitation for everyday kitchen tasks. However, the defined game lacks a daily task and reward mechanism, which rarely gives users a sense of long-term value, and is less likely to be stuck to in the long run [10].

Cognitive stimulation therapy: Carolina, Esther, Miguel, and Ana proposed a full design, implementation, and validation of an e-health service in order to improve the community health care services for patients with cognitive disorders. It allows Parkinson’s disease patients to benefit from the possibility of doing cognitive stimulation therapy (CST) at home by using a familiar device such as a TV set. This service provides therapists with the ability to conduct follow-up of therapy sessions via the web, benefiting from greater and easier control of the therapy exercises performed by patients and allowing them to customize new exercises in accordance with the particular needs of each patient. However, this system is aimed at people with Parkinson’s disease. Although the author thinks only a small change is needed, the system could be useful for patients with Alzheimer’s or mild cognitive impairment, as research on Alzheimer’s disease is ongoing and there is no relevant completed system [11].

As defined, no psychotherapy intervention is presently available for patients with Alzheimer’s. However, psychotherapy interventions have significant positive effects on patients and their families. However, psychotherapy will increase the cost of care, human resources, and financial resources, which could be considered to replace device or software techniques with similar functions.

3. Concept and Inspiration

RECALL seeks an interaction that combines cultivating plants and psychotherapy. Patients can conduct daily cognitive training by looking at family photos. Besides, by taking care of the plants, patients may gain a sense of value and responsibility.

The inspirations come from The Champa Flower (a poem of Rabindranath Tagore, Indian writer, poet, and social activist). The poem writes “If I became a Champa flower, just for fun, growing on a branch high up that tree, laughing and swaying in the wind, dancing on the new leaves, would you know me, mother?”

This is a hypothetical scene—if a child becomes a Champa flower and plays hide-and-seek with his mother, it demonstrates the fun of the family; most family pleasures come from exciting activities, such as hide-and-seek. The design of interaction indoors of RECALL is also related to hide-and-seek.

Japanese human puppets are another inspiration source. Most Japanese puppets imitate the faces, expressions, and hairstyles of lovely children, making users have the impulse to touch and caress them.

The sketch below illustrates the design process for this design goal with complexity; the basic design concept and ideas can be seen from Figure 2.

4. Expert Interview

We went to Nanchang First Affiliated Hospital of Nanchang University (Nanchang, Jiangxi province, China) and conducted 23 interviews including 10 patients, 10 caregivers (6 were employed and 4 were relatives), 2 attending doctors and a Nursing Supervisor.

When asked, “are you troubled and distressed by memory loss?”, nine patients expressed agreement; one patient kept silent due to a language disorder (a complication with Alzheimer’s); six employers said the patient’s memory decline forced them to be vigilant and long-term care made them feel very tired; four relatives expressed beyond exhaustion from nursing, and most importantly, spiritual pain, especially from close relatives.

The following interview is from a relative caregiver who was very close to the patient:

Yuxuan, son of Mr. Zhang (Alzheimer’s), is 23 years old and an art teacher. He said: “Although my dad has forgotten me, I will never forget him.”

“It pains me that he’s losing memory of who I am gradually, but when I show him some old pictures of me as a child, it’s comforting when he suddenly remembers something.”

After patient interviews, we conducted face-to-face interviews with Dr. Li, chief physician of the department of Psychiatry, and Dr. Xu, chief physician of the Department of Integrated Traditional Chinese and Western Medicine. The interviews mainly focus on three questions, following along with their respective expert answers (the repeated answers have been integrated):

Q1. Is the effect of psychological intervention on the rehabilitation or remission of Alzheimer’s disease considerable?

Correct psychological treatment and careful psychological care are more important than drug treatments for Alzheimer’s patients, especially in the early stage of the disease. If patients are given timely psychological counseling and positive encouragement and support, the patient’s condition is expected to recover. For other patients whose memory, understanding, abstract thinking and other abilities that even have been lost and cannot be completely recovered, the continuous deterioration of the condition can at least be avoided with appropriate psychological treatment, so that the patient’s behavior can be maintained at a relatively stable level.

Q2. Does looking at photos have any effects on cognitive decline in Alzheimer’s patients?

Looking at photos is one of the treatments of memory therapy. Looking at photos of patients in the past, especially those from a long time ago, has a certain function of arousing memories. Therapists often show patients photos of the past that evoke positive emotions such as happiness, beauty, pride, etc.

Looking at photos from the past can give patients the satisfaction of remembering the past, helping them reconfirm themselves and integrate their life. The most immediate effect is that recalling old photos can stimulate the residual function of the brain, alleviate the symptoms of dementia, and prevent the deterioration of dementia.

Q3. Is it possible to involve patients in psychological intervention through simple devices without the involvement of a therapist?

It is possible to implement the method of reminiscence therapy by means of a machine, such as adding the event of looking at photos to match the memory journal; petting therapies, such as having patients care for plants (which are less risky than animals) or using robots instead, can improve patients’ sociability.

Based on the interviews with patients, caregivers, and doctors, it can be concluded that: (1) Memory loss is a matter of great concern in Alzheimer’s disease. It is harmful to the mental health of both the patient and the caregiver; (2) The effect of psychological intervention on the rehabilitation and remission of Alzheimer’s disease is considerable; (3) Looking at photos from the past has a positive effect on Alzheimer’s disease; (4) Instead of including therapists in psychotherapy, reminiscence therapy (like looking at past photos) or pet therapy on machines or devices might be considered.

5. Users Segmentation

According to a report from the Alzheimer’s Association in 2019, one in 10 people age 65 and older (10 percent) has Alzheimer’s dementia. Almost two-thirds of Americans with Alzheimer’s are women. Therefore, most of the target users are older than 65, especially women.

Patients with children and good parent–child relationships are the best users of the system. The system can be used as a good psychological treatment auxiliary device to help patients alleviate memory decline, and alleviate annoyance from both their children and themselves.

Although, being older does not mean that a person has children or has happy memories with children.

For the following three type of users:

(i)

People without children;

(ii)

People with fraught or damaged relationships with their children;

(iii)

People who have experienced painful situations around children (infertility, miscarriage, child death, and so on).

RECALL aims to help patients recall happy experiences from the past by looking at old photos (including their best friends, favorite pets, pleasant trips, and past moments of glory, and so on). There is no need for child-related content, and for (ii) and (iii), child-related content should be avoided.

6. Interaction

The patient can interact with RECALL through six simple gestures. The OLED (Organic Light-Emitting Diode, or Organic Electroluminescence Display) screen in the vase will roll and play pictures of the family and friends. Figure 3 shows the flow diagram of display.

There are six movements in this interaction:

(i)

When hands move close to RECALL within 600 mm, the door opens and power turns on.

(ii)

When hands move away from RECALL outside of 600 mm, the door closes and power turns off (Figure 4 shows the display of interaction ii).

(iii)

When the user waves their hand to RECALL, it will output a greeting (a pre-recorded voice from families or friends).

(iv)

When the index finger swipes left or right, it goes back to the previous photo or forward to the next photo.

(v)

When the index finger clicks forward quickly in the air, RECALL will output a voice diary with the current photo (if the family member or friend records this voice diary while uploading this photo).

(vi)

RECALL will supervise the growth status of the plants, and remind the patients by outputting the pre-recorded family voice in case of no water or light.

7. Hand Gesture Interaction Method

In this study, the leap motion device has been used for simulation experiments, and six simple interactive gestures have been defined.

Alzheimer’s disease is characterized by progressive memory loss, cognitive dysfunction, and personality changes [12]. The decline of comprehension and memory means that it requires a more straightforward thing that allows them to understand and remember. Therefore, it is necessary to make a more straightforward interaction based on ordinary interactive gestures, which is easy for Alzheimer’s patients to understand and recognize.

Although there is no theory of touch screen gesture interaction design for Alzheimer’s patients, it is considered that 95%–96% of patients are over 65 years old. They belong to the middle-aged and elderly groups. Many current researchers have proposed the basic principles of touch-screen gesture interaction design for this user group.

The website Sap Design Guild proposed general guidelines for touch screen interface design [13]:

1. High efficiency; 2. Metaphorical; 3. Certainty; 4. Guiding; 5. Testability.

Kevin Arthur proposes a measure of the usability of touch-screen gestures [14]:

1. Easy to learn; 2. High efficiency; 3. Satisfaction.

For the middle-aged and elderly users, some researchers have proposed different design suggestions.

Stüβel proposed the importance of familiarity in touch screen design [15].

Miranda A. Farage summarized characteristics of the elderly in terms of perception, motor function and cognition, and proposed guiding principles of product design for the elderly [16].

In sum, in the touch screen gestures, the most important influencing factors are motor, function, touch, and cognition. This paper identifies the design points in

Table 1. Key points of interactive action design.

Dimension	Pressure Identify
Motor function	Reduce the difficulty of finger movement to avoid incoherent movements. The interval between operation design should not be too long.
Tactile sense	Enhance feedback stimulation and make clear distinctions.
Cognition	Use elder experience symbol, easy to learn, interactive consistent.

.

Gestures suitable for Alzheimer’s patients should be based on single-finger and click-through modes. Complex movements such as pushing multiple tasks on four fingers are not suitable for AD patients. The factors, such as the ease of learning and the high efficiency of the action, need to be paid more attention. Accordingly, it is necessary to reduce the emphasis on the metaphorical and exciting aspects of the action.

According to the particularity of Alzheimer’s users, this study designed six kinds of actions and each action input has a corresponding different featured output.

Table 2. Define gesture based on pattern recognition and pressure identify.

Gesture	Pattern Recognition
Close to	The hand shape is scaled up to x times/frame
Wave hand	Shape of hand rotates to the left and right of axis by a degree of β (specified numerical interval) and repeatedly rotates and resets x frames in succession
Away from	Shape of hand rotates to the left and right of axis by a degree of β (specified numerical interval) and repeatedly rotates and resets x frames in succession
Swipe left	The displacement of the hand in continuous x frames is larger than the threshold y, and the displacement is positive
Swipe right	The displacement of the hand in continuous x frames is greater than the threshold y, and the displacement is negative
Click	Shape of fingertip contact surface when clicking

shows define gestures based on pattern recognition and pressure identification.

The acquisition of dynamic gesture data is completed through Leap Motion. The recognition accuracy of gesture is analyzed, and the feasibility of gesture on the device is evaluated. Figure 5 shows display with gesture recognition via the Leap Motion Visualizer.

A total of 300 samples were collected, and 50 samples were collected for each gesture. The recognition accuracy was divided by the number of correct recognition times to obtain

Table 3. The posture classification and recognition accuracy of six gestures on leap motion.

Posture	Gesture	Average Recognition Accuracy %
Five fingers	Close to	97.88
	Wave hand	97.92
	Away from	92.24
	-	-
Single finger	Swipe left	92.5
	Swipe right	92.58
	Click	85.4
	Average	93.09

. The average recognition accuracy with the six hand gestures is 93.09%.

8. Conclusions and Future Work

This paper proposes RECALL, which is a touch interaction product to assist cognitive rehabilitation training for patients with AD. It integrates the functions of cultivating plants and assisting cognitive rehabilitation, mimics children’s faces, and stimulates patients’ memories, especially the memory of family members.

Compared with previous products for AD patients, this paper focus on the psychological and cognitive rehabilitation of users, rather than the physical level. In this paper, the concept of touch interaction system for AD patients is proposed, a prototype is made, and a set of simplified human–computer interaction gestures is proposed.

However, there are shortcomings in the system. Due to the ever-changing human gestures, accurate recognition cannot be achieved simply by relying on Arduino, sensors, and leap motion. In the future, it is essential to build a user’s big data gesture library and calculate the data through a machine learning algorithm in order to make human–computer interaction more accurate and intelligent.