Promoting Autonomy in Care: Combining Sensor Technology and Social Robotics for Health Monitoring ^†

Abstract

As the world’s population grows significantly older, there are not enough caregivers in many countries for all the elderly people in need of care. To promote their autonomy while also supporting their caregivers, we propose a health monitoring system comprised of a social robot, and various wearable and non-wearable sensors. Through the use of patient-reported outcome measures (PROMs), captured in conversation with the social robot, the subjective health status of the user is determined. This is supplemented by objective information gathered from wearable and non-wearable sensors used to measure numerous biosignals. By combining the subjective data obtained from interaction with the user and the objective data from the sensor network, a health report for both users and caregivers is generated. The data are visualized for the user and caregiver in a customizable and easily accessible health monitoring dashboard, which also warns the user and their caregivers when the data deviate from the expected values or ranges. The goal is to use this information to improve the quality of care, as changes in the user’s health status can be determined more quickly by themselves and their caregivers. The proposed system establishes a good base for further testing and optimization together with the user, to ensure a useful and appropriate combination of sensors and technological devices that the user is comfortable with.

1. Introduction and Related Work

Combining social robotics and health care sensors to acquire health-related data in a useful way for both elderly users and their caregivers presents a challenge. At the end of 2019, there were approximately 4 million people in need of care in Germany [1]. Due to the demographic transition, this number is expected to reach more than 6 million by 2050. To cope with the resulting increase in demand for elderly care, an additional 130 to 150 thousand trained health care workers are needed [2]. At the same time, 1.76 million people are cared for by their family members in their homes [3]. With increasing strain on caregivers and nursing personnel, support is crucial to maintain the quality of deliverable healthcare. One option to accomplish this, while also benefitting the elderly person at home, is through the use of assistive technologies. These include, among other technologies, wearable and external sensors for health monitoring, and social robots for personal interactions. The use of such technologies can support the autonomy and increase the quality of care of elderly people still living at home. Additionally, they can support and unburden caregivers by providing them with health-related data and by taking on simpler, repetitive, and tedious tasks, freeing up the caregiver’s time for valuable interpersonal contacts.

There are various approaches in the research field of assistive technologies in the care sector. In [4], Majumder et al. provide a survey on wearable sensors used for health monitoring, which includes electrocardiograms (EKGs), electrodermal activity (EDA) sensors, blood oxygen saturation (SpO₂) sensors, and activity monitoring devices. In [5,6,7], further remote health monitoring systems are presented. In their review, Marques et al. [8] detail internet of things (IoT)-related technologies used in health care, which are essential for connecting sensors to health monitoring systems. The ENRICHME project [9] uses an assistive robot to support elderly people at home. Loza-Matovelle et al. [10] introduce an architecture of robots and sensors for elderly in an ambient assisted living environment. Do et al. [11] combine smart home sensors with robots for activity monitoring, fall detection, and rescue by the robot. Robots and sensors are also used to assist elderly people with dementia [12] and depression [13]. In [14], a social robot is used to complete patient-reported outcome measures (PROMs), used to determine a patient’s health and well-being, in conversation with the patients. The participants in this study had a positive impression of the interaction with the robot.

While there are various health sensors and robotic solutions available both commercially and in research, they are still not considered state of the art in the field of elderly care. Among the largest concerns regarding such technologies are privacy, trust, and the technologies’ added value [15]. This finding highlights the importance of a user-centered design of these systems, which requires an interdisciplinary perspective. In doing so, the sensors, user interface, and interaction with the robot can be adjusted based on the preference of the users.

In this work, we propose an architecture as a base for the sociotechnical development of an ecosystem comprised of health care sensors and a social robot to promote the autonomy of elderly people in need of care. Through the integration of various sensors capable of measuring different biosignals and health-related data, such as heart rate, body temperature, or brain activity, objective information regarding the overall physical health of the elderly person living at home can be collected. This is complemented by subjective information acquired through the use of PROMs in conversation between a social robot and the elderly user, based on our work detailed in [16]. By combining the objective information gathered through the sensors and the subjective information obtained through the interaction between the robot and the user, a health assessment report is generated, which is available to both the elderly person and their caregiver or nurse.

In the remainder of this work, we present a detailed view of our sensor–robot ecosystem, provide an example for its user interface and discuss the research questions that will be investigated using the proposed ecosystem.

2. Sensor-Robot Ecosystem

In order to find an optimal sensor–robot ecosystem for the users, a complex system of hard- and software components is required. The proposed system is comprised of various health care sensors, a social robot, a data processing unit and a visualization dashboard–user interface, as shown in Figure 1. The elderly person or people at home are outfitted with wrist-worn wearable devices (pink) that incorporate sensors capable of continuous measurements of objective health-related data, such as skin temperature and heart rate. Additional information is acquired in interval measurements, when needed, with stationary sensors (green). These include an EKG sensor to determine heart health, an electroencephalogram (EEG) sensor to measure brain activity, an EDA sensor for emotional responses, an electromyography (EMG) sensor for muscle activity, and an SpO₂ sensor. Through conversation with the social robot (red) and the use of PROMs, the subjective well-being of the elderly person is determined. When sensor data are acquired that deviate from the expected values, a suitable PROMs questionnaire is chosen and the social robot approaches the user to explore the questions in a conversation. Likewise, when PROMs answers are given that indicate a negative change in the user’s health state, the sensor data are used to determine if there are correlations. If the data are highly alarming, the user and/or the caregiver/nurse/doctor are immediately notified. This acquired objective and subjective data are collected and processed in the data processing unit (blue) and then used to generate a health assessment report (purple) for the user and caregiver/nurse. Additionally, they are visualized in the health monitoring dashboard (yellow). The specific components are described in the following.

Figure 1. The proposed ecosystem consists of various health care sensors for continuous (pink) and interval (green) measurements in a local sensor network (light blue), a social robot (red), a health monitoring dashboard (yellow), a data processing unit (blue), and a health assessment report (purple).

2.1. Hardware Components

The robot used in our ecosystem is the social robot Pepper (SoftBank Robotics, Tokyo, Japan). For the continuous sensor measurements and data transfer, we use the WiFi- and Bluetooth-enabled ESP32 microcontroller (Espressif Systems, Shanghai, China). The MAX30102 is used to determine the heart rate through an optical measurement of the pulse wave, and the MAX30205 is used to measure the skin temperature (both sensors: Maxim Integrated, San Jose, CA, USA). To record an EKG, the MAX86150 (Maxim Integrated, San Jose, CA, USA) development board by Protocentral (Bangalore, India) is used, as it allows a measurement with dry electrodes using the index fingertips of each hand, while simultaneously recording a photoplethysmogram (PPG) used to determine the SpO₂ levels. For periodical measurements, i.e., EEG, EDA, and EMG, the BITalino (r)evolution Plugged Kit BLE/BT (PLUX Wireless Biosignals, Lisbon, Portugal) is used. For the visualization using the health monitoring dashboard, any internet-enabled device with a browser can be used. To facilitate the user interaction with the dashboard, a large touch screen device is recommended. Finally, to process and store the acquired data, a computer or server connected to the local WiFi network is required.

2.2. Software Components

In order to facilitate the conversation between Pepper and the user and to be ready for a variety of answers to the PROMs questions, we use Google’s natural language understanding platform Dialogflow (Google, Palo Alto, CA, USA). To control Pepper, we use the NAOqi software development kit (SDK) provided by SoftBank and the robot operating system (ROS). As an on-premise IoT platform and visualization dashboard, we use the ThingsBoard Community Edition (Thingsboard, New York, NY, USA) connected to an SQL database for storing the acquired sensor data. An example for the visualization using the health monitoring dashboard is given in Section 3. For the communication between the sensors and ThingsBoard, a local WiFi network is used. Depending on the preferences of the user, an online network is only required to send the health assessment report to the caregiver or nurse.

3. Results and Discussion

An example of the health monitoring dashboard, generated using ThingsBoard, is shown in Figure 2. It can be accessed through any browser in the same network, using the IP address and specified port of the host. The heart rate (top left) was measured using an optical measurement on the wrist. The skin temperature (top middle) was also measured by skin contact on the wrist. While this type of sensor and application can not be used to measure the core body temperature, it is nevertheless useful in determining fluctuations in skin temperature and deviations from the normal temperature values of the user, once a baseline has been established. The EKG (bottom) is shown in real time and can also be used to measure the heart rate or the heart rate variability to determine the heart health of the user. Additionally, the obtained EKG can be sent to the nurse or doctor of the user. It is also possible to visualize historical data using the dashboard, the desired interval can be specified using the top right widget. To visualize the data of other sensors, additional widgets can be created and customized based on user preferences.

Figure 2. An example of the health monitoring dashboard is shown, including heart rate (beats per minute), skin temperature (${}^{\circ }$C), and an EKG measurement. Historical data can also be chosen for visualization.

This work highlights the technical viewpoint of the proposed architecture for a sensor–robot ecosystem for the health monitoring of elderly people at home. However, it also provides a basis for integrating the interdisciplinary sociotechnical perspective by investigating the following research questions during the further development of the system together with the users:

• Which are the most important PROMs and sensors to detect specific health and care situations? How large are the differences between individual users?
• (How) can we combine PROMs and different sensors to increase the quality of care of elderly users in their homes?
• In which way, if at all, does this combination reduce the burden of care of both the elderly users and their caregivers (both trained and untrained)?
• To which degree can these results be transferred to other age groups and care situations (e.g., nursing homes)?

4. Conclusions and Future Work

As mentioned in Section 1, the biggest concerns regarding the use of assistive technologies in the care sector are privacy, trust, and the added value of such technologies. By including the user in the development of our ecosystem and thus answering the above mentioned research questions, we can address and alleviate these concerns. Additionally, through the use of a social robot, the user’s permission to store their data and/or send them to others in the health assessment report can also be periodically updated or revoked, if desired.

In the future, we will integrate more sensors into the ecosystem, such as sensors for fall detection, activity monitoring, emotion recognition, and on-wrist blood pressure measurement through the use of PPG and EKG, as explained in [17]. It is also important to test the robustness of the system through long-term measurements in realistic settings, in order to determine the effects of activities and situations not anticipated in the lab setting. In this context, we will also investigate how well the system works when only the users interact with it and deduce how to optimally reduce required maintenance to ensure long-term user satisfaction.