Assistive Robot with an AI-Based Application for the Reinforcement of Activities of Daily Living: Technical Validation with Users Affected by Neurodevelopmental Disorders

Abstract

In this work, we propose the first study of a technical validation of an assistive robotic platform, which has been designed to assist people with neurodevelopmental disorders. The platform is called LOLA2 and it is equipped with an artificial intelligence-based application to reinforce the learning of daily life activities in people with neurodevelopmental problems. LOLA2 has been integrated with an ROS-based navigation system and a user interface for healthcare professionals and their patients to interact with it. Technically, we have been able to embed all these modules into an NVIDIA Jetson Xavier board, as well as an artificial intelligence agent for online action detection (OAD). This OAD approach provides a detailed report on the degree of performance of a set of daily life activities that are being learned or reinforced by users. All the human–robot interaction process to work with users with neurodevelopmental disorders has been designed by a multidisciplinary team. Among its main features are the ability to control the robot with a joystick, a graphical user interface application that shows video tutorials with the activities to reinforce or learn, and the ability to monitor the progress of the users as they complete tasks. The main objective of the assistive robotic platform LOLA2 is to provide a system that allows therapists to track how well the users understand and perform daily tasks. This paper focuses on the technical validation of the proposed platform and its application. To do so, we have carried out a set of tests with four users with neurodevelopmental problems and special physical conditions under the supervision of the corresponding therapeutic personnel. We present detailed results of all interventions with end users, analyzing the usability, effectiveness, and limitations of the proposed technology. During its initial technical validation with real users, LOLA2 was able to detect the actions of users with disabilities with high precision. It was able to distinguish four assigned daily actions with high accuracy, but some actions were more challenging due to the physical limitations of the users. Generally, the presence of the robot in the therapy sessions received excellent feedback from medical professionals as well as patients. Overall, this study demonstrates that our developed robot is capable of assisting and monitoring people with neurodevelopmental disorders in performing their daily living tasks.

1. Introduction

Our lives have become more accommodating and improved as technology advances rapidly, and these changes affect people from all walks of life. Besides improving general aspects of human life, some research teams also focus on improving the quality of life for people with functional diversity. These advances are intended to benefit these individuals. In recent years, advances in robotics and artificial intelligence (AI) have allowed social assistive robots (SARs) to become increasingly capable of helping and assisting humans in a variety of different environments and conditions. As outlined in [1], SARs are robots that provide assistance through social mechanisms rather than physical ones. A number of studies have been conducted on SARs and the interaction between humans and robots in relation to cognitive and social skills [2,3,4,5].

In 1988, the first documented robotic-assisted surgical procedure was performed with a robotic arm and a computerized tomography (CT) scanner for a CT-guided brain tumor biopsy [6]. Since then, technological advancements have greatly increased trust in robot capabilities and they have helped people with both their mental and physical needs [7,8,9,10].

Generally, SARs are utilized for the following purposes: caring for humans, performing domestic duties, carrying out human tasks or assisting in the performance of human tasks, enhancing education systems, and assisting in the provision of medical care. Based on Dawe et al.’s [11] analysis, social robots are primarily companions, providing pleasure, entertainment, role models, buddies for playing and learning, and coaches for providing information and demonstrating exercises.

As part of the development of SARs, one of the focus areas is the provision of assistance to people with neurodevelopmental disorders (NDDs) with their activities of daily living (ADLs). It is estimated that 74% of Spanish citizens, older than 6, with disabilities have difficulty performing ADLs [12], and prevalence rates of NDDs range from 4 to 13% in school-age children depending on the country [13,14,15]. These difficulties often continue into late adolescence and adulthood [16], so it is important to assist them in overcoming their limitations and learning or reinforcing their skills in performing ADLs. According to Pivetti et al. [17], children with NDDs can benefit from interacting with and learning from robots in a more engaging and social way. It was found that SARs assisted kids in attaining learning objectives, engaging in learning activities, and interacting with others as a result of this study.

In this work, we propose the first study of a technical validation for an assistive robotic platform, which has been designed to assist people with neurodevelopmental disorders in enhancing ADL learning. This platform is referred to as LOLA2 and it is based on an artificial intelligence application that is intended to support the learning and reinforcement of daily living activities by people with disabilities. Figure 1 shows an overview of the action monitoring process with LOLA2 and users with NDDs. Our robot is equipped with an online action detection module, which allows real-time monitoring of the actions that users are performing during interventions with the platform. Those with NDDs of all ages can benefit from this work in learning or reinforcing their daily living skills. Our robot is equipped with an intuitive graphical user interface that enables users or their responsible professionals to choose the ADLs they wish to learn or practice. Upon activating the AI-based action detection application, they can begin performing the requested action in front of the robot. A report containing the details of the action monitoring process can be reviewed by the responsible professional following each session to assess the user’s progress. The main contributions of this paper are:

• We introduce the novel robotic platform called LOLA2. It is an improved version of our previous assistive platform LOLA [18]. Section 3 details all the technical improvements made.
• For the robot, a user-friendly graphical interface is designed and developed to enable efficient human–robot interaction (see Section 3.2.2). Its objective is to interact with individuals with NDDs and to assist in their therapy sessions to reinforce activities of daily living. Based on artificial intelligence techniques, we have embedded in LOLA2 an online action detection module designed for monitoring ADLs, which is detailed in Section 3.2.3.
• This work presents in Section 4 the first technical validation of the technology proposed with a set of four real final users with NDDs. The results confirm that our developed robot is capable of assisting and monitoring people with NDDs in performing their daily living tasks.

This paper is organized as follows—Section 2 reviews the literature on technological solutions for assistive robots and action detection applications. We present our assistive robot in Section 3 as well as the system for monitoring action applications based on artificial intelligence. In Section 4, experimental validation and results are presented and discussed. Finally, in Section 5, we conclude with some conclusions and suggestions for future lines of work.

2. Related Work

SARs have gained importance in the last decade and have demonstrated a variety of benefits and various spheres of life have been influenced by this. A variety of studies have been carried out in SARs, such as those investigating reductions of pain and anxiety as a result of hospitalization [19,20], consumption of meals [21], housework [22], monitoring the user’s health [23,24], teaching language skills [9], and reducing social isolation and improving well-being through social interaction with the user [1,25,26].

For children with NDDs, it has been proven that they experience difficulties taking care of themselves, playing, and moving, as well as thinking socially [27,28,29]. Gelosmini et al. [30] have determined that the use of robots will allow them to improve their quality of life, encourage human–human interaction, and develop social and cognitive skills.

In light of our robot’s ability to assist socially as well as monitoring actions, we have divided the related work into two parts.

2.1. Social Assistive Robots

A number of assistive robots have been shown to be effective in treating children with NDDs, according to [31], such as Troy [32,33], FACE [34], IROMEC [35], KASPAR [36,37,38], Lego Mindstorm [39,40], and NAO [41]. Designed to evoke one or more specific behaviors in children, each has a unique look, design element, and interaction technique. In addition, children with NDDs are particularly drawn to technological devices [42], and the presence of SARs in therapy can help them become more interested in getting treated and gain confidence and independence in their life [17].

The presence of SARs and their benefits in rehabilitation have also been the subject of several studies focusing on the robotic–human interaction [1,43]. Rehabilitation can be exhausting, painful, and difficult as its exercises must be performed according to specific guidelines. Previous works [44,45], which studied interaction with a SAR over short-term periods of time, suggest that including SARs into a practice that requires repetition may increase patients’ motivation. In some works [46,47], healthcare professionals have been surveyed to determine their perceptions of SARs and their use in therapy and rehabilitation, and their results suggest that SARs can be used to increase engagement, motivation, and compliance. Another study investigating the effectiveness of SAR feedback for rehabilitation showed that sessions with users using SARs enhanced task performance [48].

We have developed our robot based on all previous studies to assist in the rehabilitation and therapy process for people of all ages and with varying levels of NDDs so that healthcare professionals can track their improvement during the process.

While most SARs are expensive commercial platforms adapted for use with assistive objects, they are not accessible to a wide range of people of varying income levels [41,49,50,51]. However, our assistive robot is a low-cost model specially designed to assist individuals with NDDs in learning and reinforcing ADLs, and is integrated with an AI module for monitoring daily living activities [18].

2.2. AI for Monitoring ADLs

In essence, AI-based monitoring activities systems rely on action detection modules, which detect and recognize a specific sequence of sub-actions associated with a particular activity. This can be accomplished in a number of ways, including extraction of descriptive information from the environment accompanied by estimation of the human posture [52]. Sensor-based recognition is another popular method of detecting actions, which uses an algorithm to detect defined actions based on sensor information [53,54,55]. Although these methods have high accuracy, they are limited in terms of their ability to adapt to new actions with different objects as a result of the complexity of sensor placement and the possible requirement for additional sensors.

Alternatively, a vision-based method can be used for action detection, where a camera is used to capture action information. The modules integrated into PHAROS [7], RoboPhilo [56], NAO [57], and Gymmy [26] robots can be examples of vision-based methods to monitor activities. The systems mentioned above, however, are designed for the purpose of exercising and they do not include the daily activities upon which our work is based.

A number of studies have proposed a system of action monitoring that includes activities that are more similar to those that are performed on a daily basis [58,59,60]. As most of the proposed systems detect actions offline, the entire video must be fed into the action recognition model to determine when and where the specific actions have taken place [61,62,63]. The concept of early action detection has also been proposed, by which the action label of an action video is predicted prior to the end of the current action execution [64,65]. Based on the assumption that there is only one action instance in the video stream, the start and end frames are determined once the action has ended. Therefore, they are not suitable for monitoring actions according to our objectives, and we have chosen online action detection systems that are capable of detecting actions as soon as they occur. In the field of online action detection, few studies have been conducted [66,67,68,69], and the majority of those studies do not explicitly distinguish action from background [70]. This is why we have returned to our previous approach that can explicitly distinguish action from background [18].

3. Human-Robot Interaction Application for the Reinforcement of ADLs

3.1. The Assistive Robotic Platform: LOLA2

LOLA [18] is a low-cost mobile assistive robotic platform, which has been equipped with several sensors that allow it to navigate autonomously through the environment and interact with the users.

As we described in our previous work [18], the internal structure of LOLA is constructed from wood and metal, and the outer shell is entirely constructed from 3D printing. The entire mechanical and electrical design of the platform has been conceived by our research team. For more specific details about the main technical features of this differential wheeled robot, we refer the reader to [18].

For this research work, we have made some improvements with respect to the previous version. LOLA2 now integrates a new NVIDIA Jetson Xavier processing card. Contrary to the previous model, where we integrated a NIVIDA Jetson TX2. The Jetson board acts as an embedded Linux computer, capable of providing robotic platforms with the same level of functionality as a desktop computer. The Jetson Xavier, which we use for this project, is equipped with a Volta GPU with 384 CUDA Cores and 48 Tensor Cores, 8GB of RAM, and a 64-bit CPU Carmel ARM. This change in the Jetson processing board has been motivated by the need to increase the computing capacity of the platform. This has been necessary due to (1) the resources consumed by having a Robotic Operating System (ROS) [71] embedded in the system for all robot control; (2) the need to implement faster AI models for action recognition and navigation; and (3) the integration of the novel human–robot interaction applications specifically designed for the interventions with users with neurodevelopmental disorders. Other design improvements have been applied to the platform to obtain this new version: changes in the position of the sensors, elimination of ultrasonic sensors, and design changes in the casing. LOLA2 integrates a new vision sensor too. It is an Intel RealSense camera to receive high-quality RGB images, which are fed to our online action detection module. Figure 2 shows both versions of LOLA.

3.2. Human–Robot Interaction Application: Software Description

This paper describes a process of applied science in which our robotic platform interacts with both the healthcare personnel who must operate it and the end users of the platform. For this research, therefore, we have had to develop a complex human–machine interaction system, which is detailed in this section.

As is shown in Figure 1, the main objective with LOLA2 consists of the development of an AI application for the reinforcement of ADLs for users with neurodevelopmental disorders. It has therefore been necessary to provide LOLA2 with an interface to efficiently control its movement, as well as to develop a graphical user interface that allows efficient interaction with patients and healthcare professionals.

3.2.1. ROS Integration and Navigation Interface

Robot LOLA2 has been fully integrated with ROS [71] (Melodic version). The whole robotic platform, including all its sensors, is controlled via ROS. In addition, to provide the platform with advanced navigation capabilities, it was decided to use the ROS navigation libraries. Figure 3 shows the complete ROS nodes architecture we have embedded in LOLA, mainly using Python. Blue nodes represent distributed software ROS packages, and yellow nodes are those ROS nodes entirely developed by us to integrate ROS into our platform.

Recall that the ROS system is fully embedded in the NVIDIA Jetson board of LOLA2, communicating with an Arduino board that is in charge of the control of the motors and wheels. The ROS Navigation Stack, which is the backbone of the navigation system, receives input from the following sources: a map from the map server, odometry from the wheels, LIDAR data, and an estimated position, in this case provided by the Adaptive Monte Carlo Localization (AMCL) package. RVIZ can be used as the user interface. The visual environment provided by this ROS package shows the map and the estimated robot position. The user can direct the robot to move to a specific place while defining yaw rotation, as well as give it an initial position to assist it localize itself. The move_base core package (see Figure 3), which creates the linear and angular velocity commands required to move the robot to the appropriate place, will receive ROS messages from RVIZ. The information about any new potential obstacles, the odometry, and the projected localization are all continuously updated for all these commands and their parameters. We created the Hardware Interface package to fully integrate ROS into our robotic platform. It translates ROS commands to the Arduino board and provides signals back to the ROS system in the precise format required to complete the communication loop with the Arduino. The developed libraries for establishing serial port-based communication between the platform’s engines and ROS are included in the Arduino package in Figure 3. We have specifically created a communication protocol with commands that enable us to read the data from the encoders and to produce a series of speed directives to be sent to each wheel.

Finally, for enhancing the robot’s movement, as well as the usability of the platform, we have incorporated a joystick (Logitech gamepad F710) to enable professionals and users to more precisely control the robot’s displacement. With a Directional Pad or Thumbstick, the joystick can be used to navigate the robot in four different directions, depending upon the user’s position.

3.2.2. Graphical User Interfaces

Our assistive robot application was developed with a user-friendly graphical interface that includes distinct sections; see Figure 4. It is the responsibility of healthcare professionals to manage the interface shown, but it can also be managed by users who are cognitively capable of making their own decisions and taking control over the application. Any process of reinforcement of an activity of daily living requires the following steps:

• Configuration step: The healthcare professional selects a user (identified by an ID) and particular action for the monitoring.
• Activity observation: The user watches a video of the chosen action.
• Activity monitoring: The user is encouraged to replicate the observed action, and the AI online action detection module starts the automatic monitoring of the performance. The software also provides the option to generate a report of the performance of the user during therapy sessions.

The user ID, as well as the action that we want the user to perform, can be selected from the main menu of the application (see Figure 4a). Once the configuration step has been completed, the final users must watch an example video of the chosen action. This phase is started with the button in Figure 4b. For this research work, our team had to record for each of the actions of interest, up to four videos, along with step-by-step sub-action instructions for achieving the final objective of that action. Figure 5 shows some examples of two actors performing some of the actions of interest. All this dataset of videos is integrated into LOLA2 along with the application. When the professional selects an action to work with, one of these sample videos is randomly presented to the user. This option illustrates and instructs the user on the steps involved in taking a daily action. Then, the user may begin performing the chosen action in accordance with the observed video. Once the user is ready to carry out the requested action, the activity monitoring step will be activated by professionals. An AI-based online action detection (OAD) system is integrated to monitor the user’s actions, as detailed in the following section. The developed application allows the generation of a report containing a detailed log of the entire work session with the patient. In particular, the log file can be used to obtain the following statistics: How much time has the user spent on certain tasks or actions? What actions are carried out in certain time slots? Have been the actions correctly performed?

3.2.3. AI for Online Action Monitoring of ADLs

Considering the need to monitor the activity of users, the inclusion of an action detection system in our robot was a no-brainer. As an standard action detection system would require that an entire video of the action be fed into the neural network to recognize the action correctly, we believe that these models are ineligible for monitoring actions in this project. They work offline, in the sense that the action of interest must be completely covered by the video. Instead, for our robotic application, we need an action recognition system that is able to work with partial observations of the actions, or, in other words: to address the problem of localizing actions in untrimmed videos as soon as they happen, which was coined as Online Action Detection (OAD). Therefore, action detections in an OAD technique must be made over video streams, requiring the use of incomplete observations in which the action segments may be more likely to be the exception than the rule relative to the background. Furthermore, this online setting includes a crucial element: the anticipation of the action. For an OAD model, the objective is to anticipate the action even before the action is fully completed, an aspect that is fundamental to attaining an efficient human–robot interaction in the monitoring application designed for this project: LOLA must be able to recognize the actions the users are performing in an online fashion, so as to provide a real-time response.

For our robotic platform, we propose a Pytorch implementation of the 3D CNN model described in [72]. See Figure 6 for details of the deep learning architecture deployed on the NVIDIA Jetson Xavier board. Our approach uses a 3D Convolutional Neural Network (CNN) that is trained on the UCF-101 dataset containing 101 distinct categories of actions [73]. As input, our OAD model is given a 16 frame clip. The model is able to cast an action prediction for every clip, classifying the actions as soon as they appear. Using the Jetson board and a live video stream received from the robot’s camera, this method produces action estimations at a rate of more than 5 frames per second.

Considering that the robot’s camera is pointing at users, a recommended distance between users and the robot, in this case, is 1.5–2 m. The actions have been split into two groups according to whether the face needs to be viewed or whether the body needs to be viewed, as it is shown in

Table 1. The four categories of actions and their status regarding zooming.

Action Category	With Zoom/ Without Zoom
Blow Dry Hair	With zoom
Brushing Teeth	With zoom
Writing On Board	Without zoom
Mopping Floor	Without zoom

. The option of detecting the face and zooming on it is activated or not based on the requested action by the professionals. Whenever the face is required, we have implemented a traditional face detection model [74] with python and the OpenCV library [75]. Technically, our solution performs tracking of the user’s face over the captured 16 frames, creates a specific clip with the face only, and feeds it to the OAD model for the recognition of the action category. Figure 7 graphically shows how the implemented pipeline works. In the case of actions that are performed via body movements, the original video frames are prepared for feeding the OAD model.

Our OAD-3D-CNN is a heavy model containing a large number of parameters. The choice to implement this complex OAD model on the Jetson Xavier board slows down the GUI-integrated displays. The users were unable to see themselves with a mirror-like effect on the robot’s touch screen and were viewing the output of the camera with a delay. This aspect might provide a poor human–machine interaction experience. We, therefore, decided to implement a multi-threading application to solve this issue. Using different threads for the AI model and the visualizations in the GUI, we were able to provide a mirror-like effect to the users when they are performing the actions and seeing themselves on the screen of LOLA. All these improvements to the application were developed to make the human–robot interaction satisfactory.

Finally, we have recorded a video in which we show how the complete application described for the LOLA robot works https://youtu.be/JJUhtwISZaU (accessed on 19 April 2022).

4. Experiments

4.1. Experimental Validation

4.1.1. Sample of Final Users

The main objective of this work is to perform a technical validation of the robotic platform LOLA and of the application described. For doing so, a fundamental step consists in the selection of a sample of final users so that LOLA and the developed application can be experimentally validated.

This technical validation we propose includes four different patients. They were selected following the principles described in the protocol detailed in [76]. According to this study, all participants should be classified as having disabilities resulting from diseases or permanent health conditions (with the appropriate certificate, according to Spanish law [77]). Their functional independence level should be moderate-low, their functional skills-mobility level should be moderate-low, and they must be registered at one of the collaboration centers. Each of the patients we selected met these requisites. They ranged in age from 7 to 23 and all had physical limitations and cognitive deficits.

The first patient is a 7-year-old female diagnosed with idiopathic ataxia. She is level III in the Gross Motor Function Classification System Extended and Revised (GMFCS E&R), level III on Manual Ability Classification System (MACS), and level III in the Communication Function Classification System (CFCS). The nature of her condition required her to use a walker for short distances and a wheelchair for longer distances. She manipulates objects with difficulty and needs assistance in preparing or modifying her activities. She offers a challenging scenario where the AI models will be technically validated when the users need a walker and direct support from the healthcare professionals.

The second patient is a female 14-year-old who is diagnosed with Cerebral Palsy—Spastic Diparesis. She is level II on GMFCS E&R, level II on MACS, and level III on CFCS. Consequently, she can walk under most conditions and manipulate most objects, but with some reduction in quality or speed.

The third patient is a 15-year-old female who has Cerebral Palsy—Spastic Triparesis with the predominance on the right side. She is level II on GMFCS E&R, level III on MACS in the right upper limb, and level II on CFCS. She is capable of walking in most conditions, manipulating objects with difficulty, and needs assistance preparing and/or modifying activities.

A female 23-year-old, the fourth participant in this technical validation study, is diagnosed with metabolic syndrome due to GLUT 1 deficiency. She is level II in GMFCS E&R, level II in MACS, and level II in CFCS. Thus, she is capable of walking under most conditions and manipulating most objects, though with a corresponding reduction in quality or speed.

In light of the fact that the level of autonomy regarding movement varies from patient 1 to patient 4, our proposed SAR will be validated in all these different situations with a certain granularity.

4.1.2. Design of the Interventions

Before starting interventions with end users, health professionals in the therapy center were given a manual on how to operate and use the LOLA platform. A briefing session was held with the personnel to answer any questions they may have. In therapy, the patient’s physiotherapist is responsible for all evaluation processes involving the assistive robot. We aim that they can autonomously work with the platform and the applications designed. In the context of this technical validation, this is an important aspect, as it is crucial for us to identify what users actually think about the implemented application: is the proposed graphical user interface easy to use? Does the LOLA platform move and interact with the users in an effective way?

Once the health personnel were trained, we were able to start the interventions with the 4 users that made up the study sample.

Each patient is expected to take part in the therapy sessions with the robot for a maximum period of one month (four weeks) to study the results of the system. Only patients 1 and 2 could complete the 4 sessions. Patients 3 and 4 participated for two weeks and one week, respectively.

In each week of the trial, the patients had a 30 min session with LOLA during which they interacted with it and were monitored while performing ADLs. Almost five minutes were required to complete each action, divided into two parts that lasted around two minutes each. For this technical validation, the healthcare professionals selected 4 particular ADLs to be used during the interventions. They are the following: Blow Dry Hair; Brushing Teeth; Writing On Board; and Mopping Floor. All these 4 actions can be recognized by the AI OAD model integrated in LOLA.

Before each session of action monitoring with the robot, all the processes were explained to the patients. After selecting the patient user’s ID and the requested action by the professionals, one of the sample videos is shown to the patients with all sub-actions included. After that, the patients were asked to wait some moments for the AI-based model to be loaded and then start performing the action that they had watched in the videos. When the patients performed certain actions, such as blowing dry hair and brushing their teeth, it was not necessary for them to stand to accomplish the task. Instead, they sat in front of the robot, watching the action and trying to mimic what had been demonstrated. These actions were implemented with the aid of face detection, so the patients were instructed to remove their masks, which were mandatory due to the COVID-19 restrictions. In the remainder of the actions, it was necessary for the patients to stand up and perform the actions so that they maintained an appropriate distance from the robot. Figure 8 shows some patients developing some of the ADLs during the intervention sessions.

Since the main purpose of LOLA is to assist and monitor users, activating the OAD model will generate a report to inform professionals of the progress. The report contains information about the details provided by the AI model for action recognition. For every action recognition by the AI model, the report includes its starting and ending times, as well as an image of the user performing the action. The system also specifies whether the users performed the correct actions or not. All these details included in the report are a fundamental part that will allow the healthcare professionals to follow the protocol in [76]. Specifically, all the variables included in the reports will allow one to validate the level of matching between user and technology, the psychosocial impact of the assistive device on the user’s life, the level of satisfaction in different domains, and the level of independence in the performance for activities of daily living. After each session, these variables can be evaluated alongside the robot’s report to follow the patient’s progress.

4.2. Results

An overview of the progress and results for each of the four patients during this technical validation is presented in

Table 2. This table details the results of the entire process of technical validation of the platform. We report the average accuracy of the AI model when recognizing the actions during the different sessions. Our OAD model has detected the actions of brushing teeth and mopping the floor with the highest accuracy during the sessions. Considering the patients’ physical limitations, blow drying hair, and writing on board were challenging activities for them to undertake.

User ID	Actions	Session 1	Session 2	Session 3	Session 4
UAH-1	Blow Dry Hair	0%	0%	50%	50%
	Brushing Teeth	100%	0%	100%	100%
	Writing On Board	0%	0%	0%	50%
	Mopping Floor	0%	0%	100%	100%
UAH-2	Blow Dry Hair	50%	50%	50%	50%
	Brushing Teeth	100%	100%	100%	100%
	Writing On Board	100%	0%	0%	100%
	Mopping Floor	100%	100%	100%	100%
UAH-3	Blow Dry Hair	100%	50%	—	—
	Brushing Teeth	100%	100%	—	—
	Writing On Board	0%	0%	—	—
	Mopping Floor	100%	100%	—	—
UAH-4	Blow Dry Hair	50%	—	—	—
	Brushing Teeth	100%	—	—	—
	Writing On Board	0%	—	—	—
	Mopping Floor	100%	—	—	—

. The average accuracy of each action in each session for all patients are reported in the table. As we previously mentioned, each action was carried out in two separate attempts to better understand the precision of the OAD model with respect to patients with NDDs. Due to the fact that the OAD model is trained with videos with healthy subjects, depending on the physical limitations of the patients, certain actions were harder for the model to recognize.

With respect to the first patient, here are our conclusions. Due to her condition, she can not walk without a walker and is unable to control her movements. For the first two weeks, she assisted the sessions with her walker. Since she was using the walker, the system was not able to correctly recognize those actions which required an image of her entire body. This is a clear limitation of our OAD model, which would need an update to deal with such situations to attend to all possible patients. Overall, the accuracy of most of the actions she performed was 0% in the first two weeks. In terms of the actions which were implemented with face detection and zooming, the system was able to detect that she was brushing her teeth in one session despite all of her movement limitations. Another two sessions were conducted without her walker, but with a direct assistance of her physiotherapist in holding her. Under these circumstances, Table 2 shows how the accuracy of the AI model increases for all the actions. When the patient brushed her teeth and mopped the floor, LOLA recognized the activity with 100% accuracy. Due to her inability to write or hold the hair dryer in her hand, the system had difficulty detecting these two actions, which resulted in mainly 50% error rates. Figure 9 shows the average accuracy for all detected actions over all sessions for the first patient.

The second case of study is a patient that is capable of walking, although she experiences some difficulty with her balance and has mostly cognitive impairment. Her first encounter with LOLA was pleasant. The experience of seeing herself on the screen while performing an action in front of the camera of the robot appeared enjoyable to her. Table 2 shows the following numbers. There were no errors while the patient was brushing her teeth, and the Al-based module was 100% accurate in detecting this activity. During blow drying hair, there were sometimes conflicts between brushing teeth and blow drying hair because the patient was moving the mouth while performing other activities, and the results were only 50% accurate. When writing on the board, the patient, due to her balance difficulties, relied on the board and covered what she was writing. Several times she was advised to write something visible for the robot’s camera, and LOLA was able to detect her action with an accuracy of 50% over the course of four sessions. A perfect performance was achieved by the patient and the system detected the mopping of the floor with a high accuracy. An average accuracy for each action is shown in Figure 10 based on the detection of actions in all sessions.

The third patient in this evaluation trial had physical and cognitive disabilities, and as a consequence, she had difficulty maintaining an object in her hand. She could only participate in the study for two weeks, covering two complete sessions for the technical validation. She was able to brush her teeth despite all of her limitations, and the system detected this action category with a 100% of accuracy. It was difficult for her to hold a blow dryer and perform the corresponding action, but nevertheless, she was able to perform the action in less quality, and the system was able to recognize it with an average accuracy of 75%. Her difficulty in manipulating the whiteboard marker also made writing on the board a challenging process. Despite her best efforts, she was unable to hold the marker and write on the board at the same time, resulting in the system not recognizing that she had performed the action. Mopping the floor was not a particularly difficult task for her and the system consistently detected her actions correctly with 100% accuracy. Based on the detection of actions in all sessions, Figure 11 shows the average accuracy for each action.

Our last patient was able to grasp and hold objects in her hand but she primarily suffered from cognitive impairments. In addition to only participating in one session, she was not able to focus on her actions and did not show any interest in participating in other sessions. In Table 2, it can be seen that blow drying hair was a challenging task for her as she had short hair and the system was only capable of recognizing the action with 50% accuracy. In terms of brushing teeth and mopping the floor, she was able to perform them perfectly, and the system was able to correctly detect them 100% of the time (see Figure 12 for an example of user 4 performing an action). As she was tall, writing on the whiteboard proved to be the most complicated action, as we had to put a greater distance between the robot and the whiteboard to ensure that she was in the field of view of the camera. The patient was asked to write on the whiteboard in a larger size so that the robot could see the action, but she ignored the instructions. The system, therefore, could not correctly detect this action.

All of the results discussed above were extracted from detailed reports generated by the platform following each session. These reports are primarily intended to be distributed to health professionals at the conclusion of each session to allow them to monitor the progress of their patients during their treatment. With these reports, they will have access to all relevant information such as the action requested, the action detected by the system, the duration of performing the action, and an image of the patient while the action is being performed. From these reports, the health professionals can determine whether the patient understood the process of action correctly, whether he or she performed the action independently in a manner that the robot could detect, and how long the patient remained focused on the action. Additionally, it is also an opportunity for them to observe the progress of the patient’s independence and their ability to interact with new technologies as time progresses [76].

Our general observation is that all patients were interested in interacting with the robot and enjoyed seeing themselves on the screen. In most cases, LOLA is able to correctly detect all of the actions, but we note that there are some actions that need to be refined for the system to detect them with greater accuracy. These are Blow dry hair and Writing on Board. It is important to note that all these trials were performed on individuals with disabilities and impairments. Despite all these issues, LOLA received positive feedback from healthcare professionals and patients and demonstrated promising results during its first technical validation with real users. We can therefore conclude that LOLA and the developed application have successfully passed the technical validation proposed.

5. Conclusions

A study of the first technical validation of a low-cost robotic platform designed to monitor and assist people with neurodevelopmental disorders in their learning of daily living activities is presented in this paper. Our developed robot includes a user-friendly graphical interface, coupled with an online action detection module that helps users learn or reinforce daily living activities and lets health professionals observe their progress through reports of each session.

The evaluation trial included four users performing four daily routine activities in each session. As these users had varying degrees of physical and cognitive impairment, we were able to assess the functionality of our proposed platform in the context of real users with disabilities. In this trial period, we achieved acceptable results which demonstrated that our developed robot is capable of assisting and monitoring people with neurodevelopmental disorders in performing their daily living tasks.

Our future work consists of two different lines. The first line will be to deliver the assistive robot to a care center for people with NDDs for a long-term clinical study with a greater number of participants. In the second line, improvements will be made to the robot’s technical aspects. This will entail enriching the robot’s ability to recognize actions to accommodate a broader range of daily living actions appropriate for people with NDDs. A second improvement will be the incorporation of vocal commands into the graphical user interface for users with high cognitive skills and physical limitations to facilitate a more independent use of the platform.