Exercises that are prescribed, regulated and controlled, help people with motor difficulties to recover mobility in the affected limbs and, therefore, quality of life. A frequent practice is to have specialized personnel in the rehab sessions (i.e., physiotherapists), who supervise the quantity and quality of the exercises. An alternative to specialized personnel is the use of robots designed to assist patients in performing rehabilitation exercises [1
], which is usually possible only in specialized or clinical environments (e.g., hospitals). However, these are expensive solutions and may imply the displacement of patients with reduced mobility.
A cheaper but less effective approach is to perform the exercises prescribed independently at home, without the supervision of a professional. However, a study indicates that only 31% of the people that suffer from a mobility disorder execute the exercises correctly [2
]. In addition, loss of patient motivation is frequent.
The proliferation of increasingly cheaper sensors has allowed for the creation and development of systems capable of automatically controlling the frequency, duration and correction of the prescribed exercises [3
]. In addition, the use of multimodal user interfaces, together with the gaming technologies, could help to keep the user motivated and engaged during the execution of the exercises. The ultimate goal is to maintain patient adherence to treatment, which is particularly important in long-term rehabilitation due to the need to reduce recovery periods [6
This work introduces KineActiv, a system designed to replace physiotherapists in supervising upper limb exercises, helping the patient to achieve a more effective and faster rehabilitation. The system relies on an RGBD (Red, Green, Blue, Depth) sensor (MS-Kinect V2) and on a friendly, interactive, gaming-based user interface, aimed at making the execution of the exercises easier and enjoyable. The system quantitatively measures upper-limb movements and compares them with expected goals previously established by the physiotherapist. It also allows the specialist to perform real-time monitoring of the progress of the patients through a web-based system, with statistics that summarize patients′ performance.
Experiments were conducted on a real dataset consisting of quantitative measurements obtained from 10 patients, when they performed a number of sessions of a prescribed rehabilitation routine. Statistical analysis allowed us to evaluate the accuracy and reliability of the system, as well as its sensitivity at measuring the progress of patients.
This manuscript is an extended version of a previous conference paper [7
], which was limited to a usability study on the interactive user interface of KineActiv aimed at measuring the patient′s immersion experience. Unlike, Chang, Y.J. [4
], this paper provides a comprehensive evaluation of KineActiv, bringing together a quantitative analysis of its effectiveness as a rehabilitation tool, a detailed description of its operational workflow, and the aforementioned usability study. The effectiveness study is intended to establish the accuracy and sensitivity of KineActiv, while the workflow description stresses the opportunities for real-time monitoring of the degree of correction of prescribed exercises. Finally, this work includes a much broader review of the state of the art development.
2. Related Work
The performance of Microsoft Kinect as a tool to evaluate kinetic variables, as compared to more conventional solutions, is a subject of great interest [5
]. Chang [4
] compared Kinect against the high-fidelity OptiTrack system, showing that the former can achieve competitive motion tracking performance. Kinect robustness at modelling the human skeleton in the presence of partial occlusions has promoted its use as a marker-free motion monitoring system, being a low-cost solution as regards more specialized vision-based technologies (BTS, Vicon, OptiTrack, etc.). Naeimabadi et al. [8
] evaluated the accuracy and usability of the Kinect 1.0, 2.0 and wearable devices for tele-rehabilitation of the knee. A routine of exercises was defined to measure the angle of the knee and determines the accuracy. Their findings showed that the second generation of Kinect and wearable sensors have acceptable accuracy, with average errors of 2.09°, 3.11° and 4.93° for Kinect 2.0, accelerometers and inertial measurement units, respectively. Tanaka et al. [9
] conducted a study aimed at assessing the accuracy of Kinect compared to a marker-based motion capture system. They concluded that, in spite of differences with marker-based systems, Kinect could be useful for accurately classifying movements. Zhou and Hu [10
] conducted a review of movement systems for rehabilitation. Six criteria were considered: cost, size, weight, function, operation and automation. Marking-free visual systems were stressed because of their small dimensions, robust performance and low cost.
Other studies have shown that Kinect is able to measure gross movements, which makes it suitable for stroke rehabilitation, Napoli et al. [11
] or for measuring motion disorders in people with Parkinson′s disease [12
]. Varona, J. et al. [14
] and Zanatta, J. et al [15
] study, a Kinect-based system with an interactive virtual environment was successfully used for rehabilitation of upper limbs in stroke patients. Based on motion data and signals obtained from ergonomic measuring devices, the system monitors and assesses the rehabilitation progress. It has also been determined that Kinect device can track body movement with the precision required for standard equilibrium tests [16
], such as foot balance assessment [17
]. Kinect sensitivity was also stated in, Obdrzalek, S. et al. [18
], where postures of elderly people in standing and sitting positions were accurately estimated. Results proved to be very useful in active therapy exercises.
Rehabilitation of brain injuries has also been addressed through virtual reality [19
]. Da Costa and de Carvalho [20
] and Edmans [21
] showed the positive results of a virtual reality device for cognitive rehabilitation. The potential of virtual reality in stroke patients was investigated by Rand [3
] and Broëren [22
]. Following these results, Pyk et al. [23
] presented a virtual reality therapy system for the rehabilitation of arms and hands in children. They verified that the system could reduce the therapy cost, increase patient motivation and objectively evaluate the progress made. Several studies have shown that by offering virtual rehabilitation exercises designed as games, it is possible to motivate patients to perform rehabilitation exercises, and also to increase their adherence to treatment, as we can see in, Lange, B., Flynn, S., Lozano, J. et al. [24
Examples of integration of Kinect, gaming and virtual reality for motor rehabilitation purposes of patients with brain injuries can be found in, Pyk, P., Cabrera, R., Jung, I., Jonsdottir, J. et al. [23
]. They have been designed to simply encourage patients to do exercises, without keeping a precise control of the movement. That is, they are intended to educate the brain to recover a lost function, or to mitigate movement degeneration, by roughly requiring a patient to reach a goal.
Unlike the vast majority of previous works, which focus on neurorehabilitation, this paper concerns the use of a Kinect device to accurately measure kinematic parameters of upper limbs, within a gamified and augmented environment. There exist commercial systems with some similarities to the one described here. Kinovea [30
] is a software of sports biomechanics, which is also used in physical rehabilitation. It is a video analyzer that evaluates, corrects and keeps track of movements. It measures times, angles, trajectories, perspectives and coordinates. Despite being a powerful software, the movement must be recorded and then studied. Skill Spector [31
] also records videos first to perform later offline analyses. The system requires the user to manually locate the joints, from which it generates a model for the analysis. Another Kinect-based example is Diaple [32
], in which the patient must imitate video-recorded exercises done by physiotherapists. Then, the system measures how similar the patient′s movements are to those of the physiotherapist. This similarity score can be considered as general feedback, that is to say, no visual cues are given to the patient to correct their personal execution in real time. Other commercial systems are based on sophisticated multi-camera systems that record patients wearing special suits within a controlled scenario. However, they are generally costly and require calibration tasks, thus their use is usually constrained to specialized environments.
Two studies were conducted to evaluate KineActiv. Firstly, a usability study was performed with the enrolled patients by means of a questionnaire Section 4.2
. Secondly, a performance study was carried out to empirically prove the system accuracy and the system sensitivity at assessing upper-limb disorders Section 4.3
Samples were recorded using a MS Kinect device under a spatial resolution of 1920 × 1080 pixels, a mean distance to sensor of 2.5 m, a horizontal field of view of 70 degrees, a vertical field of view of 60 degrees, and a frame rate of 30 frames/sec.
Ten adult patients with some diagnosed arm injury were involved in experiments, distributed into six males and four females with ages ranging from 38 to 83 years old. They were recruited in the SEAP Polyclinics (http://www.policlinicasseap.com/centros/policlinica-teruel/
), a collaborating center that is evaluating KineActiv in its Physiotherapy service. Data consists of series of quantitative measurements obtained from each patient, when performing four upper-limb exercises. Eligible patients were those who showed a clear pattern of health improvement throughout the sessions, which was important to assess the measurements provided by the system when dealing with different health conditions. Data were also acquired from a professional physiotherapist doing the same exercises prescribed to patients, to help establish the accuracy of the system and to be used as control data. Participants signed a written consent form, agreeing to take part in the study subject to their personal data remain confidential. Table 1
shows the patients′ profiles.
Recording took place in a research laboratory where background and illumination were constant. Patients were instructed about the exercise layout (3 weeks × 2 weekly sessions × 4 exercises × 3 series), and they were asked to let themselves be guided through the user interface. The four exercises are combinations of the types of upper-limb motion (abduction, flexion) and the two dynamics (isometric, concentric). Each exercise led to a particular measure.
4.2. Usability Study
After analyzing questionnaires used for similar purposes in the field of rehabilitation and virtual health [36
], our usability questionnaire was designed by combining the items present in two questionnaires that match the goals pursued in this study. Firstly, we adopted the ten items defined in the SUS (System Usability Scale) questionnaire [38
], that have been used in the usability assessment of a rehabilitation system for the upper limbs [39
]. Besides, in order to gather more information, we added the six items used in, Shin, J. et al. [40
] for evaluating flow in a virtual reality rehabilitation system, adopted from, Park, J. et al. [41
]. These items were taken into account since the gamification approach is expected to cause a full immersion in the system, guiding a patient to achieve the goals set out by the physiotherapist. In this way, we have been able to test both usability and flow, when playing with the games included in our system. A total of 16 items were put together in one questionnaire, where every item needs to be scored between 1 and 5, with 1 and 5 being the lowest and the highest values, respectively. Table 2
shows the average values and standard deviations of all items. The items from 1 to 10 are those from the SUS questionnaire, while items from 11 to 16 are those corresponding to flow. Based on the results of the first ten items, the SUS score of the system, a usability measure ranging from 0 to 100, was computed. The overall result was 84.5, a value that establishes the suitability of KineActiv as a usable system.
The second part of the questionnaire, as we have already stated, is about flow when using the system. The flow items (11–16) evaluate three different factors. The first one is attentional focus (items 11 and 12). In this case, the values obtained were not the ones desired, which suggests that the game may not maintain the attention in a strong manner. The second factor is intrinsic interest or pleasure (items 13 and 14). In this case, the values received by the items are good (1.6 in a question where boredom was valued and 4.4 in the opposite one, where they were asked by fun). Thus, the system was considered enjoyable. Lastly, the control was also evaluated (items 15 and 16). Here, values were also good, showing that the use of KineActiv did not cause any negative feelings.
According to all above discussion, the validation of KineActiv usability yielded satisfactory results. The system proved to be easy to use and the flow experience was considered interesting and fun. Also, considering the results of this study, we can state that the gamified activities included in the system attract users and catch their attention. This can be concluded as the items about attentional focus and intrinsic interest received acceptable values in the study.
4.3. Performance Study
This study is intended to establish the accuracy and the sensitivity of KineActiv, by examining the distributions of four measures over patients as a function of the rehabilitation session. As referred above, these measures were defined to quantify the progress of patients in goal achievement when performing the four upper-limb exercises. These four measures were formulated as follows:
Isometric-based measures. Two measures were defined from the isometric evaluation of both types of motion: The Isometric Abduction Index (IAI) and the Isometric Flexion Index (IFI). In each case, the participant was asked to keep the hand within the computer-generated 3D region (the cage) for 45 s, while the system provided real-time visual feedback on exercise correction and execution time. The amount of seconds in which this goal was verified was given as the measure value.
Concentric-based measures. Two measures were defined from the concentric evaluation of the two motions: The Concentric Abduction Index (CAI) and the Concentric Flexion Index (CFI). The participant was asked to repeat the corresponding exercise 20 times, and the angle of the arm with respect to the body was measured in each repetition.
Summarizing, each patient′s data is composed of the following pieces:
IAI Six daily sessions of three series of a single isometric abduction each, resulting in 18 IAI measurements in total.
IFI Six daily sessions of three series of a single isometric flexion each, resulting in 18 IFI measurements in total.
CAI Six daily sessions of three series of 20 concentric abductions each, resulting in 360 CAI measurements in total.
CFI Six daily sessions of three series of 20 concentric flexions each, resulting in 360 CFI measurements in total.
4.3.1. System Accuracy
This section includes two analyses aiming at providing evidence in favor of the accuracy of KineActiv. The first one involves a healthy physiotherapist, who was asked to perform an abduction and a flexion up to 80°, as established by a goniometer. Both exercises were also measured by the Kinect-based system, yielding 80.12° and 80.06° in abduction and flexion, respectively. That is, the relative errors of the KineActiv with respect to the goniometer were 0.15% and 0.075% in each case. The physiotherapist also carried out the same exercises done by the patients, carefully following the instructions. Mean results, measured by the system, were IAI = 43.65 s, IFI = 44.05 s, CAI = 120° and CFI = 120°. That is, measurements were very close to expectations, which can be considered a successful first performance test on the system.
The second perspective of analysis is based on the distribution of standard deviations of CAI and CFI across patients, as functions of the rehabilitation session. These measures were chosen due to their numerous repetitions, which makes them more appropriate for descriptive statistics.
The rationale of using distributions of standard deviations is that narrow distributions of small deviations would empirically demonstrate the stability of the system, while measuring different executions of the same action. Note that each single measurement is the result of a particular execution of a given action, unique by nature, and the system acquisition error. Thus, small deviations from the average can reasonably be assumed to reflect a reliable system.
shows box plots representing the distributions of standard deviations across patients for each measure and daily session. As observed, they are very tight distributions of small deviations, most of them between 1 and 2 degrees in angle measurements in the range 80–120 degrees. This level of disagreement is consistent with the Kinect error of 2.09 degrees reported by Naeemabadi, M. et al. [8
]. Deviations tend to decrease with the progress of rehabilitation (with performance improvement), with the exception of the last session, possibly the most demanding one. This second analysis also shows the very reliable behavior of the system.
4.3.2. System Sensitivity
depicts the distributions of mean values of patients′ isometric-based measurements as a function of the session. The domain of these measures ranges from 0 to 45 s, with the latter being the goal established by the physiotherapist in this study. Thus, the higher the value, the better the execution. Resulting box plots can be considered narrow, except for the lower whisker, suggesting a high level of agreement among the majority of patients. In particular, the lower whisker is stretched by patient number eight, who performed considerably below the rest. When looking more closely at results, most distributions (box plots) are strongly left-skewed. This means that the half of patients with poorer performance (below the median) progress more unequally, while those patients who are closest to goals, progress more evenly.
This result is somewhat consistent with intuition: there is more dispersion among patients with greater disorders. Figure 9
shows similar distribution patterns of the two concentric indexes. The domain of these indexes ranges from 0 to 120°, with the latter being the goal established by the physiotherapist in this study. Thus, the higher the value, the better the execution.
This work has presented a RGBD-based interactive system (KineActiv) which proposes a holistic approach to the rehabilitation of upper limbs. KineActiv provides a gamified and augmented user interface designed to replace physiotherapists in the supervision of upper limb exercises. The system guides patients through customized games on augmented scenes, which aims at encouraging patients to achieve a number of goals established by the physiotherapist. KineActiv tracks and measures limb motions, compares them against goals, and provides real-time feedback to patients about whether the exercise is meeting the expected goals. Our approach includes a web platform that allows the specialist to monitor the progress of patients by means of statistics, tables and charts. The main goal has been to create interactive, simple to use and fun environments that favor more effective and faster rehabilitation processes, while keeping patients engaged.
Two studies were conducted to evaluate KineActiv based on the experience of ten patients. Firstly, a usability questionnaire was designed to measure both usability and flow when using the gamified environments. Secondly, a functionality study was performed to establish the accuracy and the sensitivity of KineActiv, by examining the distribution of four measures of limb exercises over patients as a function of the rehabilitation session. Results empirically prove that KineActiv is a usable, enjoyable and effective system, that does not cause any negative feeling.
Several issues remain for future research. The system is open to new exercises and associated games to rehabilitate any part of the body, as well as to improve the quality of games with better graphics and more engaging goals. For example, it would be interesting to conduct competitions among patients with similar injuries, in order to maintain adherence to treatment.