Wrist Rehabilitation System Using Augmented Reality for Hemiplegic Stroke Patient Rehabilitation: A Feasibility Study

Objective: Our objective was to investigate the effect of the rehabilitation system using augmented reality (AR) on upper extremity motor performance of patients with stroke. Methods: The system using AR applying mirror therapy mechanism provides the intervention protocol for the patient with hemiplegia after stroke. The system consists of a patient positioning tool (a chair), a white surface table, an image acquisition unit, an image processing unit, an image displaying unit, an arm holder, a Velcro-strap, and two blue circle stickers. To assess the feasibility of our system in motor function recovery, a stroke patient was recruited to receive the AR intervention. The treatment was performed two times a day for ten minutes over two weeks (ten days of treating weeks), except for the time of installation, calibration, and three minute breaks. Jebsen Taylor hand function test and Arm Motor Fugl-Meyer assessment were used as primary and secondary outcome measures, respectively, to evaluate the effect of motor function recovery. Additionally, stroke impact scale, Korean version-Modified Barthel Index (K-MBI), active range of motion of wrist joint (ROM), and the grasp force in Newtons were measured. Participants’ feedback and adverse effects were recorded as well. Results: Motor function improvements were exhibited in wrist and hand subtest of Arm Motor Fugl-Meyer (baseline: 19; post-intervention: 23), proximal arm subtest of Fugl-Meyer (baseline: 31; post-intervention: 34), ROM (extending ROM: 10◦ and 3◦ for flexion and extension, repeatedly), stroke impact scale (baseline: 46; post-intervention: 54), K-MBI (baseline: 92; post-intervention: 95), nine-hole pegboard (baseline: 30 s; post-intervention: 25 s), and grasp force in Newtons (baseline: 12.7; post-intervention: 17.7). However, the adverse effects were reported after the intervention. Conclusion: The system using AR applying mirror therapy mechanism demonstrated the feasibility in motor function recovery for the stroke patient.


Introduction
Stroke is one of the main causes of long-term disability in adults [1]. The most general stroke symptom is the limitation of upper limb function which significantly impacts on a person's level of independence and daily living activities [2]. Some rehabilitation interventions which may involve different exercises, type of equipment or techniques have developed to improve the upper limb function and reduce the disability [3][4][5][6]. Particularly, traditional rehabilitation techniques are effective approaches in motor function recovery. However, these methods are time-consuming, expensive, and intensive resources as well as specialized facilities are needed [7,8]. Therefore, post-stroke rehabilitation needs new approaches that allow individuals to enhance motor function.
Recently, mirror therapy has been considered as an effective adjunct to improve upper extremity motor function for stroke patients [9,10]. The principle of the mirror therapy techniques is the reflective superimposition of healthy extremity movements on the affected upper limb through a mirror [11]. This induces a visual illusion of increased movement ability of the unhealthy extremity for the patient [12]. Due to the simplicity of mirror therapy systems, they become difficult to provide a perfectly synchronized mirrored movement for the patient [13].
To address limitations of the mirror therapy, avatar images or virtual objects have been constructed by image processing techniques in virtual reality (VR) systems. These systems create a challenging and motivating environment to provide enhanced feedback of movement characteristics and improved motor learning tasks for the patients [14,15]. However, the VR therapy systems have been widely used for the upper limb rehabilitation and have been adapted for individual patients after stroke [16]. Furthermore, some clinical trial reports figured out the effectiveness of VR in upper limb function improvement [17][18][19]. However, the systems could not provide a high sense of presence and embodiment to the patients due to using the rendered avatar images instead of real photos. Hence, the patients only obtain short-term effects from these systems. Meanwhile, augmented reality (AR) has emerged as one of the effective methods in rehabilitation. The AR combines real and virtual objects to run interactively in a common real-time environment [20]. Despite exhibiting advantages similar to the VR in the upper limb function recovery, the AR could provide a higher degree of presence and embodiment than the VR [21].
In order to overcome drawbacks of the VR technology, an AR system applying the mirror therapy principle has been reported to provide a higher quality of presence and embodiment for the patients [13,22,23]. Moreover, the system could be feasible for long-term neuromodulation. In this study, we proposed an AR system applying mirror therapy mechanism for hemiplegic patients and evaluated its effectiveness in eliciting upper-limb motor recovery.

Subjects
Participants were recruited from an inpatient rehabilitation unit in Korea according to the selection criteria. The inclusion criteria for the participants was (1) above 18 years old; (2) wrist/hand subscales of Fugl-Meyer Assessment for the upper extremity [24] score of above 2-20; (3) nine-hole pegboard [25] test above 25% of contra-lateral hand. Patients were excluded if they had (1) Alexander apraxia scale score of below 2 [26]; (2) National Institutes of Health (NIH) stroke scale question score of above 1 [27]; (3) Nottingham sensory scale less than 75% of contra-lateral hand [28]; (4) presented with spasticity score of more than 3 in the affected limb quantified by the Ashworth scale; (5) NIH stroke scale question IX higher than 2; (6) Beck depression inventory score of above 21 [29]; (7) significant upper extremity disease-causing hand or below elbow dysfunction; (8) visual disturbance; (9) refusal of participation. This study was a randomized, controlled trial. The present study was approved by the Ulsan university hospital institutional review board. The participants were given an explanation of the objective of the study and its requirements.

Hardware
The AR System consists of a patient positioning tool (a chair), a white surface table, an image acquisition unit, an image processing unit, an image displaying unit, an arm holder, a Velcro-strap, and two blue circle stickers as shown in Figure 1a,b. The image displaying unit is a 20-inch screen (Samsung S23C340 1920 × 1080 @ 60 Hz) placed on the table and in front of a user with a monitor arm, which allowed the user to put their hands underneath so that the user only saw their wrists through the monitor. In addition, their hands were placed on the arm holder and immobilized by the Velcro-strap. The image acquisition unit equipped a Webcam C920 of Logitech (Max resolution: 1080 at 30 fps, 78 • Field of view) which was mounted behind the screen. The computer (Intel ® Core i5-7500 CPU 3.4 GHz with 8 Gb RAM) was utilized as the image processing unit to provide an image processing program. The two blue circle stickers were attached on the Velcro-strap and the user's hands to recognize the wrist position.
Appl. Sci. 2019, 9, x FOR PEER REVIEW 3 of 11 Velcro-strap. The image acquisition unit equipped a Webcam C920 of Logitech (Max resolution: 1080 at 30 fps, 78° Field of view) which was mounted behind the screen. The computer (Intel ® Core i5-7500 CPU 3.4 GHz with 8 Gb RAM) was utilized as the image processing unit to provide an image processing program. The two blue circle stickers were attached on the Velcro-strap and the user's hands to recognize the wrist position.

Image Processing Program
Our image processing program was developed based on Microsoft Foundation Classes (MFC, C++) with OpenCV libraries in Windows operating system. Our program was composed of a color detecting function, a counting function, angle and velocity data managing functions, an image flipping function, and an image merging function. The program created a flexible and extensible exercise for the user.
The user was seated on a chair opposite the monitor. Then, his/her hands were placed on the arm holder of the table and his/her wrists were immobilized by the Velcro-strap. The two blue circle stickers were attached to the Velcro-strap and the user's hands (Figure 1b). In order to complete the exercise, the user had to undergo three sections. In the first section, the user performed the flexible and extensible task with his/her unhealthy wrist 5 times. The number of time was automatically counted down by the counting function. Figure 2a shows the preparation of the user's unhealthy wrist prior to the task's beginning. The user had to put the blue circle sticker on the hands into the red circle and keep this position for 5 s. After 5 s, the user carried out the flexible and extensible movement 5 times as far as possible (Figure 2b,c). The moving images of the unhealthy wrist were stored as angles and velocities data by recognizing the flexible and extensible positions of the two blue circle stickers. The image processing unit provided movements of the unhealthy wrist to the image displaying unit without any image processing which allows the users to watch their real wrist movement in real-time for embodiment augment of the displayed images. In the second section, the user implemented a task similar to the first section with their healthy wrist as described in Figure  2d-f. After this section, the moving images of the healthy wrist were mirrored by the flipping image function. In the final section, the user repeated the task one more time. During this section, our system presented the speed-up movement of the unhealthy wrist, which showed the mirrored movements of the stored healthy wrist movements by the real-time image merging function. It meant that the flipping images of healthy wrist movements were processed and transferred to the

Image Processing Program
Our image processing program was developed based on Microsoft Foundation Classes (MFC, C++) with OpenCV libraries in Windows operating system. Our program was composed of a color detecting function, a counting function, angle and velocity data managing functions, an image flipping function, and an image merging function. The program created a flexible and extensible exercise for the user.
The user was seated on a chair opposite the monitor. Then, his/her hands were placed on the arm holder of the table and his/her wrists were immobilized by the Velcro-strap. The two blue circle stickers were attached to the Velcro-strap and the user's hands ( Figure 1b). In order to complete the exercise, the user had to undergo three sections. In the first section, the user performed the flexible and extensible task with his/her unhealthy wrist 5 times. The number of time was automatically counted down by the counting function. Figure 2a shows the preparation of the user's unhealthy wrist prior to the task's beginning. The user had to put the blue circle sticker on the hands into the red circle and keep this position for 5 s. After 5 s, the user carried out the flexible and extensible movement 5 times as far as possible (Figure 2b,c). The moving images of the unhealthy wrist were stored as angles and velocities data by recognizing the flexible and extensible positions of the two blue circle stickers. The image processing unit provided movements of the unhealthy wrist to the image displaying unit without any image processing which allows the users to watch their real wrist movement in real-time for embodiment augment of the displayed images. In the second section, the user implemented a task similar to the first section with their healthy wrist as described in Figure 2d-f. After this section, the moving images of the healthy wrist were mirrored by the flipping image function. In the final section, the user repeated the task one more time. During this section, our system presented the speed-up movement of the unhealthy wrist, which showed the mirrored movements of the stored healthy wrist movements by the real-time image merging function. It meant that the flipping images of healthy wrist movements were processed and transferred to the displaying unit by processing the unit. Processing time was short enough that the user could not feel any movement delay in the flipping movement images. The processing unit replaced the unhealthy wrist movement images by the flipping healthy wrist movement images. For instance, if the unhealthy wrist is at a 20 • position related to the extensible task, the image processing program processed and transferred an unhealthy wrist image at 30 • position of the extensible task to the displaying unit. More specifically, when the real-unhealthy wrist moves to the maximal flexible or extensible position as the position of the white sticker on the screen, the program shows the movement of the unhealthy wrist that moves two red-flag positions as shown in Figure 2g,h. The two red-flag points indicate the maximal range of motion of the normal wrist.

Intervention and Procedure
The intervention was based on the flexible and extensible task which was provided by our AR system. The intervention consisted of three sequences. The first and second were the acquisition of a real-time hand and the third was of virtual hand presence. After each exercise, the validation of virtual reality and augmented reality was checked to assess the ability of the system to provide the degree of presence and embodiment for the patients (Appendix A) [30]. The patients carried out the task two times a day for ten minutes over two weeks (ten days of treatment weeks), except for the time of installation, calibration, and 3-min breaks. Assessments were performed before the first and last intervention.

Outcome Measures
Outcome measures were evaluated before the first and after the last intervention for all patients. The primary outcome was Jebsen Taylor test which was used to evaluate unilateral hand skill and provide an objective test of hand functions involved in daily living activities [31]. This test comprised of a series of seven items that consisted of the following: (1) Writing a 24-letter sentence; (2) simulated page turning; (3) picking up small, common objects such as a coin or a bottle cap; (4) simulated feeding; (5) stacking checkers; (6) lifting large, light objects such as an empty tin can; (7) picking up and moving large weighted cans. The results were measured objectively by a stopwatch after completing each task. The secondary outcome was designed to assess motor functioning, sensation, and joint range of motion in the upper extremity of patients with post-stroke hemiplegia, namely Fugl-Meyer scale [24] (wrist and hand upper extremity Fugl-Meyer includes 12-items [32]) and proximal arm subscale (score on a 0-34 point scale). Furthermore, Ashworth and modified Ashworth Scales, score of nine-hole pegboard [33], stroke impact scale [34], Korean version-Modified Barthel Index (K-MBI) [35], active range of motion (ROM) of wrist joint, grasp force in Newtons, motor evoke potential, and adverse effects (dizziness, psychological anxiety, boredom, gloomy feeling, muscle twist, motion sickness) were recorded after completion of each session.

Results
A hemiplegic patient who satisfied our criteria was recruited to assess the feasibility of our AR system in a clinical trial. Table 1 shows the detailed information of the patient.

Intervention and Procedure
The intervention was based on the flexible and extensible task which was provided by our AR system. The intervention consisted of three sequences. The first and second were the acquisition of a real-time hand and the third was of virtual hand presence. After each exercise, the validation of virtual reality and augmented reality was checked to assess the ability of the system to provide the degree of presence and embodiment for the patients (Appendix A) [30]. The patients carried out the task two times a day for ten minutes over two weeks (ten days of treatment weeks), except for the time of installation, calibration, and 3-min breaks. Assessments were performed before the first and last intervention.

Outcome Measures
Outcome measures were evaluated before the first and after the last intervention for all patients. The primary outcome was Jebsen Taylor test which was used to evaluate unilateral hand skill and provide an objective test of hand functions involved in daily living activities [31]. This test comprised of a series of seven items that consisted of the following: (1) Writing a 24-letter sentence; (2) simulated page turning; (3) picking up small, common objects such as a coin or a bottle cap; (4) simulated feeding; (5) stacking checkers; (6) lifting large, light objects such as an empty tin can; (7) picking up and moving large weighted cans. The results were measured objectively by a stopwatch after completing each task. The secondary outcome was designed to assess motor functioning, sensation, and joint range of motion in the upper extremity of patients with post-stroke hemiplegia, namely Fugl-Meyer scale [24] (wrist and hand upper extremity Fugl-Meyer includes 12-items [32]) and proximal arm subscale (score on a 0-34 point scale). Furthermore, Ashworth and modified Ashworth Scales, score of nine-hole pegboard [33], stroke impact scale [34], Korean version-Modified Barthel Index (K-MBI) [35], active range of motion (ROM) of wrist joint, grasp force in Newtons, motor evoke potential, and adverse effects (dizziness, psychological anxiety, boredom, gloomy feeling, muscle twist, motion sickness) were recorded after completion of each session.

Results
A hemiplegic patient who satisfied our criteria was recruited to assess the feasibility of our AR system in a clinical trial. Table 1 shows the detailed information of the patient.

Motor Function Improvement
To the primary outcome measure, there were small improvements in the writing (2 s of improvement) and stacking checkers (1 s of improvement) subtest of Jebsen Taylor test between pre-and post-intervention. However, no difference was found in the time total of Jebsen Taylor test in both evaluations (baseline: 65 s; post-treatment: 65 s). Noticeably, there was 4 points and 3 points improvement related to the wrist/hand and proximal arm subscale of Arm Motor Fugl-Meyer scale in the second outcome measure, respectively, after 10 days of intervention (Table 2). Furthermore, apparent enhancements were exhibited in the stroke impact, hand function scale, grasp force in Newtons scale, active ROM of wrist joint scale (flexion and extension), in nine-hole pegboard, and Korean version-Modified Barthel Index scale from (improvement from pre-to post-: intervention) 9 points, 5 (kg), 10 • for flexion, 3 • for extension, 5 s, and 3 points, respectively. Nevertheless, there were negligible decreases in both the wrist and elbow subscale of the Ashworth scale (pre-intervention: 1; post-intervention: 0) and the peak to peak Motor Evoked Potential was not evaluated after intervention due to the poor patient condition.

Validation of Virtual Reality and Agmented Reality
The authors used questionnaires consisting of six items with a scale rating of −5 (not at all) to 5 (perfect) for the validation of VR and AR (more details in Appendix A). Table 3 shows the validation results after 10 days of intervention. From the patient's feedback, he did not feel the moving hand on the screen similar to his hand. Table 3. Validation of virtual reality and augmented reality.

Adverse Effects
The authors utilized questionnaires including six items (dizziness, psychological anxiety, boredom, gloomy feeling, muscle twist, and motion sickness) (more detail in Appendix B) with a scale rating of 0 (not at all), 2 (nervous), 4 (little uncomfortable), 6 (very uncomfortable), 8 (disturbed), and 10 (extremely disturbed) to assess the adverse effects after the intervention. The items dizziness, muscle twist, and motion sickness were reported after the intervention as shown in Table 4. Therefore, the patient felt a little uncomfortable with our system after the intervention. Values are presented as mean ± standard deviation.

Discussion
The purpose of this study was to investigate the effects of AR system for the upper limbs of patients who experienced a stroke. Positive effects after the intervention were detected in the participant.
A few studies have chosen the Jebsen-Taylor hand function test (JTHFT) to evaluate the effect of motor function recovery because of (1) the standardized tasks relative to norms; (2) everyday activities tasks; (3) using readily available materials [36]. In our results, there was no change in the total scores of JTHFT, but the results show the small improvements in writing and stacking checkers test of JTHFT after 10 days of intervention. Merians et al. used JTHFT to assess the fine motor dexterity of patients with brain damage. The clinical results showed the improvements in the speed and precision of fine movement in two of three patients. However, there was one patient who did not transfer the improvement to the functional activities after the intervention [37]. These improvements of writing, stacking checker, and speed movement were consistent with the need for fine, sequential, and bimanual coordination skills of the finger and thumb.
In addition, the Arm Motor Fugl-Meyer Test is most commonly used to assess the upper limb function in the stroke patient [38]. In this study, we used the wrist/hand and proximal arm Fugl-Meyer to assess the effect of functional recovery for the mildly impaired stroke survivor. After 10 days of intervention, the results showed the improvement in the scores of wrist/hand and proximal arm Fugl-Meyer. Particularly, the patient was able to reach the highest score of the proximal arm (34/34) and wrist/hand (23/24). Compared to the other methods, Faria et al. reported that both VR and control group showed significant improvements at the end of treatment in wrist/hand Fugl-Meyer (p = 0.034 and p = 0.04) [39]. However, these improvements were significant with respect to baseline, but they were modest for both groups at end of treatment (0.8 ± 1.4 in VR versus 1.3 ± 2.3 in control). This indicated that the improvement of our system in wrist/hand Fugl-Meyer is superior to the improvement in VR. The prominence could be explained by using patients with better functions in wrist/hand Fugl-Meyer in the current study. Furthermore, the specific task of the flexion and extension of the wrist joint promoted by our system could also explain the improvement detected in the active ROM and the nine-hole pegboard test, stroke impact scale, and grasp force in Newtons. This is also consistent with the previous report of robotic rehabilitation [40]. Hence, our system showed to be as effective as other methods since it can promote hand dexterity, as measured by the nine-hole pegboard test and active ROM while being simpler than the robotic systems.
There were several limitations to our study. Firstly, although the results showed the improvement of the functional motor, they are not powered to detect significant changes due to the small sample size. Secondly, our system could not induce a high sense of the presence and embodiment for the patient. In order to assess the validity of presence of the system, most of VR or AR applying mirror therapy trials used a questionnaire. For instance, Koo et al. used a questionnaire including six items to check for real-time embodiment and virtual limb presence in which the mean scores were 4.0 and 4.0 (−5 to 5 rating), respectively [13]. These scores indicated that their system could provide a high degree of presence and might induce a long-term effect. Compared to our study, the results exhibited lower scores of validation of virtual reality and augmented reality (mean 0.8 and −4.16, respectively) which is not enough to provide a high presence, indicating that our system could not induce a long-term effect for the patient in motor function recovery. Finally, some adverse events were noticed after the intervention following dizziness, muscle twist, and motion sickness. This is similar to the VR [41,42] and mirror therapy [43,44].
A future study involving a larger sample size and long-term for follow-up should be performed to enhance significant improvements in outcome measures. Besides, more exercises should be added into the system to establish a standardized program for post-stroke rehabilitation. Furthermore, we should design the system in three-dimension (3D) to provide a higher sense of presence and embodiment of the upper limb for the patient which might reduce the adverse effects and induce long-term effects for the patient.
In conclusion, we developed the AR system applying mirror therapy mechanism to supply the wrist rehabilitation exercise for hemiplegic stroke patients using real-time image processing techniques. Our findings demonstrated that the AR system was feasible in the improvement of motor function rehabilitation for a patient with hemiplegia.