Review of the Augmented Reality Systems for Shoulder Rehabilitation

Abstract

Literature shows an increasing interest for the development of augmented reality (AR) applications in several fields, including rehabilitation. Current studies show the need for new rehabilitation tools for upper extremity, since traditional interventions are less effective than in other body regions. This review aims at: Studying to what extent AR applications are used in shoulder rehabilitation, examining wearable/non-wearable technologies employed, and investigating the evidence supporting AR effectiveness. Nine AR systems were identified and analyzed in terms of: Tracking methods, visualization technologies, integrated feedback, rehabilitation setting, and clinical evaluation. Our findings show that all these systems utilize vision-based registration, mainly with wearable marker-based tracking, and spatial displays. No system uses head-mounted displays, and only one system (11%) integrates a wearable interface (for tactile feedback). Three systems (33%) provide only visual feedback; 66% present visual-audio feedback, and only 33% of these provide visual-audio feedback, 22% visual-audio with biofeedback, and 11% visual-audio with haptic feedback. Moreover, several systems (44%) are designed primarily for home settings. Three systems (33%) have been successfully evaluated in clinical trials with more than 10 patients, showing advantages over traditional rehabilitation methods. Further clinical studies are needed to generalize the obtained findings, supporting the effectiveness of the AR applications.

1. Introduction

Shoulder disorders affect 16–26% of the community [1,2] causing pain and functional loss. The symptoms can be so severe and persistent to have self-reported impact on activities of daily life (ADLs) and on the patient’s quality of life, both mental and physical. In addition to the aforementioned consequences, shoulder disorders have repercussions on health and social care systems, due to: Economic burden (associated with significant direct and indirect costs of healthcare), increased demands on healthcare, poor performance at work, sickness absence, and early retirement [3].

Shoulder disorders have many causes. In addition to musculoskeletal diseases (e.g., rotator cuff, biceps tendinopathy and glenohumeral disorders, alterations in the deep fascia [4,5]), which are the most common causes of shoulder pain, shoulder disorders can be manifestations of gastrointestinal (e.g., gallstone disease, liver cancer or abscess), neurological (e.g., Parkinson’s disease), cardiological (e.g., stroke), or rheumatological (e.g., Rheumatic polymyalgia) diseases [1,2,6].

Treatments depend on the underlying cause of the disorder, and can range from anti-inflammatories to physiotherapy; however, sometimes they can also involve surgery.

Rehabilitation plays a key role in maximizing recovery and improving function to increase performance and autonomy in ADLs for both neurological and orthopedic patients [7]. In particular, shoulder rehabilitation focuses on two main aspects: Flexibility, to regain range of motion (ROM); and strength, to fortify the muscles around the shoulder. Traditional rehabilitation methods are based on a therapist-patient one-to-one activity, and on repeated exercises: Usually carried out with external devices to strengthen a target body area. However, the traditional rehabilitation process is time-consuming and difficult, especially for patients who suffered particularly severe strokes. Additionally, the repetitiveness of exercises reduces the patient’s motivation; consequently, the patient is bored, and tends to neglect the exercises prescribed [8].

The application of virtual and augmented reality (VR-AR) is gaining popularity and scholarly interest because of the possibility to generate environmental and perceptual stimuli [9], which transform the user experience offering high levels of “personal efficacy” (beliefs about own capability to accomplish challenging goals), and “self-reflectiveness” (intense focus on the particular instance or experience), generated by the user sense of presence and emotional engagement [10].

AR is in the exploratory stage in many applications, including the rehabilitation field where it shows advantages over VR. In fact, AR supplements reality but does not replace it, providing also a better sense of presence and reality judgments of the environment: As the elements the patient uses to interact with the application are real [11].

However, while in VR the user is completely immersed in a computer-generated environment, in AR the user sees both the real world and the virtual objects superimposed on it [12].

As defined by Azuma et al. [13], a system must fulfil the requirements to be considered an AR system: to combine real and virtual objects in a common environment, to run in real-time and interactively, and to align both spatially and coherently (i.e., register [12,14]) real and virtual objects. AR may overcome the difficulties related to traditional rehabilitation methods, by creating interactive and immersive environments. Additionally, AR can provide reliable and accurate feedback to guide and correct the patient during an exercise, enhancing the individual motor learning as showed in [15,16]. In this way, the rehabilitation process can occur outside of a clinical setting, and without requiring the therapist supervision; however, in case of an unsupervised rehabilitation, the AR system should provide the therapist with the user performance data, allowing an offline physiotherapeutic evaluation.

Current literature shows the need for new rehabilitation tools for upper extremity. Indeed, even if several studies have demonstrated that physical therapy can improve the recovery rate of upper extremity function, the effectiveness of traditional interventions in this human body region has resulted to be less pronounced than in other areas (such as the lower extremity) [17]. AR technology seems promising, offering an alternative approach to traditional rehabilitation. AR-based rehabilitation increases patient motivation, gives clinicians full control over specific parameters (e.g., movement distance), and allows for adjustment to individual capabilities.

The aim of this review is to provide an overview of the AR systems developed in the field of shoulder rehabilitation, and to understand the added value of AR. In particular, we aim to classify the AR rehabilitation systems reviewed (AR Rehab Systems) on the basis of: Wearable and non-wearable technology, integrated feedback, rehabilitation setting, and type of evaluation conducted.

2. Overview, Technical and Clinical Aspects of AR-Based Systems for Shoulder Rehabilitation

The literature search was conducted using the following databases: PubMed, Web of Science and Scopus up to September 2018. The search terms used were “Rehabilitation AND Augmented Reality” or “Rehabilitation AND Mixed Reality (MR)”, selecting only reports using AR-based systems for shoulder rehabilitation. The following data were extracted from each paper: Wearable and non-wearable technology, feedback provided, rehabilitation setting, clinical usefulness, and feasibility of the system.

Our literature review identified 212 articles. After removing duplicates, 143 studies remained. Of these, 123 were discarded since they did not fulfil the inclusion criteria: Seven papers were not written in English, three were book chapters, 112 were not related to shoulder rehabilitation. A total of 12 articles remained relevant for inclusion, corresponding to nine different AR-based systems for shoulder rehabilitation (see Figure 1): NeuroR [18], ARS [19], RehaBio [20], MirrARbilitation [21,22], ARIS [23,24], AR Games by De Leon et al. [25], SleeveAR [26,27], AR Fruit Ninja [28], AR System by Colomer et al. [29].

The following section presents an overview of the AR-based reviewed systems for shoulder rehabilitation, as well as an analysis and discussion of their technical and clinical aspects.

2.1. Overview

NeuroR [18] (Figure 1, top-left) is a post-stroke rehabilitation system based on AR technology. The system provides motor imagery (which can be defined as “the mental execution of a movement without any overt movement or without any peripheral activation” [30]) by means of a virtual 3D arm that replaces the paralyzed arm in a virtual avatar of the patient. It can be used either in rehabilitation centers as well as at home. The system includes exercises to achieve ROM of flexion and abduction of the shoulder.

ARS [19] (Figure 1, top-center) is a simple AR system for home rehabilitation after stroke. The system uses AR technology to provide a motivational environment, based on a “butterfly catching” game, to perform reaching tasks. The authors presented two versions: A basic modality based on a computer game, and a more advanced version which incorporates a control tool (a computer mouse/arm skate) to increase the physical effort associated with reaching tasks. Additionally, the system includes a game score.

RehaBio [20] (Figure 1, top-right) is an AR system for home rehabilitation that can be used to restore the upper-limb lost functions of patients who have hemiparesis from stroke, traumatic brain injury (TBI), or spinal cord injury (SCI). The system includes three modules: (1) a database module to store patient profile and training information, (2) a module for AR-based rehabilitation comprising four exercises (pin pong rehab, balloon collection rehab, transfer object rehab, and feeding animal rehab), and (3) a biofeedback simulation module (as described in Section 2.3) to monitor and to visualize the trained muscles performance. The system includes the user performance evaluation.

MirrARbilitation [21,22] (Figure 1, middle-left) is an AR rehabilitation system with gesture recognition based on marker-less body tracking technology, such as the Kinect™ motion sensing input device (Microsoft^®, Redmond, WA, USA [31]). The system guides and motivates the user during the execution of a reaching task, allowing the physiotherapist to set the target object angle. In addition, it provides score points and gives instructions to avoid the incorrect execution of movements. This system is specifically designed to fit within a home environment.

ARIS (Augmented Reality based Illusion System) [23,24] (Figure 1, middle-center) is a system designed for clinical rehabilitation after stroke. The system uses: AR technology to develop a rehabilitation game-exercise with visual and audio feedback, computer vision technology to create the illusory environment, and signal processing to monitor the performance of trained muscles via EMG signal (electromyography signal). The game consists in a circular motion exercise (CME) to increase the shoulder joint ROM.

The AR Games by De Leon et al. [25] (Figure 1, middle-right) are developed to rehabilitate stroke patients. This system provides an interactive and motivational augmented reality environment and is designed for a home setting. The proposed AR games are “color ball” and “mole attack”. Both can be played with a hand-held roller-ball device, that gives the possibility to focus the exercises on different movements and muscles. The system provides a game score.

SleeveAR [26,27] (Figure 1, bottom-left) is a system that integrates multimodal feedback (visual-audio-haptic) to guide the patient through therapeutic rehabilitation exercises (abduction-adduction, elevation-depression, flexion-extension) prescribed by a physical therapist. The system provides the patient a guidance for movements together with a report on the exercise progress by means of AR projections on their arm and on the floor. Specifically, motion capture technology, such as Optitrack [32] is used to track peoples’ arm movement. AR Sleeve system is designed for home settings; however, because of its complexity and high cost, at this time it can be used in rehabilitation gyms with multiple and concurrent therapeutic sessions.

AR Fruit Ninja [28] (Figure 1, bottom-center) is an AR version of the popular “Fruit Ninja” game [33] for patients with chronic stroke. The system uses a projection-based AR technology and provides a game score.

The AR System by Colomer et al. [29] (Figure 1, bottom-right) is a portable and low-cost mixed reality tabletop system for upper limb rehabilitation which is based on marker-less tracking technology (Kinect™ Microsoft^®, Redmond, WA, USA [31]). This system allows multi-touch interaction with the hands or via manipulation of tangible objects, depending on the exercise selected. The exercises cover a wide range of hand and arm movements and involve ADLs (e.g., sweeping the crumbs from the table). In addition, the system provides audio-visual feedback of the user performance.

Table 1. A summary of the AR-based systems for shoulder rehabilitation, reviewed in this paper.

AR Rehab System	Physiotherapy Applications	Rehab Settings	Target Population
NeuroR [18]	Shoulder abduction-flexion Shoulder horizontal flexion Finger extension Hand grasping	Home setting Clinical setting	Patients after stroke
ARS [19]	Shoulder-elbow movements	Home setting	Patients after stroke
RehaBio [20]	Shoulder flexion Shoulder horizontal adduction Elbow-wrist flexion	Clinical setting	Patients after stroke
MirrARbilitation [21,22]	Shoulder abduction	Home setting	Patients after stroke and mastectomy
ARIS [23,24]	Shoulder flexion Shoulder adduction-abduction Shoulder int.-ext. rotation	Clinical setting	Patients after stroke
AR Games by De Leon et al. [25]	Elbow flexion -extension Shoulder abduction-flexion Shoulder medial-lateral rot Wrist radial deviation Forearm pronation-supination	Home setting	Patients after stroke
SleeveAR [26,27]	Shoulder abduction-adduction Shoulder elevation-depression Shoulder flexion-extension Elbow-wrist flexion-extension	Home setting Rehab gym	Patients after stroke
AR Fruit Ninja [28]	Hand movements Shoulder movements	-	Patients after stroke
AR System by Colomer et al. [29]	Wrist flexion-extension Elbow flexion-extension Fingers flexion-extension Grasping objects Shoulder rotation	Clinical setting	Chronic stroke patients

shows a summary of the AR Rehab Systems for shoulder rehabilitation.

2.2. Wearable Technologies for Tracking, Enabling Augmented Reality

Accurate tracking between virtual objects and real world objects is a crucial challenges for developing AR applications [34]. The alignment of virtual content whit the real-world depends on a precise tracking of the viewing pose (relative to the real environment), and the annotated objects. Zhou et al. [35] categorized AR tracking techniques in: Sensor-based methods, employing sensors placed in the environment; vision-based methods, using image information to track the position and orientation of the desired object/subject; and hybrid tracking techniques, able to utilize both previous methods [34].

According to the literature [36], there is a wide variety of wearable sensing devices proposed for data collection of human body motion. Although many publications report effective tracking of body motion, most of the proposed methods rely on sensor/peripheral interconnection cables, not always flexible, and other components that have to be worn. These devices are usually not comfortable or easily wearable, and hamper the patient’s movements; thus, they are neither suitable for physical rehabilitation activities, nor for training programs in sports rehabilitation [36].

All the AR Rehab Systems reviewed use vision-based methods, which do not require the user to wear cabled devices; these methods can be further categorized in: Marker-based, or marker-less (“location-based position”). Marker-based AR technology uses the recognition of a distinguishable element, which can be identified apart from other objects in the environment by means of a camera. The virtual object is displayed only if the corresponding marker is visible. Although this technology uses visually obtrusive markers, it can offer a better tracking (more precise and faster), thus it is used in numerous AR applications [37].

The wearable markers employed in the AR Rehab Systems reviewed are inexpensive passive markers, extremely light to wear, that can be used by everybody, including children. More in particular, the markers employed consist of: black-and-white (B/W) fiducial markers in the form of a pattern, colored fiducial markers in the shape of strips or finger glove, and fiducial markers covered with retroreflective materials to reflect infrared (IR) light emitted from IR LEDs of the tracking camera.

The main issues of marker-based AR are related to possible difficulties in marker recognition due to poor lighting conditions (that can make the makers unrecognizable), and marker occlusions. The use of IR retroreflective markers improves the reliability of tracking, reducing the effects of ambient illumination.

Placing markers is not always practical, especially when they should be placed on the subject body. Therefore, with the recent advances in hardware (modern GPUs and low-cost motion capture devices), marker-less based systems now represent attractive solutions to solve the problems associated with marker-based tools.

In November 2010, Microsoft launched the low-cost marker-less camera-based Kinect™ motion sensing input device (Microsoft^®, Redmond, WA, USA [31]) which originally was used for the Xbox 360 gaming console, to track the users generating the 3D position and orientation of each body segment. This system requires minimal calibration with no marker to be positioned on the body. As disadvantage, its cameras are only capable of capturing 30 frames per second, thus are only able to track certain basic motions (e.g., walking or jumping), rather than fast movements [38].

Table 2. Tracking methods utilized by the AR Rehab Systems reviewed.

AR Rehab System	Tracking Methods
	Tracking Type	Trackable Items
NeuroR	Wearable marker-based AR Marker-less AR	Fiducial markers (B/W pattern) Natural features (skin-face)
ARS	Wearable marker-based AR	Fiducial markers (B/W pattern) IR Fiducia markers
RehaBio	Wearable marker-based AR	Fiducial markers (colored finger glove)
MirrARbilitation	Marker-less AR (Microsoft®, Kinect™ [31])	None (full-body tracking)
ARIS	Wearable marker-based AR	Fiducial markers (colored strips)
AR Games by De Leon et al.	Wearable marker-based AR	Fiducial markers (B/W pattern)
SleeveAR	Wearable marker-based AR (OptiTrack Motion Capture [32])	IR Fiducial markers
AR Fruit Ninja	Non-wearable marker-based AR	Colored Fiducial markers (strips)
AR System by Colomer et al.	Marker-less AR (Microsoft®, Kinect™ [31])	None (full-body tracking)

summarizes the tracking methods used by the AR Rehab Systems reviewed.

2.3. Augmented Reality Interfaces: Wearable and Non-Wearable Technology Used

Besides motion tracking technology, the primary components of the AR Rehab Systems reviewed are the augmented reality interfaces providing the desired feedback to the user.

In general, feedback can provide: information for guidance and error detection during the exercise execution, and information about performance outcomes; additionally, feedback can be used for motivational purposes. Augmented (extrinsic) feedback, if provided in an appropriate manner, facilitates the motor (re-) learning. It can be delivered by a trainer, a therapist, or technical displays according to a chosen feedback modality, such as: Visual, auditory, haptic, a combination of these (multimodal feedback), and/or from biofeedback devices [15]. Biofeedback provides biomechanical measurements (related to the movement, postural control, and forces), and measurements of the body physiological systems (neuromuscular, respiratory, and cardiovascular). Specifically, in physical rehabilitation, biofeedback based on EMG is the most used and provides information about muscle activity. Sigrist et al. [15] indicate that the multimodal feedback is more efficient than unimodal feedback, because it prevents the cognitive overload enhancing the learning effectiveness.

Table 3. Feedback modalities and interfaces of the AR Rehab Systems reviewed.

Integrated Feedback	AR Rehab System	AR Interface
		Visual	Audio	Haptic	Bio
Unimodal: Visual	ARS	Screen	/	/	/
	MirrARbilitation	Screen	/	/	/
	AR Games by De Leon et al.	Screen	/	/	/
Bimodal: Visual-Audio	NeuroR	Projector	PC audio speakers	/	/
	AR Fruit Ninja	Projector	PC audio speakers	/	/
	AR System by Colomer et al.	Projector	PC audio speakers	/	/
Trimodal: Visual-Audio-Haptic	SleeveAR	Projector	Ext. audio speakers	MYO wearable tactile interface	/
Trimodal: Visual-Audio-Bio	RehaBio	Screen	PC audio speakers	/	FlexComp Infiniti™ System [40]
	ARIS	Screen	PC audio speakers	/	FlexComp Infiniti™ System [40]

summarizes the types of feedback implemented in each AR Rehab System reviewed and the AR interfaces.

Regarding the visual feedback, available display technologies for AR include: Spatial displays; hand-held displays (mobile phones and tablets), and head-mounted displays (HMDs) [39]. The AR Rehab Systems reviewed use spatial displays, which can be further categorized in: Screen-based, and projection-based.

Screen-based AR technologies use video mixing and display the merged images on a regular monitor. In this type of AR, the field of view (FOV) is limited by: The monitor size, the spatial alignment of the display relative to the observer, and the distance between monitor and observer.

Projector-based spatial displays use front-projection to display images directly on physical surfaces. These systems overcome some of the shortcomings that are related to wearable HMDs, providing: Improved ergonomics, a theoretically unlimited FOV, a scalable resolution, and an easier eye accommodation. However, they also have several issues, such as: Shadow-casting of physical objects and interacting users; restrictions of the display area (constrained to the size, shape, and colors of physical surfaces); and restriction to a single user when virtual data is displayed with non-zero parallax. Moreover, conventional projectors only focus on a single focal plane located at a constant distance, and projecting images onto non-planar surfaces causes blur [41].

Until now, wearable visual feedback was challenging due to practical limitations. Despite their popularity, HMDs are sometimes avoided in clinical applications because of side effects such as simulator sickness and visual fatigue [42]. Besides, HMDs still present technological and perceptual limitations. Some technological issues are: A small FOV, the obtrusiveness and weight of the device, and the low luminance of micro-displays. Additionally, examples of perceptual conflicts are: The vergence-accommodation conflict [43], and the mismatched accommodation between virtual content and real-world scene for optical see-through (OST) HMDs [44].

All these limitations are the reason why HMDs have not been used yet in shoulder rehabilitation. However, upcoming developments in AR headsets are expected to improve their functionalities, guaranteeing an unobtrusive use in the future. HMDs deserve future attention since they can provide a more immersive experience than other non-wearable AR display technologies, and they intrinsically provide the user with an egocentric viewpoint. Furthermore, HMDs can integrate both sensing capabilities (to derive their position and orientation in 3D) and computing capabilities (to track external environments and self-localization). Finally, HMDs as the Microsoft HoloLens [45], can also provide multi-modal interactions for a more intuitive and enjoyable AR experience [46].

Visual feedback is incorporated, as exclusive feedback modality, in only three of the nine AR Rehab Systems reviewed: Two computer games (ARS, and AR Games by De Leon et al.), and MirrARbilitation. Specifically, in these two games, visual feedback is used as a motivational tool to provide a funnier experience for patients. The motivational aspect is very important during therapy sessions, because it can increase the chances of a fast recovery [47]. The visual feedback integrated in MirrARbilitation is used to guide and correct patient movements during rehab sessions, to evaluate user performance, and also for motivational purposes. Exercise instructions, as well as scores and positive feedbacks, are displayed on a standard screen during therapy sessions. In addition, once more for motivational purposes, X-ray images of the shoulder anatomy are superimposed during the interval of motion.

A combination of visual and other feedbacks is used by the other AR Rehab Systems reviewed. Particularly, five systems (NeuroR, RehaBio, ARIS, AR Fruit Ninja and AR System by Colomer et al.) incorporate both audio and visual feedback. Additionally, two of them (ARIS and RehaBio) also include biofeedback in real-time. The audio-visual feedback is utilized in these systems to motivate and encourage users during long-term training sessions, and also to notify patients of specific events. The visualization of content ranges from simple abstract in form of basic objects (i.e., EMG line graph, muscle animation, and score, all integrated in both ARIS and RehaBio), to natural visualization of a 3D virtual arm (included in NeuroR and ARIS). The latter is superimposed on the user impaired arm to create an illusory scene of a real arm, speeding up in this way the recovery.

Audio feedback is used as an alarm during the exercise execution, for example: to detect a collision, to provide a score, or to indicate the beginning of movement. External audio speakers (Sleeve AR), and PC audio speakers (Neuro, AR Fruit Ninja, AR System by Colomer et al., RehaBio, ARIS), are employed as audio interfaces in the AR Rehab Systems reviewed.

ARIS and RehaBio also integrate biofeedback, which has a twofold purpose: it monitors and tracks the trained muscles providing performance information, and it motivates the user during the exercise execution showing visual reflections of currently activated muscles. Many studies have shown that the inclusion of biofeedback improves the quality of rehabilitation therapy, because it provides higher motivation especially for long-term training [26]. In both these AR Rehab Systems, FlexComp Infiniti™ System from Thought Technology [40], with four wearable EMG MyoScan™ Sensors for the user’s upper limb muscles, is used to acquire the sEMG (surface electromyography) data in real time. Extracted EMG data are then displayed in real-time on a screen.

Finally, SleeveAR incorporates audio-visual-haptic trimodal feedback that is applied to guide and motivate the user during a task. In particular, tactile feedback is used to notify patients of specific events (e.g., current state of the exercise). The AR interface in this system is a wearable device (MYO bracelet, North Inc., Canada) able to provide multiple ranges of vibrations in an easy and simple way via wireless communication.

2.4. Rehabilitation Settings

Rehabilitation services may be conducted by therapists, physicians, community rehabilitation workers, and family members. Thus, they can be provided in a wide variety of settings, such as: Rehabilitation centers, hospitals, clinics, community facilities, and homes.

The choice of a rehabilitation setting is based on patient characteristics, for example: Overall health, age, and degree of disability. However, patients with mobility difficulties who live in rural and remote areas have issues related to accessibility, availability, and utilization of rehabilitation services [48]. Furthermore, since rehabilitation can be very expensive and not widely available in healthcare sectors, a way to overcome these problems could be to implement user-friendly, portable, and low-cost systems that do not require the supervision of a therapist for guiding the patient during the rehabilitation task execution.

Four AR rehabilitation systems (MirrARbilitation, SleeveAR, ARS, AR Games proposed by De Leon et al.) are designed to support a home rehabilitation process without the direct supervision of the therapist. However, only two systems (MirrARbilitation and SleeveAR) provide a real time feedback to guide the patient during the rehabilitation exercise and to maintain a quality in the execution of the task; the others (ARS and AR Games by De Leon et al.) only provide a performance score which can be useful for motivational purpose. Moreover, at this time, SleeveAR can be only used in rehabilitation gyms with multiple and concurrent therapeutic sessions, because of its complexity and high cost.

In all four cases, the possibility of registering user performance for (online or offline) evaluation by physiotherapists is not explicitly mentioned: The latter functionality is of fundamental importance to support a home rehabilitation process and should therefore be integrated in the future.

As for AR Fruit Ninja, the authors do not specify the type of rehabilitation setting, and the system only provides score points as in ARS and AR Games by De Leon et al.

NeuroR can be considered suitable for both home rehabilitation without direct therapist supervision and rehabilitation centers.

Finally, ARIS, RehaBio and AR System by Colomer et al. are designed for clinical settings.

2.5. Evaluations

All AR Rehab Systems reviewed, except from AR Fruit Ninja, have been evaluated in terms of usability. A technical evaluation has been carried out for two systems (RehaBio and ARIS); and a clinical evaluation, involving real patients, has been performed for five systems (MirrARbilitation, NeuroR, AR System by Colomer et al., ARS, AR Fruit Ninja). All the other AR Rehab Systems reviewed have been tested by healthy subjects: RehaBio, ARIS, AR Games by De Leon et al., and SleeveAR.

Table 4. Evaluation of the AR Rehab Systems reviewed.

AR Rehab System	Evaluation Type	Evaluation Method
NeuroR	Usability and clinical (n = 8; stroke patients)	Objective (performance analysis) and subjective measures (via questionnaire)
ARS	Usability and clinical (n = 4; chronic stroke patients)	Objective (performance analysis) and subjective measures (interviews)
RehaBio	Usability and technical (n = 10; normal people)	Objective (performance analysis) and subjective measures (via questionnaire)
MirrARbilitation	Usability (n = 33; 11 patients;11 therapists; 11 developers)	Objective (performance analysis) and subjective measures (via observations)
ARIS	Usability (n = 10; normal people)	Objective (performance analysis) and subjective measures (via questionnaire)
AR Games by De Leon et al.	Usability (n = 4; normal people)	Subjective measures (via questionnaire)
SleeveAR	Usability (n = 18; normal people)	Objective (performance analysis) and subjective measures (via questionnaire)
AR Fruit Ninja	Usability and clinical (n = 18; patients with chronic hemiparesis)	Objective (performance analysis) measures
AR System by Colomer et al.	Usability and clinical (n = 30; patients with residual hemiparesis)	Objective (performance analysis) and subjective measures

shows the types of evaluation performed on each AR Rehab System reviewed with details on the number of subjects involved in the testing, and the evaluation methods.

The authors of NeuroR conducted two studies. In the first study, they evaluated the effectiveness of the system on upper limb motor function using the Fugl-Meyer scale [49]. The results of Fugl-Meyer test suggested a trend for greater upper limb motor improvement in the augmented reality group (from 17% to 62%) than in the control group (from 4% to 14%). In the second study, they measured the increase of shoulder ROM with computerized photogrammetry. All participants showed an increase of the ROM for the shoulder flexion (61.3% to 90%) and abduction (46.7% to 73.9%). However, a further large randomized study is needed to support observed improvements.

The ARS system was evaluated as a good tool for home rehabilitation after stroke. Of the 4 participants in the study, 2 showed improvement in the execution on the timed game-play and in arm function. The system was evaluated using: The Fugl-Meyer test [49] to assess motor performance, the Wolf Motor Function Test to assess functional movement, and the DASH (Disabilities of the Arm, Shoulder and Hand questionnaire) to acquire information about patients symptoms and abilities. At the end of session, participants were asked to express their opinion regarding the system. The participants considered the game score to be a very important motivational tool and they suggested to make it visible on the screen while playing. However, the results of the study are tentative since the sample size was very small thus limiting the ability to generalize the obtained findings.

Authors of RehaBio and of ARIS showed encouraging results, in terms of effectiveness, user motivation, and enjoyment. For both systems, patients’ opinions were collected using a five-point Likert questionnaire to investigate if the system is motivating, fun, ease to use, and to express opinions regarding the system interface and the AR characteristics. However, also in this case, clinical trials are needed to validate the effectiveness of the systems.

The AR Games by De Leon et al. was evaluated positively as a motivational rehabilitation tool. Also, for this system a qualitative evaluation was performed; at the end of game sessions, each participant completed a five-point Likert scale questionnaire, comprising 6 items, focused on the users’ opinion (i.e., item 2: “I felt the game was intuitive and enjoyable”). However, the results are tentative since only four healthy volunteers participated in the tests.

Only four systems were evaluated by more than ten subjects: MirrARbilitation, SleeveAR, AR Fruit Ninja, and AR System by Colomer et al.

The authors of AR Fruit Ninja, MirrARbilitation and SleeveAR and AR System by Colomer et al. conducted a comparative study showing the efficacy of their AR application. The former compared the user motor performances obtained whit their system to those obtained using the original videogame “Fruit Ninja” without AR. The results were very positive: scores were higher (21%), reaching times were faster (19%) and less movement variability was observed (15%). In addition, the authors of the MirrARbilitation system showed that their application is more efficient than traditional rehabilitation methods because it improves the user engagement and the exercise performance outcomes. The motivational and usability aspects were evaluated by means of a five-point Likert questionnaire. In addition, the increased motivation of the users was showed by the number of exercise repetitions that was improved from 34.06% to 66.09%. On the other hand, the exercise performance outcomes were showed from increase percentage of correct exercises from 69.02% to 93.73%.

The authors of SleeveAR conducted a usability study to evaluate user performance: The results showed that the system can successfully guide subjects through an exercise prescribed (and demonstrated) by a physical therapist, with performance improvements between consecutive executions. The study included a comparison of two different approaches: AR system and video observation. Each test session involved the execution of five different exercises in the two approaches as the guidance: The video and with the SleeveAR. At the end of test sessions, all participants filled a six-point Likert questionnaire; results showed a statistically significant difference for the item “It was easy to see if the arm was in the wrong position”, whereby it is noteworthy that participants gave higher assessment to the AR system.

Finally, Colomer et al. demonstrated the clinical effectiveness and the acceptance of their AR system, comparing it to conventional physical therapy.

Patients treated with the AR system showed positive results in terms of body activity (assessed using the following tests: The Wolf Motor Function Test, the Box and Blocks Test, and the Nine Hole Peg Test) and participation (assessed using the subscales of Quality of Movement and Amount of Use of the Motor Activity Log). Additionally, the authors have evaluated the acceptance of their AR system using: the System Usability Scale (a Likert scale with scores ranging from 0 to 100) and four (seven-point Likert) subscales of the Intrinsic Motivation Inventory. The former scale was used to provide a global assessment of subjective usability; the results have showed a good acceptance (79.13 ± 7.54 from a total score of 100) of AR system. The latter scales were used to assess perceived value/usefulness of the system, participant interest/enjoyment, perceived competence and pressure/tension.

Results have showed that the system is highly usable (6.17 ± 0.69), enjoyable (5.73 ± 0.79 of 7), and the participants have found themselves competent (5.21 ± 0.98) but not pressured (1.98 ± 0.58).

3. Conclusions

Augmented reality allows to merge the natural perception of real environment with computer-generated information, and it is considered an enabling technology of the fourth industrial revolution, aimed at aiding specific tasks as a tool in the manufacturing industry. Additionally, AR has proven to be a promising technology in healthcare as demonstrated by the increasing number of publications in medical training [50], surgery [51,52,53,54], and rehabilitation [55].

The goal of this review was to investigate to which extent augmented reality applications are currently used in shoulder rehabilitation. A literature search was conducted and a total of nine different AR-based systems for shoulder rehabilitation were identified. This review has shown that current systems use vision-based methods for registration, and primarily wearable marker based- tracking (avoiding sensor-based methods). Moreover, all of them employ spatial displays (both screen-based, and projection-based) for the visualization of AR content. The use of HMDs has not been explored yet for this specific type of rehabilitation (shoulder); and a wearable myoelectric remote-control interface been only used to provide haptic (tactile) feedback. Probably, until now, HMDs have not been used because of technical impediments (small FOV, obtrusiveness and weight, low luminance of micro-displays) and perceptual limitations (vergence-accommodation conflict, mismatched accommodation between virtual content and real-world). However, HMDs deserve future attention since can deliver a more immersive experience than other non-wearable AR display technologies, and they intrinsically provide the user with an egocentric viewpoint. Additionally, modern HMDs such as the Microsoft HoloLens can integrate tracking capabilities (for external environments and self-localization) and provide multi-modal interactions for a more intuitive, and enjoyable AR experience.

Concerning the incorporated feedback, three systems (33%) present visual feedback only, the remaining systems (66%) provide also audio feedback: alone (33%), coupled with biofeedback (22%), or together haptic feedback (11%).

As for the rehabilitation setting, most of the systems reviewed are designed for are designed to support a home rehabilitation process without the direct supervision of the therapist (44%), one systems is suitable for both home settings and rehabilitation centers (11%), three are designed only for clinical settings (33%), however, the authors of one system do not specify the type of rehabilitation setting (11%).

However, only two out of four systems (MirrARbilitation and SleeveAR) can be considered suitable for home sitting because they provide feedbacks, normally given by physiotherapist, to guide patients on how to perform the movements.

The other two systems, (ARS and AR Games by De Leon et al.) together to AR Fruit Ninja, at the time, can be considered suitable only for motivational purposes because they provide only score game.

Most systems have reached the maturity to support test with the final user, but only four AR Rehab Systems have been evaluated in clinical trials: of these, only three systems have been evaluated by more than ten subjects. Clinical studies conducted show clear benefit over traditional rehabilitation methods in terms of usability, enjoyability, and motivation and in terms of improving of exercise performance outcomes.

To conclude, this review shows that the use of AR technologies in shoulder rehabilitation is promising and deserve future attention, but today the number of clinical studies conducted is low thus limiting the ability to generalize the obtained findings to support the effectiveness of the AR applications for shoulder rehabilitation.