Wearable Augmented Reality Application for Shoulder Rehabilitation

Abstract

Augmented reality (AR) technology is gaining popularity and scholarly interest in the rehabilitation sector because of the possibility to generate controlled, user-specific environmental and perceptual stimuli which motivate the patient, while still preserving the possibility to interact with the real environment and other subjects, including the rehabilitation specialist. The paper presents the first wearable AR application for shoulder rehabilitation, based on Microsoft HoloLens, with real-time markerless tracking of the user’s hand. Potentialities and current limits of commercial head-mounted displays (HMDs) are described for the target medical field, and details of the proposed application are reported. A serious game was designed starting from the analysis of a traditional rehabilitation exercise, taking into account HoloLens specifications to maximize user comfort during the AR rehabilitation session. The AR application implemented consistently meets the recommended target frame rate for immersive applications with HoloLens device: 60 fps. Moreover, the ergonomics and the motivational value of the proposed application were positively evaluated by a group of five rehabilitation specialists and 20 healthy subjects. Even if a larger study, including real patients, is necessary for a clinical validation of the proposed application, the results obtained encourage further investigations and the integration of additional technical features for the proposed AR application.

1. Introduction

The application of virtual (VR) and augmented reality (AR) is gaining popularity and scholarly interest thanks to the possibility to generate environmental and perceptual stimuli which can transform the user experience: enhancing the emotional engagement, easing the acquisition of knowledge/skills, and the achievement of objective performance goals while staying in a controlled and safe environment. Literature studies show the potential of AR and VR for enhancing personal and clinical change, offering high levels of “personal efficacy” (beliefs about own capability to accomplish challenging goals), and “self-reflectiveness” (intense focus on the particular instance or experience) [1]. Additionally, there is a growing interest in the use of VR and AR devices allowing automatic recording and objective measurement of the user’s performance, which is particularly important in several medical fields including medical training [2] and rehabilitation medicine [3,4], since most current program are based on subjective progress evaluation and they the lack objective performance goals.

AR is still in the exploratory stage in many medical applications, including the rehabilitation field where it shows advantages over VR. In fact, AR supplements reality but does not replace it [5], it provides the user with a better sense of presence and reality judgment of the environment, preserving the possibility to directly interact with real instrumentation and other subjects, such as rehabilitation specialists [6].

In this paper, we present a wearable AR application for shoulder rehabilitation. As has been previously reported in the literature, there is the need for new rehabilitation tools for the upper extremity since the effectiveness of traditional interventions in this body region is less pronounced than in other areas, such as the lower extremity [7]. According to literature, shoulder pain affects 18–26% of adults [8,9], making it one of the most common pain syndromes. It can be due to local pathologies (e.g., rotator cuff syndrome) but it can also be linked to several other disorders (e.g., abdominal pathologies, alterations in the deep fascia [10,11] and malignancy [9]). Regardless of the specific pathology, shoulder pain has several impacts on activities of daily living (ADLs) and, thus, on the patient quality of life.

A number of authors [12,13,14,15,16,17,18,19,20,21] have recognized the potentialities of AR-based applications in this specific field. The virtual content is used to: guide the patient arm during the rehabilitation session, give a visual feedback to correct in real-time the movement, furnish scores as well as positive feedback to motivate the patient.

For example, Sleeve AR [20,22] provides the patient with a real-time guidance for upper limb exercise (abduction–adduction, elevation–depression, flexion–extension) by means of AR information projected on the patient arm and on the floor. Another example is the mixed reality portable system by Colomer et al. [21], which consists of a “projective tabletop system” to guide planar gamified tasks. Furthermore, Fruit Ninja [23] and the augmented reality system (ARS) by King et al. [13] use gamified exercises (the “butterfly-catching” game and the popular Fruit Ninja game, respectively), to motivate the patient during the rehabilitation session. While Fruit Ninja is based on projective AR display technologies, ARS is a screen-based AR system. This latter can also incorporate a hardware device (a computer mouse/arm skate) to increase the physical exercise effort during the reaching tasks.

According to a recent literature review [24], first clinical studies show clear benefits of AR-based rehabilitation over traditional methods not only in terms of usability, enjoyability, and user motivation, but also in improving patient performance outcomes. Even if additional clinical studies are needed to generalize these finding, the results obtained encourage further investigations and technical development in this field.

Available display technologies for AR include spatial displays, hand-held displays (i.e., tablets), and head-mounted displays (HMDs). The previously mentioned literature review [24] shows that the use of HMDs has not been explored yet for shoulder rehabilitation: All the developed systems indeed employ spatial displays (screen-based or projection-based) for the visualization of AR content.

Despite their popularity, HMDs are sometimes avoided in clinical applications because they still present technological and perceptual limitations (i.e., the vergence accommodation conflict [25], and “the focal rivalry” between virtual content and real-world scene for optical see-through (OST) HMDs [26,27]) bringing side effects such as simulator sickness and visual fatigue [28]. Some technological issues are: a small field of view (FOV), the obtrusiveness and weight of the device, and the low luminance of micro displays. Photometric consistency in HMD applications is important to understand the AR space and to provide realistic experiences. In bright environments, the VR content displayed by OST-HMDs may appear transparent and dim, and, therefore, the perception of the AR scene may be compromised by the gap between the brightness of real and virtual content [29]. This issue restricts the usage of commercial OST-HMDs to an indoor scenario where current OST-HMD technology can match the brightness of the scene.

However, HMDs deserve attention since: They are deemed as the most ergonomic solution for applications including manual tasks performed by the user under direct vision [30]; they can provide a more immersive experience than screen-based AR display technologies; finally, compared to projection-based displays, they do not present issues such as shadow casting of physical objects and interacting users, and constraints of the display area imposed by the size, shape, and colors of physical surfaces on which the virtual content is projected [31,32].

Furthermore, recent HMDs can integrate sensing and computing capabilities for self-localization and external environment tracking.

This paper presents an evolution of a VR serious game we have designed for Nintendo Wii Remote MotionPlus [33]. In this work, we have explored the potentialities offered by Microsoft HoloLens, an OST-HMD, for the development of a stand-alone AR application for indoor use (i.e., no direct sunlight), with no need of markers/sensors for arm/hand tracking. The game was developed using the Unity game engine and the Microsoft HoloToolkit.

2. Materials and Methods

2.1. Rationale

The system is designed to improve the shoulder range of motion (ROM) delivering more specific, intensive, and enjoyable therapy with real-time feedback of performance, to overcome the limitation of traditional rehabilitation methods.

Traditional rehabilitation techniques to recover the motor function rely on standard exercises, carried out by a therapist during inpatient hospital care and continued at home, with no monitoring. Hospital sessions entail human and financial resources, with especially high costs for demanding patients such as those with traumatic brain injury or spinal cord injury [25]. Moreover, the repetitiveness of traditional rehabilitation exercises can bore the patient, reducing his/her motivation, and negatively affecting the outcomes of the therapy [23].

For this reason, in this work, AR and gamification are used to deliver therapy, motivate the patient, and make him/her pro-active in performing the rehabilitation tasks: More particularly, AR is used for guiding the patient arm along pre-programmed trajectories to improve related sensory motor functions through repetitive practices, and visual feedbacks, enriched with audio signals, are provided throughout the rehabilitation session for entertainment purposes and to inform the patient about his/her performance. The AR information supplements reality but does not replace it, therefore, compared to virtual reality applications, it offers the advantage of allowing a seamless interaction with medical personnel and the surrounding environment, providing a better sense of presence (physical presence, social presence, and “self-presence” intended as the sense of “feeling one’s own body”).

Moreover, unlike conventional therapist-led sessions, where progresses in physical/cognitive functions may be subjective and/or difficult for patients to identify [24], the AR application is designed to provide an objective, quantitative measure of the subject performance. Acquired data and performance results can be stored, remotely accessed by the therapist, and used not only to evaluate patient progress, but also to predict rehabilitation potential, and to assist in rehabilitation program planning. Indeed, our AR application is designed to provide clinicians the ability to individualize training programs based upon the user’s personal performance.

2.2. Selection of the Head-Mounted Display

The selected HMD is the Microsoft HoloLens, an OST-HMD with self-contained computing power, wireless communication system, and no physical tethering constrains that can hinder the patient movements during rehabilitation tasks.

More in particular, the HoloLens technology is based on an undisclosed Intel 32-bit processor, with a custom-built Microsoft Holographic Processing Unit (HPU 1.0) which supports Universal Windows Platform (UWP) apps. It is equipped with 2 GB of RAM (1 GB for CPU, and 1 GB for HPU) and 64 GB of flash memory; it features network connectivity via Wi-Fi 802.11ac and Bluetooth 4.1 LE wireless technology. The HPU processes data from multiple sensors: including four grayscale tracking cameras; one depth camera; and one world-facing photo/video camera (2 MP); one ambient light sensor; one inertial measurement unit (IMU) to track head movements; and four microphones.

The four grayscale tracking cameras and the depth camera are used to sense the environment and capture user gestures. Two grayscale cameras are configured as a stereo rig capturing the area in front of the HMD, thus, the 3D position of tracked features can be determined through triangulation; the other cameras allow for a wider field of view to keep track of features [34]. These functionalities allow the real-time tracking of the user’s hand positions without any sensor/peripheral interconnection cable or marker (that can be uncomfortable for the user and can hamper his/her movements) [35].

Being an OST system, the HoloLens offers an unhindered and instantaneous full-resolution view of the real environment, which assures that visual and proprioception information is synchronized.

As emerges from literature [36], HoloLens provides considerable benefits over other commercial HMDs from human factors and ergonomics standpoints; it outperforms other currently available OST-HMDs (Epson Moverio BT 200, Osterhout Design Group (ODG) R 7), in terms of: contrast perception, task load, frame rate, and system lag.

However, some technical issues have to be considered: HoloLens weighs 579 g, it features a diagonal field of view of only approximately 35°, and it can only display virtual images at a fixed focus of 2.0 m; thus, it lacks the ability to correctly render the naturally coupled accommodation and convergence cues [27]. The first two issues are mitigated in the next Microsoft HMD generation; indeed, the Microsoft HoloLens 2 features an improved ergonomics allowing for an extended use and it has a wider field of view (52°). Discomfort from the vergence accommodation conflict can be avoided by keeping the VR content at 2.0 m.

2.3. Design of the AR Rehabilitation Serious Game

The serious game [33] was designed starting from traditional rehabilitation exercises performed with the “Rolyan Range of Motion Shoulder Arc”: This is used to treat any upper extremity deficit that impairs ROM and consists of a curved tube with movable colored tabs. Moving the tabs from one side of the tube to the other achieves full upper extremity ROM and improved motor planning and visual tracking skills. In a similar way, the goal of the AR game, which is called “Painting Discovery”, is to move a virtual cursor (“Virtual Magnetic 3D Cursor”), along a predefined AR trajectory (in green) (Figure 1) on a semitransparent panel 2 m in front of the user. The user controls the cursor through the movement of his/her hand (the game can be played by using either the left or right arm) and moves his/her head to explore the AR graphical interface (the area of the panel visible at any time using the HoloLens lens is highlighted with a black border in Figure 1).

The semitransparent panel consists of a grid of tiles covering a painting (Figure 2): As the virtual cursor moves to follow the trajectory, any tile “touched” changes. If the tile is part of the trajectory, it turns transparent to show the portion of the painting below; if the tile is not part of the trajectory, it turns red to show that an error has been made by going off trajectory.

As the user progresses through the game to complete the trajectory, all the “untouched” tiles gradually become transparent. Once the trajectory is completed, a gratifying message is displayed on the screen, the tile grid is destroyed, and the painting is shown.

At present, the subject’s performance is expressed in terms of: the number of errors, the score (calculated as the ratio of the tiles marked as errors to the total number of tiles in the trajectory), and the completion time. These data are automatically saved and exported to a text file at the end of each rehabilitation session. The top right section of the user interface (UI) includes: a countdown timer showing the time remaining to complete the game, the total number of errors performed so far, and the score.

Our AR rehab game application enables the physiotherapist to configure: the trajectory path and its width, the sequence of trajectories presented to the user, the number of times each trajectory is presented, the maximum completion time for each trajectory. This allows the therapist to adjust the difficulty of the game according to the individual needs of each patient.

The virtual environment for the serious game was sized considering: the HoloLens field of view, the game playability, and the resulting range of motion of the shoulder. These three factors were considered to ensure that the user can easily and comfortably play the rehabilitation serious game, and that the obtainable upper arm excursion range was appropriate.

The HoloLens features a 16:9 screen aspect ratio (Figure 3), thus the width (w), and height (h) of the viewing area at a distance (d) of 2.0 m, can be derived as follows:

$$\mathrm{h}=\frac{9\mathrm{s}}{\sqrt{{16}^2+9^2}},$$

(1)

$$\mathrm{w}=\frac{16\mathrm{s}}{\sqrt{{16}^2+9^2}}.$$

(2)

In Equations (1) and (2), the diagonal size of the visible area (s) can be derived from the diagonal field of view (α) as follows:

$$\mathrm{s}=2\mathrm{d}\tan \frac{\mathsf{\alpha }}{2}.$$

(3)

Considering a diagonal FOV of 35° and an aspect ratio of 16:9, Equations (1) and (2) predict, respectively, a 0.60 m height and 1.10 m width at a distance of 2.0 m in front of the HoloLens.

The ideal shoulder ROM to be trained can be easily defined from the size of the traditional “Rolyan Range of Motion Shoulder Arc”, which is ~0.9 m wide and ~0.6 m high: Considering a standard arm length of 0.5 m, the horizontal ROM (angular excursion in the transverse plane) and the vertical ROM (angular excursion in the sagittal plane) are ~61° and ~50°, respectively.

Given these considerations, different sizes of the virtual trajectory panel (tile grid) were evaluated, ranging from 100% to 300% of the size of the HoloLens viewing area (

Table 1. Virtual trajectory sizing and performance.

Virtual Trajectory			Shoulder Movements
% Visible with HoloLens	Width [m]	Height [m]	Horizontal Flexion *		Vertical Flexion **
			ROM [°]	% of ROM Target	ROM [°]	% of ROM Target	Number of Grid Cells
100%	1.1	0.6	29	47%	17	33%	66
67%	1.7	0.9	40	66%	24	47%	153
50%	2.2	1.2	48	78%	31	61%	264
44%	2.5	1.4	51	84%	35	69%	350
40%	2.8	1.5	54	89%	37	72%	420
38%	3.0	1.7	56	92%	40	79%	510
33%	3.3	1.8	59	96%	42	82%	594

* Plane of motion: horizontal (i.e., transverse plane). ** Plane of motion: vertical (i.e., sagittal plane).

).

Table 1 summarizes the obtainable shoulder ROM (expressed as excursion in degrees) for each size tested for the virtual trajectory panel. Only sizes allowing more than 50% of the target horizontal and vertical ROM (five in total) were considered and tested (see Section 2.6), whereas all the other sizes were discarded (and not tested).

Tiles were sized to have a sub-decimeter accuracy in detecting hand movement. Particularly, a 0.1 m × 0.1 m size was tested, allowing a resolution of ~3° of ROM (corresponding to ~3 cm of hand excursion).

2.4. Design of the AR Rehabilitation Serious Game

This section describes the software architecture developed with Unity3D (version 2017.4.27f1) using the Microsoft HoloToolkit (version 2017.4.3.0): a collection of C# scripts and Unity components to develop mixed-reality applications. The main elements of the software architecture are depicted in Figure 4.

The patient interacts with our AR rehab game app trough head and hand movements, moving the “Virtual Magnetic 3D Cursor” along the visualized AR trajectory, and receives visual and audio feedback based on his/her performance.

The therapist can use the Windows Device Portal to connect to the HoloLens via Wi-Fi or USB, and interact with the app: by setting up the configuration of rehabilitation exercises (at this time only maximum completion time, number of repetitions, and sequence of trajectories), or to retrieve performance data of a rehabilitation session (at the moment only number of errors and completion time).

The user interactions are managed using the functionalities offered by the Microsoft HoloToolkit, using its “HoloLens Mixed Reality Camera” and its “HoloLens Input Manager”. The former handles the visual rendering and moves in sync with the user’s head movements. The latter is responsible for handling inputs (e.g., user’s hand movements and gestures) and dispatching input events to the appropriate input handlers.

In our AR rehab game application, the HoloLens Input Manager is extended with a custom script “Custom Hand Tracking”: to use in real time the tracked user’s hand position updating the “Virtual Magnetic 3D Cursor”.

The “Virtual Magnetic 3D Cursor” is a custom script to implement a hand-controlled cursor for our AR rehab game app. In game, this cursor is visualized as a 3D sphere with configurable radius, and it is implemented using both a “Mesh Filter” and a “Mesh Renderer” component in Unity. Additionally, the cursor integrates a “Kinematic Rigidbody Trigger Collider” to detect its interactions with UI controls and the tiles of the “Game Grid”, both implemented using “Trigger Collider” components (see Figure 4 and Figure 5).

The “Game Grid” is a custom Unity GameObject implementing a 2D matrix of square tiles, fully configurable in terms of resolution and size. This grid is configurable at runtime by cloning a custom Unity Prefab “Game Grid Cell”, with each clone representing a single tile of the grid. Moreover, each clone/tile can be marked as “On Path” or “Off Path” (depending on the current trajectory path), as well as an error (whenever the cursor collides with an off-path tile at runtime). Finally, each clone/tile includes a “Static Trigger Collider” component to enable collision detection with the virtual cursor.

The heart of our AR rehab game app is the “Game Manager” script, which handles the game logic and the import–export functionalities. The game logic is implemented using a simple finite state machine (FSM) model, and includes: game setup, game play, winning losing conditions, and the relative updates to the “Game Grid” and UI.

The import export management is implemented using file I/O operations, and includes: setting up the game according to a config file (text file edited by the therapist), saving into a text file the performance data after each rehabilitation session (to be reviewed by the therapist).

2.5. Hand Tracking and Virtual Cursor Control

The Microsoft HoloLens detects the user’s hands when they are in either the “ready” state (back of the hand facing the user with index finger up) or the “pressed” state (back of the hand facing the user with the index finger down) [30].

The Microsoft HoloToolkit provides an easy to use interface to allow developers to access hand tracking information: in our case the detected real-time 3D position of the user’s hand.

The tracked hand 3D position is used to update in real time the “Virtual 3D Magnetic Cursor” performing these steps:

• Derive the 3D vector, $\overrightarrow{V}$, starting from the Mixed Reality Camera 3D position and ending at the 3D position of the tracked hand.
• Update the 3D rotation of the “Virtual 3D Magnetic Cursor” to align it with the 3D vector $\overrightarrow{V}$.
• Translate by, t (see magnetic behavior below), the virtual cursor 3D model along the 3D vector $\overrightarrow{V}$.

Additionally, our virtual cursor integrates a magnetic behavior to facilitate user interactions with the “Game Grid” and the UI in AR. Whenever a collision is detected between the virtual cursor and a game element (a tile or a UI control), the translation, t (see step 3 above), is calculated so that the virtual cursor is moved on top of the colliding object (with an offset equal to its radius); otherwise, if no collision is detected, the translation, t, is set to keep the virtual cursor at a constant distance (1.9 m) in front of the user.

Finally, the virtual cursor changes in color depending on active interactions: If the cursor is colliding/interacting with a game element, its color is red; otherwise, the cursor color is gray (Figure 5).

2.6. System Testing: Performance and User Evaluation

Both the memory and frame rate of our AR rehab game app were tested for each of five different resolutions for the virtual trajectory panel, to take into account possible performance degradations with an increase in the number of tiles (namely the number of colliders).

Performance data was acquired using the Windows Device Portal, connecting the HoloLens over Wi-Fi. Moreover, all these tests were also carried out to select the optimal dimensioning of the virtual trajectory panel and the game UI as a compromise among: obtainable ROM, game playability, and performance (in terms of frame rate and memory).

After this preliminary evaluation, 5 physiotherapists and 20 healthy volunteers, 2 left-handed and 23 right-handed subjects, with 10/10 vision or corrected (contact lenses) to 10/10 vision, were recruited to use and then evaluate out AR rehab game app using the selected configuration as resulted from the preliminary tests (Figure 5).

Table 2. Participant’s demographics.

	Number of Subjects
Physiotherapists (yes, no)	5, 20
Gender (male, female, non-binary)	14, 11, 0
Age (min, max, mean, standard deviation)	23, 52, 32, 7.4
Handedness (left, right, ambidextrous)	2, 23, 0
Vision (10/10 naked eyes, corrected to 10/10 with lenses)	17, 8
Experience with Video games (none, limited, familiar, experienced) *	5, 4, 8, 8
Experience with AR (none, limited, familiar, experienced) *	8, 9, 6, 6
Experience with HoloLens (none, limited, familiar, experienced) *	16, 6, 1, 2

* none—technology never used; limited—technology used less than once a month; familiar—technology used about once a month; experienced—technology used several times a month.

reports the demographics of the participants; besides demographic data we also asked the participants to rate their experience with video games, AR methods, and HoloLens.

Three different trajectory paths were implemented (“single arch”, “double arch”, “infinity symbol”) (Figure 6), the number of repetitions of each trajectory was set at three (for a total of nine shoulder ROM rehabilitation exercises), the trajectory width was set at 10 cm (equal to the virtual cursor diameter), and the maximum completion time for each trajectory was set at 70 seconds. A different background painting was chosen for each of the nine shoulder ROM rehabilitation exercises.

The study protocol for each participant included the following steps:

• Administration of a “Consent and a Demographic Form” (Table 2), including information about the subject’s previous experience with videogames, AR, and HoloLens.
• Introduction to the AR rehabilitation game app (game goals, UI, and virtual cursor control) with a standardized 8–10 min oral explanation.
• User-specific HoloLens calibration using the Microsoft Calibration app.
• Introduction to the HoloLens 3D tracking (workspace, gestures) using the Microsoft Learn Gestures app.
• Testing of our AR rehab game app until the completion of all nine shoulder ROM rehabilitation exercises.
• Administration of a “Likert Questionnaire” to collect the subject evaluation on the engagement and ergonomics of our AR rehab game app.

The “Likert Questionnaire” (

Table 3. Results of the Likert Questionnaire.

Item		Median (25°~75°)			p-Value (All)
		Ph	N-Ph	All	VG	AR	HL
Engagement	1. The game goal (discovering the painting) is motivating, interesting, and engaging.	4 (4–4)	5 (4–5)	4 (4–5)	0.618	0.175	0.439
	2. The visual feedback such as countdown timer and scoring system are motivating.	4 (3–4)	4 (3–4.25)	4 (3–4)	0.243	0.723	0.730
	3. The game visuals and audio are enjoyable.	4 (3–4)	4 (3–5)	4 (3–5)	0.748	0.380	0.301
	4. The game goal is clear.	4 (4–4)	5 (4–5)	4 (4–5)	0.414	0.246	0.337
	5. Likely to play again.	4 (4–5)	4 (4–5)	4 (4–5)	0.330	0.466	0.447
Ergonomics	6. The graphical user interface (buttons) is intuitive and user-friendly.	4 (3–4)	4 (4–5)	4 (4–4)	0.643	0.324	0.997
	7. The text instructions, buttons, and counters are readable and clear.	4 (3–4)	4 (4–5)	4 (4–5)	0.445	0.270	0.612
	8. The trajectory thickness and the panel size allow good playability of the game.	3 (3–4)	3.5 (2.75–4)	3 (3–4)	0.522	0.436	0.218
	9. The virtual 3D cursor is intuitive and easy to control.	3 (2–4)	4 (4–5)	4 (3–5)	0.647	0.740	0.407
	10. The latency (lag, delay) between real hand movement and virtual 3D cursor displacement is acceptable.	4 (4–4)	4 (4–5)	4 (4–5)	0.070	0.727	0.837
	11. No visual discomfort is perceived during the game session.	4 (3–4)	5 (4–5)	5 (4–5)	0.253	0.169	0.283
	12. Arm–shoulder fatigue is perceived during the game session.	4 (3–4)	4 (3–4)	4 (3–4)	0.555	0.760	0.586
	13. No postural discomfort (arm–shoulder excluded) is perceived (e.g., neck fatigue).	3 (2–3)	5 (3.75–5)	4 (3–5)	0.018	0.353	0.140

* Ph—physiotherapists; N-Ph—non-physiotherapists; VG—video game; HL—Microsoft HoloLens.

) comprises 13 items, each evaluated using a five-point Likert scale (from 1 = strongly disagree to 5 = strongly agree) addressing the motivational value and the ergonomics of our wearable AR rehab game.

Statistical analysis of questionnaire results was performed using the SPSS^® Statistics Base 19 software. The central tendencies of responses to a single Likert item were summarized by using median, with dispersion measured by interquartile range.

The Kruskal–Wallis test was used to understand whether the answering tendencies (with respect to each Likert item) differed based on “Experience with Videogames”/“Experience with AR”/“Experience with HoloLens”. A p-value < 0.05 was considered statistically significant.

3. Results

For all the tested sizes, the app consistently met the recommended target frame rate goals for the Microsoft HoloLens: 60 fps [37]. The memory required to run the application was ~220 MB, independently of the number of tiles. A size of 2.5 m × 1.4 m was selected according to the obtainable ROM and a qualitative evaluation on the game playability. Table 3 summarizes the data collected with the “Likert Questionnaire”.

Overall, the participants agreed with all the statements addressing the AR rehab game motivational value and the ergonomics with the exception of item 8 (“The trajectory thickness and the panel size allow good playability of the game”) for which the subjects expressed a neutral opinion. The rehabilitation specialists agreed with the motivational value and the ergonomics with the exception of: item 8 (“The trajectory thickness and the panel size allow good playability of the game”), item 9 (“The virtual 3D cursor is intuitive and easy to control”), and item 13 (“No postural discomfort is perceived”), for which the subjects expressed a neutral opinion. As for item 8, the restriction of the HoloLens field of view did not compromise the successful completion of the gamified task: All the subjects were able to complete the nine trajectories within the given time of 70 seconds. Moreover, no repercussion on the application enjoyability was found.

As for items 9 and 13, rehabilitation specialists suggested to limit trunk movements (for example asking the subjects to sit on an armless, high back chair during the rehabilitation session, using an elastic restraint around their torsos) which can negatively affect both the control of the virtual 3D cursor and postural comfort. Moreover, they stressed the need of a further evolution of the HMD technology, for an improved field of view and ergonomics in terms of HMD weight.

Finally, results reveal that the users experienced arm–shoulder fatigue, showing that the application difficulty level could be adjusted to setup an intensive exercise for the shoulder ROM rehabilitation. For all items, there was no statistically significant difference (p > 0.05) in answering tendencies among participants with different levels of experience with video games (VG), AR, and Microsoft HoloLens (HL) (see Table 3 for p-values).

4. Discussion

AR has proven to be a promising technology in healthcare, as demonstrated by the increasing number of publications in medical and surgical training [38,39,40,41,42,43], surgical navigation [44,45,46,47], and also rehabilitation [48,49,50,51,52].

Recently AR solutions based on spatial displays have been proposed in the literature to increase the effectiveness of shoulder rehabilitation. These systems, which employ vision-based methods for the real-time tracking of patient arm/hand, are mainly based on the use of wearable markers [24].

To the best of the author’s knowledge, this paper presents the first wearable AR rehab game app for shoulder rehabilitation. The potentialities of Microsoft HoloLens are explored to develop a portable system without the need for any adjunctive sensor/peripheral interconnection cable and/or marker that has to be worn. Indeed, these devices can hamper the patient’s movements, making them unsuitable for certain types of activities, such as the training programs in sports rehabilitation [47].

Our software application allows a real-time markerless tracking of the user’s hand, and it has an optimal frame rate performance (60 fps) for an immersive AR experience (according to Microsoft guidelines).

A user study with 20 healthy subjects and 5 physiotherapists was performed to test the app performance in technical terms, and to obtain a preliminary feedback on the motivational value and the ergonomics of the proposed AR application. Overall, participants agreed that the application is motivating, interesting, and engaging. It is widely accepted that motivation can increase the chances of a fast recovery and can lead to more substantial levels of achievement. The traditional rehabilitation process can be time consuming, and the repetitiveness of exercises can bore the patients, sometimes leading them to neglect the prescribed exercises. For all these reasons the gamification of the rehabilitation task could substantially improve the result of the therapy.

Moreover, subjects overall agreed with the ergonomics of the user interface and text instructions regardless of their experience with videogames, AR technologies, and Microsoft HoloLens. Results also show the absence of visual discomfort, however, as stressed by rehabilitation specialists, there is the need of a further evolution of the HMD technology, for an improved field of view and ergonomics in terms of HMD weight: The use of HoloLens 2 will be a step forward in this direction. Moreover, physiotherapists suggested to use supports during the rehabilitation session to help the patient maintain the correct posture.

A limitation of the present study is that no elderly subjects were recruited in the trial and we have not investigated the impact of subject age on questionnaire results: This restricts the generalizability of our findings. Reasonably the proposed technology could be used for the rehabilitation of young-to-middle aged subjects: e.g., for training programs in sport rehabilitation to strength the shoulder muscles and regain the natural ROM.

Future work will focus on testing the system on a large cohort of patients with shoulder pain, performing a comparative study with traditional rehabilitation methods. Subsequently, work will focus on the implementation of advanced metrics for patient performance analysis, the use of Microsoft HoloLens 2 to further improve ergonomics and to mitigate current technological limitations related to the narrow field of view, and a comprehensive cost-effectiveness assessment.

Compared to other low-cost AR systems proposed in the literature (such as the AR platform by Colomer et al. [21]), our wearable AR application requires a higher initial investment for instrumentation (Microsoft HoloLens 2 costs about $3500). However, it could reasonably allow for a reduction of total rehabilitation cost since the application does not require the supervision of a therapist for guiding the patient during the rehabilitation.