A Projected AR Serious Game for Shoulder Rehabilitation Using Hand-Finger Tracking and Performance Metrics: A Preliminary Study on Healthy Subjects

Abstract

Research studies show that serious games can increase patient motivation regardless of age or illness and be an affordable and promising solution with respect to conventional physiotherapy. In this paper, we present the latest evolution of our system for shoulder rehabilitation based on hand-finger tracking and projected augmented reality. This version integrates metrics to assess patient performance, monitors the game progress, and allows the selection of the game visualization mode (standard on-screen or projected augmented reality). Additionally, the new software tracks the velocity, acceleration, and normalized jerk of the arm-hand movements of the user. Specifically, sixteen healthy volunteers (eight technical and eight rehabilitation experts) tested our current prototype. The results showed that the serious game is engaging, its design is ergonomically sound, and the overall system could be a useful tool in shoulder rehabilitation. However, clinical validation is needed to assess that the serious game has the same effects as the selected therapy. This is the preliminary step toward laying the foundation for future studies that investigate abnormalities in shoulder movements by using hand-finger tracking.

1. Introduction

Serious games (SGs) are videogames whose primary purpose is to teach the user something or permit them to develop skills in an interactive environment that promotes the user’s interest in terms of engagement and fun [1]. Applications of SGs range from healthcare (where the focus is on treatment, recovery, and rehabilitation) to the business environment, and up to the military sector [1,2].

Unlike traditional teacher-centric learning environments, SGs present a learner-centric educational approach where the learner has more control over the learning process through interactivity.

Several studies [3,4,5,6] have shown that SGs, specifically exergames (i.e., SGs involving physical exertion to play), can be an inexpensive and promising solution to unengaging conventional physiotherapy. These studies have confirmed that exergames in rehabilitation increase patient motivation regardless of age or health condition.

As reported in the literature [7], only 55% (patients with upper limb issues) to 76% (patients with lower limb issues) of rehabilitation patients perform the exercises prescribed by medical staff due to costs, logistical reasons, and lack of motivation.

According to Rego et al. [1], the main criteria for the classification of serious games in rehabilitation are:

• The application area (e.g., cognitive and motor rehabilitation);
• The game interface (e.g., two-dimensional or three-dimensional);
• The number of players (e.g., single or multi-player);
• The game genre (e.g., general videogames, serious games, etc.);
• The adaptability to the user competence level (e.g., novice versus proficient);
• The feedback on performance (e.g., no feedback or performance metrics);
• The monitoring of game progress (e.g., in-game monitoring or not);
• The portability of the game (e.g., special hardware required or not);
• The accessibility of the game (e.g., eliminating rehab barriers such as cost, etc.);
• The interaction technology (e.g., mouse/keyboard, gestures, touch screens, etc.).

The interaction modalities in serious games for upper limb rehabilitation can range from traditional mouse/keyboard to novel virtual reality (VR) and augmented reality (AR) interfaces: from VR/AR controllers to gesture-based interactions, and even haptic interfaces [8,9]. However, one of the disadvantages of these innovative VR/AR interfaces is the necessary technology (i.e., sensors, controllers, etc.) that patients with different levels of motor impairment must use or wear. Furthermore, most of these interfaces are expensive and require specific expertise to be properly configured, limiting their use to specialized rehabilitation centers [10,11,12,13].

Recently, researchers and the rehabilitation community have paid attention to videogame technology: full-body tracking devices such as the Microsoft Kinect (Microsoft, Redmond, WA, USA [14]) and hand-finger tracking systems such as the Leap Motion Controller (LMC) (Leap Motion Inc., San Francisco, CA, USA [15]). This technology has proven to be valuable for practicing arm and hand rehabilitation [16].

For example, “VAST.Rehab” [17] is a commercial VR rehabilitation platform for patients with impaired function resulting from neurogenic, cerebral, muscular, spinal, or bone-related disorders. This platform was developed to integrate a wide variety of tracking systems, including the Microsoft Kinect and the Leap Motion Controller (LMC).

Overall, the Microsoft Kinect v2 can track the whole body, but gesture recognition is far less accurate and reliable than the detection performed by the LMC. This latter device can track hand and finger movements with declared submillimeter accuracy. Specifically, the detection range of the Microsoft Kinect v2 is significantly bigger than the tracking span of the LMC (80 cm–4 m vs. 25–60 cm of LMC) but its field of view (FOV) is smaller than that of the LMC (horizontal and vertical FOV of 70° and 60°, respectively, for the Microsoft Kinect v2, versus 150° and 120°, respectively, for the LMC) [18,19].

Several authors [20,21,22,23,24] focused on the effectiveness of the LMC and virtual environments in upper limb rehabilitation. In particular, the LMC is used in systems oriented to the rehabilitation of fine and gross manual dexterity. This paper presents an evolution of a serious game we have designed previously [25,26,27]. In the past, we used AR technology for the development of two different versions of our serious game (both versions were preliminarily tested by healthy subjects and rehabilitation experts):

• A wearable application “AR Rehab Game App” for training shoulder horizontal and vertical flexion, based on head-mounted display (HMD) technology (i.e., the Microsoft HoloLens). Our preliminary tests showed that AR technology allows promising results in terms of user motivation but needs further evolution to improve the FOV and reduce the physical discomfort (e.g., the weight of the HMD) [25];
• A non-wearable application for training shoulder horizontal adduction, based on a standard desktop computer, a screen, and the LMC as a hand-tracking system. A limitation of this version, compared to the HMD-based version, is the reduced portability. However, this limitation was balanced by other advantages such as improved ergonomics and lower cost [27].

The key features of our serious game (previous and current versions) are software and hardware customization according to patients’ needs and skill levels, portability, and versatility (because it is suitable for children and adults). The aim of our rehabilitation system is to allow individuals with shoulder disorders to practice movement training at home or the clinic, without requiring the therapist to be always present.

In this paper, we present the latest version of our system as a non-wearable projected AR serious game for shoulder rehabilitation: an evolution of our previous prototypes [25,26,27]. The new system integrates metrics for patient performance assessment, allows the monitoring of game progress, and implements different game visualization modes. In particular, the user can choose between a conventional screen-based mode or the projected AR mode. As shown in [28], the choice of human–computer interface influences the demands of rehabilitation therapy and can be individualized to patients according to their needs (for example, the cognitive demand is different for each visualization mode, etc.). In projected AR modality, the game scenario is projected directly onto a desk pad, the system can be adjusted to use a tilt-top table, and the desk pad material can be changed to modify the hand-pad friction during a rehabilitation exercise. In addition, at the end of each session, the user can check their performance report expressed in terms of hand speed, hand acceleration, normalized jerk, and completion time. Whenever a rehabilitation exercise is completed, the performance report is automatically stored in a database for clinical analysis and progress monitoring.

Finally, a preliminary study with 16 healthy subjects, including rehabilitation and technical experts, was conducted to prove the system is functional, anticipating potential issues.

In particular, the main goal of this evaluation was the assessment of our system before its introduction in rehabilitation therapies with real patients; our secondary objective was the investigation of the effects on user motivation and enjoyment, as well as the overall ergonomics of the system. In addition, during these preliminary tests, data were collected to define performance metrics (e.g., the thresholds for maximization of the game score).

2. Materials and Methods

The “Painting Discovery” serious game was conceived to provide a playful rehabilitation process of the upper limb, using concepts of video games and AR system design [29]. The development of our serious game involved a multidisciplinary team with diverse stakeholders (technical and rehabilitation experts), and it is based on the concept of iterative and incremental development. An iterative and incremental method is useful not only for managing communication between clinical and technical experts but also for supporting incremental development that considers the validation process. For example, an important aspect is the testing of critical non-functional requirements, such as the safety of interaction mechanisms [30].

Considering the existing interaction technology and based on the technical observations and knowledge of the therapists involved in our study, we defined the desirable features of the interaction mechanism:

• The tracking device must be able to detect the patient’s hand during the trial;
• The selected device must be non-invasive, easy-use, and affordable.

We excluded a priori glove-based tracking technology because of wearability issues for users with physical disabilities or who are recovering from hand surgery, hygiene issues due to the Lycra fabric, short durability, and high cost. In addition, because our serious game is intended for therapy in clinical practice, especially in home settings, we ruled out technology based on passive markers and cameras because of their complex configuration.

We analyzed three different possibilities as shown in

Table 1. Motion-based device analysis.

Device	Contact-Free	Hand Tracking	Low Cost
Nintendo Wii Remote MotionPlus	no	yes	yes
Microsoft HoloLens	no	yes	no
Leap Motion Controller (LMC)	yes	yes	yes

.

The results of this analysis led us to reject Wii Remote MotionPlus (Nintendo of America Inc., Redmond, WA, USA) because is invasive for patients who have difficulty holding physical devices. The HoloLens-based solution was also rejected mainly because it is invasive for patients [31,32] and also outside the target budget.

Finally, LMC was selected as the best option because it fulfilled all the requirements. The LMC was used as a natural user interface that does not require the user to hold any device for interactions. A PicoPix projector (Philips PPX4010) was used to display the game on the desk. The experimental setup is shown in Figure 1. A positive feature of our system is that it does not require to be in a dimly lit environment; in fact, the projector and desk pad allow the rehabilitation session to be performed in environments with normal artificial or natural light. The software was developed in the Unity 3D game engine (version 2019.4.18f1) for the Microsoft Windows 10 platform, with all scripts coded in C#, and using the Unity Plugin for the Leap Motion Orion Beta (version 4.4.0).

2.1. Apparatus

The rehabilitation system setup consists of:

• A laptop (ASUS UX303UB, Intel Core i5-6200U @ 2.30 GHz processor, 8 GB RAM memory, and NVIDIA GeForce 940 M graphics board) running Microsoft Windows 10 Pro;
• The LMC (range detection 25–60 cm, Vertical FOV 120°, Horizontal FOV 150°, sampling rate 50–200 Hz) tracks in real-time the hand movements of the user by using infrared (IR) cameras and emitters (IR LEDs) [15,19];
• A tilt-top table (range of tilt angle: 0–50°);
• A black rubber desk pad (60 × 40 cm), as the projection surface for the game;
• A portable PicoPix projector (Philips PPX4010, resolution 1280 × 720, screen distance 50–500 cm, aspect ratio 1.47:1, size 6.8 × 6.6 × 2.2 cm, weight 81.6 g) [33].

Note that the serious game GUI (graphical user interface) is visualized in part on the laptop display (e.g., for login, configuration, etc.), and in part on the projected AR (i.e., the game panel).

2.2. Technical Details

The LMC presents important advantages over other motion tracking systems thanks to its portability, ease of use, commercial availability, low cost, and contact-free interaction. This latter feature is particularly critical for upper limb rehabilitation because these patients often have physical impairments preventing the use of wearable sensors or interfaces.

In our setup, the LMC is enclosed in an ad-hoc 3D-printed support, and it is mounted upside down on an adjustable desk arm orthogonally to the desktop, close enough (42 cm) to the user workspace on the desk pad to track the user’s hand.

The black rubber desk pad is used to prevent IR interferences (e.g., IR reflections, etc.), that could degrade the accuracy of the LMC hand tracking. The conventional configurations of the LMC are placed on a desk pointing upward or mounted on the front face of a VR HMD pointing forward. In both these standard setups, the LMC tracks hand-finger movements looking at an empty volume; on the contrary, in our upside-down configuration, the LMC tracking has to face hand movements on a solid background (i.e., the desk pad surface). For these reasons, the black rubber desk pad has proven to be valuable in improving both the reliability and accuracy of the LMC tracking. Additionally, by selecting materials with different mechanical properties (i.e., surface friction), our system allows a therapist to tailor the friction (desk pad vs. user’s hand) to the patients’ motor or sensory deficits.

The mini projector, enclosed in an ad hoc 3D-printed box, is mounted to an adjustable desk arm, and oriented orthogonally to the desk pad at a fixed distance (70 cm). The physical size of the projected image is 50 × 28 cm², so the projector field of view overlaps on the user’s workspace on the desk pad.

2.3. Serious Game Design

Our shoulder rehabilitation system consists of a single-player serious game. The exercise consists of a planar unimanual task performed on a desk pad that involves arm and hand movements and aims to train shoulder horizontal adduction within the average normal range of motion (ROM) from 0° to 130°.

The game requires the user to move a virtual cursor (VC), controlled by hand movements, along a predefined 2D trajectory visualized on the screen or projected on the desk pad. This path consists of several tiles highlighted on a 2D grid occupying most of the game area and covering a painting. Whenever a path tile in Figure 2 is “touched” by the VC, its transparency changes, uncovering part of the hidden painting (“touching” off-path tiles results in an error highlighted by coloring the tile in red). Once the trajectory is completed (i.e., all path tiles have been “touched”), the painting will be completely uncovered (Figure 2). In the current prototype, the game includes three trajectories: single arc (difficulty: easy), double arc (difficulty: medium), and infinity symbol (difficulty: hard).

Specifically, the single- and double-arc trajectories are based on the “Rolyan Range of Motion Shoulder Arc” device [34]: a standard rehabilitation tool used to treat any upper limb deficit that impairs ROM.

The painting pictures can be customized to make the game more engaging/suitable for adults or children [25,27]. A full rehabilitation session consists of successfully completing each trajectory three times in a row, for a total of nine rehabilitation exercises.

Visual feedback is incorporated into our game to provide the patient a measure of their progress in terms of achieving goals, and to notify specific events such as instructions, completion time, as well as gratification messages during rehabilitation training sessions. In addition, background music is integrated for comfort [25,27].

The user (the therapist or the patient based on the advice of the therapist) can adjust the complexity of the session configuring the game via software and/or hardware.

At the hardware level, it is possible to choose the game presentation mode between screen-based or projected AR modes (by disconnecting the projector) and vary both the inclination of the table and the friction between the user’s hand and the desk pad by replacing the desk pad with one of a different material.

At the software level, before starting a game session, the user can load a configuration file including settings for the trajectory path, the tile size, the sequence of exercises/trajectories presented, and the maximum completion time for each exercise. After loading a configuration file selected with the “File Browser Panel”, the user has to log in by using the “Login Panel” (Figure 3). Then, the system automatically creates a file to store all the performance data of the patient when the rehabilitation session is over.

The “Config Panel” (Figure 3) allows the calibration of the projector and the virtual cursor, these settings can be saved into a file and loaded as needed. In addition, the user is also able to adjust the volume of the background music and the brightness of the projected game scenario. After these adjustments, the user can start playing/exercising.

One of the novel features of our system is the capability to monitor the patient’s progress over time in terms of hand speed, hand acceleration, normalized jerk, and completion time.

At the end of each session (three exercises/trajectories repeated three times), the “Report Panel” (Figure 3) shows the performance data to the user. Then, each report is stored in a database for clinical analysis and progress monitoring.

Figure 4 shows the complete system architecture of our projected AR serious game, including all software and hardware modules, all data transfers, and all the user and module interactions.

2.4. Performance Metrics

As explored in the literature, several studies [35,36,37] considered velocity, acceleration, jerk, and normalized jerk as key parameters in diagnosing upper limb movements. For example, Elgendi et al. [35] used instantaneous velocity and acceleration to study the movements of the finger, elbow, and shoulder.

The jerk, defined as the rate at which the acceleration changes over time, is a measure of the smoothness of the movement [38]. Several studies used the jerk as a measure of motor performance of both healthy and unhealthy subjects [37,39].

Specifically, we will measure the velocity (as average and max value), acceleration (as average and max value), jerk, and normalized jerk. The collected results will be used as a benchmark for effective speed, acceleration, jerk, and normalized jerk detection of upper limb movements. The average and max values of velocity and acceleration are computed starting from the corresponding instantaneous values. The instantaneous velocity of motion for the hand is calculated as the resultant of the VC positions ($x_{vc}$, $y_{vc}$, $z_{vc}$) between two consecutive frames (i.e., game images visualized on screen). The 3D coordinates of the VC positions are computed in Equations (1)–(3), their values are expressed in the coordinate system of the LMC and measured in meters [40].

$$x_{vc}\left(t\right)=x_{hand}\left(t\right)\times \alpha$$

(1)

$$y_{vc}\left(t\right)=y_{hand}\left(t\right)\times \beta$$

(2)

$$z_{vc}\left(t\right)=0$$

(3)

In these equations, x_hand and y_hand (and z_hand) are the real-time 3D coordinates of the position of the user hand (i.e., palm), whereas α and β are control-display (CD) gain factors used for the X and Y axes, respectively (usually the same CD gain is used for both axes, so α = β). If the system is properly configured, the z_hand coordinate is negligible, so, only x_hand and y_hand are used to update the 3D position of the VC.

The instantaneous velocity ($V_{inst}$) is computed in Equation (4):

$$V_{inst}\left(n\right)=\frac{d{x}_{vc}{,y}_{vc}}{dt}{|}_{t=nT}=\frac{1}{∆T}\sqrt{{{[x}_{vc}\left(n\right)-x_{vc}\left(n-1\right)]}^2+{{[y}_{vc}\left(n\right)-y_{vc}\left(n-1\right)]}^2}$$

(4)

where $∆T$ is the sampling interval (equal to the reciprocal of the fixed sampling frequency $f_s$ = 100 Hz), and n is a sequential counter identifying a data sample in the dataset of a processed motion (i.e., an exercise) with N data samples in total.

The instantaneous acceleration ($A_{inst})$ is defined in Equation (5):

$$A_{inst}(n)=\frac{d{V}_{inst}}{dt}{|}_{t=nT}=\frac{1}{∆T}{[V}_{inst}(n)-V_{inst}(n-1)]$$

(5)

Equations (6) and (7) specify the formulas to compute the average velocity $\left(V_{avg}\right)$ and the average acceleration $\left(A_{avg}\right)$.

$$V_{avg}=\frac{1}{N}\sum _{n=1}^N{V}_{inst}\left(n\right)$$

(6)

$$A_{avg}=\frac{1}{N}\sum _{n=1}^N{A}_{inst}\left(n\right)$$

(7)

The instantaneous jerk ${(J}_{inst})$ is defined in Equation (8):

$$J_{inst}(n)=\frac{d{VA}_{inst}}{dt}{|}_{t=nT}=\frac{1}{∆T}{[A}_{inst}(n)-A_{inst}(n-1)]$$

(8)

where N refers to the total number of samples in the processed motion.

The normalized jerk ($J_{norm}$) is expressed in Equation (9) [41,42]:

$$J_{norm}=\sqrt{\frac{{T_{comp}}^5}{2{L_{hand}}^2}\sum _{t=0}^{t=T_{comp}}{\left[J_{inst}\right(n\left)\right]}^2}$$

(9)

where $T_{comp}$ refers to the time employed to complete a predefined trajectory (measured in seconds), as defined by Equation (10):

$$T_{comp}=T_{max}-T_{diff}$$

(10)

where $T_{max}$ is the maximum completion time (i.e., the maximum time to complete the trajectory), and $T_{diff}$ is the remaining time after the trajectory is completed (i.e., the time from the completion of the trajectory to the maximum completion time).

$L_{hand}$ is the path length carried out by the user’s hand during the trial (measured in meters).

2.5. System Testing

Our projected AR serious game for shoulder rehabilitation underwent preliminary tests to measure its performance and to collect users’ evaluations.

Both the memory and frame rate of our rehab serious game were tested.

A specialist validation was performed by two types of users: rehabilitation and technical experts. The former specialists tested the acceptance and feasibility of the serious game in rehabilitation therapies as well as the consistency of the collected metrics. Instead, the latter specialists assessed the technical aspects such as the performance, design, and safety of our serious game. The main goal was to analyze whether the game was easy to use, whether it could contribute to improved patient performance, and whether physical therapists would use it in real rehabilitation processes.

Sixteen healthy adult volunteers (8 physiotherapists, 8 engineers) and 16 right-handed subjects with 10/10 vision or corrected (contact lenses) to 10/10 vision participated in the study. No subject had a disease affecting upper limb motor function.

Table 2. Demographics of participants.

	Number of Subjects
Gender (male, female, non-binary)	5, 11, 0
Age (min, max, mean, STD)	21, 46, 31, 7
Physiotherapists (yes, no)	7, 9
Handedness (left, right, ambidextrous)	0, 16, 0
Vision (10/10 naked eyes, corrected to 10/10 with lenses)	7, 9
Experience with videogames (none, limited, familiar, experienced)	2, 5, 8, 1
Experience with AR (none, limited, familiar, experienced)	4, 3, 7, 2
Diagnosed with a shoulder disorder (no, yes)	16, 0
Perceived shoulder pain (no, yes)	16, 0

reports the demographics of the participants; in addition to demographic data, we also asked the participants to rate their experience with video games and AR methods.

This study was conducted following the Declaration of Helsinki, and it was approved by the bioethics committee of Pisa University. After being informed about the study, all participants signed an informed consent form. Each subject performed nine exercises. The $T_{max}$ for each trajectory was set at 300 s. The CD gain factors, α and β, were both set to 1.5 position, velocity (as average and maximum velocity), acceleration (as average and maximum acceleration), and normalized jerk of the hand were recorded and saved on file for each exercise, in addition to completion time and game level. The collected data will be analyzed to define performance metrics to represent the user progress in a rehabilitation program. At the end of a trial, each candidate completed a 5-point Likert questionnaire (from 1 = strongly disagree to 5 = strongly agree) that included 14 questions addressing the motivational value and the ergonomics of our serious game. A modified questionnaire was used for the expert rehabilitation, with 7 additional items targeting: the usefulness of the game as a shoulder rehabilitation tool; its portability; its advantage over traditional rehabilitation processes; its personalization over traditional rehabilitation treatment; the game as a tool to speed up recovery time; its suitability in terms of the correct posture; and implemented trajectories to recover the shoulder ROM.

The statistical analysis of questionnaire results was performed using the SPSS Statistics Base 22 software. The central tendencies of responses to a single Likert item were summarized by using the median, with dispersion measured by the interquartile range. The Kruskal–Wallis test was used to understand whether the answering tendencies (for each Likert item) differ based on the subject “Profession”, “Videogames Experience” and “AR Experience”. A p-value < 0.05 was considered statistically significant.

3. Results

During our preliminary tests, the memory required to run the serious game was ~170 MB, whereas the frame rate was always well above 100 fps. Overall, the participants (strongly) agreed with all the items addressing the motivational value and ergonomics of our rehab serious game, as can be seen in

Table 3. Likert questionnaire results (from 1 = strongly disagree to 5 = strongly agree).

Item		Median (25°~75°)			p-Value (All)
		All	Ph *	Eng *	Profession *	VG *	AR *
Engagement	The game goal (discovering the painting) is motivating, interesting, and engaging.	4.5 (5–4)	4 (5–4)	5 (5–4)	0.418	0.103	0.885
	The game goal is clear.	5 (5–5)	5 (5–4.25)	5 (5–5)	0.535	0.442	0.312
	The visual feedback such as countdown timer and scoring system is motivating.	4 (4–3)	4 (4.75–3.25)	4 (4–3)	0.427	0.115	0.268
	The game visuals and audio are enjoyable.	4.5 (5–4)	4 (4.75–3.25)	5 (5–4.25)	0.077	0.169	0.834
	Likely to play again.	4 (5–4)	4 (5–4)	4.5 (5–4)	0.480	0.228	0.790
Ergonomics	The graphical user interface (buttons) is intuitive and user-friendly.	5 (5–4)	5 (5–4.25)	5 (5–4)	0.653	0.387	0.634
	The text instructions, buttons, and counters are readable and clear.	4.5 (5–4)	4 (5–4)	4 (5–4)	0.418	0.226	0.228
	Adjusting the projected image brightness and volume improves playability.	4.5 (5–3.25)	5 (5–3)	5 (5–4)	0.254	0.521	0.816
	The trajectory thickness and the panel size allow good playability of the game.	4 (5–4)	4 (4.75–4)	4 (5–4)	0.637	0.659	0.932
	The projected image is well contrasted to allow for good playability.	4 (5–4)	4 (5–4)	4.5 (5–4)	0.626	0.350	0.218
	The projected image has a good resolution to enable good playability.	4.5 (5–4)	4.5 (5–3.25)	4.5 (5–4)	0.643	0.168	0.200
	The latency (lag, delay) between real hand movement and virtual 3D cursor displacement is acceptable.	4 (5–4)	4.5 (5–4)	4 (4–4)	0.239	0.256	0.932
	Interaction with the game does not require mental effort.	4 (5–4)	4 (4.75–3.25)	5 (5–4)	0.085	0.361	0.929
	No postural discomfort (arm–shoulder excluded) is perceived during the game session.	5 (5–4)	4 (5–4)	5 (5–4.25)	0.195	0.763	0.612
Rehabilitation Experts Evaluation	The experimental setup allows the user to perform the task with the correct posture.	5 (5–4)		-	-	-	-
	The system could bring more benefits than a traditional rehabilitation process.	3.5(4–3)		-	-	-	-
	System can help speed up patient recovery.	4 (4–3.25)		-	-	-	-
	The proposed system is useful for upper-arm rehabilitation.	4 (4–4)		-	-	-	-
	The proposed system allows the user to perform the rehabilitation task without the need for a supervisor.	4 (5–4)		-	-	-	-
	The system is easier to customize than traditional rehabilitation treatment (e.g., Rolyan’s arch).	4 (4.75–3)		-	-	-	-
	The implemented trajectories are suitable for a range of motion rehabilitation.	4 (4.75–3)		-	-	-	-

* Ph—physiotherapists; Eng—engineers; VG—video game; AR—augmented reality.

and Figure 5, Figure 6 and Figure 7. All physiotherapists (strongly) agreed on our serious game’s usefulness as a shoulder rehabilitation tool, game portability, game suitability in terms of correct posture and implemented trajectories, and greater personalization of treatment over the traditional method.

Figure 5, Figure 6 and Figure 7 show the results for all questions presented in Table 3; each of them shows the results for all participants in this experiment.

In addition, physical therapists agree that our serious game can speed up recovery time. Half of the physiotherapists agreed, including one who fully agreed, that the use of “Discovery Painting” could bring more benefits than the traditional rehabilitation process; however, three of them expressed a neutral opinion, while one disagreed (3.5 median); their answer was mainly related to the fact that they underlined that rehabilitation movement was not complete and only allowed in the horizontal plane. This indicates that physiotherapists tend to prefer new technologies in therapies with the aim of making the rehabilitation treatment more engaging, although the serious game still needs some improvements. Additionally, some physiotherapists provided useful remarks after completing the preliminary tests, such as, for example: making the projection image larger, allowing wider movements on the horizontal plane to allow the patient to move their hand in the vertical plane as well so that the range of motion can be completed, providing a measure of movement accuracy. Additionally, some validators/testers suggested replacing the background of level 8 (“Les Amoureux et Luna Park” by Joan Miró) because the drawing could be mistaken with the trajectory.

For all items, there was no statistically significant difference (p > 0.05) in answering tendencies among participants with different levels of experience with video games and AR (see Table 3 for p-values). This result shows that all participants were able to use our rehab system regardless of their experience with video games and AR; in other words, the game is accessible to everyone and does not require specific skills.

The average execution time was 351 ± 193 s with Max_time and Min_time of 957 and 182 s, respectively.

A representative example of the speed, acceleration, and jerk data of a participant performing the rehabilitation exercise is shown in Figure 8. The original data are displayed on the left, whereas the corresponding filtered data are represented on the right. A first-order Butterworth low-pass filter with a cut-off frequency of 6 Hz was applied to eliminate the noise from the environment and the small body movements. The first-order filter was selected to avoid over-smoothing the acquired motion [35].

A clinical evaluation of the values obtained is not significant in our group of healthy subjects. In any case, the values obtained are consistent with the literature and what is shown in Figure 9. All average velocity values follow a normal distribution, confirming that all the subjects in the pool are healthy.

A preliminary analysis of the collected data on two subjects showed that our serious game can discriminate between users who execute the trajectory correctly (Subject 1) and users who do not (Subject 2). Figure 10 illustrates the trajectories of two participants performing the double arc exercise.

4. Conclusions

Research studies [3,4,5,6] show that serious games, specifically exergames, can increase patient motivation regardless of age or illness and be an inexpensive and promising solution with respect to unengaging conventional physiotherapy.

This paper presents a non-wearable projected AR serious game for shoulder rehabilitation, an evolution of our previous prototypes [25,26,27]. This new version integrates advanced metrics to assess patient performance, monitors the game progress, and allows the selection of the game visualization mode: standard on-screen or projected augmented reality.

A user study with eight healthy subjects and eight physiotherapists was conducted to investigate the effects of our game on players’ motivation and enjoyment, test the game’s ergonomics, and verify the game’s usefulness, portability, and suitability as a shoulder rehabilitation tool.

The results show that our rehab serious game is attractive, ergonomic, clinically useful, and does not require the therapist to be always present during the entire rehabilitation session. Specifically, physical therapists agree that our serious game can speed up recovery time and provide benefits over traditional methods, as well as the increased ease of customizing treatment. The findings demonstrate the opportunities offered by our serious game in the upper limb rehabilitation process and confirm the opportunity for clinical studies.

In comparing this version of the system to its previous version (screen-based system [27]), the kinematics of the upper limb movement to control the game is the same, but the level of cognitive demand is lower (as the user’s eye and hand movements are coupled and do not require the user to perform a visuospatial transformation). This aspect (i.e., the cognitive demand) is of utmost importance in designing computer-based therapies. In this context, Mousavi Hondori et al. [28] demonstrated how the choice of human–computer interface influences the demands of rehabilitation therapy and can be individualized to patients according to the patient’s needs. For example, the screen-based version can be useful to promote cognitive recovery after a stroke [28].

However, clinical trials will be conducted to demonstrate the efficacy of our game as a complementary tool for upper limb rehabilitation. A clinical validation will be performed to (1) verify that the interaction mechanism safely facilitates therapy; (2) determine the effects of physiotherapy treatment; and (3) compare them with the effects of standard therapeutic methods. In this way, we will understand if our serious game is appropriate for integration in shoulder rehabilitation therapies with real patients.

Overall, our system is highly adaptable and flexible to the patient’s needs. For example, in addition to changing the tilt angle of the table and the friction characteristics between the user’s hand and the desk pad, it is possible to integrate a hand-held device for patients with hand spasticity. From a technical point of view, the next version of our serious game will include a few new features, such as: (1) adding to the horizontal ROM (angular excursion in the transverse plane) the vertical ROM (angular excursion in the sagittal plane) to facilitate the complete recovery of ROM; (2) simplify the system installation by replacing the two arms holding the projector and LMC, respectively, with a single arm able to hold both devices.

Future works will also include the definition of performance metrics by analyzing the data collected and the creation of the clinical trial protocol. The performance metrics will be designed to evaluate the progress of a patient following a rehabilitation program. This is the preliminary step toward laying the foundation for future studies that investigate abnormality in shoulder movement via the use of a hand-tracking system.