Most studies are quite specialised and tend to cover the same groups of largely elderly patients (e.g., stroke and Parkinson’s), which do not constitute a credible target group per se for gaming among the population. In addition, the impression is that the same functionalities are being tested repeatedly, without any evolution. Above all, other groups like children and adolescents with chronic diseases are rarely addressed, even though they are an excellent target group and would probably benefit greatly from using exergames as they need to move like any other child but are mostly limited to performing their exercises with a physiotherapist. This is generally boring, time-consuming and prevents them from playing with friends during this time. If instead they could play games involving physical exercises, without it feeling like rehabilitation, due to proper immersion and motivation, they would possibly need fewer sessions with the therapist, which may in turn improve their social life. Commercially available games would be good enough for many children with physical disabilities, if only they were configurable and adaptive to their potential and needs. Remote controls (RC) are typically not sufficiently configurable (button functions cannot be changed or the RC cannot be used with one hand) and are only made for hands (why not for feet or the mouth?) Some RCs are not sufficiently precise in detection, and so the user ends up tired and loses motivation. Motion capture devices like the Kinect sensor seem to provide better prerequisites for exergaming purposes but feature important limitations too, (e.g., detection of fine movements and rotations) such that the needs of many people are still not be covered by commercial solutions.
To fill these gaps, the authors of the work presented here are pursuing the overall aim (as part of a long-term project) of creating an entertaining exergaming environment for adventure games that immerses the players into a virtual world and makes them forget their physical impairments. Knowledge of the gaming industry is applied to create motivating challenges that the users have to solve, which are sufficiently addictive to make the exercises pass to an unconscious plane. The gaming environment is configurable to the user’s potential and requirements. Challenges will be programmable by a therapist and will also adapt themselves to the players automatically real-time, by observing their fatigue or emotional state (lowering the difficulty or switching to more relaxing exercises when needed).
The gaming environment is built in Blender and combined with a middleware implemented in C# that passes motion information from the Kinect sensor to the game. The middleware is modular and incorporates support for an Android device and a head mounted display (HMD), but in this work, we focus only on the motion capture with the Kinect.
1.1. Related Work
Given that this work is based on the usage of the Kinect motion capture device, related work on systems that apply other devices is not reviewed. The reason is to present an overview of what has been achieved using this sensor in particular for rehabilitative games.
After the Kinect camera was brought onto the market in 2010 and Microsoft released the Software Development Kit (SDK) in 2011, an exponential boom in development took place, with serious contributions in many areas. The vast majority of research belonged to the e-health sector as the Kinect sensor opened new ways to capture human motion in a non-intrusive way. A peak of contributions appeared in 2013, but since then, less has been published each year. After assessing the literature it appears that despite trying out many options, a point of saturation has been reached, without achieving any generally applicable or commercial usage of the Kinect for rehabilitation purposes.
Many usability studies concerning the accuracy of human motion analysis, feasibility and safety prove that the sensor is generally useful for exercises at home, but has certain limitations detecting fine movements and rotations of the hand, foot, wrist, ankle, shoulder and head, which impedes its ad hoc applicability to any kind of exercises needed and hinders precise supervision [1
]. The diseases studied have been mostly focused on similar diseases (stroke, Parkinson’s, cerebral palsy (CP), multiple sclerosis (MS)), with stroke patients being the most common (28% [5
]). Furthermore, the inclusion criteria generally excluded severely disabled patients, as their movements present detection difficulties [2
]. Wheelchairs usually present a special detection problem due to metallic reflections [6
], although the authors of publications for wheelchair exercises surprisingly fail to highlight this issue [7
]. Gaming environments are not evolving for real usage in health applications, as they…
…are limited to non-occluding movements
] which restrict applications to a frontal view perspective. Other individuals can interfere negatively by walking into the camera’s view.
…lack exactness and may be prone to undesired compensating movements
often used to “cheat” the system [2
], although [6
] is convinced that here the Kinect has advantages over hand-held remote controls, as it can measure movements more exactly.
…may have few positive effects when the system’s performance is poor
, resulting in demotivation and frustration when the environment does not respond correctly to movements, lacks feedback, or is boring [2
…generally lack interesting design and motivating stories
. The authors of [7
] published extensive results of their survey of volunteers. Some of them mentioned monotony due to repetitive exercises, the absence of variety and challenges, as well as some frustration when the game did not react as expected.
…are mostly addressed to user groups of elderly patients
, which are the main affected group by stroke and Parkinson’s [2
], to the detriment of children and adolescents [4
], who form a group likely to be more easily responsive to gaming.
, e.g., in [7
], the surveyed participants stated that the games were too difficult when they suffered greater impairments, and found them boring when they had high motor functionality. In [37
], screens had to be adapted to the range the users could reach with their arms due to muscle weaknesses. Appropriate players are those with a wide range of possible arm movements (spreading), as resting arms near the body impede any true detection of the person [6
…lack configurability and well-designed monitoring systems
. The authors of [7
] point out especially that it is very important to implement monitoring systems and the possibility to vary and graduate the intensity and duration of practice, in particular, in cases of very repetitive activities and when individuals ignore pain symptoms due to high levels of engagement and motivation.
The newest approaches found are also still tentative, preliminary and drive small sized test series [7
]—only commercial applications have been tested exhaustively [1
]. Design guidelines are appearing but are still very limited [6
]. In [6
], the authors present a guide to applications for clinicians, where controller-based and motion-capture-based systems, as well as commercial games, are compared. The general outcome is that they cannot be played by everybody, due to the required wide arm movements, standing position etc. Publications with recommendations for wheelchairs are missing and no work has been found that applies gamification guidelines, which are well known in game designer circles, such as the Octalysis approach [40
], to the design of exergames.
Nevertheless, the evaluation of serious games for physical exercises leads to generally positive results. Compared to EyeToy or Wii, the Kinect seems to be the most natural device and offers great promise for creating enjoyable exercises with a low budget [2
]. It also has the potential to track specific movements that should be exploited by incorporating complex and adaptive exercises [13
]. Here, fine-tuning, user adaptability, monitoring (to prevent over-exertion) and personalized feedback on how movements should be performed, are highly needed elements [2
Hereinafter, some applications found in literature are analyzed regarding the deficiencies mentioned above.
] is especially addressed to wheelchair
users. The gaming environment is similar to “JeWheels” [9
]: two hand icons have to be moved to grab objects for recycling. To improve balance training and motor coordination, five upper limb movements are captured: weight transfer, rotation, and moving sideways, back and forth. No user tests were performed and no photos of wheelchair users are shown in the publication. As no detection problems inherent to wheelchairs are mentioned, our impression is that this system has not been fully investigated.
, the European Rehabilitative Way out In Responsive home Environments (REWIRE) project [21
] stands out because of its special focus on intelligent systems. Particular publications of the consortium members can be found about “Adaptive Games for Rehabilitation at Home” [22
] or “An Intelligent Game Engine for the At-Home Rehabilitation of Stroke Patients” [23
]. The authors state the importance of combining effective exercises with compelling games and implemented, similar to us, a set of mini-games that are included using a common theme, in this case, farming. Nevertheless, no results can be found about user tests or the final state achieved in the project, which finished in 2014. In [26
], another user-adaptive game is presented that also wraps the exercises in a farming environment: the user is placed on a moving tractor and has to catch apples from the trees that are passing by. The game was tested by patients affected by mild to moderate Parkinson’s disease. It adapts to the player during the play by slowing down and speeding up according to the player’s success.
To enhance the playing experience
and motivation, various high quality graphic implementations can be found, mostly using Unity 3D [15
], in one case Blender [12
]. “Kinect-o-Therapy” [15
] is a complex environment consisting of four mini-games, which are gamified versions of the generally prescribed exercises for arms and body. According to the newspaper article found in [24
], this system seems to be available on the market, but no updated web page has been found to discover the end results. In the “Apple catcher” [25
], the authors present a game, especially designed for patients with hemiplegia as a possible consequence of stroke. The motivation for the player is to reach a score in a limited time, but scenes are neither changing nor have a story. The work in [12
] presents a simple game in a medieval setting, aimed at elderly stroke affected users. This is the only contribution found that mentions the use of Blender, specifically for the modelling of the environment. In our opinion, the motivation (capture coins doing the right movements) is not strong enough, as it has nothing to do with the created environment and no story is involved. “ReaKing” [27
] (Rehabilitation using Kinect-based Games) is the only system found that uses the Kinect V2. It comprises two different environments with adventure-like scenes, but also lacks a story. Here, the user, represented by a character in third-person view, is walking through different landscapes where, from time to time, objects appear which have to be dodged, while the user performs a walking exercise (on the spot). The motivation is driven by the collection of coins. The “Upper Extremity Rehabilitation Gardening (UERG) Game” presented by [36
] is interesting due to the story and user feedback. It has a charming environment to practice every-day movements. User feedback is included nicely into the story, as seeds change colours according to the correctness of movements.
Few systems have been found that are designed for the physical rehabilitation of disabled children
, and here, most are addressed to CP. The search for publications about gaming systems used for CP lead to many studies that test the functionality and efficiency of using games in general. For instance, [38
] presents a preliminary study of the usefulness of commercially available Kinect games. Although the outcomes are positive, the authors state that future studies require the development of specific
videogames targeted at the treatment of motor symptoms in children with CP. The same conclusion can be drawn from the review presented in [41
] concerning the use of commercial video games in rehabilitation: from a total amount of 4240 studies, only 8% were about CP, 25% about weight, 22% about balance and the remaining 45% about stroke, Parkinson’s and aging. The 8% CP studies include [38
] and two others that do not present applied games. Only one proposal has been found that directly targets game design: [17
] presents an authoring tool based on event recognition, which is applicable to the Kinect and other devices and includes different mini-games, but does not rely directly on the motion capture data. Furthermore, the authors of [36
] propose a system to assist patients with spastic diplegia and hemiparesis in their rehabilitation process. Unfortunately, it is actually not a game, as the user is only required to mimic the movements of an avatar. However, the conclusions are positive, stating that impaired children with disorders equivalent to those of the tests’ participants could benefit from the Kinect’s capacities.
The newest proposal found in [28
] at the time of writing (January 2017), does not present anything new compared with the formerly presented systems. Here, three exergames evaluate different psychophysical rehabilitation exercises of neurological patients, by means of an avatar on the screen that follows the movements of the patient. A quantitative analysis of the movements is carried out by saving the joint positions of the skeleton in a database, which allows a follow-up of the evolution of the therapy. Here, no real gaming environment is presented; the user is just required to avoid some virtual balls or to step on coloured rectangles. The exercises can be configured by the therapist by changing e.g., the angles for the required arm rising. No seated mode for wheelchair users is supported and only three patients were tested.
1.2. Motivation for this Work
Our impression when analysing the state of the art to-date is that development has stagnated. No developments have been found beyond the simple measurement of joint displacements. All approaches, regardless of whether they are implementing simple or more sophisticated gaming environments, are just analysing movements by reading skeleton positions and movements. By doing this, the only basis for exercise design is the mirroring of the patient’s movements and their visualisation by an avatar (a character or a limb shown on the screen). Here, accuracy and speed could be obtained as metrics. The only motivations for the user are to copy the movements of a demonstrating animated character or to move an avatar to reach objects like coins, apples, or kitchen devices. The purpose is always obvious and users notice quite consciously that they are required to do a certain movement. This possibly hurts or at least is very tiring, as it is aimed at an affected limb. In our opinion, this effect itself probably lowers motivation considerably and should be avoided by varying the type of exercise from time to time, inserting relaxing moments and presenting challenges that make the users forget their pain.
Everything should be included in the same gaming environment and make sense, i.e., if one has to reach for objects, this should be for a reason given in the game. Therefore, a realistic storyline is needed and an environment that provides the transition between different scenes. To the best of our knowledge, this kind of environment has not yet been presented, and our purpose is to create a meaningful gaming environment that addresses all mentioned deficiencies. It should be generally accessible by a wide range of users, adaptable to their potential and needs, and above all, during the gameplay, configurable by the therapist, be funny and motivating such that the patient is not aware of doing exercises. To achieve all these objectives, this very ambitious project will need a long time and the involvement of many experts from different areas such as e-health, vision, telematics, machine learning, etc. Therefore, we are still looking for partners and invite interested readers to contact us.
In the initial study presented here, we report the first set of results about a possible method to achieve this, where users with physical limitations can have the same experiences as others inside a virtual environment, which is an indispensable basis to provide full immersion. Section 2
explains the proposed environment and new approaches to handle game motion control. In Section 3
, the results of our feasibility tests are presented. Section 4
closes the paper with some conclusions and an outline of our planned future work.