Digital Technologies for Innovative Mental Health Rehabilitation

Abstract

Schizophrenia is a chronic mental illness, characterized by the loss of the notion of reality, failing to distinguish it from the imaginary. It affects the patient in life’s major areas, such as work, interpersonal relationships, or self-care, and the usual treatment is performed with the help of anti-psychotic medication, which targets primarily the hallucinations, delirium, etc. Other symptoms, such as the decreased emotional expression or avolition, require a multidisciplinary approach, including psychopharmacology, cognitive training, and many forms of therapy. In this context, this paper addresses the use of digital technologies to design and develop innovative rehabilitation techniques, particularly focusing on mental health rehabilitation, and contributing for the promotion of well-being and health from a holistic perspective. In this context, serious games and virtual reality allows for creation of immersive environments that contribute to a more effective and lasting recovery, with improvements in terms of quality of life. The use of machine learning techniques will allow the real-time analysis of the data collected during the execution of the rehabilitation procedures, as well as enable their dynamic and automatic adaptation according to the profile and performance of the patients, by increasing or reducing the exercises’ difficulty. It relies on the acquisition of biometric and physiological signals, such as voice, heart rate, and game performance, to estimate the stress level, thus adapting the difficulty of the experience to the skills of the patient. The system described in this paper is currently in development, in collaboration with a health unit, and is an engineering effort that combines hardware and software to develop a rehabilitation tool for schizophrenic patients. A clinical trial is also planned for assessing the effectiveness of the system among negative symptoms in schizophrenia patients.

1. Introduction

Schizophrenia is a psychiatric disorder that, according to the World Health Organization (WHO), affects about 20 million of the world’s population, which makes it the most common psychotic disorder [1]. Of multifactorial etiology, it is characterized by “loss of contact with reality”, implying alterations at the level of thought, perception, affect, and behavior. It is a complex and heterogeneous clinical syndrome, associated with a constellation of signs and symptoms with repercussions on the patient’s occupational and social functioning.

Cognitive and social rehabilitation is performed by repetition and simulation, in sessions supervised by rehabilitation technicians or medical staff. This implies that the patient moves frequently to the rehabilitation facilities and adjusts the schedule accordingly. Technology emerges as one of the possible solutions to assist in the rehabilitation of patients. As a complement, it can promote solutions that improve accessibility to health care. Moreover, it allows crossing geographical barriers, schedule and time inconvenience, or lack of human resources. Potentially, they also provide economically viable solutions, which allow for measurability (data collection and cross-referencing) and scalability (through easier perception of ideas that work, allowing for their replication).

Rehabilitation is comprised of interventions required when someone faces limitations in the performance of their daily activities, due to aging or acute/chronic conditions, which may be manifested by difficulty in thinking, seeing, hearing, communicating, moving, interacting, or maintaining employment. Rehabilitation supported by digital technologies emerges as a solution to support health professionals by providing high-intensity, repetitive, task-specific exercises to enhance the rehabilitation process. One of the problems is the massive use of this technology, often developed in controlled laboratory environments.

Virtual Reality (VR) technology has been shown to be acceptable, safe, and engaging for both physical and psychological rehabilitation [2]. This technology allows for multiple possible therapeutic solutions, such as: reproduction of daily life, adaptation techniques for adjustment in daily routines, training regimens to improvement of cognitive and social skills, reproduction of classical cognitive-behavioral therapy techniques, and exposure-based therapy, among others.

The popularity of the VR technology for rehabilitation has been growing, with confirmed benefits. The rehabilitation of gait disorders in patients with multiple sclerosis and stroke is often based on conventional treadmill training. VR-based treadmill training can increase motivation and improve therapy outcomes [3]. The same is confirmed for upper limb motor recovery, when combined with arm cycling [4]. Many more examples can be found, focusing on older adults and physical training, functional rehabilitation, etc. [5].

Mental health rehabilitation is a dynamic evidence-based model that provides comprehensive and continuous care plans focused on the person with severe and persistent psychiatric illnesses, closely linked to community integration and increased quality of life. In recent years, several studies have started to apply VR tools to try and contrast patients’ cognitive decay over time, particularly in neurodegenerative diseases, such as Alzheimer’s disease, frontotemporal dementia, Parkinson’s disease, and multiple sclerosis [6]. Age is an important factor for neurorehabilitation, in both normal and pathological aging. Deficits in spatial cognition and decline in body-related information is observed in aging, contributing to impairments in navigation, memory, and space perception [7,8].

Because of the importance motivation has in the rehabilitation process, it is also important to consider tools and approaches that foster motivation and involvement. Gamification and Game-Based Learning (GBL) are known for their potential in increasing the motivation and provide rich learning experiences [9,10,11]. The long-term engagement with therapeutic interventions is the norm, and researchers have been incorporating best-practices in game design to achieve engaging experiences [12,13].

Gamification is a broad concept, that hosts many uses and applications. It departs from serious games, being applied in the use of game design elements in non-game contexts [14]. On the other hand, serious games is a term that is usually used to indicate that fun and its positive effects on motivation are not the main motives for using games, aiming at training and learning, instead [15].

Thus, the current potential of VR environments allows the design and development of simulations with the aim of assessing and rehabilitating patients’ performance, while game design elements enable the introduction, in therapy sessions, of systematic, motivating, and engaging forms of training and testing, which enable tight control of complex, dynamic, and immersive representations, of sophisticated interaction, and with the possibility of recording patient performance.

Having this in mind, this paper addresses the use of digital technologies, namely VR, GBL, Internet of Things (IoT), Artificial Intelligence (AI), and Advanced Data Analytics (ADA), in the development of innovative rehabilitation techniques, contributing to the promotion of well-being and health in a holistic perspective, focusing, mainly on mental health rehabilitation in people diagnosed on the schizophrenia spectrum.

The paper is organized as follows: Section 2 presents the related work, and Section 3 discusses the functional and non-functional requirements for the immersive and intelligent rehabilitation system. Section 4 describes the system architecture, and Section 5 overviews the study design where the rehabilitation system based on VR and serious games will be deployed. Finally, Section 6 rounds up the paper with the conclusions, particularly analyzing the challenges associated to the implementation of such system architecture, and points out the future work.

2. Related Work

Schizophrenia corresponds to a group of disorders with heterogeneous etiologies and includes people with variable clinical presentations, response to treatment, and course of illness [16].

The Diagnostic and Statistical Manual (DSM) of mental disorders [17] suggests that its diagnosis depend on at least two or more of the following symptoms, each for a significant portion over a period of 1 month (or less if successfully treated). At least 1 of these should be (1), (2), or (3):

(1)

Delusions.

(2)

Hallucinations.

(3)

Disorganized speech.

(4)

Grossly disorganized or catatonic behavior.

(5)

Negative symptoms (e.g., decreased emotional expression or avolition).

The DSM suggests that, from the onset of the disorder and for a significant period of time, the level of functioning in one or more major areas, such as work, interpersonal relationships, or self-care, is markedly below the level previously achieved.

The symptoms can be structured in three different groups: (a) positive symptoms, including delusions, hallucinations, formal thought disturbances, and disorganized behavior; (b) negative symptoms, with alogia, isolation and social impoverishment, apathy and abulia, affective flattening, and psychomotor slowing; and (c) cognitive symptoms, described by altered attention, memory loses, reducing intellectual quotient, language difficulties, and degradation of executive functions [18].

This disease is chronic and of continuous evolution. Although there is a premorbid and prodromal phase with some indicative signs, the diagnosis of this disease can only be made after the first psychotic episode. The initial stages are characterized by repeated exacerbation of positive symptoms and remissions. Simultaneously, the disease progresses with continuously worsening cognitive and negative symptoms, which are primarily responsible for the decrease in functional capacity, loss of autonomy, with impairment in daily activities, social relationships, and emotional processing. These symptoms translate into a loss of quality of life, affecting long-term outcomes [19].

Negative symptoms and neurocognitive dysfunction affect about 60% of sufferers, respectively [20]. Although these are considered separate domains of psychopathology in schizophrenia, shared features suggest that symptoms in one domain may reinforce the other.

With the evolution of psychopharmacology since the late 1950s, anti-psychotic medication is the mainstay of treatment for the schizophrenic patients. This form of treatment acts primarily on the positive symptoms of the illness, with a marked reduction in these symptoms and a reduced risk of relapse. However, most of the available medications have limited effects on the negative and cognitive symptoms of schizophrenia, that are not secondary to the positive symptoms, and, so far, none have been approved for the treatment of these symptoms.

Addressing the negative and cognitive symptoms can be difficult, requiring a multidisciplinary approach that includes psychopharmacology, individual psychotherapy, and psycho-social approaches, such as social and life skills training, family therapy, vocational therapy, art therapy, occupational therapy, cognitive training, etc. [21]. These approaches aim to promote social and interpersonal functioning, promote independent living in the community, lessen the severity of symptoms and associated co-morbidity, improve cognition, and improve personal coping with the disease. However, access to these therapeutic modalities is not granted to all people. The place of residence, the different resources, the discrepancy in access, the characteristics of the disease, and the number and training of human resources in the hospital are examples of elements that influence access to treatment and, therefore, the course of the disease.

Rehabilitation is comprised of interventions required when a person faces limitations in performing daily activities due to aging or due to acute or chronic pathologies, which may be manifested by difficulty in thinking, seeing, hearing, communicating, moving, interacting, or maintaining employment. Rehabilitation is an essential component of universal health coverage [22].

Mental health rehabilitation is a dynamic evidence-based model that provides comprehensive and continuous care plans focused on the person with severe and persistent psychiatric illnesses, closely linked to community integration and increased quality of life [23]. Social skill training is, in fact, an effective intervention to improve the patient’s function and quality of life [24].

Currently, VR systems are increasingly used as a powerful auxiliary tool in rehabilitation. They allow an immersive audiovisual environment with maximum clinical relevance and robust ecological validity unattainable through traditional digital simulations. They appear to improve rehabilitation interventions, leading to improved motor function and bio-mechanics or diagnostic ability superior to traditional methods in patients with suspected cognitive impairment [25,26]. The rehabilitation process, regardless of the health area it refers to, remains a challenge for professionals, patients and their families. In an attempt to overcome the limitations of traditional interventions, VR technology has been increasingly applied to rehabilitation and is beginning to provide important tools that, however, generate debate and divergent positions. The possibility of its application to a diversity of domains, cognitive functions, behaviors, neurological diseases, and physical disabilities are advantages of VR [27].

Rehabilitation supported by digital technologies emerges as a solution to support health professionals by offering high-intensity, repetitive, and task-specific exercises to enhance the rehabilitation process [28]. However, these approaches are often developed and used in controlled laboratory environments. The availability and implementation of this technology outside of laboratories presents substantial clinical challenges [29].

VR and Non-Invasive Brain Stimulation (NIBS) are innovative technologies that have potential in this area [30]. Current social cognition-related research suggests VR has potential to facilitate more direct translation of improvements from VR training to functional outcomes by immersing individuals in virtual environments that can feel like real-world encounters [31]. Compared to traditional cognitive social therapies, which lasted an average of six months to show results, the use of VR in these interventions proved to be effective, since it only took five weeks to demonstrate positive effects.

Some experiences and prototypes have been in discussion within the scientific community. GameChange was developed with the aim of helping to combat mental illness through the use of automated VR cognitive therapy, anxiously avoiding everyday social situations. With six events, in six everyday scenarios: a street, a bus, a café, a bar, a doctor’s waiting room, and a shop [32]. Possessing five levels of difficulty, each level provides an opportunity to test fear cognitions while limiting the use of safety-seeking behavior, allowing patients to gain confidence in coping skills. In each scenario, patients must perform important tasks, such as ordering a drink, finding items in the store, or calling someone across the room.

Freeman et al.’s treatment study demonstrated that it is not simply exposure to a problematic situation that is the key to clinical change but actively testing fears by abandoning defensive behaviors [33].

Oliver and Stéphane used immersive VR, as the growing evidence suggests that immersion in a VR system increases both the sense of presence and the emotional excitement. The results propose that VR is a highly promising intervention for refractory auditory verbal hallucinations in schizophrenia, reduction in depressive symptomatology, and enabling patients to gain a better quality of life [34].

The use of games to aid in the rehabilitation of patients with schizophrenia has had some expression in recent years, stimulating physical activity, social integration, attention, and memory. Specific exercise-based games (exergames) have been found to contribute to improved psychomotor coordination and increased walking speed [35]. Additionally, the feedback presented by games on consoles, such as the Nintendo Wii, allows patients to improve their sense of body and movement. The ability to play more than one person also contributes to improved social interaction [36].

Additionally, VR allows simulating environments that patients can explore and experience actions that have a significant degree of difficulty in their lives, such as shopping, memorizing the way to the supermarket, or being in a certain place at a certain time [37]. In patients with positive symptomatology, the exercise of playing on the Internet contributes to a decrease in symptoms, such as hallucinations, among others [38].

The effectiveness of games, both in terms of development and motivation, strongly depends on the quality and difficulty of the challenges within. The individuality of the player requires that the obstacles and experiences are continuously adapted to face his skills. Only through adequate difficulty challenges can the player learn and be motivated to continue learning [39]. GBL helps the player to learn in different and attractive scenarios, based on the mechanics of overcoming challenges and through positive and interesting learning experiences. It also introduces the possibility for individualization and the flexibility of e-learning.

Nevertheless, dynamic adaptation of the game to face the player’s characteristics requires a multidisciplinary approach, that includes measuring the player’s behavior signals and his biometric signals to recognize the player’s emotional states and stress levels, in order to adapt specific game features in a dynamic manner and to improve player’s engagement, immersion, excitement, and challenge [40].

3. System Requirements

As previously mentioned, the proposed system is designed to support the psycho-social rehabilitation of patients diagnosed with the schizophrenia spectrum, namely targeting the negative and cognitive symptomatology. It integrates functionality that allows for greater flexibility and comprehensiveness of the rehabilitation experience, as well as consists of several hardware and software components, which support data acquisition, processing, storage, and visualization.

In terms of requirements, the system is independent and completely autonomous from other rehabilitation or hospital systems. It needs a physical place to host the system that safeguards the user experience and safety. The caregiver will need a workstation, where he can monitor the dashboard with real time information about the rehabilitation session. In this context, the caregiver is a health professional, that will support the rehabilitation sessions. The patient will be immersed in a VR environment, so it is important to ensure a 4 m × 4 m area, free of objects and obstacles, for the patient to stand. Besides the computer system, the visualization and interaction with the system by the patient is done through a set of VR Head Mounted Display (HMD) and associated controllers. It is through these that the player receive directions from the game and acts on the elements and objects represented therein. The communication between the HMD, the sensors, and the computer should be wireless, so as not to limit the patient’s movements. This should also allow obtaining the patient’s displacement in the room, in order to measure the translation and rotation movements to be replicated in the virtual environment. The availability of the physical area may incur in costs, depending on the facilities and infrastructure of the rehabilitation institution.

The interaction with the system is performed through a serious game in VR that, simulating spaces and tasks of daily life activities, proposes problem-solving as the basis of psycho-social rehabilitation, with the aim of recovering autonomy and gaining quality of life. Therefore, the environment is anchored in three-dimensional spaces and objects with realistic proportions and color palette, and obeying the laws of physics, such as inertia, gravity, or light reflection. The area is explored through a 1st person camera perspective.

Besides the direct interaction with the game through the HMD and the controllers, biometric and physiological data is also read during the session (Figure 1). This data includes the Heart Rate (HR) and Heart Rate Variation (HRV), read by a wristband or similar device. This device must allow the data to be read in real time and sent to the system via a wireless technology, such as Bluetooth. Additionally, the body posture and facial expression is also gathered through image processing and machine learning. Software probes gather game interaction statistics (through game instrumentation). Finally, the patient’s voice features (intensity, frequency band, pitch, formant frequencies) and their variations are acquired by a microphone. The signal is transcribed, and, with assistance of language models and deep learning models, the phrases sentiment is analyzed.

Figure 1. Biometric and physiological data sources for dynamic game adaptation.

The caregiver will have access to a web-based application, similar to a dashboard, that will present various real-time data in visual format, that will help him manage the rehabilitation session. The data includes information related to the stress level, performance, and progress in the session, gathered by hardware and software probes. Finally, the medical team will have both read and write access to all the information, from the medical record and data obtained during rehabilitation sessions, as well as the ability to add recommendations, notes, and observations regarding each patient. The caregivers and medical staff use a regular computer, with keyboard, mouse, and monitor, to run web-based applications.

4. System Architecture

Similar to the system presented by Amado et al., the basis is a simulation environment in VR [37]. It is intended to improve several aspects, namely the possibility to adapt, in real time, the challenges and the degree of difficulty that the patient encounters in the simulated environment. It is also intended to provide a platform for monitoring the rehabilitation session, through the representation in a dashboard of various values of performance and physiological condition of the patient. Finally, it is intended to store all the data that, after processing, will be used by the medical team to evaluate the evolution of the rehabilitation process.

The proposed system is currently in development and assembly in two locations: in a research lab, at the university, and in a rehabilitation room, in a hospital. Although the hardware is of-the-shelf and widely available in the market, software is specific and in development. It is composed of several modules and technologies, which include 3D assets design, VR game design and development, REST API in the backend, and web frontends development. Software development follows an Agile methodology and the source code, as well as tickets, requirements, issues and versions are kept in a Git version control system. Overall, the system is an engineering effort that combines hardware and software to develop a rehabilitation tool for schizophrenic patients. Initially, its use will be restricted to the hospital, in supervised rehabilitation sessions. These will allow the organization of clinical trials, that seek to understand its effectiveness.

In this context, there are several challenges that require research and development, namely the design of the game environment, which has to be safe and suitable for the patient, collection and processing of biometric and physiological data, such as HR, HRV, voice, facial expressions and body posture, data analysis and processing, and dynamic adaptation of the environment, based on AI.

The proposed system architecture comprises 3 main modules, namely the data integration and processing for patient profiling and real-time adaptation of rehabilitation procedures, game design and development for mental health rehabilitation, and data visualization (Figure 2).

Figure 2. System architecture.

These modules are developed and articulated based on the requirements and specification documentation. The system will be validated and tested in clinical trials, during the development and optimization and, finally, with the conclusion of the development process.

Briefly, each of the modules is designed and developed in parallel, starting at the same time. Each will focus on complementary aspects, that will converge into the virtual environment for mental health rehabilitation. The virtual environment will be based on a digital game, with the mechanics, rules, graphics, constraints, and narrative developed in articulation and with the collaboration of the medical team. During the rehabilitation sessions, it is expected that the patient vocalizes some reactions, and, as such, the voice signal will be captured and analyzed, to assess the evolution and the changes during the session, that will be used also as feedback for the virtual environment. The patient profiling and real-time adaptation of the rehabilitation procedures is responsible for collecting other signals, such as HR, HRV, facial expressions, and body posture, and, through reinforcement learning, to automatically adapt the game challenges to the ability and skills of the patient.

Considering the related work and the experiences described by the scientific community, we believe that the system is useful and has the potential to help both health personal and schizophrenia patients. The effectiveness will be assessed in a clinical trial, described in Section 5.

4.1. Data Integration and Processing

The dynamic and real-time adaptation of the rehabilitation procedures to the individuality of each patient requires gathering information from multiple sources (see Figure 3).

Figure 3. Sensor integration.

Each sensor is responsible for collecting, processing, and analyzing different features. Game instrumentation will output the timestamped log of actions the patient is doing in the game. Body posture will assess, through image processing, the behavior of the patient. It is known that body posture communicate social cues, relevant for successful social interactions [41]. Several approaches, based on AI, can be used for assessing the body posture, such as References [42,43].

Facial expression is also a good vehicle of emotional signals [44,45]. Since part of the face is covered with the VR goggles, the inference of emotion has to be made through mouth and lips shape [46,47].

Other parameters, such as HR, HRV, and breathing rate, are associated with the stress level of the patient [48]. Monitoring these parameters allows for indirectly identifying the challenge the patient is facing and his ability to overcome it. If the stress level is growing, the game should take actions to calm down the patient by reducing the difficulty or allowing him to go to a resting place in the simulation.

Speech contains relevant information about human thought, including semantic and emotional content, semantic coherence, and syntactic structure and complexity [49]. It is expected that speech features, such as prosodic voice parameters, semantic and emotional content and coherence, sentence length, number of words or syllables, and number and duration of pauses, are involved and will change with time [50]. For example, exploring nonverbal speech parameters (pauses in speech, response time, among others) allowed identification of schizophrenic voices with an accuracy of 81.3% [51], between 80–93% (repeated study after 12 weeks) [52] and 87.51% [53]. Reading an excerpt from a children’s story allowed to identify schizophrenic voices with an accuracy of 79.4% in a set of 57 subjects [54]. In this study, they analyzed parameters, such as number of pauses, average duration of pauses, and amount of silence, among others.

The VR goggles embedded microphone will be used to capture the speech signals generated during the rehabilitation session and the features will be extracted and recorded for continuous analysis and storage. The features include the transcription of the speech signal, frequency, pitch, pauses, cadence, etc.

Finally, according to the medical staff, it is also important to get the body temperature before and after the simulation take place.

All of these data are time sequence samples. Each sample is sent to a centralized module through a set of message queues, that will store the information and notify other modules through a distributed events service, of the Publish/Subscribe type (Pub/Sub). The information manager also exports a gRPC-based Application Programming Interface (API), to be able to be queried by other modules.

The game is conditioned through a dynamic adapter, powered by a reinforcement learning algorithm, which learns by interacting with the environment with no explicit training process, towards maximizing a numerical reward signal. This process works in closed-loop problems, such as this, since the learning system’s actions influence its later inputs. The algorithm discovers which actions yield the most reward by trying them out, pursuing a best case scenario.

The inputs to the dynamic adapter are the data gathered from the game instrumentation (the patient’s actions in the game), the body posture, lips expression, HR, HRV, breathing pattern, etc. As the patient interacts with the game, these signals evolve, contributing to the adjustment of the game parameters and difficulty.

4.2. Game Design and Development

The game design is responsible for specifying the narrative, experiences and sequences of tasks in the rehabilitation experiences. With the collaboration of the medical team, the game design follows an interactive approach for designing daily activities, that will be presented to the patient. The game starts with the selection and customization of the initial domestic environment. In the first session, the system will generate a home environment, according to the profile of the patient. The home can be traditional or modern, with 2 to 5 rooms. The challenges will be designed following a level design approach, including the assets, sequences, and obstacles. Each sequence reproduces daily life activities, such as waking up, brushing the teeth, and preparing breakfast, among others. Within each challenge, the difficulty of the obstacles will be adapted in a smooth and discrete way, for example, having the toothpaste run out, which forces the patient to look for a replacement in a drawer, for example.

The patient will advance through the experiences and, according to his ability, physiological, and emotional conditions, the experiences may transition to more complex environments, such as the street, public transports, or the supermarket.

Within the design process, it is also fundamental not to provide situations that could exacerbate positive symptoms, such as hallucinations. So, voice-over indication, force-feedback (that could be mistaken by “phantom touches”), and floating signs (visual hallucinations) are strictly forbidden. All signs and indications that are not a natural part of the narrative and that are not expected by the patient cannot be used. This is a fundamental step, requiring constant validation and participation by the medical staff.

The game design provides the base for the implementation of the virtual environment simulations, challenging the patient with experiences close to the real world, and allowing an intensive repetition of essential tasks during mental-health rehabilitation process, with real-time feedback in a controlled and safe environment. VR is increasingly used as a powerful auxiliary tool in rehabilitation. It allows an immersive audiovisual environment with high clinical relevance and robust validity, modernizing rehabilitation interventions, leading to improve motor function and bio-mechanics or better diagnostic capacity than traditional methods. The simulations are implemented in the Unity game engine, and an API will be exported to allow game monitoring and dynamic adaptation of the game simulations.

The VR environments incorporates basically three key concepts that generate rehabilitative learning: repetition, feedback, and motivation. As these factors are independent, and, when combined in an intelligent environment, they can achieve promising goals. Rehabilitation in mental health aims to help patients to find a way that allows them to interact and function in society. Interaction with people and the experience of everyday situations can contribute to this goal, thus allowing control of the experience in order to enhance the effects of therapy. As a preparation and anticipation of some potential real circumstances during the game, it is also the objective of this module to produce and test the implementations of some VR scenarios, where the patient will be confronted with some condition of real life simulations, to be implemented in future activities.

If the patient stress level is low, the average difficulty is increased, requiring him to fulfill additional steps, such as retrieving a new toothpaste from the closet, change the cover of the teeth brush, etc. Nevertheless, if the stress starts to rise, the patient is taken to a quiet and peaceful environment, such as an indoor garden, to relax and recover.

4.3. Data Visualization

The medical staff and the caregivers interact with the system through web-based applications: the medical dashboard and the session dashboard (see Figure 4).

Figure 4. Active modules.

Both the medical and session dashboard are supported by a SpringBoot REST backend, with web interface in Angular. This provide a visualization of the full patient’s medical record and the real-time information of the session, including the physiological parameters and game visualization.

The modules described in this section constitute the main blocks for the development of a cognitive and psycho-social rehabilitation system based on serious games and VR. The connections between them are based on indirect communication approaches, that ensure a loose binding and time and space independent communication process. This contributes to a flexible architecture, in which different programming languages and different operating systems can be used transparently. However, the design and development of this system also requires a deep understanding of the impact and effects in the rehabilitation process.

5. Study Design

The main purpose of the work described in this paper is the development of a rehabilitation system based on VR and serious games. The experience cannot exacerbate symptoms, and, as such, it is important to design experiences that do not provide any unexpected stimuli or effects that could be misinterpreted as hallucinations. For example, there should be no narrator voice to guide the patient, since it would look like audio hallucinations, just like there should not be any “force-feedback” device, since it could be confused with tactile hallucinations, or even put arrows or other elements in the VR environment, for the same reasons. This is a challenge, and, to mitigate the possibility of this happening, a four stage study procedure should be done:

• Internal testing, with the developers.
• External testing, with caregivers and medical staff.
• Testing with a control group (healthy adults).
• Testing with patients.

This process requires that a clinical trial be assembled, of the responsibility of the medical staff, partner in this project. Broadly, it will be based on a multi-case study, aimed at people diagnosed with the schizophrenia spectrum disorder for at least 6 months and who are psychopathologically stabilized. They should be adults, with age between 18 and 65 years old, classified as: minimally ill, mildly ill, moderately ill, markedly ill, severely ill, among the most severely ill (scale to be defined in the course of the project, according to international standards and its application validated in Portugal) (Table 1). The patient will receive information about the study and will decide to participate (informed consent).

Table 1. CGI-SCH Severity of Illness: English Version [55].

Gravity of illness
Considering all your clinical experience, with schizophrenic patients, what is the severity of the patient’s illness over the past week?
	Not ill	Minimally ill	Mildly ill	Moderately ill	Markedly ill	Severely ill	Most severely ill
Positive symptoms (e.g., hallucinations, delusions, or bizarre behavior)	1	2	3	4	5	6	7
Negative symptoms (e.g., affective blunting, avolia, and anhedonia)	1	2	3	4	5	6	7
Depressive symptoms (e.g., sadness, depressed mood, or hopelessness)	1	2	3	4	5	6	7
Cognitive symptoms (e.g., impairment of attention, concentration, or memory)	1	2	3	4	5	6	7
Total Severity	1	2	3	4	5	6	7

The rehabilitation sessions promoted with the serious game in virtual reality will take place at the healthcare facilities. The patient will be accompanied by a caregiver, who monitors the session and may intervene if necessary. The progress throughout the sessions is monitored by the medical team and allows for assessing the adequacy of the rehabilitation treatment and possible adjustments.

It is important, in order to consider the needs of the patient as a user of the serious game, that prior data collection takes place, namely in pre-rehabilitation sessions to identify the difficulties experienced in organizing and planning their daily life. This list contributes to the game design and complements, with direct testimonies, the game narrative.

6. Conclusions

Schizophrenia is the most common psychotic disorder that affects approximately 20 million people worldwide. Although the positive symptoms can be addressed with anti-psychotic medication, cognitive and negative synptoms require a multidisciplinary approach that includes psychopharmacology, individual psychotherapy, and psycho-social approaches, such as social and life skills training, family therapy, vocational therapy, art therapy, occupational therapy, cognitive training, etc. Due to the individual, social and geographical conditions, not everyone has easy access to treatment, worsening the symptoms during the course of the disease.

In this paper, a virtual reality serious game is proposed, as an auxiliary tool for rehabilitation. It supports three types of users, namely the patient, the caregiver, and the medical team, that will have different tools to assist on the rehabilitation procedure. Patients rely on a safe and controlled VR environment that allows them to repeat, with constant feedback and motivating challenges, simulations of daily activities. The rehabilitation session is constantly monitored, collecting the patient’s performance in the game, several biometric and physiological signals, and voice signal and body pose, to automatically and dynamically adapt the game challenges and environment to the patient’s stress level and skills.

The communication between the modules uses loose binding techniques, allowing for a flexible and distributed architecture, independent of the sensors, networks, and programming languages.

Some challenges remain, though. The use of game elements and mechanics should not exacerbate positive symptoms, so audio and visual clues must be carefully designed. In addition, the rehabilitation room, should be clear of obstacles and provide a safe area for the patient during the rehabilitation sessions. At least a 16 m² should be prepared, and all the wearable equipment (smart band, microphone, VR goggles) should be wireless, to allow freedom of movements and reduced discomfort to the patient.

The design and development of the system is currently ongoing, with intense evolution in the data collection and processing through AI algorithms, careful game design and asset development, and VR environments prototyping.