1. Introduction
The control of the postural system is one of the fundamental neurophysiological mechanisms of the human body. It is fundamental to ensuring balance against gravity and fixing body orientation, and functions as a reference frame for perception–action coupling while efficiently dealing with the external world. Postural control is a dynamic process that requires sensory detection of body motions and integration of sensorimotor information within the central nervous system. In more detail, the central nervous system triggers the execution of appropriate musculoskeletal responses in order to obtain an equilibrium between destabilizing and stabilizing forces [
1].
It has been shown, in the reference literature of physiatric medicine, that measurements of postural stability are critical for determining predictors of performance [
2], for evaluating musculoskeletal injuries [
3], for determining the effectiveness of physical training and rehabilitation treatments [
4], and to provide injury prevention through the study of injury risk-factor analysis [
3,
5]. The body’s motion is mainly based on the integration of the proprioceptive, visual, and vestibular inputs [
6]. Afferent proprioceptive inputs are conveyed to different levels of the central nervous system [
7,
8,
9]; however, most of them remain unconscious. The joint positions and movement sensing (kinaesthesia) are the expressions of the conscious component, but postural control is primarily based on the unconscious component [
9]. Specifically for the antigravity movements, proprioceptive control represents the expression of the effectiveness of the stabilizing reflexes in controlling vertical stability [
8]. In fact, antigravity movements are the activities that counteract gravity and postural instability with at least a phase of single-limb stance [
6]. In this way, proprioceptive input represents the most relevant sensory system in the maintenance of static postural stability at all ages and fitness levels [
10]. This topic is also relevant in neurodegenerative diseases such as Parkinson’s disease (PD). For example, PD patients with postural instability have worse reactions to brief perturbations, more stance sway, and trouble switching between tasks. Moreover, quantifying balance changes in early and moderate-stage PD and the comparison to healthy subjects using clinical assessments of balance and musculoskeletal activation is paramount, primarily if performed through less invasive and costly systems [
11,
12].
Many tools to detect musculoskeletal activation have been used in sport and rehabilitative medicine. Mainly, electromyography (EMG) is employed for this purpose. However, EMGs are not yet widely used in combination with accelerometers for forecasting and customizing measures for the analysis of the human body’s motion to achieve different goals. This is mainly due to the limited number of investigations that have been focusing on the nature of musculoskeletal response to a broad spectrum of stimuli able to identify the thresholds that establish the standard/ideal status of the postural system. Hence, there is a lack of low-cost technology that habilitates these measurements.
Furthermore, in the last decade, with the advent of the Internet of Things (IoT), embedded sensors have been integrated into personal devices such as smartphones and smartwatches. In several applications, sensors are integrated into clothes or other equipment/objects of daily life, becoming a central research topic due to their importance in many areas, including healthcare, interactive gaming, sports, and monitoring systems for general purposes in controlled and uncontrolled settings [
13,
14,
15].
The primary purpose of the investigation we carried out in this project was to present a preliminary study on the design of a portable and reliable postural system prototype composed of HW and SW, adapted to diverse individual profiles concerning the performance viewpoint, from patients needing rehabilitation to top-level athletes. It is widely accepted in the community of psychiatric medicine that proper quantification of the postural system efficiency represents an essential assessment for improving the quality of life. However, most of the actual measurements are developed in a laboratory environment where natural movements are usually constrained by the instruments applied to subjects’ bodies and the environment. This process is performed to distinguish, as precisely as possible, in the limits of the experimental setting, the roles of proprioception, visual and vestibular input using a low-cost and portable instrument. This habilitates the individuals to move freely and perform activities at home or in other uncontrolled environments (e.g., gyms or sports facilities) [
5,
16].
The technology integrated into such project includes state-of-the-art apparatus (i.e., force platform, EMG, and motion capture cameras) and wireless three-dimensional inertial units performing computation and data analysis over low-cost devices (i.e., sensors themselves, smartphones, and tablets) or cloud platforms.
In the project, we considered variants of these configurations’ implementations to evaluate possible solutions to different settings. Subsequently, another essential goal will be the usability of the envisioned technology. Thus, the possibility to perform a comparative study with existing state-of-the-art validated systems is needed to obtain a well-ground assessment. Moreover, fundamental analysis of the specific characteristics of the exercises existing in the literature and a combinational calculation will define the subsequent exercise prescriptions; a multitude of exercise combinations will be available to satisfy the actual needs of the different clinical conditions. To identify these exercises, in future works, we are going to investigate the potential of reinforcement learning as a tool for customization (in order to adapt the response of the portable tool to the needs of the setting, such as medical diagnosis, performance test, training evaluation, and training progress measurements) and personalizing (in order to adapt to the individual variability of the applications mentioned above).
The project has a threefold scope:
- (a)
providing a theoretical foundation of a set of motor tasks employed to evaluate the performance of posture and gait in at least three contexts: rehabilitation, sports performance, and good health training at different ages;
- (b)
devising and testing a set of sensors that allow obtaining the above measures;
- (c)
defining a preliminary set of experiments and algorithms to identify individual profiles and compute, via machine learning methods, the correct set of functional exercises.
Finally, this paper presents the experimental apparatuses and describes the experimental design strictly related to the goals mentioned above. The experiments conducted were structured for a final purpose: obtaining an experimental system for identifying the correct patterns to be devised in the project’s future developments.
The rest of the paper is organized as follows.
Section 2 provides an overview of the relevant references to similar studies.
Section 3 discusses the project’s expected research outcome, and
Section 4 introduces the overall experimental design.
Section 5 presents a preliminary evaluation of the data collected by the designed setup. Finally,
Section 6 presents some conclusions and sketches further work.
2. Related Work
Posture is studied in two aspects: (a) postural control and (b) postural orientation. The former involves studying the positional control of the body in space and orientation. Instead, the latter involves studying the relationships between body segments. A neutral state (also known as neutral posture) is observed when the upper trunk and head are at zero degrees concerning the vertebral column. Subjects deviating from this neutral posture are said to have low stability that can provoke accelerated intervertebral disc (IVD) degeneration, damage, and misalignment of vertebrae, producing nerve compression that can cause radicular manifestations, such as sensorimotor deficiencies and pain in the involved regions [
17]. When considering the maintenance of the vertical posture in everyday life situations, postural control might become a complex task that requires the ability to anticipate and compensate postural strategies when fast actions are performed and when environmental perturbations are applied [
18]. How individuals control their preparatory and compensatory postural adjustments is still under debate [
19]. Several mechanisms help individuals to keep their posture when task conditions change due to self-inflicted perturbation (e.g., I am suddenly moving my upper arm forward [
20] or when somebody is pushing me [
21]). These mechanisms are represented by changes in the activation levels of postural muscles called early postural adjustments (EPAs), starting up to 1000ms prior to the impact [
22], and anticipatory postural adjustments (APAs), starting 0–150ms prior to the impact [
23]. The primary role played by the EPAs is to adjust the posture and facilitate action planning. Typical examples are seen in preparation for making a step [
24] or to avoid contact with an approaching object [
25]. On the other hand, the function of APAs is to generate forces that act against an effect (mechanical) of a predictable perturbation [
26]. Here we concentrate on these ecological motor tasks where individuals are challenged to control posture when facing a highly dynamic situation. The tasks selected in this experimental design involve sequences of actions that require the maintenance of stable posture while standing on an unstable proprioceptive platform and receiving in an unexpected or expected way a perturbation requiring sudden balance recovery. These motor tasks will help unveil the individual strategies adopted given the individual’s level of skill. Based on the literature background, a single limb stance is regularly used to examine the postural system [
16,
27,
28,
29],
EMG, electrocardiography (ECG), and inertial sensors integrated into wearables are emerging as promising low-cost and easily usable solutions in everyday life [
13] and health care contexts [
30]. Inertial measurement unit (IMU)-based movement identification can be achieved by statistical classification or be threshold-based [
31]. Such statistical methods utilize supervised machine learning, which links features of a movement to possible movement states in terms of the observation’s possibility [
32]. Many of these studies are devoted explicitly to disabled people with diminished gait/posture abilities. This holds for multiple sclerosis patients [
33] and Parkinson’s disease sufferers [
30,
34,
35]. The ability to monitor the gait of multiple sclerosis patients and provide correct biofeedback can help prevent falls and detect freezing (an aspect that can be fruitful also for Parkinson’s patients) [
33,
36]. Prototype systems often include integrated sensors located on the ankles to track gait movements. Body sensors are positioned near the cervical vertebra or on the shoulders to monitor body posture [
13]. Many systems can also measure parameters that might be difficult to provide manually, such as the maximum acceleration of the patients during standing up, or the time it takes from sitting to standing [
37,
38].
Moreover, the current diffusion of machine learning methods employed in gait, posture analysis, and feedback is not comprehensive, but a few significant results have already been achieved. A relevant group of investigations has been designed for decoding algorithms for brain–machine interactions (BMIs) that use the spiking activity as their control signal [
39]. These approaches are powerful in devising usable technologies. Specifically, feedback for reinforcement-learning-based brain–machine interfaces using confidence metrics has been addressed [
40]. Some studies show how to derive the required evaluative feedback from a biological source, using both the feedback’s quantity and quality, and incorporate it into reinforcement learning controller architecture to maximize performance. Analogously, the Berlin BCI has developed an accurate system that works from the first session in BCI-naive Subjects [
41].
An overview of the various steps in the brain–computer interface (BCI) cycle, i.e., the loop from the measurement of brain activity, classification of data, feedback to the subject, and the effect of feedback on brain activity, is the focus of [
30,
42]. On the other hand, the role of technology for accelerated motor learning in sports is investigated in [
43]. Finally, parallel man–machine training in ECG-based cursor control development is the subject of [
30,
44]. Some references should be given to smart environment previous investigations as a foundation of the method developed here, emphasizing some development related to energy management [
45] and concerning the design of energy-efficient transmission protocols for wireless body area networks [
46,
47]. However, the systems mentioned above present many limitations related to accessibility, economic cost, size, battery life, intrusiveness, and usability (i.e., controlled and uncontrolled home or working context) environments.
3. Expected Research Outcomes of the Project
This section illustrates the project goals, reporting the most suitable application scenarios of the technology we envision and an overview of the presented architecture design. For better comprehension, the goals are presented from a top-down perspective.
- (a)
basic application scenarios;
- (b)
envisioned technology;
- (c)
system architecture workflow.
3.1. Application Scenarios
Three different potential application scenarios have been devised under the supervision and collaboration of psychiatric medicine personal and sports training experts. Scenario 1 was a controlled environment enriched with a set of sensors to the extent that it makes this environment smart. Scenario 2 was set without specifying whether the performed activities were to be carried out indoors or outdoors. It is legitimate to suppose that the environmental setting shall be relatively poor regarding available interactions, including the potential unavailability of an Internet connection. Scenario 3 identified variations determined by post-traumatic rehabilitation, personalized performance control over the evolution of illnesses with harmful consequences on the patient’s stability, and training of athletes with special needs.
In the context of a psychiatric medical practice, a patient who suffers from postural instability due to a traumatic event (e.g., car accident) or neurodegenerative disorder is visited by medical personnel. The diagnostic process is assisted with the envisioned technology. The patient is asked to execute a sequence of three exercises: single-stance stability test, forward movement of arms with weights, and step-on gait on a free-range. During the exercises, the patient wears a jacket equipped with a set of sensors and interacts with a visual focus tool that helps her to identify a fixed point at a given distance. The jacket interacts with an application that works on the cloud, measures the reaction time or other variables, including anticipation’s effectiveness in the movements, and provides the operator the possibility of marking progress in performance quality based on a fixed threshold that the operator can define concerning age, sex, and the clinical condition of the patient. The whole process is recorded on video, and the instrumental measures are saved on the patient’s profile.
An athlete training for a sporting event is monitored by her coach. He provides her with a performance benchmark in line with the event requirements and expectations. The athlete has a given training period for preparing for the event and a performance level she has to accomplish in order to be competitive. The performance benchmarks have been defined by the coach based on the athletic preparation path of the athlete. While following the coach’s requests, the athlete executes some training exercises while wearing the jacket described in Scenario 1. Every athlete’s exercise is compared against the benchmark performance and consequently identified in terms of a negative gradient concerning the benchmark itself.
During the rehabilitation period, a patient wearing the jacket described in Scenario 1 attends a program consisting of a series of exercises. Each step in the series requires comparing the performance with the provided reference benchmark defined by the psychiatrist during the diagnostic process. The patient measures are the same as in Scenario 2 and represent how a patient uses the jacket in a medical context.
3.2. Envisioned Technology
Our project’s aim is to design an accessible, low-cost, small, dedicated low-cost technological solution (i.e., Gait and Posture Smart Jacket (GPSJack)) suitable for the previously introduced application scenarios.
Figure 1 shows a graphic representation of the envisioned technology where the subject is wearing the GPSJack and is immersed in the Gait and Posture Smart Environment (GPSEnv). Numbers 1–5 mark the sensors attached to the jacket, and number 6 marks the tablet application used by the top-level user. Medical and potentially sports training staff can interact with the GPSJack/GPSEnv through a tablet to guarantee total portability and versatility of the envisioned technological solutions.
Therefore, the GPSJack system represents the low-cost inertial sensor-based system, which is the project’s final goal. GPSEnv represents the adaption of the existing state-of-the-art technologies to the presented scenarios for validation purposes. The final GSPJack/GPSEnv architecture will provide to medical/sports staff the capability to perform the following actions:
register a GPSJack/GPSEnv administrator;
start setup of a GPSJack/GPSEnv;
registration of a new user;
definition of the profile of a registered user;
eliminate a profile;
execution of a base test for a registered user;
registration of a new exercise;
registration of a new benchmark for an existing exercise;
assignment of a benchmark for a given exercise to a specific user;
execution of an exercise;
registration of a sequence of exercises as a training path;
association of benchmark values for a training path;
assignment of benchmark values for a training path to a given user;
visualization of a single test progresses along a temporal interval;
visualization of progresses with respect to a given benchmark;
Based on the above-defined functions, several background software instruments are required. The technologies for managing and analyzing the data from the sensors, which we may name the GPSJack Framework, have been envisioned. Nevertheless, machine-learning-based algorithms will guide the personalization of the benchmark process by employing intelligent reinforcement learning methods. Finally, since the proposed technology has its main applications in healthcare, it will provide, in addition to the classical data protection techniques, a physical protection layer that, based on the radio signals’ propagation patterns, will habilitate the possibility of utilizing the tablet if and only if the tablet is under a certain distance (e.g., 5 m) from the GPSJack nodes.
3.3. Architecture Workflow
The system’s architecture is composed of several modules, each one with a single responsibility, the logic model of which is reported in
Figure 2.
Every module is related to at least another one:
- User Interface:
allows the user to input “environments” and provides visualization of all data/results returned by the Machine Learning module;
- Configurator:
takes as input a set of environments and gives as output a set of feasible “exercises” to be used by the Trainer, according to user input and machine learning algorithms’ indications;
- Trainer:
uses the exercises and determines the “should be” paths, meaning which exercises shall be executed and with which environment;
- Evaluator:
responsible for the evaluation of data and automatic comparison with benchmarks;
- Sensor:
deals with wearable sensors, collecting and normalizing data;
- Machine Learning:
gathers data and induces models.
The User Interface’s output is an ordered set of “environments”, which are one of the inputs of the Configurator. The Configurator continuously computes, and at given times uses models from the Machine Learning module to produce a set of feasible “exercises”. These exercises are to be given to the Trainer, whose outputs are chains of exercises, also named “paths”, to be executed by patients (or athletes). At different times, different paths are possible due to the work of the Trainer. The data are then gathered from the Sensor module and sent to the Evaluator to be stored, visualized, and compared with benchmarks. They are also used to devise possible “paths” and exercises to be delivered as hints to the user. This is the responsibility of Machine Learning, which acts as a feedback generator for the whole system, enabling the system to enhance performances continuously.
5. Preliminary Experimental Evaluation
Following the experimental design of
Section 4, we collected data from 12 different male subjects, whose physical characteristic are shown in
Table 2.
At the end of the data collection phase, each subject had performed a total of 84 7 tasks (i.e., do not rotate, rotate of an angle of 30
or 60
or 90
, both right and left) × 2 statuses (i.e., stable/unstable) × 6 repetitions) data collection sessions. Each session involved the five movements phases (i.e., rotate, in position, APAs, drop, and recover) described in
Section 4.1 and shown in
Figure 5.
In this preliminary evaluation, the collected data are divided based on the data collection technologies we utilized (force platform, EMGs, or acceleration data) into three different datasets, as shown in
Table 3.
Subsequently, for each dataset type (i.e., A, B, and C), we applied the data-processing pipeline defined in
Section 4.3 and shown in
Figure 10. In particular, the data, in segmentation and feature representation forms were segmented in time windows of 150 ms as the hypothetical duration of the APA movement phase), were used to train three different machine learning models, whose performances in recognizing the movement phases of
Figure 5 were measured in terms of accuracy
A), precision (
P), recall (
R), and F1-score (
F1), defined as follows [
57]:
Here, tp represents the number of true positives, n represents the number of true negatives, fn represents the number of false negatives, and fp represents the number of false positives.
5.1. Preliminary Results
Table 4 and
Table 5 present the results of our preliminary analysis.
Table 4 presents the results for datasets A, B, and C from all subjects simultaneously by performing a k-fold test (k = 5) on all subjects’ data.
Table 5 presents the results for datasets A, B, and C for every single subject by performing a k-fold test (k = 5) on each subject’s data.
The results of
Table 4 show that the random forest performed the best on all three datasets (i.e., A, B, and C), and for both data-treatment types (i.e., segmentation and feature extraction). Moreover, when differentiating by dataset type, the results show that the conjunct information of dataset A (i.e., EMGs and Force Platform) and dataset B (i.e., acceleration values) was put to use significantly better than when used separately. In particular, in terms of F1-score, there was an increment of 12.2% from dataset A to dataset C and 39.9% from dataset B to dataset C. Such results indicate that the acceleration provides precious information concerning the recognition of the studied activities. Overall, the achieved F1-scores for all subjects using the random forest model and dataset C were 88.2% and 89.6%, respectively, in segmentation and feature representation modes.
Table 5 shows that when training and testing the models with one specific subject, the models’ performances are subject-dependent. As shown from the statistics of each model for all subjects (bottom of the table), the RF model performed much better (F1 score > 91.6%) in both segmentation and feature representation modes, showing that the recognition accuracy, in terms of F1-score, is on average 5% higher than when training and testing with all 12 subjects’ data (see
Table 4). Again, differentiating by dataset type, the results show that the conjunct information of dataset A (i.e., EMGs and Force Platform) and dataset B (i.e., acceleration values) performed significantly better than when used separately. In particular, on average, there was an increment of 7.3% in the F1-score from dataset A to dataset C and 31.1% from dataset B to dataset C.
5.2. GPSJack Evaluation
Concerning the suitability and principal characteristics of the GPSJack system, this section presents its evaluation in terms of RAM, storage, CPU, battery consumption, and data loss. In particular, a Samsung Galaxy Tab A7 with the following HW/SW characteristics has been tested in the setup described below.
Processor: Qualcomm Snapdragon Octo-Core (4 × 2 GHz + 4 × 1.8 GHz);
Operating system: Android 10;
RAM: 3 GB;
Storage: 32 GB;
Connectivity: Bluetooth 5.0, Wi-Fi a/b/g/n/ac Dual Band;
Battery: 7040 mAh;
Dimension: 247.6 × 157.4 × 7 mm per 476 g;
Price: $140.
Three Nordic Thingy 52 were connected to the Samsung Galaxy for three consecutive hours. The setup was tested once for each sampling frequency: (i) 50 Hz, (ii) 100 Hz, and (iii) 200 Hz.
Table 6 presents the evaluation results obtained using the Android Studio Profiler suit.
As observed, the designed system can work on various setups with no data loss and low storage, RAM, and CPU usage. Moreover, its battery consumption allows a data collection phase of almost 8 h. Based on the tests performed during the project, the Nordic Thingy 52 can efficiently compute for more than 48 consecutive hours at 200 Hz.
5.3. Discussion
The results of these preliminary experiments clearly show that the conjugate of EMG, Force Platform, and acceleration data performs considerably better than their separate utilization. This improvement enables a better understanding and in-depth study of human motion. In fact, the GPSJack prototype has good potential to capture relevant information, enabling the possibility to recognize the studied motion classes with an average F1-score of 89.6% when using all the subjects’ data at once. Furthermore, when tested on single subjects, the F1-score ranged from a minimum of 83.2% to a maximum of 100%, outperforming the usage of only one of the aforementioned data sources. Even though the collected acceleration data present precious information, further work must be conducted to increase the overall performance and reduce the dependence on state-of-the-art technology. This can be done by: (i) implementing more complex recognition models than the used k-NN, RF, and LDA; and (ii) exploiting the utilization of a larger number and different positions of data collection nodes on the human body. Nevertheless, the information generated by the EMG and Force Platforms is paramount; thus, a possible next step in addition to those mentioned above would be the integration of EGM sensors into the same GPSJack nodes.
6. Conclusions and Further Development
This paper has dealt with defining the experimental design of the “Biofeedback Wearable and Environmental Technologies for Postural Correction” project. We illustrated the target technology, described the project’s evaluation workflow (i.e., state-of-the-art instruments and low-cost wearable sensors, data processing flow, and machine learning-based analysis), and provided a high-level description of the context in which the envisioned technologies are forecasted to operate. In particular, we devised a methodology investigating how to build the parameters that allow the physiatric medical staff to evaluate the patient. Three challenging motor tasks were identified to, on the one hand, train fundamental motor skills such as action adaptation, compensation, and anticipation, and on the other hand, to measure the performance levels fir such skills. To measure the quality of the performed motor tasks, we evaluated a low-cost body area network (aka GPSJack). GPSJack uses at most eleven data collection nodes integrating an accelerometer, gyroscope, and magnetometer sensors and can compute for 48 h at a sampling frequency of 200 Hz. Moreover, we used an android mobile application that works as a data aggregator and controller for the GPSJack system, through which the user can observe the collected data and the posture and gait quality indicators. Nevertheless, in conjunction with the GPSJack, we integrated the GPSEnv based on state-of-the-art gait and posture evaluation systems.
Tests on data collected from 12 subjects for a total of 84 data collection sessions each showed that the designed system could highly accurately recognize the phases of the defined motor tasks. In particular, in a subject-independent setup, we achieved an F1-score of 89.6% in recognizing the five studied movement states (i.e., rotate, in position, APAs, drop, and recover). With a subject-dependent setup, the F1-score ranged from 100% for subjects 23 and 26 to 83.2% for subject 22. These results show that the acceleration information that we will add to the state-of-the-art systems significantly increases recognition capabilities.
It is widely accepted in the community of psychiatric medicine that proper quantification of the postural system’s efficiency represents an essential assessment for improving quality of life in the elderly, patients with neurological pathologies, and athletes. Moreover, since most of the actual measurements are made in a laboratory environment where natural movements are constrained by the instruments applied to subjects’ bodies and the environment, a system usable in uncontrolled and unconstrained environments (e.g., home, gym, or sports facilities) habilitates the individuals to move freely in their natural environment and perform the required motor tasks. Thus, the designed system will evaluate profiles from the performance viewpoint of individuals ranging from patients undergoing rehabilitation to top-level elite athletes in controlled and uncontrolled environments.
Since the final goal of the project is performing the defined task in uncontrolled environments and using only the acceleration information provided by a system such as GPSJack, the next step will concern the exploitation of the data collection phase while making use of a large number of nodes positioned on different body parts and of more complex pattern-recognition models than the used k-NN, RF, and LDA. In particular, deep learning models such as recurrent neural networks (RNN) and long short-term memory (LSTM) have shown optimal results in such fields. Moreover, since the information captured by the EMG sensors is paramount, integrating an EMG sensor into the GPSJack nodes should be considered.