Development of Machine Learning Algorithms for the Determination of the Centre of Mass

Abstract

The study of the human body and its movements is still a matter of great interest today. Most of these issues have as their fulcrum the study of the balance characteristics of the human body and the determination of its Centre of Mass. In sports, a lot of attention is paid to improving and analysing the athlete’s performance. Almost all the techniques for determining the Centre of Mass make use of special sensors, which allow determining the physical magnitudes related to the different movements made by athletes. In this paper, a markerless method for determining the Centre of Mass of a subject has been studied, comparing it with a direct widely validated equipment such as the Wii Balance Board, which allows determining the coordinates of the Centre of Pressure. The Motion Capture technique was applied with the OpenPose software, a Computer Vision method boosted with the use of Convolution Neural Networks. Ten quasi-static analyses have been carried out. The results have shown an error of the Centre of Mass position, compared to that obtained from the Wii Balance Board, which has been considered acceptable given the complexity of the analysis. Furthermore, this method, despite the traditional methods based on the use of balances, can be used also for prediction of the vertical position of the Centre of Mass.

1. Introduction

The study of the human body and its movements has acquired a key role in scientific research in recent years, particularly in the biomedical fields. The technological evolution allowed significant steps forward, especially in motor rehabilitation techniques [1,2], in the study of motor problems [3] related to ageing [4], pregnancy [5], sport rehabilitation [6,7], neuromuscular diseases [8], and in the analysis of dynamic systems, in which man interacts with the surrounding environment, be it real or virtual [9,10]. Most of these issues focus on the study of the balance characteristics of the human body and the determination of its Centre of Mass (CoM) [11,12,13,14,15,16]. In sports, much attention is paid to improving and analysing the athlete’s performance [17,18,19]. This is done by means of advanced techniques that allow detecting, through the use of special sensors [20,21], what are the physical quantities (strength, speed, acceleration, and displacement) related to the different movements performed by the athletes [22,23,24].

Many biomedical studies are made through the use of tools such as the Nintendo™ Wii Balance Board™ (BB) and Wii-Fit, the playful video game that allows performing aerobic exercises and balance games using the multi-sensor platform. Following the execution of each exercise, on the basis of the detections made by the sensors, it is possible to calculate the weight but also the body mass index and the shift of the Centre of Mass on the Ground (CoG) of the individual placed on the board. Clark et al. validated the reliability of the BB as a standing balance in order to evaluate the gait and posture for clinical uses [25]. Recent studies have shown how the BB can be a valid tool to help individuals suffering from different pathologies and to improve their cognitive, balance, and motor skills [26,27,28,29]. Gil-Gomez et al. [30] have presented eBaViR (easy Balance Virtual Rehabilitation), a system based on the BB, which has been designed by clinical therapists to improve the standing balance in patients with acquired brain injury (ABI) through motivational and adaptative exercises. Gonçalves et al. [31] and Mhatre et al. [32] have studied the application of the BB in the motor rehabilitation of individuals with Parkinson’s disease (PD), requiring a simultaneous interaction to develop strategies for physical, visual, auditory, cognitive, psychological, and social activities in the performing of virtual activities, resulting in improvement in the functional performance and gait. Miller et al. [33] have analysed the effects of a balance training program utilizing the BB and body-weight supported gait training on aerobic capacity, balance, gait, and fear of falling in two persons with trans-femoral amputation.

In recent years, the interest in markerless Motion Capture (MoCap) systems has been increased, given the fact that it is an inexpensive and non-invasive technique [34,35]. Several authors adopted the Microsoft Kinect™ system as a low cost device in order to study important kinetic parameters [36,37,38,39]. Another software method for automatically identifying anatomical landmarks is OpenPose (OP) [40]. It is capable of performing real-time skeleton tracking on a large number of subjects analysing 2D images [40,41,42]. A novel approach is reported in the study by Liaqat et al. [43], which proposes a hybrid approach based on machine learning classifiers and deep learning classifiers to identify the posture detection. The results achieved an accuracy of more than 98%.

In this work, the OP system accuracy is evaluated to predict the CoM of a subject under different poses from the analysis of 2D images. This technique could be a useful aid in the study of the motion in such applications where the impossibility of adopting markers, the large number of subjects under study, and real-time performance are of fundamental importance. The OP method for the determination of the CoM has been validated by comparing it with measurements from a BB, performed on the same subject. The innovative aspect concerns the possibility of applying the OP method for the dynamic detection of the CoM, with consequent applications: From the aforementioned biomedical field to motorsport applications, where the instantaneous positioning of the driver’s CoM is strictly correlated to vehicle performances.

2. Materials and Methods

The experimental phase was divided into several tests, in which the data from the BB were recorded and, at the same time, a video of the subject in the various positions taken was recorded. A total number of 10 positions were analysed and for each of them, 10 acquisitions were made with a duration of 5 s each. From the BB measurements, the Centre of Pressure (CoP) was evaluated (i.e., the point where the resultant vector of the body constraint reaction is located), while from the OP measurement, the CoM and its projection on the X-Y plane were evaluated, i.e., the CoG, according to the definition in [44] (Figure 1).

2.1. Wii Balance Board

The BB is equipped with four sensors, each placed at the four corners of the platform (Figure 2a). It was connected to a PC to acquire its data via the Bluetooth interface, using the BlueSoleil and WiiMoteLib software libraries. The data from the BB were acquired through the BrainBlox software, which provides a graphical interface to acquire, record, and display in real-time the force recorded by the four sensors and the position of the CoP (Figure 2b). At the beginning of each acquisition, the BB tare was carried out using the “Zero Sensors” command in order to reset the force values recorded by the sensors during an interval of 4 s. This procedure is required to suppress possible vibrations and/or inclinations of the support base.

The data coming from the BB and acquired by the BrainBlox software were postprocessed through an algorithm developed in MATLAB^®. Data from BrainBlox in text format were imported into MATLAB^® in the form of column vectors and reorganized into a matrix so that it can be subsequently plotted. A “scatter” plot is performed (Figure 2b) obtaining the dispersion of the CoP values (in mm) for each single of the 10 acquisitions made for each test in the X-Y plane.

The ellipse of confidence (or sway area) represents a measure of the amplitude of the surface described by the dispersion of the positions of the CoP and was defined as the surface that contains, with 95% probability, the single points of the calculated CoP [45], with an average value of the measurements and standard deviations, corresponding to the ellipse major axes (Figure 2c). The adopted right-leaning reference system has its origin O_BB corresponding with the geometric centre of the BB and the axis oriented as in Figure 2a.

2.2. Video Acquisition

For the video acquisitions, a Logitech C270 webcam was used (max. resolution 720p, fixed focus, FoV 60°), with an acquisition frequency equal to 30 fps, in order to verify the operation of the point acquisition system with a video of medium-low quality. The positioning of the webcam has been carried out using a professional tripod with the possibility of adjusting the height and inclination of the webcam. The videos were acquired through the OBS Studio software in order to be post processed. Since photo frames and mass captures are quasi-static, the problem of signals triggering was quite simple and not impactful.

2.3. OpenPose

OP is an open source software for real-time multi-person key point detection. It is able to jointly detect the key points of the human body, as well as the hands, the face, and feet on single images, for a total of 135 key points. It accepts single images or videos as input and outputs the basic image with the detected key points. OP has been developed within a project of the Carnegie Mellon University and currently is one of the most accurate and performant methods for human posture detection.

It is based on the multi-stage convolution neural network (CNN). In the first stage, the algorithm assigns key points to the image, each one with a confidence score. In each subsequent stage, the algorithm couples the new detected key points with the previous stage ones, generating a confidence map. This enhances the forecast step by step, and after only four stages the confidence map is already very reliable.

The convolutional neural network is organized with two branches. The first one predicts a set of 18 confidence maps related to parts of the human body. The second one predicts a set of 38 part affinity fields (PAFs), which investigate the relative position of the various parts of the body, in order to verify their reliability [40].

For the recognition of the key points, OP uses a pre-trained CNN VGGNet. The network accepts as input a colour image (Figure 3a) and outputs the 2D positions of the key points for each person present in the image (Figure 3b). Once the individual output files, divided by the number of frames, are obtained, it is possible to proceed with their processing and representation. OP returns a file which contains the cartesian coordinates in millimeters of the detected key points. If the OP algorithm does not recognize a specific point, this is indicated with coordinates (0;0). Due to the presence of zeroes, there is a drift phenomenon in the calculated CoM. To solve this problem, some points (Figure 3c) which are not necessary for calculating the CoM (eyes, nose, and some points of the feet) were eliminated, simplifying the processing in computational terms, and making the calculation of the CoM considerably more precise adopting 25 key points (Figure 3d). The data coming from the OP software is processed according to the following flow chart. The key point coordinates from the OP for each frame, in a JavaScript Object Notation format (json), are decoded through MATLAB^® by means of a script. The decoding returns a “structure” which contains the key points identified during the processing of the video.

A cleaning of the data can be carried out according to the number of unrecognized key points present in each single frame. This was done by replacing the missing point with the symmetric equivalent with respect to the sagittal plane passing through key points 2–9 (Figure 3d), according to the following logic, shown in Equations (1) and (2):

$$y_i=2y_2-y_s$$

(1)

$$z_i=z_s$$

(2)

where $y_i$ and $z_i$ are the coordinates of the reconstructed key point, $y_s$ and $z_s$ are the coordinates of the acquired key point with respect to the sagittal plane, and $y_2$ the coordinate of key point 2.

The adopted reference system has the origin O_OP located at the same X and Y coordinates of O_BB, but with the Z coordinate assumed zero in correspondence of key point 9 (Figure 3a). In order to represent the data with an adequate unit of measure, a suitable scale factor was used, appropriately calculated since a fixed measurement was detected on the subject. This made it possible to obtain the measurement scale expressed in millimetres (mm). The chosen scale factor must be varied according to the subject and the type of shot. For an accurate determination of the true measurements, it is necessary to use a properly calibrated stereo camera system. Following the cleaning and correction of the data, it is possible to continue determining the CoM which is divided into the following phases: It is necessary to evaluate the distance between the key points detected in order to reconstruct the structure of the human body and to be able to assign the mass properties of each segment. With reference to Clerval et al. [46], the vector was constructed containing the percentages in length relating to the segments making up the body, through which it is possible to define the position of the single segment CoM. The total weight of the individual is defined. Subsequently, the vector can be defined containing the respective percentages by weight relating to the total mass and referring to the single segment CoM previously identified. Finally, the kinematic method for calculating the CoM has been applied.

2.4. Assessment of the Centre of Mass

The calculation of the CoM of an object can be easily done geometrically if it is a body with homogeneous mass density and relatively simple geometry. In the case of the human body the calculation is more complicated, due to the inhomogeneous mass distribution. Hence, it is important to find an experimental method that gives a precise estimate of this position. In this study, the kinematic method was adopted.

The kinematic (or segmentation) method is based on a simple principle which states that the sum of the moments of the single part of the body defined with respect to an arbitrary axis must be equal to the sum of the moments (i.e., the moment of the total body mass) with respect to the same axis. Through this method, we obtain the cartesian coordinates of the CoM and to its corresponding projection on the X-Y plane, the CoG, shown in Equations (3) and (4):

$$x_G=\frac{\sum m_i{x}_i}{M}$$

(3)

$$y_G=\frac{\sum m_i{y}_i}{M}$$

(4)

where $x_G$ $y_G$ represent the coordinates of the CoG; mi represents the percentage masses of the segments making up the body, obtained through tabulated values; $x_i$ and $y_i$ represent the coordinates of the respective CoGs of the segments constituting the body; M is the total mass of the body.

In order to use this method, it is necessary to know the position of the CoG of each individual part of the human body. The data used in this paper refer to the publication of Clerval et al. [46]. This method is consistent with the mass distribution of the human body and allows varying the position of the CoG segments so that it can adapt to specific cases by changing the influence of a single part of the body on the calculation of the overall human body CoG.

3. Results

The experimental part of the work is characterized by several tests in which the data coming from the BB have been recorded and, at the same time, the video recording of the subject placed in 10 different positions have been carried out. For each individual test, 10 quasi-static acquisitions of 5 s each have been made adopting the methods of Section 2. The data acquired through the BB and processed using the BrainBlox software return a map of points on the X-Y plane, allowing an estimate of the CoP subject. Data acquisition through video recording, and the subsequent processing with the OP software, return a map of points on the Y-Z plane, allowing an estimate of the CoM projection on that plane. The comparison between the two types of data made it possible to validate the OP method for determining the CoM with video acquisitions. In Figure 4, several adopted subject test configurations are reported. Test A was carried out with the subject in an upright position, placed in front of the camera frame (0° position), as shown in Figure 4a. Test B was carried out with the subject turned away from the camera (Figure 4b), turned 180° with respect to the camera frame. Tests C, D, E, and F were carried out with the subject in an upright position, rotating clockwise and counterclockwise with the BB, respectively by ±30, ±45, ±60, and ±90° with respect to the camera frame (Figure 4c–f).

A fixed reference for the positioning of the BB was done by creating a paper template in which the corresponding positioning points of the BB are represented for each chosen angle. The data acquired through BrainBlox were processed using an algorithm developed in MATLAB™, and subsequently plotted on a graph showing the coordinates of the CoP. By matching the data obtained from the 10 different acquisitions, the absolute position of the CoP along the X-Y plane is identified and the ellipse of confidence is plotted in red (Figure 5). For each position, the average value of the CoP and its standard deviation are reported. It is possible to notice how the points obtained are concentrated in a specific part of the quadrant with respect to the reference system of the BB, and this is due to the characteristics of the subject and his tendency to shift the weight according to the posture assumed during the test. A variation of the subject position implies a variation of the CoP coordinates, with respect to the X and Y axes, in the frontal (0°, Figure 5a) and rear position (180°, Figure 5b). On the other hand, considering the rotated BB positions, the average CoP of the subject is approximately located in the same position.

During the performed acquisition of the CoP by the BB, video recordings of the subject have been conducted in order to estimate the projection of the CoM on the Y-Z plane. In Figure 6, the detected key points with the estimated CoM projection and the video frame with the detected body segments are reported. As it is possible to see, OP is able to detect the majority of the key points and recognize when the subject is placed in a frontal (Figure 6a) or rear position (Figure 6b). The software detects in a clear way the key points of the angulated positions (Figure 6c–h), while it has some difficulties to correctly recognize the body segments of the subject orientated ±90° with respect to the camera frame. However, the developed algorithm is able to assess the CoM of the subject.

It is possible to compare the Y coordinate of the CoP, assessed by the BB, with the same coordinate of the CoG, evaluated as the projection of the CoM on the X-Y plane. In Figure 7, the average Y coordinate value of the CoP and CoG are reported with the standard deviation. In the same figure, the absolute difference (ΔOP-BB) between those values is also reported. It is possible to observe how this difference is, in the majority of the performed tests, below 20 mm. Only Test D (−45°) and Test E (−60°) show a greater difference, but still below 40 mm.

The quality of the results obtained is directly linked to the accuracy in which the OP software recognizes the key points of the analyzed subject. In particular, limits in the recognition of key points in some positions have been noticed. To solve this problem, some key points which are not necessary for the direct evaluation of the CoM have been eliminated, making a more precise and simpler CoM estimation. If one or more of the 25 acquired key points are not correctly detected, a reconstruction based on the symmetry of the human body must be performed. In the case of non-symmetrical positions, a small error is introduced in the calculation of the CoM, quantifiable as a function of the position assumed by the subject. In addition, the calibration procedure to assess a scale factor must be performed in order to correctly represent the coordinates of the acquired key points.

The estimation of the CoM is severely affected by the methodology to assess it. The CoM of the body segment coincides with the midpoint of that segment. To calculate the static moment, the body segment is equivalent to a point mass concentrated in the CoM of each individual part of the body, having the total length of the segment by intensity. However, this method is inconsistent with the mass distribution of the human body as it considers the CoM of each segment as located in its midpoint. Furthermore, it does not allow managing the different mass characteristics of the different subjects under study. For a better accuracy, the kinematic method has been adopted in the present work. The two methods have been compared by the authors, observing a different position of the CoM of the body segments, as well as a different number of CoM. This is addressed by the fact that the geometric method considers every single detected segment, while the kinematic method condenses the mass of some segments into a unique one (e.g., the trunk segment includes also the masses relative to the shoulders and to the pelvis). The standard deviation of the CoM Z coordinate is reduced with the kinematic method and the determined values are consistent with the experimental study that reports a CoM falling at about 53% of the height of the individual [47].

The comparative analysis performed between the CoG, projection of the CoM on the X-Y plane, and the CoP, assessed on the same plane by the BB, returns very small differences for the majority of the performed tests. These differences are consistent with other markerless MoCap systems, based on the trained neural network, which have uncertainties in the marker position of the order of 10 mm [35]. For a more precise estimation of the CoM with OP, a stereo video acquisition should be adopted to also estimate the coordinates in the X-Z plane.

4. Conclusions

In this paper, the CoP measurement method using Wii BB has been compared with a CoG markerless optical measurement method, using the OP software. The results demonstrated an error of the optical method of less than 20 mm, for the majority of the angles, and of less than 40 mm for the highest angulation (45 and 60°). Given the complexity of the measurement and the use of a method which is contactless and markerless, the method has been considered reliable. The use of inexpensive video facilities, with relatively low resolution and frame rates, and an open source software has enabled really low-cost results.

A particular and interesting aspect of this experimental campaign lies in its novelty, since not many comparative studies have been carried out between the direct methods (Wii BB, force plates) and indirect methods (Kinect, OP) for the determination of the CoG.

Furthermore, an interesting aspect of this methodology consists of the fact that it is possible to make a prediction of the vertical coordinate of the CoG, which is not possible with traditional methods.

The validation of the use of OP for the CoG calculation can become decisive for the development of different sectors such as sport or medical rehabilitation.