Evaluation of a Framework for Visual-Feedback Training Based on a Modified Self-Organizing Map Using Sensing Information Including Muscle Activity ^†

Abstract

In this research, we propose a visual-feedback system and evaluate it based on motion-sensing and computational technologies. This system will help amateur athletes imitate the motor skills of professionals. Using a self-organizing map (SOM) to visualize high-dimensional time-series motion data, we recorded the cyclic motion information, including the muscle activities, of a male subject as he pedaled a bicycle ergometer. To clarify the difference between the subject’s motor skill and the target motor skill in a cyclic movement, we used the modified SOM algorithm; a visual-feedback system was developed, which displayed the target motion as a circular trajectory on a two-dimensional motor skills map. The subject trained by observing only the displayed static target trajectory; the subject’s real-time trajectory was constructed from the subject’s real-time motion. We validated our proposed framework for the visual-feedback system by evaluating the motion performance of a subject using feedback training.

1. Introduction

Advances in motion-sensing technologies have made it possible to obtain a detailed analysis of human movements; this has facilitated the quantification of motor functions in sports science. In the last few decades, sports training systems equipped with motion sensors are being widely used to improve the motor skills of amateurs. Amateurs can improve their performances by attempting to imitate the performances of professionals by using motion data collected from professionals [1,2]. In such training sessions, the recorded motion is usually presented by visualizing intuitive information using stop-motion images or videos or simply signal waveforms. Training using this simple information is valid when it comes to imitating limited human motion, such as a kinematic movement in body parts or part of a professional’s movement. However, in an overall complex movement, it is difficult to imitate motor skill, which includes the use of various muscles, because of the large data volume (e.g., signal waveforms of various joint angles and various muscle activations). For gaining a deeper understanding of complex movement, a learner must perceive this information simultaneously and develop an internal model, i.e., a correlation map between body part motion and muscle activity as motor command, through trial and error. Therefore, to imitate an expert’s motor skills, it is extremely important to simultaneously integrate both kinematic movement in body parts and various muscle activities, and then provide feedback to the leaner during training sessions.

In our previous research, we proposed a method for visualizing motor skills that included various motion data presented in a self-organizing map (SOM), which could integrate significant quantities of data describing human motion [3,4]. In this study, we use a modified SOM algorithm to visualize high-dimensional data for time-series motion [4]. The data were visualized as a trajectory on a two-dimensional (2D) map called the motor skills map (MSM) constructed by applying the SOM algorithm. The MSM shows the difference between the integrated target signals (e.g., expert information or ideal motion data for a learner, produced by optimization calculation) and the measured signals in the form of trajectories on the MSM that correspond to the given motion. Subjects can perceive differences with respect to the target motion by monitoring the trajectories; they can imitate the target motion through trial and error and confirm their trajectory on the MSM using the trajectory of the expert.

In this study, we aim to evaluate a proposed framework for visual-feedback training by a verification experiment using motion information such as muscular activity. In this paper, we focus on the pedaling function as a cyclic motion. This motion was executed on a bicycle ergometer so that the subject could easily refine his motion based on the feedback received from a circular target trajectory displayed by the proposed algorithm on the MSM. To verify the validity of the framework for the visual-feedback system, we quantified the lower limb movements in the pedaling motion of a subject with electromyogram (EMG) sensors and inertial measurement units (IMUs) using the proposed feedback training.

2. Materials and Methods

2.1. Framework for Visual-Feedback Training

To make it easier to understand the difference between the subject’s and target’s motor skills in a cyclic movement, we used the modified SOM algorithm and developed a visual-feedback system that displayed the target motion as a circular trajectory on the MSM (see Figure 1). For feedback training, the subject observed only the displayed static target circle and the subject’s real-time (i.e., varying) trajectory constructed from the subject’s real-time motion. The subject imitated the desired rhythmical motion through trial and error by observing cyclic trajectories even when the target cyclical motion included high-dimensional time-series information. Details of the original and modified SOM algorithms are described in our previous studies [3,4,5,6].

2.2. Experimental Conditions

In this study, we focused on the pedaling motion to evaluate the proposed training system. For training using visual feedback, a bicycle ergometer (Mirage-PRO, Fujimori Co., Ltd., Toyama, Japan) was used to imitate the target motor skills. The experiment involved one participant, who was an adult male (age: 23 years, height: 167 cm, weight: 50 kg). To detect the participant’s pedaling motion, we fixed wireless IMUs (SS-WS1103, Sports Sensing Co., Ltd., Fukuoka, Japan) at the following positions: the crank axis, the femur, a lower leg, and a pedal (see Figure 2). Furthermore, to quantify muscle activities, wireless surface EMG sensors (LP-IW2PADW, Logical Product Co., Fukuoka, Japan) were fixed to the following positions: the rectus femoris, the hamstrings, and the lateral head of the gastrocnemius muscle (see Figure 2). The angles of the crank and body parts were calculated in the world coordinates by using the acceleration detected by the inertial sensors. The recorded EMG signals were processed for the integrated EMG (IEMG) signals. All sensors recorded at the sampling rate of 100 Hz and were able to process/transfer motion data in real time. The seat position was adjusted for the subject to enable comfortable pedaling. All pedaling was performed at the same slow cadence (30 rpm) so that feedback training was performed with an awareness of one’s own motions.

To construct the MSM for feedback training, the subject wore sensors and performed pedaling motions in two ways before the training session: (i) spontaneous pedaling called “target” and (ii) heel-raised pedaling called “Poor”. The recorded data of the two motion patterns were inputted for the modified SOM algorithm, as described in Section 2.3. In the training session, the subject aimed to change his measured motion from Poor to target using only those trajectories for reference, which were capable of projecting the measured motion data successively on the MSM in real time.

2.3. Constructing the Motor Skill Map

To visualize the subject’s pedaling motion as trajectories on a two-dimensional map, the MSM was constructed using the modified SOM algorithm described in [3,4]. To build the MSM, we used 6400 neurons arranged as an 80 × 80 grid. The input vector x from the subject motion contained the angle of the crank and the lower limbs, as well as the IEMG signals of the rectus femoris, hamstrings, and gastrocnemius muscle. The motor skill can be expressed as a function of the information about these seven motions as follows:

$$x\left(t\right)=\left\{w_1{\theta }_{crank}\left(t\right),w_2{\theta }_{femur}\left(t\right),w_3{\theta }_{leg}\left(t\right),w_4{\theta }_{pedal}\left(t\right),w_5{A}_{RF}\left(t\right),w_6{A}_{HAM}\left(t\right),w_7{A}_{GAS}\left(t\right)\right\},$$

(1)

where θ_crank is the crank angle; θ_femur is the femoral angle; θ_leg is the angle of the lower leg; θ_foot is the pedal angle; A_RF is the IEMG value of the rectus femoris; A_HAM is the IEMG value of the hamstrings; A_GAS is the IEMG value of the gastrocnemius muscle; and w_i are the weight parameters that serve to normalize the motion information from 0 to 1 using the minimum/maximum values in pre-training. Using the w_i parameters, we can consider the priority of motor skills recorded using sensors as weighting parameters and simultaneously deal with various motion data even when the input vector included different quantities, such as the angle, velocity, force, and muscle activity. The learning sample data (set for the SOM algorithm) were collected from one cycle by the subject’s two pedaling motions, target and Poor, at the sampling frequency of 18 Hz. Thus, the learning samples from each motion filled 36 vectors. All 72 vectors were used in the learning process. The total number of iterations in the learning phase was set to 1 × 10⁴. The learning rate and the neighborhood radius were set to 0.8 and 20, respectively. (The diameter of a neuron is defined as unity.) After the SOM learning, the input vectors of the target motion used in the SOM learning phase were projected as a circular trajectory before training, and the input vectors of the Poor motion were projected as outside the target circle according to the Euclidean distance. The two circles were divided into 36 sections of 10° each. Next, the subject wearing the sensors was trained with visual feedback using the proposed system, and we tried to make his Poor motion approach the target motion in real time by referring to the two cyclic trajectories on the MSM.

3. Results and Discussion

We evaluated the motor skills by varying the seven pedaling movements during visual-feedback training. Figure 3 shows the trajectories on the MSM of the feedback system during the training session. The obtained MSM background was colored by the vector component of each neuron. Figure 3a shows the projection of the motion data. Initially, the subject consciously performed the Poor motion. Figure 3b shows the motion data obtained when the subject tried to bring his trajectory closer to the target circle in the final stage of training. We confirmed that the trajectory obtained for performing the Poor motion in the initial stage of training was projected outside the target circle. The trajectory in the final stage was similar to the target circle. Thus, the subject succeeded in making his trajectory approach the target circle on the MSM using feedback training. It is assumed that the cyclic movement makes it as easy as possible for the subject to align his trajectory with the target one because the subject can repeat his movement continually without intermission. Figure 4 shows two sets of motor skills information for the initial (see Figure 4a) and final stages (see Figure 4b) of the experiment corresponding to Figure 3a and Figure 3b, respectively. When the trajectory on the MSM was projected far from the target circle, the motion information was also recorded far from the target information (see Figure 4a). However, when the trajectory was close to the target circle, the motion information was closer to the target information (see Figure 4b). Therefore, we proved that approaching the subject’s trajectory toward the target circle on MSM meant that the subject’s motion mimicked the target motion, including the use of muscles. Specifically, the subject can learn hidden information (i.e., target motion comprising high-dimensional motion data) to refer to two-dimensional information as trajectories on an MSM through trial and error. It is expected that the subject is successful in developing his internal model, i.e., a correlation map between body part motion and muscle activity. We concluded that the proposed framework for the visual-feedback training could help a subject imitate the desired motor skills even if the motion information was composed of high-dimensional time-series motion data. In future work, the target motion can comprise optimal motion produced by a numerical simulation of human movement. To adopt the optimized motion for the proposed system, subjects can perceive their optimal movement from the projected target trajectory on the MSM. Furthermore, the target motion can be adopted in the proposed framework, not only for expert motion but also for inefficient motion. To imitate the inefficient motion based on MSM trajectories, a subject can further exert his body and perform various muscular exercises.

4. Conclusions

We evaluated the motion performance of a subject using the proposed feedback training. To make it easier for subjects to imitate target motor skills, our proposed system uses a modified SOM algorithm capable of reducing high-dimensional motion data. The difference between the target motion and subject motion was visualized in terms of a separation between the two motion trajectories displayed on the MSM. To facilitate reproduction of the complete cyclic motion, the target motion was displayed on the MSM as a circular trajectory. By including invisible information such as muscular activity, we confirmed that when the trajectory was close to the target circle on the MSM, the actual motion information was also close to the target motion. In our future work, this system will be further developed by measuring additional motion data and adapting the method to additional complex motions.