Human Gait Data Augmentation and Trajectory Prediction for Lower-Limb Rehabilitation Robot Control Using GANs and Attention Mechanism

Abstract

To date, several alterations in the gait pattern can be treated through rehabilitative approaches and robot assisted therapy (RAT). Gait data and gait trajectories are essential in specific exoskeleton control strategies. Nevertheless, the scarcity of human gait data due to the high cost of data collection or privacy concerns can hinder the performance of controllers or models. This paper thus first creates a GANs-based (Generative Adversarial Networks) data augmentation method to generate synthetic human gait data while still retaining the dynamics of the real gait data. Then, both the real collected and the synthesized gait data are fed to our constructed two-stage attention model for gait trajectories prediction. The real human gait data are collected with the five healthy subjects recruited from an optical motion capture platform. Experimental results indicate that the created GANs-based data augmentation model can synthesize realistic-looking multi-dimensional human gait data. Also, the two-stage attention model performs better compared with the LSTM model; the attention mechanism shows a higher capacity of learning dependencies between the historical gait data to accurately predict the current values of the hip joint angles and knee joint angles in the gait trajectory. The predicted gait trajectories depending on the historical gait data can be further used for gait trajectory tracking strategies.

1. Introduction

Gait disturbances affect autonomy but above all peoples’ quality of life. Walking recovery after gait disorders can be a long, labor-intensive process, but robotic assisted therapy (RAT) can help [1,2,3]. Robotic devices are designed for lower-limb rehabilitation, powered orthoses with computer-controlled motors, and to support the joint movement by improving patients’ locomotor ability, balance impairments, or muscle control ability [4,5]. In this way, the RAT can increase rehabilitation intensity and frequency, enhancing functional recovery even with minimal guidance from therapists and without the association with another rehabilitative approach [6].

There are already a range of robotic devices commercially available [7]. The robotic devices for gait rehabilitation can be grouped into three categories: body weight-supported treadmill (BWST) exoskeleton devices, end-effector devices and wearable lower-limb exoskeletons (WLLEs) [8]. BWST exoskeletons involve a harness that supports an adjusted percentage of a patient’s body weight, while robotic orthoses control hip, knee, and/or ankle movement patterns during gait. The most popular BWSTT exoskeleton can be Lokomat, which has been used for over 280 gait rehabilitation studies with different patient populations [9]. End-effector devices instead provide bodyweight support with the use of a harness by strapping the patient’s feet and ankles onto foot-plates that mimic the trajectory of gait [10]. A typical end-effector device is the Gait-trainer GT II that provides functional electrical stimulation to up to eight muscle groups. A GT II could offer practical gait training for early-stage rehabilitation of orthopedic and impaired neurological patients [11].

Wearable lower-limb exoskeletons used in gait training aiding recoveries can be over ground, which provide support, protection and therapy to reduce the burden on the limbs and assisting patients to complete daily living activities, like walking, squatting, sitting-to-standing, going up and down stairs [12]. Wearable exoskeleton Phoenix is designed for clinic and community ambulation. It has motors that control hip and knee movements, allowing for a couple of hours’ continuous walking [13]. Whilst, WLLEs for gait rehabilitation are still in their early stages of development, partially due to the following underlying reasons. First, bulky wearable devices hinder mobility and independence in the use of walking aids; second, there is a need for standardised measures for therapy protocol and assessment. Consequently, precise and timely control trials are required to demonstrate WLLEs’ clinical evidence for patient improvement.

The main objective of lower-limb rehabilitation training is to restore the ambulatory functions of patients to normal levels [14]. To assist gait in WLLEs, researchers explore multiple control strategies for rehabilitation. One of the main identified trends in WLLEs’ control strategies is trajectory tracking control [15]. The related investigations in gait rehabilitation suggest the patients with similar gait disorders have homogeneous gait patterns [1]. Patients often follow a predetermined trajectory in their rehabilitation. Thus, in the process of rehabilitation training, a normal gait pattern is required as a reference input to the control system, whilst the normal gait pattern of patients cannot be directly measured due to their impaired motor functions. The predetermined trajectories can be obtained from normal gait data collection. The relevant characteristics of normal human gait are used for human walking gait rehabilitation. However, the high cost of data collection and privacy concerns make human gait data scarce. Due to the limited amount of human gait data, matching the obtained gait data with different rehabilitation trajectories tracking is difficult. It can explain why data augmentation methods, like Generative Adversarial Networks (GANs), are developed to generate additional synthetic data based on original data [16].

The predefined gait trajectories, that is, joint angles, angular velocities, angular accelerations, joint torques and so on [17] are often used as the reference input of the controller in trajectory tracking control. Predicting the evolution of the gait trajectory can be critical to contribute to intended movements, which can make the control more timely in online use. Meanwhile, a larger amount of human gait training data benefits from improving the prediction performance and reducing overfitting. Based on the above two issues, this paper explores creating a Generative Adversarial Network (GAN)-based augmentation model to generate more gait data while retaining the dynamics of the real collected data. Based on the augmented gait data, we also create an attention-based model for gait trajectory prediction to provide timely data for the controller of a trajectory tracking strategy in WLLEs.

2. Related Work

The conventional data augmentation methods are usually based on transformations, such as flipping, rotating, adding random noise, cropping, scaling, random warping, etc. [18]. Authors in [19] proposed an approach using Dynamic Time Warp Distance (DTW) to alleviate the overfitting problem of small time-series data sets. Researchers in [20] applied multiple methods augmenting wearable sensor data to automatically classify the motion state of patients. The results suggested the combination of rotation and arrangement augmentation improved the classification accuracy to 86.88% from 77.52%. Most of the above transformation-based methods are designed according to intuitive experiences, thus only slightly generating new data. There also exists a diverse amount of time series with different properties, and each transformation is not applicable to all datasets.

The generative adversarial networks (GANs) approach, in turn, is promising especially in dealing with scarce image data by generating realistic-looking images [16]. In order to generate new data, based on the real data and sampled random vectors, GANs utilise an adversarial training mechanism to jointly optimize two neural networks, that is, a generator and a discriminator. There is already a range of time series GANs proposed for data augmentation. A certain author has proposed a GAN structure for wearable sensor data augmentation in [21], which was established based on Gaussian mixture density networks and recurrent neural networks. The generator failed to learn from the discriminator well since the generator and the discriminator were trained separately. Also, unlabeled data limited the supervised learning task. The authors of [22] designed a recurrent generative adversarial network for generating medical time series data, whilst no supervision during training made the training unstable.

The authors of [23] proposed a novel method generating realistic time-series data. They combined the unsupervised GAN with the supervised autoregressive models to encourage their designed network to preserve temporal dynamics of the real time series data. They empirically evaluated the superiority of their method in terms of measures of predictive ability and similarity. Researchers in [24] introduced a novel method to reduce high data collection costs in fingerprint-based localization tasks. They employed GANs to learn the distribution of the limited collected data and to generate synthetic data. Experimental results showed they obtained essentially similar positioning accuracy from 10% of collected data and 90% of synthetic data as that from the whole collected samples. It implied that, with GAN-generated synthetic data, the acceptable accuracy was achieved only using 10% of the real data, thereby reducing data-collection costs. The above GANs-based methods demonstrate that data augmentation on the one hand is beneficial for data collection cost reduction. Additionally, the augmented data help to obtain the competitive performance in classification or prediction tasks with much fewer observations.

To appropriately guide patients following their gait trajectory in WLLEs, understanding their normal gait is needed to analyze the characteristics of gait patterns [25]. Human gait analysis is commonly performed using foot contacting force/pressure sensors [26], electromyography (EMG) sensors [27], inertial measurement units (IMU) [28] or optical motion capture devices [29]. Gait analyses are carried out using machine learning methods. The authors of [30] employed a decision tree estimating the gait phrase based on the feet loads and segmented IMU data. The authors of [31] presented a novel method for gait phase classification. They modified a Random Forest algorithm to foresee an initial phase of gait data, classifying the gait phase (stance or swing). The gait data used were acquired from a power gait orthosis: the cameras embedded in the crutches used by a user. The above two applications for gait analysis using machine learning methods focused on the uni-variate gait phrase data without considering multi-dimensional gait data.

Differently, data trajectories required in WLLEs, such as joint angles and joint torques, are usually deduced from multi-variate gait data [17]. The prediction of multivariate gait trajectories can be difficult, since it adds the complexity of order or temporal dependencies between observations. A statistical continuous variables model in [32] was developed. The model estimated and predicted the evolution of the joint trajectories by taking both the variations on the trajectories and the nominal joint trajectories into account. Yet, the model highly relied on the observations, unable to comply with joints. In [33], an anthropometric features-based prediction method was presented for patient-specific gait trajectories. The method applied Fourier coefficients to represent gait patterns and then deployed a Random Forest algorithm to learn the dependencies between human gaits and the anthropometric features for gait prediction. The above machine learning methods considered the dependencies between multi-dimensional time series gait data, whilst they shared a common problem of requiring manually extracted and carefully selected features to fit prediction models.

Deep neural networks (NNs) can instead learn complex mappings automatically from input to output, providing the potential for multidimentional time series prediction tasks without manually extracted features. Particularly, the modified version of recurrent neural network (RNN), that is, Long Short-Term Memory (LSTM), is designed to learn mapping functions from input over time to output [34,35]. LSTM has been widely employed for gait analysis especially with time-series data. For predicting the falls of older people, study in [36] applied the LSTM models for gait sequence prediction. The gait sequences were from nine-axis accelerator gyroscope angle sensors. Experimental results indicated the LSTM model performed well for predicting of both multi-variate and uni-variate gait data since it considered the dependencies between the gait data over time. Gait event detection can help identify and assess gait abnormalities. Authors in [37] used a modified LSTM network to detect toe offs and heel strikes during the user’s gait cycle based on accelerometer-based data.

Meanwhile, encoder–decoder architectures have become one popular way of organizing RNN or LSTM networks for sequence-to-sequence prediction problems [38,39]. An encoder–decoder architecture can be developed where a source sequence is read in entirety and encoded to a fixed-length vector. A decoder network then uses the compressed representation that is expected to be a good summary of the entire input sequence to generate the transformed output (the target sentence). RNN or LSTM networks can be used for both the encoder and the decoder. One problem with the architecture above is that the performance will deteriorate in long input or output sequences. The reason is because the fixed-length vector design causes the system to be incapable of remembering longer sequences. The attention mechanism is an extension that addresses this limitation of the encoder-decoder architecture on long sequences [40,41]. It was originally developed for machine translation problems and it has been proven successful at related sequence-to-sequence prediction problems. Attention-based encoder-decoder networks provide a richer context from the encoder to the decoder and a learning mechanism. The decoder can learn where to pay attention in the richer encoding when predicting each time step in the output sequence. A novel multivariate time series prediction model, composed of an influence attention mechanism-based encoder and a temporal attention mechanism-based decoder, was proposed [42]. It was proved that the model could outperform some baseline models. The study in [43] combined a temporal attention mechanism with a graph convolution network. The model was used to capture the dynamic temporal feature and the inner relationship for bike-sharing prediction.

Back to the exoskeleton robot tracking control systems, the dynamics of a two-link rigid robot can be written as Equation (1), where $D\left(q\right)$ represents the inertial matrix, $C(q,\dot{q})$ is the centrifugal and Coriolis torque vector matrix, and the gravity matrix is demoted as $G\left(q\right)$ [44,45]. Normally, the collected human gait data are used to compute the gait trajectory values, for example, the joint angle q, the angular velocity $\dot{q}$, the angular acceleration $\ddot{q}$ and the corresponding joint torque value $\tau$. These trajectory values can be further used as the reference input of the designed controller to drive the actuator of the rehabilitation robot. To maintain the continuous action of the robot, the joints target torques are needed to compute at each time step of the main control loop. This mechanism cannot drive the robot until the values in Equation (1) are obtained after acquiring humans’ gait trajectories, which can lead to a lack of timely control for online control.

$$D\left(q\right)\ddot{q}+C(q,\dot{q})q+G\left(q\right)=\tau$$

(1)

Our previous work [46] employed LSTM models for the prediction of univariate gait data one dimension be one. The prediction results can be used to timely compute the gait trajectory values in Equation (1). This paper aims to extend the previous work by first augmenting the collected real gait data and then predicting the current gait trajectory values needed in Equation (1) from the historical multivariate human gait data. The human gait data collected from our data capture platform are multivariate time sequences with complex correlations and dynamic changes. To augment the gait data, we create a GAN-based model to generate more gait data while retain the dynamics. To improve the performance of gait trajectory prediction, we also construct a two-stage attention model and a baseline LSTM model. The predicted gait trajectory values can also be used for other control schemes, such as reinforcement learning or optimal controls.

3. Materials and Methods

3.1. Framework

Figure 1 illustrates the framework of human gait data augmentation and gait trajectory prediction in the paper. The core of the design is the GAN-based gait data augmentation model and the attention-based gait trajectory prediction. Specifically, the collected limited real gait data from healthy subjects are first augmented by the created GAN-based model to generate more synthetic gait data for the gait trajectory prediction task. Then both the real and the synthesized gait data are preprocessed and fed to the constructed attention-based and baseline LSTM prediction models respectively. The best prediction results are presented accordingly after optimizing the parameters of the prediction models. The predicted gait trajectory values required in Equation (1) can be transformed to the corresponding joint torque $\tau$ values in further work for rehabilitation robot controls.

3.2. Data Acquisition

The human gait data in the work are acquired from a NOKOV passive infrared optical motion capture platform [47], referring to the data collection setting in Figure 1. There are six cameras uniformly spaced in two lines along the walls in the lab environment. The data collection associated procedures were approved by Zhongyuan University of Technology Research Ethics Committee (Project identification code ZUTSEI202008-001). All subjects gave their informed consent for inclusion before they participated in the data collection. During the data collection, each participant is instructed to walk at his/her own normal way under the setting of NOKOV. The captured gait data are recorded in the format of three-axis position values from each of the six mark points placed on human body (see Figure 1).

Walking is achieved by coordination between the pelvis, hip, knee, and ankle [15]. Also, this paper only explores the prediction of certain trajectory values in Equation (1), which are from the recorded gait data on the hip, knee and ankle. Therefore, we only attached six marks to the hip, the knee and the ankle of each body side of a subject in the study for gait data collection. As long as two lenses capture a mark point, the corresponding axis-based position values are measured referring to the calibration point on the ground. Figure 2 shows the ground truth for a certain duration from each of five participants’ three mark points on the right body side. Thus, in each subfigure, the nine ordinate-axis values denote the position coordinate values in terms of X,Y,Z axis at each of the three mark points (the hip, knee, angle) on the right lower extremity.The horizontal-axis value is the time sequence orders for all the nine channels. The five healthy subjects in Figure 2 are aged 22 to 26, with an average height of 1.722 cm. Four of them are male except subject one. However, they show the homogeneous gait patterns with similar stride cycles in terms of the axis position values at the corresponding marks.

3.3. Models

3.3.1. GAN-Based Gait Data Augmentation Model

The fundamental networks of GANs for time series include GANs with temporal CNNs, GANs based on fully-connected networks, recurrent GANs, and so forth [18]. The synthetic data by GANs should preserve temporal dynamics of the real data, thus the augmentation model should not only be able to capture the feature distributions of the real data, but to capture the complex dependencies of the features over time. Inspired by the work in [23,24], we complete a GAN-based multi-dimensional time series model to learn and generate new synthetic gait data. The pipeline of the data augmentation model is shown at the lower part in Figure 3. The augmentation model comprises four sub networks: a generator, a discriminator, with an extra embedding network and an extra recovery network. Each of the four networks play different roles in modelling the data. Both the generator and the discriminator are implemented with a multi-layer GRU network, they generate and distinguish synthesized data respectively. The supervised and adversarial objectives are jointly optimized through the learning embedding network and recovery network to remain the dynamics of the training data, which are implemented with an auto-encoder structure.

In the data augmentation model, there are three losses included: the reconstruction loss, the supervised loss and the adversarial loss. The reconstruction loss is based on the auto-encoder network (embedding and recovery), responsible for comparing how well the reconstruction of the encoded data is compared with real data. The supervised loss captures how well the generator approximates the next time step. The adversarial loss reflects the relation between the generator and discriminator networks. The corresponding training phases include: (1) training the autoencoder on the provided sequential gait data for optimal reconstruction; (2) training the supervisor using the real sequence gait data to capture the temporal behavior of the historical information, and (3) the combined training of four networks while minimizing all the three loss functions mentioned previously.

3.3.2. LSTM Prediction Model

For the gait trajectory prediction based on both the real and augmented synthetic gait data, we construct two prediction models. We first establish a multi-variate temporal LSTM prediction model as the baseline prediction model. LSTM maintains RNN’s ability to learn the temporal dynamics of sequential data but also retain the long-term information in the prediction of time series. The two-layer LSTM prediction network is shown in

Table 1. The structure information of the LSTM model.

Layer (Type)	Output Shape	# Parameters
lstm (LSTM)	(None, 1, 64)	17,664
lstm_1 (LSTM)	(None, 32)	12,416
dense (Dense)	(None, 1)	33

with its basic details. The parameters of LSTMs are further optimized according to the specific prediction tasks detailed in the following sections.

3.3.3. Attention-Based Gait Trajectory Prediction Model

Conventional encoder–decoder structures ignore the intermediate states of the encoder. Only the output of the final state initializes the decoder for decoding and prediction after the encoder processes the input data into a fixed-length vector. Consequently, inspired by how human brain processes input information with attention mechanism, we add two attention layers into the LSTMs-based encoder–decoder architecture, referring to the upper part in Figure 3. The underlying idea is to introduce a layer of feature attention in the encoder, this enables the encoder to handle the long-term dependencies of drive sequences over historical time steps, also learn and select more correlated features in the input data. At the same time, the introduction of a temporal attention in the decoder is to evaluate all relevant encoder states and and to learn information for the prediction of the gait trajectory values by performing learning with different weights.

As shown in Figure 3, given the time-series gait data (the real $X_0$ or the synthetic gait data $X_1$), $X={(x^1,x^2,\cdots ,x^m)}^T=(x^1,x^2,\cdots ,x^T)\in R^{m\times T}$ and the target series $y=(y_1,y_2,\cdots ,y_{T-1}),y_t\in R$, the attention-based model aims to find the nonlinear mapping by retrieving the long-term temporal information from the encoded inputs to predict the current value of the target series ${\widehat{y}}_T$. Compared to a conventional encoder-decoder structure that discards all the intermediate states of the encoder only use its final states to initialize the decoder, the model in Figure 3 first utilizes a feature attention (multi-layer MLPs) in the encoder to learn the relevant features from the input time series ($x_t$) and map them to new representations ${\overline{x}}_t$, rather than treating all the input series equally. To predict the target series ${\widehat{y}}_T$, in the decoder a temporal attention operation (multi-layer MLPs) is designed to select the relevant encoder hidden states required by decoder across all the states ($h_1$,$h_2$,...,$h_T$) instead of only the final time steps ($h_T$).The output of the second attention is the weighted summed context vectors ($c_t$), which are concatenated with the decoder input ($y_{t-1}$) as the updated decoder input.

3.3.4. Experimental Setup

The critical values in Equation (1) are needed for rehabilitation control design. This study only picks the joint angle q for prediction tasks to explore a potential method future online control use. We use five healthy subjects’ gait data captured by the cameras with the marks on body. The real data examples are plotted in Figure 2. Due to the space limitation in the lab setting, each data collection trial only can last a couple of seconds. Thus, in this work, we establish a GAN-based augmentation method to generate more synthetic gait data based on the collected real data. According to the inverse kinematics analysis method [45], the human lower-limb hip joint angle $q_{hip}$ and the knee joint angle $q_{knee}$ can be obtained from expressions (2) and (3):

$$q_{hip}=arctan\frac{kne{e}_x-hi{p}_x}{kne{e}_y-hi{p}_y}$$

(2)

$$q_{knee}=q_{hip}-arctan\frac{ankl{e}_x-kne{e}_x}{ankl{e}_y-kne{e}_y}$$

(3)

where x represents the x-axis value and y represents the y-axis value on the corresponding mark point, for example, $kne{e}_x$ means the x-axis value based on the mark point placed on the knee on one body side. Thus, the hip joint angle is based on four time series in expression (2), and the knee joint angle is derived from the six time series in expression (3).

Table 2. Details of the datasets.

Dataset	Subject	# Real Data	# Augmented Data	# Features	Target Value
Hip joint	Subject 1	448	1776	4	Hip joint angle
	Subject 2	429	1700	4	Hip joint angle
	Subject 3	394	1558	4	Hip joint angle
	Subject 4	606	2408	4	Hip joint angle
	Subject 5	288	1136	4	Hip joint angle
Knee joint	Subject 1	448	1776	6	Knee joint angle
	Subject 2	429	1700	6	Knee joint angle
	Subject 3	394	1558	6	Knee joint angle
	Subject 4	606	2408	6	Knee joint angle
	Subject 5	288	1136	6	Knee joint angle

lists the real gait datasets from the five subjects. For the data augmentation tasks, we separately augment the five subjects’ gait data according to the prediction tasks shown in Table 2. Both the synthetic and real multi-dimensional gait data will be used to for gait trajectories prediction. For the prediction of the hip joint angles and the knee joint angles, the details about the real datasets are listed in Table 2. For different subjects, the sample numbers of the real data sequences are different, referring to Figure 1 and Table 2. The sequence numbers of the augmented data are different accordingly. We will explain the details of the augmented data in the next section. For all datasets in Table 2, we use 70% of the instances for train and 30% for test in all the prediction tasks. It is noted that all the above models and the following evaluation measures are coded with Python 3.7.1 and tested in the environment of Torch 1.8.1

3.3.5. Evaluation

t-SNE

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear technique for dimensionality reduction, which is particularly well suited for the visualization of high-dimensional datasets. We use t-SNE to project the augmented high-dimensional gait data into a low-dimensional space for the similarity visualization.

MAE

Root mean square error (RMSE) and mean absolute error (MAE) are two of the most common metrics to measure accuracy for continuous variables. We use them to score the prediction performance in our work. MAE measures the average over the verification sample of the absolute values of the errors between paired predicted and actual values. It is defined as:

$$MAE=\frac{1}{n}{\sum }_{j=1}^n\left|y_j-{\widehat{y}}_j\right|$$

(4)

RMSE

The RMSE gives a relatively high weight to large errors, which measures the averaged magnitude of each squared difference between paired actual value and corresponding prediction. It is calculated as:

$$RMSE=\sqrt{\frac{1}{n}{\sum }_{j=1}^n(y_j-{\widehat{y}}_j)}$$

(5)

where n indicates the number of prediction samples, ${\widehat{y}}_j$ represents the predicted target values, and $y_j$ represents the actual data.

4. Results

4.1. Gait Data Augmentation

We use the developed GAN-based model in Figure 3 to augment the limited gait data from five subjects by synthesizing new data based on the real data. Figure 4 shows the comparisons of the real and synthetic gait data from three of the five subjects. The six subfigures in each column represent the six-channel values based on Equation (3) used for the prediction of the knee angles. The curves in green are the real gait data and the curves in orange are the synthetic data. We can see that the orange curves can basically follow the dynamics of the green ones. Also, the six values in each column in Figure 4 are all augmented approximately four times in terms of the number of the data sequences. The specific number for each augmented dataset can refer to Table 2.

The GAN-based data augmentation models aim to synthesize the data that have the similar distribution with the real data. To illustrate this, in Figure 5 we also present a visual comparison of the real and synthetic gait data after dimensionality reduction by the t-SNE algorithm. We can observe in Figure 5 that the synthetic gait data (the orange samples) and the real data (the green samples) have similar distributions and they show prominent overlap with each other. Figure 5 and Table 2 together indicate that synthetic datasets generated and the real datasets are basically in sync and the GAN-based data augmentation model for multidimensional time series can synthesize realistic-looking human gait data. The synthetic data plus the real gait data will be used for the prediction tasks of gait trajectories.

4.2. Gait Trajectory Prediction

In this work, we only perform the gait trajectory prediction for one body side of each subject. Each subject’s real and augmented gait data from the hip and the knee mark points are used. According to Equations (2) and (3), four values are needed for hip angle prediction and six values for knee angle prediction. The details of each dataset can refer to Table 2. Time series prediction is usually performed through sliding time-window feature and make prediction depend on the order. We develop the baseline prediction model LSTM and the attention-based models to predict the current values of the target series in Table 2.

The structure of the LSTM is shown in Table 1, which is optimized by tuning the key parameters apart from the number of the LSTM layer. The optimizer of Adam (lr = 0.01, beta_1 = 0.9, beta_2 = 0.999, epsilon = 1 × ${10}^{-8}$, decay = 0.0) works well for all the datasets in Table 2. As to the attention-based models, the parameters of encoder_hidden_size = 128, decoder_hidden_size = 128, timestep = 16, learning_rate = 0.01, batch_size = 32 work best for the three hip angle datasets; and the parameters of encoder_hidden_size = 32, decoder_hid den_size = 32, timestep = 16, learning_rate = 0.01, batch_size = 32 perform well for the three knee angle datasets. We firstly compare the gait trajectory prediction performance of the LSTM and attention models on real datasets from subject 1 in Figure 6, in which we can see that the red curves (test values) based on the attention models seem more perfectly follow the actual data on both the hip and knee datasets. The LSTM model performs well on the hip dataset whilst it cannot well handle the knee dataset. The prediction results on the synthetic datasets from subject 1 is shown in Figure 7. Similarly, the attention-based models perform better than LSTM models on the synthetic data since the red curve (test results) can follow the actual data better especially on the knee angle prediction.

The specific prediction results in terms of RMSE and MAE for all subjects are listed in

Table 3. Prediction performance of models on real data.

Model	Dataset	Hip Joint Angle		Knee Joint Angle
		RMSE	MAE	RMSE	MAE
LSTM	Subject 1	0.0620	0.0430	0.0190	0.0160
	Subject 2	0.1700	0.1530	0.0530	0.0360
	Subject 3	0.2700	0.2090	0.0790	0.0570
	Subject 4	0.0800	0.0660	0.1230	0.1020
	Subject 5	0.1380	0.1100	0.2650	0.2390
Attention-based	Subject 1	0.0129	0.0112	0.0116	0.0101
	Subject 2	0.0031	0.0027	0.0074	0.0069
	Subject 3	0.0083	0.0066	0.0280	0.0221
	Subject 4	0.0056	0.0039	0.0069	0.0051
	Subject 5	0.0062	0.0050	0.0270	0.0236

and

Table 4. Prediction performance of models on augmented data.

Model	Dataset	Hip Joint Angle		Knee Joint Angle
		RMSE	MAE	RMSE	MAE
LSTM	Subject 1	0.0620	0.0430	0.0190	0.0160
	Subject 2	0.1700	0.1530	0.0530	0.0360
	Subject 3	0.2700	0.2090	0.0790	0.0570
	Subject 4	0.0360	0.0260	0.0560	0.0270
	Subject 5	0.0780	0.0640	0.0670	0.0460
Attention-based	Subject 1	0.0035	0.0032	0.0092	0.0084
	Subject 2	0.0039	0.0033	0.0074	0.0069
	Subject 3	0.0067	0.0060	0.0320	0.0284
	Subject 4	0.0131	0.0051	0.0218	0.0079
	Subject 5	0.0049	0.0037	0.0242	0.0570

. Table 3 exhibits the two models’ prediction performance on the real dadasets, and Table 4 presents the performance on the augmented data. The two tables give the same results that the attention-based prediction model perform better than the LSTM models on both the real and synthetic gait data. Figure 8 and Figure 9 further detail the prediction results of hip joint angles and knee joint angles across all the subjects. The attention-based models markedly beat all the LSTM models apart from the dataset of the third subject’s augmented data.

5. Discussion

The GAN-based model augments the scarce gait data and the attention-based prediction model acquires the current values of gait trajectory values from the historical data. Based on both the real and synthetic data, we compare the attention-based model with the LSTM baseline. All the models are evaluated with the same datasets and same measures. Figure 6, Figure 7, Figure 8 and Figure 9, Table 3 and Table 4 together illustrate that more accurate prediction results are provided by the attention-based model across all datasets from the five subjects. This could verify that LSTM is less capable of capturing the dependencies between long-range sequences; whilst in the attention-based model the feature attention is able to pick the relevant series for driving the prediction and the attention linking the encoder and decoder contributes to the acquisition of the long-range dependencies from the encoded representations. We use both MAE and RMSE to measure the variation in the errors for the prediction performance. We can conclude that all the RMSE results in Table 3 are larger or close to the MAE for both models, the greater difference between the models, the greater variance in the individual errors in the sample. Yet, comparing Table 3 with Table 4, Figure 8 with Figure 9, we observed that the performance on the augmented data is slightly worse than that on the real data for both LSTM and attention models. This might be due to that the amount of the augmented data are four times of the real data thus the prediction errors may accumulate more on the larger dataset.

It is noted that the X-axis values in Figure 2 decrease gradually instead of giving a periodic pattern, like the Y or Z values. This can be explained that the space equipped with the camera array for data collecting was limited and the participants only walked forward during data collection without turning back to the calibration point, which causes the decrease of X values only in one direction. In this case, the x values might be less relevant to the target series, although they are needed according to expressions (2) and (3).

This work still has some limitations. First, we only demonstrated the results based on five subjects’ gait data in the augmentation and prediction tasks. The future work should consider using more subjects’ gait data for a general model’s training or more data from one specific subject for a customized model’s training. Second, we only completed the prediction of part of the trajectory variables in Equation (1). We thus could not obtain all the current gait trajectory values needed for gait trajectory tracking control. This leads to the missing evaluation on the conjunction with RAT and the usefulness in the rehabilitation of the gait. The prediction of all the required trajectory variables should be completed in future work to further verity the role of gait trajectory prediction in RAT.

6. Conclusions

In typical rehabilitation control, values of the joint angle, the angular velocity, the angular acceleration and the joint torque in Equation (1) are required in the control loop to be the inputs of the designed controller; the controller then produces outputs based on the inputs to drive the actuator of the rehabilitation robot. The control mechanism could work well only if the gait trajectory values are accurately obtained first, otherwise, it may cause control delays for online use. On the other hand, the scarcity of gait data may hinder the accurate acquisition of gait trajectory values. This paper thus completes a framework that can first generate more human gait data and then use them for gait trajectory prediction. The generated new data maintain a similar distribution and dynamics to that the real gait data. We construct the LSTM and the attention-based models to capture the dynamic dependencies among the gait data time series for accurate prediction. The experimental results show the satisfied performance of the attention-based model on predicting the target values from the multivariate gait data. Our future work will be further extended by: (1) generating and training more gait data; (2) predicting all the variables needed for real control use in Equation (1). In addition, we will also focus on reducing the dimensionality of the input data to see the contribution of different axis values on different body parts.