Time–Frequency Feature Fusion Approach for Hemiplegic Gait Recognition

Abstract

Accurately distinguishing hemiplegic gait from healthy gait is significant for alleviating clinicians’ diagnostic workloads and enhancing rehabilitation efficiency. The center of pressure (CoP) trajectory extracted from pressure sensor arrays can be utilized for hemiplegic gait recognition. Existing research studies on hemiplegic gait recognition based on plantar pressure have paid limited attention to the differences in recognition performance offered by CoP trajectories along different directions. To address this, this paper proposes a neural network model based on time–frequency domain feature interaction—the temporal–frequency domain interaction network (TFDI-Net)—to achieve efficient hemiplegic gait recognition. The work encompasses: (1) collecting CoP trajectory data using a pressure sensor array from 19 hemiplegic patients and 29 healthy subjects; (2) designing and implementing the TFDI-Net architecture, which extracts frequency domain features of the CoP trajectory via fast Fourier transform (FFT) and interacts or fuses them with time domain features to construct a discriminative joint representation; (3) conducting five-fold cross-validation comparisons with traditional machine learning methods and deep learning methods. Intra-fold data augmentation was performed by adding Gaussian noise to each training fold during partitioning. Box plots were employed to visualize and analyze the performance metrics of different models across test folds, revealing their stability and advantages. The results demonstrate that the proposed TFDI-Net outperforms traditional machine learning models, achieving improvements of 2.89% in recognition rate, 4.6% in F1-score, and 8.25% in recall.

1. Introduction

Hemiplegia is one of the most common sequelae following a stroke, typically presenting as unilateral motor dysfunction that significantly impacts a patient’s daily quality of life [1]. As such, accurately identifying gait characteristics in hemiplegic patients is essential for assessing the severity of the condition, developing personalized rehabilitation plans, and tracking rehabilitation progress [2]. However, current clinical evaluations largely rely on subjective physician observations and scale-based assessments, which are inherently biased and lack objective, quantifiable metrics to support the evaluation [3].

Recent advancements in sensor technology and artificial intelligence have led to the rise of hemiplegic gait recognition methods based on various sensing technologies. Researchers have employed single-point pressure sensors [4,5,6] and pressure sensor arrays [7,8] to capture gait-related data, enabling the objective quantification of gait parameters in patients. By combining these gait data with traditional machine learning or deep learning approaches, efficient models for recognizing hemiplegic gait can be developed, allowing for the automatic differentiation between hemiplegic and healthy gaits. This method offers valuable technical support for objective evaluation and auxiliary diagnosis.

Studies have shown that compared to healthy individuals, hemiplegic patients exhibit significant differences in plantar pressure distribution between the affected and unaffected sides, and these differences can be utilized to distinguish hemiplegic gait from healthy gait [9,10]. Although research on the gait differences between the affected and unaffected sides in hemiplegic patients has made some progress, studies focusing on the use of plantar pressure features for effectively differentiating between hemiplegic patients and healthy individuals remain relatively limited [7]. Among various sensing technologies, pressure sensor arrays are considered the ideal choice for plantar pressure measurements, owing to their high spatial and temporal resolution, as well as their excellent data stability. The plantar pressure distribution data collected by these sensor arrays not only capture dynamic changes in plantar pressure throughout the walking cycle but also enable the extraction of key spatiotemporal features, such as the pressure center trajectory. This offers a solid foundation for high-precision hemiplegic gait recognition. However, systematic explorations of how to leverage the pressure center trajectory extracted from pressure sensor arrays for effective hemiplegic gait identification remain insufficient.

To analyze the differences in the performance of time–frequency features extracted from center-of-pressure (CoP) trajectories in different directions for hemiplegic gait recognition, this study aims to develop a gait recognition model that integrates both time domain and frequency domain features. In addition, it compares the classification performance and computational efficiency of several deep learning models and traditional machine learning algorithms commonly adopted in related research studies. The main contributions of this work are as follows:

(1)

CDUT Plantar Pressure Dataset—Middle-Aged and Elderly Group (CDUT-PP-MAE): A dataset of plantar pressure data was collected from 48 middle-aged and elderly individuals, including both hemiplegic and healthy participants. Based on these data, a center of pressure trajectory dataset was constructed to validate the effectiveness of the proposed hemiplegic gait recognition method.

(2)

The study compares the performance of hemiplegic gait recognition using plantar pressure center trajectories in different directional orientations.

(3)

A novel temporal–frequency domain interaction network (TFDI-Net) is introduced. Its superiority is demonstrated through comparisons with traditional machine learning methods, as well as state-of-the-art deep learning techniques.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the methodology, including data collection, feature engineering, and data augmentation. Section 4 reports experimental results, comparing the model performance, conducting ablation studies for TFDI-Net, and evaluating computational metrics such as the parameter count and training time. Finally, Section 5 concludes the study.

2. Related Work

In recent years, significant progress has been made in the research of hemiplegic gait recognition and analysis using pressure sensor arrays and wearable plantar pressure sensor systems. However, studies focusing on the use of plantar pressure center trajectories in different directional orientations for hemiplegic gait recognition remain relatively limited. Most wearable-based studies place pressure sensors inside shoes to collect plantar pressure data, which are then directly used in machine learning and deep learning methods for accurate classification and anomaly detection of different gait types.

For instance, Xie et al. [4] designed a gait analysis system based on a self-developed pressure insole and recruited six hemiplegic patients (including both left and right side hemiplegia) and six healthy volunteers to collect plantar pressure data. By dividing the gait phases and extracting the plantar pressure distribution and CoP trajectory, they constructed a 54-dimensional gait feature set. Features were selected using a Spearman correlation analysis and the Random Forest (RF) algorithm, and the RF classifier was employed to identify different gait types. Experimental results showed that this method could distinguish normal, left hemiplegic, and right hemiplegic gaits with 99% accuracy, demonstrating its efficient recognition ability in small sample scenarios. Potluri et al. [5] introduced a deep learning approach by using a stacked long short-term memory (LSTM) network to build a temporal model for recognizing various abnormal gait types associated with neurological disorders. The study recruited ten healthy volunteers, who were instructed to simulate typical gait disorders, including hemiplegic, Parkinsonian, and sensory–ataxic gaits. The model’s training and validation results indicated high discriminative performance in abnormal gait recognition, with significant differences in the R² regression metric between normal and abnormal gaits. Additionally, gait parameters such as the stance phase ratio, step frequency, and step time also exhibited consistent differences.

Cui et al. [8] combined the Bertec force platform, Qualisys motion capture system, and Biomonitor ME6000 system to collect tri-modal data from 21 stroke patients and 21 healthy individuals. Using random forest and support vector machine (SVM) algorithms, they achieved a high recognition rate of 98.21%, demonstrating the advantages of multimodal fusion. Lemoyne et al. [7] collected approximately 25 gait data samples from one hemiplegic patient using an AMTI pressure array. Despite the limited sample size, the study achieved 100% accuracy under leave-one-out cross-validation, highlighting the practical potential of logistic regression models in differentiating between the affected and unaffected sides. Rastegarpana et al. [11] further combined the VICON MX motion capture system with the Kistler 3D force platform to analyze the differences between “targeted steps” (goal-directed steps) and regular walking in stroke patients and healthy individuals. The results revealed that target steps were characterized by reduced step speed and shortened step length, suggesting that hemiplegic patients may adopt different motor control strategies in specific situations.

Kokolevich et al. [12] embedded plantar pressure sensors into the shoes of patients to collect ground reaction force (GRF) data during gait. The study involved two ischemic stroke patients (aged 59 and 73) who were both in the post-stroke recovery phase, six months after the stroke. The focus of the study was to compare the weight-bearing differences between the affected and unaffected sides in the forefoot and hindfoot regions. The analysis revealed that the ground reaction forces in the fore and hindfoot regions of the affected side were lower than those on the unaffected side, with a particularly noticeable asymmetry during the toe-off and heel-strike phases. Notably, some patients exhibited a degree of insufficient weight-bearing even on the unaffected side. This suggests that stroke-induced gait abnormalities not only affect the weight distribution on the affected side but may also impair the weight-bearing mechanism of the contralateral lower limb. Hayashi et al. [6] designed a wearable plantar pressure biofeedback system (PEBF) and recruited six hemiplegic patients (mean age 56.8 ± 11.4 years) at the Tokyo General Hospital for a 3 week walking intervention, consisting of three 20 min sessions per week. Post-intervention, the patients showed a significant increase in the step length of the affected side (p = 0.0277) and a marginal improvement in the plantar flexion angle of the affected side (p = 0.0747), indicating the potential of the PEBF system in improving the push-off function in gait.

In rehabilitation intervention studies for hemiplegic patients, Park et al. [13] used the GAITRite gait analysis system to collect plantar pressure data from hemiplegic patients while wearing shoes. Through an analysis of gait parameters and plantar pressure distribution, they suggested that changes in plantar pressure distribution induced by the intervention may play a positive role in improving gait symmetry and regulating postural balance. Kim et al. [14] conducted a comparative analysis of initial and follow-up gait data from two groups of patients to assess their gait recovery progress. The study found that left- and right-side hemiplegic patients exhibited differences in the rate of gait recovery, which might be related to the habitual use of body balance and compensatory mechanisms in right-handed patients. Therefore, rehabilitation training for hemiplegic patients should fully consider the differences between the affected sides and develop more individualized gait intervention strategies to further enhance recovery outcomes and improve the patients’ quality of life.

3. Materials and Methods

In this study, the Footscan pressure sensor array system was used to collect plantar pressure data from 48 middle-aged and elderly participants at the Second Affiliated Hospital of Chengdu Medical College. The plantar pressure center trajectories were then extracted from the data, and data augmentation techniques were applied to expand the training set samples. Additionally, the effectiveness of the proposed TFDI-Net was validated through comparisons with recent deep learning and machine learning methods. The overall workflow of the study is shown in Figure 1.

3.1. Data Acquisition from Student Participants

The data used in this experiment were collected using the Footscan plantar pressure measurement system (RSscan, Paal, Belgium), with a sampling rate of 253 Hz. The dataset, comprising both healthy and hemiplegic middle-aged and elderly individuals, was obtained from participants at the Second Affiliated Hospital of Chengdu Medical College under a 7 m straight-line walking protocol. The subjects included 19 hemiplegic patients and 29 healthy individuals, all of whom were able to walk independently without assistive devices or support. Detailed information about the dataset is provided in

Table 1. Demographic information of middle-aged and elderly participants.

	Healthy	Hemiplegic	Total
Age	58.9 ± 8.9	61.6 ± 10.9	60.0 ± 9.8
Sex	8/21	16/5	24/26
Height (cm)	164 ± 14	161 ± 11	164 ± 14
Weight (kg)	65 ± 22	65 ± 17	65 ± 22
Shoe Size (EU)	39 ± 3	39 ± 3	39 ± 3
Hemiplegic Side	0/0	11/10	11/10

. The participants ranged in age from 50 to 77 years, with heights ranging between 1.5 and 1.78 m, weights ranging between 43 and 87 kg, and shoe sizes ranging from EU 36 to 42.

The experimental procedure adhered to the principles of the Declaration of Helsinki. Prior to participation, each subject was informed of the experimental protocol and provided written consent for data collection. Medical personnel were present throughout the entire procedure to ensure participant safety. Given the age-related decline in physical function, particularly among some hemiplegic patients who exhibited unstable gait, the data collection sessions for middle-aged and elderly participants were shorter in duration compared to those typically conducted with younger individuals. To minimize the influence of external factors on the experimental data, all participants walked on a standardized mat surface and were instructed to complete six straight-line walking trials at their natural, self-selected walking speed along a designated path. The data collection was conducted indoors, with the walking path set to a straight 7 m distance, following the 7 m walk test protocol. The experimental setup is illustrated in Figure 2.

3.2. Feature Engineering

To compare the performance of machine learning and deep learning methods in hemiplegic gait recognition, statistical features were separately extracted from the anterior–posterior (AP) and medial–lateral (ML) CoP trajectories in both the time and frequency domains. Equations (1) and (2) present the calculation methods for the CoP trajectories in the ML and AP directions, respectively, while Figure 3 and Figure 4 illustrate the corresponding trajectories collected from the subjects. The time domain features included the mean, standard deviation, maximum, and minimum values. The frequency domain features included the dominant frequency, low-frequency energy, mid-frequency energy, high-frequency energy, and spectral entropy. The detailed computation processes are presented in Equations (3)–(11).

$$Co{P}_{ML}={\displaystyle \frac{\sum P_{i,j}{X}_i}{\sum P_{i,j}}}$$

(1)

$$Co{P}_{AP}={\displaystyle \frac{\sum P_{i,j}{Y}_i}{\sum P_{i,j}}}$$

(2)

$$\overline{x}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{x}_i$$

(3)

$$std=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(x_i-\overline{x})}^2}$$

(4)

$$Amplitude=abs(max(CoP Trajctory)-min(CoP Trajctory))$$

(5)

$$Zero Crossing Rate={\displaystyle \frac{1}{2N}}\sum _{n=1}^{N-1}|sgn(x[n])-sgn(x[n-1])|$$

(6)

$$Sample Entropy(m,r)=-ln\left({\displaystyle \frac{A}{B}}\right)$$

(7)

$$\overline{v}={\displaystyle \frac{1}{(N-1)\cdot \mathsf{\Delta }t}}\sum _{i=1}^{N-1}|x_{i+1}-x_i|$$

(8)

$$RMS=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{x}_i^2}$$

(9)

$$f_{dominant}=argma{x}_{f}P(f)$$

(10)

$$E_{low}={\displaystyle \frac{{\displaystyle \sum _{f_i<1}}P(f_i)}{{\displaystyle \sum _{i=1}^N}P(f_i)}}$$

(11)

$$E_{mid}={\displaystyle \frac{{\displaystyle \sum _{1\le f_i<3}}P(f_i)}{{\displaystyle \sum _{i=1}^N}P(f_i)}}$$

(12)

$$E_{high}={\displaystyle \frac{{\displaystyle \sum _{f_i\ge 3}}P(f_i)}{{\displaystyle \sum _{i=1}^N}P(f_i)}}$$

(13)

$$H=-{\displaystyle \sum _{i=1}^N}\widehat{P}(f_i){\log }_2({\displaystyle \frac{P(f_i)}{{\displaystyle \sum _{j=1}^N}P(f_j)}})$$

(14)

In Equations (1) and (2), $P_{ij}$ represents the pressure value of the sensor unit located in the i-th row and j-th column on the FootScan pressure plate, while $X_i$ and $Y_i$ denote the row and column indices, respectively. In Equations (3) and (4), $x_i$ denotes the i-th point in the CoP trajectory, and N represents the total number of samples. In Equation (6), x[n] represents the signal value of the n-th sample point in the CoP trajectory, sgn(⋅) denotes the sign function, and N is the total length of the signal. In Equation (7), A represents the proportion of sample pairs with a similarity distance less than r when the template length is m + 1, and B represents the proportion when the template length is m. The tolerance threshold r is defined as $r_{ratio}\cdot \sigma$, where σ is the standard deviation of the signal, and m is the embedding dimension. In Equation (8), N denotes the total number of sampling points in the time series, $x_i$ represents the value at the i-th sampling point, and $\mathsf{\Delta }t$ is the time interval between two adjacent sampling points. In Equation (9), $x_i$ represents a time point in the CoP trajectory.

In Equations (10)–(14), $P(f_i)$ represents the power spectral density, $f_{dominant}$ the dominant frequency, $E_{low}$ the low-frequency energy, $E_{mid}$ the mid-frequency energy, $E_{high}$ the high-frequency energy, and $H$ the spectral entropy.

3.3. Experimental Setup

A total of 50 participants were initially recruited for the study. After excluding one patient with bilateral hemiplegia and another with an insufficient number of samples, the final dataset included 29 healthy individuals and 19 hemiplegic patients. A subject-wise strategy was employed to partition the dataset into training and testing sets. To ensure an objective evaluation of the model’s ability to distinguish between hemiplegic and healthy gait patterns, five-fold cross-validation was performed, and the average values of all performance metrics across the folds were reported as the final results. During model training, the number of epochs was set to 100, and an early stopping strategy was implemented to prevent overfitting. For deep learning models, the initial learning rate was set to 1 × 10⁻⁴, and the Adam optimizer was used. A ReduceLROnPlateau scheduler was applied with a learning rate decay factor of 0.1 and a weight decay of 1 × 10⁻³. For machine learning models, optimal hyperparameters were determined using a grid search. The model was trained using an NVIDIA 3070 Ti GPU, Intel Core i7 12th-generation CPU, with PyTorch version 2.1.0 and Python version 3.8.

Six evaluation metrics were used to assess the performance of the models, including the accuracy, precision, F1-score, area under the ROC curve (AUC-ROC), specificity, and recall. The formulas for each metric are provided in Equations (15)–(20). In addition, confusion matrices were used to provide a more intuitive visualization of the differences in hemiplegic gait recognition performance across different methods.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(15)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(16)

$$F1 Score=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(17)

$$Specificity={\displaystyle \frac{TN}{TN+FP}}$$

(18)

$$AUC={\displaystyle {\int }_0^{1}TPR(FPR)d(FPR)}$$

(19)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(20)

To comprehensively validate the effectiveness of the proposed method, the following representative traditional machine learning and deep learning approaches were selected as baseline models for comparison: support vector machine (SVM) [15], random forest (RF) [16], XGBoost [17], EfficientNetV2 [18], transformer [19], ResMLP [20], and long short-term memory (LSTM) network [21].

3.4. Data Augmentation

To enhance the model’s robustness to outliers and perturbations, a Gaussian noise-based data augmentation strategy was incorporated during training set construction. Considering that each fold in the cross-validation process involves a different subset of participants, an in-fold augmentation strategy was adopted to prevent data leakage. Specifically, data augmentation was performed independently within each training fold, rather than applying augmentation to the entire dataset prior to cross-validation. This approach ensured complete separation between training and validation data, thereby improving the reliability during model evaluation. For each original training sample, N augmented replicas were generated by introducing small perturbations. Gaussian noise sampled from a normal distribution was added to the original sequences to simulate real-world variability in data acquisition and sensor errors. The detailed procedure is illustrated in Algorithm 1.

Algorithm 1: Gaussian Noise-Based Data Augmentation

Input:

X: Original time series sample ∈ ${ℝ}^{T\times D}$

N: 5//Number of augmented copies

${\sigma }_{min}$, ${\sigma }_{max}$: 0.01, 0.05//Minimum and maximum noise standard deviation

Output:

Augmented dataset ${\mathcal{D}}_{aug}$ containing N noisy copies of X

1 Initialize ${\mathcal{D}}_{aug}$ ← ∅ 2 for i = 1 to N do 3     Sample σ ∼ Uniform(${\sigma }_{min}$, ${\sigma }_{max}$) 4     Generate Gaussian noise: ε ∼ $𝒩$ (0, σ2), shape = (T, D) 5     Add noise: $\tilde{\mathit{X}}$ ← X + ε 6     Compute A ← max(abs(X)) 7     Clip $\tilde{\mathit{X}}$ within [−1.5A, 1.5A] 8     if ∃ $x$ ∈ $\tilde{\mathit{X}}$ such that $x$ ∉ $ℝ$ then 9         Replace $\tilde{\mathit{X}}$ ← X 10     end if 11     Append $\tilde{\mathit{X}}$ to ${\mathcal{D}}_{aug}$ 12 end for 13 return ${\mathcal{D}}_{aug}$

3.5. TFDI-Net

In this study, we propose a novel hemiplegic gait recognition network architecture that integrates temporal and frequency domain information, termed TFDI-Net (temporal–frequency domain interaction network). The network comprises three primary functional modules: a temporal domain convolution module, a frequency domain feature extraction module, and a temporal–frequency interaction module, which collaboratively enables multiscale feature fusion and complementary enhancement. The overall architecture of the TFDI-Net is illustrated in Figure 5 and its parameter settings are provided in

Table 2. TFDI-Net structure settings.

Module	Layer Name	Input Channels	Output Channels	Kernel Size	Activation	Pool
time feature	conv1	500	64	5	ReLU	Maxpool
	conv2	64	128	5	ReLU	Maxpool
	conv3	128	256	5	ReLU	Maxpool
	fft	-	-	-	-	-
freq feature	conv1	2	32	3	ReLU	Maxpool
	conv2	32	64	3	ReLU	Maxpool
	conv3	64	128	3	ReLU	Maxpool
time2freq mag	conv	256	64	1	ReLU	-
	conv	64	256	1	Segmoid	-
freq2time mag	conv	128	64	1	ReLU	-
	conv	64	128	1	Segmoid	-
freq_enhancement	conv	384	64	1	ReLU	-
	conv	64	384	1	Tanh	-
time2freq correlation	Linear1	384	64	-	ReLU	-
	Linear2	64	1	-	Sigmoid	-
feature fusion	conv	512	62	1	ReLU	-
	conv	62	512	1	Sigmoid	-
	adaptive pool	-	62	-	-	-
	fc1		-	-	ReLU	-
	dropout	-	-	-	-	-
	fc2	-	-	-	-	-

.

3.5.1. Temporal Domain Convolution Module

To effectively capture the temporal structure of gait signals, TFDI-Net incorporates a temporal convolution module, as illustrated in the upper-right part of Figure 5, to model local dependencies and dynamic variation patterns within time series data. This module consists of a three-layer stacked one-dimensional convolutional architecture, where each convolutional layer is followed by a ReLU activation function and a max-pooling operation. This design enhances nonlinear feature extraction and facilitates efficient temporal downsampling.

Specifically, the input gait time series signal is first passed through a one-dimensional convolutional layer to extract low-level temporal features. These features are then further refined through a second and third convolutional layer, progressively deepening the temporal representation. Each convolutional layer uses a kernel size of 5, a stride of 1, and a padding of 2 to preserve the original sequence length. The max pooling operation following each convolution not only reduces the computational complexity but also enhances the robustness to temporal shifts. The computation process of this module can be formally expressed as follows:

$$H_{i+1}=Maxpool(ReLU(Con{v}_i(x)))$$

(21)

Here, $Con{v}_i$ denotes the i-th (i = 1, 2, 3) one-dimensional convolutional operation, with the numbers of output channels set to 64, 128, and 256, respectively. The final output of the temporal convolution module is fed into the subsequent attention module and the temporal–frequency interaction module for further feature fusion and modeling.

The parameter configuration of this module draws inspiration from the classical architectural design of the temporal convolutional network (TCN) [22], which has demonstrated stability and strong generalization performance across various biosignal processing and time-series classification tasks. Accordingly, this study adopts TCN’s convolutional kernel size and hierarchical channel configuration strategy. Preliminary experiments further confirmed the suitability of this structure for the gait recognition task.

3.5.2. Frequency Domain Feature Extraction Module

Temporal signals often contain rich frequency domain information that is not readily accessible in the time domain. To fully exploit the spectral characteristics of the gait signals, a frequency domain feature extraction module based on the fast Fourier transform (FFT) was designed in this study, as illustrated in the upper-left part of Figure 5. The core idea of this module is to transform the input from the time domain to the frequency domain using FFT, thereby extracting informative spectral features. Given an input time domain signal $x(t)\in {ℝ}^{B\times T}$, where B is the batch size and T is the sequence length, the module applies the discrete Fourier transform to each sample as follows:

$$X(k)=\sum _{n=0}^{T-1}x(n)e^{-j2\pi kn/T}, k=0,1,\dots ,T-1$$

(22)

To fully preserve the complex-valued information from the Fourier transform, the module decomposes the complex frequency domain coefficients into their real and imaginary parts:

$$F_{real}(k)=\mathit{\Re }(X(k)), F_{imag}(k)=\mathit{\Im }(X(k))$$

(23)

The final output of the frequency domain feature extraction module is denoted as $F\in {ℝ}^{B\times 2\times K}$, where the second dimension corresponds to the real and imaginary components of the frequency domain features, respectively. This design allows the module to effectively capture the spectral characteristics of the signal, providing valuable frequency domain information for subsequent feature fusion and classification tasks.

3.5.3. Temporal–Frequency Modal Interaction Module

To model the intrinsic correlation between time and frequency domain features of gait signals, we designed a cross-modal interaction module that enables bi-directional guidance and gated fusion between the two domains. The overall structure of the interaction module is illustrated on the middle of Figure 5, and the parameter settings of its submodules are provided in Table 2. The module consists of five key components:

(1)

Time-to-Frequency Guidance

The time domain features are first transformed using the fast Fourier transform (FFT) to obtain the magnitude and phase spectrum. Two modulation branches are then applied—one using a sigmoid function for magnitude and the other using a tanh function for phase—to generate adaptive modulation weights, enabling the dynamic refinement of frequency domain information.

(2)

Frequency-to-Time Guidance

The frequency domain features are interpolated to match the temporal resolution and passed through a learnable modulation branch to guide the reconstruction of time domain signals from the modulated spectral features.

(3)

Frequency Feature Enhancement

The magnitude spectrum derived from the time domain FFT is concatenated with original frequency features and passed through nonlinear transformations to produce enhanced frequency domain representations.

(4)

Global Time–Frequency Correlation Learning

Global average pooling is applied to both time and frequency features to obtain compact representations. These are fused via a lightweight MLP to compute a scalar correlation weight, representing the semantic alignment between the two modalities.

(5)

Gated Fusion

The original and modulated time domain features are concatenated and passed through a gating network. The final time domain representation is computed by weighting the fusion result using the learned correlation.

This module enables dynamic, bidirectional interactions between temporal and spectral cues, allowing the network to extract more discriminative and robust features for hemiplegic gait recognition.

4. Results

4.1. Hemiplegic Gait Recognition Results Based on CoP Trajectories

To investigate the differences in hemiplegic gait recognition performance between AP and ML CoP trajectories, experiments were conducted using both traditional machine learning methods and recent deep learning approaches. The results for accuracy, precision, and other metrics are presented in

Table 3. Hemiplegic gait recognition results based on AP direction CoP trajectories.

Model	Accuracy	Precision	F1-Score	AUC-ROC	Specificity	Recall
SVM	86.99	89.23	82.76	92.75	91.7	77.17
RF	84.81	83.73	80.48	91.7	87.9	77.48
XGBoost	84.75	84.81	79.65	90.44	89.53	75.05
LSTM	87.66	90.46	83.26	93.60	92.85	77.13
Transformer	85.47	88.78	79.60	92.31	92.37	72.15
EfficientNetV2	85.88	83.01	82.18	93.12	87.56	81.36
ResMLP	86.07	89.56	81.03	93.24	92.09	73.98
Ours	89.88	89.39	87.36	94.9	91.93	85.42

and

Table 4. Hemiplegic gait recognition results based on ML direction CoP trajectories.

Model	Accuracy	Precision	F1-Score	AUC-ROC	Specificity	Recall
SVM	81.13	83.02	73.21	86.33	89.27	65.48
RF	79.24	76.42	71.12	86.56	85.82	66.51
XGBoost	79.98	77.79	72.5	87.59	85.98	67.88
LSTM	77.85	87.21	65.26	83.76	91.84	52.14
Transformer	73.29	72.75	71.81	74.39	71.87	70.89
EfficientNetV2	86.91	80.75	83.63	92.44	86.81	86.73
ResMLP	85.54	87.51	80.01	91.07	92	73.7
Ours	88.61	88.73	85.06	91.97	92.65	81.71

, respectively. The results demonstrate that TFDI-Net consistently achieved the best performance on both the AP and ML CoP trajectory datasets. Moreover, the experiments indicate that CoP trajectories in the AP direction are more informative and effective for recognizing hemiplegic gait.

4.1.1. Analysis of CoP Trajectory Recognition Results Using Machine Learning Methods

In this study, three machine learning algorithms—a support vector machine (SVM), random forest (RF), and XGBoost—were employed to classify hemiplegic gait based on time domain and frequency domain features extracted from AP and ML CoP trajectories. Through feature engineering, the raw trajectory data were transformed into 12-dimensional feature vectors, encompassing both statistical characteristics in the time domain and spectral features in the frequency domain. A five-fold cross-validation strategy was used to evaluate model performance.

The results revealed significant differences in recognition performance across the three algorithms when applied to AP and ML CoP trajectory data. Overall, for the AP direction, the SVM algorithm outperformed the others across all evaluation metrics, followed by RF, with XGBoost performing the worst. In contrast, for the ML direction, the SVM algorithm achieved the best performance on all metrics, followed by XGBoost, while RF yielded the poorest results.

SVM demonstrated the best performance on the AP CoP trajectory, which can be primarily attributed to the algorithm’s strong compatibility with the 12-dimensional feature space. First, SVM is known for its excellent classification capability in low- to medium-dimensional feature spaces, where it can identify the optimal separating hyperplane. Second, given the relatively limited sample size in this study (29 healthy subjects and 19 hemiplegic patients), SVM maintained strong generalization performance—an important advantage in small-sample settings. The use of kernel functions enables the SVM to effectively capture potential nonlinear relationships between time domain and frequency domain features. As shown in the box plots for the AP CoP trajectory, the SVM exhibited the smallest variability across all evaluation metrics, indicating the highest performance stability under different data splits. In particular, for the AUC-ROC metric, both the median and mean values of the SVM remained high, with a narrow interquartile range, suggesting consistent classification performance based on the 12-dimensional features. Moreover, the SVM’s built-in regularization mechanism effectively mitigated overfitting, which is especially beneficial given the relatively small dataset.

Random forest (RF) exhibited moderate performance within the 12-dimensional feature space, reflecting the algorithm’s adaptability to the current feature dimensionality. One of RF’s key advantages lies in its ability to automatically assess feature importance, which is valuable for understanding the relative contribution of time and frequency domain features to hemiplegic gait classification. Through bootstrap sampling and random feature selection, RF can reduce the risk of overfitting to some extent. However, in a relatively low-dimensional space such as the 12-feature setting used here, the benefits of random feature selection may be less pronounced than in high-dimensional scenarios. RF achieved a recall of 77.48%, comparable to that of SVM, but its specificity (87.9%) was slightly lower, indicating a greater risk of false negatives in identifying hemiplegic patients. This may be related to the decision-making mechanism of RF when handling combined time and frequency domain features; certain critical patterns in feature combinations may not have been fully captured or exploited.

The XGBoost algorithm, while generally known for its strong performance in classification tasks, exhibited certain limitations in the context of hemiplegic gait recognition based on time–frequency features. Despite its ensemble structure, XGBoost may overfit when the dataset is relatively small and the feature space is complex, as is the case with the 12-dimensional feature set derived from both time and frequency domains. In our experiments, XGBoost achieved a recall rate of only 75.05%, which although better than RF still lagged behind the performance of deep learning models. This relatively lower recall suggests that XGBoost may struggle to detect some subtle gait abnormalities associated with hemiplegia. Additionally, as shown in the box plots, XGBoost demonstrated noticeable variability in F1-scores and recall across different folds, indicating limited robustness and generalizability. These findings suggest that while XGBoost can model nonlinear relationships, it may not fully capture the intricate temporal–spectral dependencies present in gait dynamics, especially when training data is limited.

The results presented in Table 3 and Table 4, along with the box plots in Figure 6 and Figure 7, indicate that all three machine learning algorithms performed better on the AP CoP trajectory than on the ML CoP trajectory. This suggests that the AP direction plantar pressure center trajectories are more suitable for hemiplegic gait recognition and possess greater potential clinical diagnostic value.

4.1.2. Analysis of CoP Trajectory Recognition Results Using Deep Learning Methods

In this study, five deep learning methods were employed to recognize hemiplegic gait based on plantar pressure center trajectories, including the LSTM, transformer, EfficientNetV2, ResMLP, and proposed temporal–frequency domain interaction network (TFDI-Net) methods. Their performance was evaluated using five-fold cross-validation on plantar pressure data from 48 participants (29 healthy individuals and 19 hemiplegic patients). The results revealed significant performance differences among the models. Overall, the proposed TFDI-Net achieved the best performance, followed by ResMLP. EfficientNetV2 ranked third, with the LSTM and transformer methods performing slightly worse in sequence.

As a deep learning model specifically designed for sequence data, LSTM demonstrated the fourth performance in this study, following EfficientNetV2. It achieved an accuracy rate of 87.66% and an AUC-ROC of 93.6%, indicating its effectiveness in capturing the temporal dependencies within plantar pressure center trajectories. The gated architecture of LSTM enables it to learn long-term dependency patterns in gait signals, which is particularly valuable for identifying abnormal temporal characteristics associated with hemiplegic gait. However, the recall of LSTM was relatively low (77.13%), suggesting a certain risk of missed diagnoses when identifying hemiplegic patients. This limitation may stem from LSTM’s reliance solely on temporal features, without incorporating frequency domain information, which could reduce its effectiveness in recognizing certain types of hemiplegic gait patterns.

The transformer model, with its self-attention mechanism, is theoretically capable of capturing dependencies between any positions within a sequence, offering stronger modeling capacity than LSTM. However, in this study, the transformer achieved slightly lower performance, with an accuracy of 85.47%, compared to LSTM. This may be attributed to the fact that transformer architectures typically require large-scale datasets to effectively train their complex attention mechanisms, whereas the number of subjects in this study (48 participants) may not have been sufficient to fully exploit the model’s potential. As shown in the box plots, the transformer exhibited relatively high variability across evaluation metrics, particularly in recall, indicating that its performance stability remains suboptimal in this small-sample setting.

The recognition performance of ResMLP can be explained by its architectural design. The model adopts a patch embedding mechanism that divides the CoP spectrogram into patches, followed by 12 MLP-based residual blocks for feature learning. Its high specificity (92.09%) can be attributed to the regularity of plantar pressure control patterns in healthy individuals. These dynamic patterns are consistent within each patch and can be effectively captured through deep feature abstraction, resulting in clearly clustered feature distributions. However, the model’s recall is relatively low (73.98%), which may be due to the use of a global average pooling strategy that tends to dilute localized abnormal signals from hemiplegic patients—especially in cases of mild hemiplegia, where abnormalities typically occur only during specific phases of the gait cycle. Nevertheless, the model achieved an excellent AUC-ROC score (93.24%), indicating that the 384-dimensional embedding space it constructs is highly expressive and enables clear linear separability between hemiplegic and healthy samples in the high-dimensional representation space.

EfficientNetV2 exhibited moderate performance in the analysis of anterior–posterior (AP) CoP trajectories. Specifically, the model achieved an accuracy rate of 85.88%, a precision rate of 83.01%, an F1-score of 82.18%, and an AUC-ROC of 93.12%, slightly underperforming compared to ResMLP. Notably, its specificity surpassed that of ResMLP, reaching 81.36%. However, this still implies that approximately 18.64% of healthy individuals were misclassified as hemiplegic patients—an outcome that remains unacceptable in clinical applications. The core module of EfficientNetV2 is the mobile inverted bottleneck convolution (MBConv) block, which may struggle to capture critical features when extracting information from spectrograms. The use of depthwise separable convolutions and channel attention mechanisms is better suited for capturing spatial correlations, and may require further structural optimization to effectively extract meaningful frequency domain features. In the AP direction CoP trajectory analysis, crucial information lies in dynamic temporal patterns and long-range dependencies within the time series. However, EfficientNetV2 focuses more on local feature extraction and inter-channel weighting, which limits its ability to effectively model the temporal dynamics essential for accurate gait recognition.

The proposed TFDI-Net achieved the highest hemiplegic gait recognition accuracy on the AP CoP trajectory dataset, which can be attributed to several key architectural designs. First, the temporal feature extraction module utilizes only three 1D convolutional layers to capture local temporal dependencies. Then, frequency domain features are extracted via Fourier transform and further processed using three additional convolutional layers. Finally, the temporal and frequency features are integrated through a domain interaction mechanism, resulting in more discriminative representations. An analysis of the five-fold cross-validation box plots reveals that TFDI-Net demonstrates excellent stability and generalization capability. The precision metric is consistently concentrated within the 0.95–1.00 range, indicating high consistency and low variance. The AUC-ROC values remain stably within the 0.95–1.00 range as well, reflecting the model’s strong discriminative power across different data splits. The specificity metric also performs remarkably well, with all folds exceeding 0.95. Although the recall metric shows some variability (ranging from 0.75 to 0.95), the overall performance still surpasses that of the other models.

The performance of both machine learning and deep learning methods on AP and ML CoP trajectories suggests that AP direction CoP data hold greater clinical diagnostic value. Moreover, deep learning models do not rely on handcrafted feature extraction, and with the integration of frequency domain information, can achieve superior performance. These findings indicate that deep learning approaches based on temporal–frequency feature fusion offer promising potential for the clinical application of hemiplegic gait recognition.

4.2. Ablation Experiment

To evaluate the contribution of temporal, frequency, and temporal–frequency domain features to hemiplegic gait recognition, ablation experiments were conducted on the temporal convolution module, frequency convolution module, and temporal–frequency interaction fusion module, with the results shown in

Table 5. Results of ablation experiments.

T	F	Fusion	Accuracy	Precision	F1-Score	AUC-ROC	Specificity	Recall
✓			88.81	89.77	85.14	95.16	92.70	80.97
	✓		88.56	88.51	85.74	94.16	90.72	83.13
✓	✓		89.4	88.24	86.74	94.97	90.82	85.29
✓	✓	✓	89.88	89.39	87.36	94.9	91.93	85.42

’T’ represents the temporal feature extraction module, ‘F’ represents the frequency domain feature extraction module, and ‘fusion’ represents the temporal–frequency interaction module.

. First, by removing the frequency domain feature module from the proposed TFDI-Net model, the baseline performance using only temporal features was obtained. When only the frequency domain feature module was retained, the performance in terms of the F1-score and recall outperformed the results of the temporal feature module. This indicates that models trained with frequency domain features are better at capturing subtle pattern differences, especially those that are not prominent in the time domain but are significant in the frequency domain, thereby reducing the incidence of missed diagnoses.

The accuracy, F1-score, and recall improved after directly merging temporal and frequency features, compared to using only temporal or frequency features individually. This indicates that the fused features provide a richer decision boundary in the high-dimensional feature space. Samples that were difficult to distinguish in a single feature domain were effectively separated in the fused feature space. When temporal and frequency features were further integrated through interaction, the accuracy, F1-score, and recall improved again, suggesting the presence of temporal–frequency coupling features in the CoP trajectory. This nonlinear synergistic effect not only validates the multi-scale dynamic characteristics of the balance control system but also provides an important technical pathway for developing more accurate models for balance function assessments.

4.3. Computational Efficiency

We not only compared the computational efficiency of our method with existing approaches such as SVM, RF, and LSTM, but also with XGBoost, transformer, ResMLP, and EfficientNetV2 methods. The results are presented in

Table 6. Computational efficiency of machine learning and deep learning models.

Model	Parameters (M)	Size (MB)	Training Time (s)	Inference Time (ms)
SVM	-	-	35	-
RF	-	-	59	-
XGBoost	-	-	25	-
LSTM	0.17	0.69	55.4	0.5
Transformer	31.9	62.7	606	2
EfficientNetV2	20.3	78.08	1055	13.48
ResMLP	14.36	54.77	874	3.35
Ours	12.77	48.73	195	2.3

.

Traditional machine learning models are typically trained on CPUs, so only the training time can be reasonably measured; all other metrics are denoted by ‘-‘. Moreover, their parameter counts and model sizes are significantly smaller than those of deep learning models, making direct comparisons less meaningful. Compared with models such as LSTM, transformer, ResMLP, and EfficientNetV2, TFDI-Net achieved competitive performance results, ranking second only to LSTM across all evaluation metrics. This demonstrates its strong potential for real-world clinical deployment.

5. Conclusions

This study proposes a hemiplegic gait recognition method based on temporal–frequency feature interaction, namely the temporal–frequency domain interaction network (TFDI-Net). Comparative experiments with traditional machine learning methods and recent deep learning models validated the potential and effectiveness of temporal–frequency domain fusion in hemiplegic gait recognition. In addition, ablation studies further revealed the existence of temporal–frequency coupling features within the CoP trajectory and demonstrated that feature fusion can effectively improve the recognition performance.

The current dataset consists mainly of healthy individuals and a limited number of hemiplegic patients. Due to safety concerns during data collection, only data from mildly hemiplegic patients were included, which restricts the model’s generalizability to broader pathological gait scenarios. Therefore, further validation on more diverse and representative datasets is required. Future work will focus on collecting plantar pressure data from patients with varying degrees of hemiplegia to optimize the algorithm, and integrating it into an automated gait assessment system based on plantar pressure, with the aim of improving its robustness and clinical applicability.