Next Article in Journal
Realization of Modified Electrical Equivalent of Memristor-Based Pavlov’s Associative Learning to Avoid Training Fallacies
Next Article in Special Issue
Empowering Recovery: The T-Rehab System’s Semi-Immersive Approach to Emotional and Physical Well-Being in Tele-Rehabilitation
Previous Article in Journal
Adaptive Step Size Linear Multistep Method for Small Signal Stability Analysis in Power Systems with Time Delays
Previous Article in Special Issue
TheraSense: Deep Learning for Facial Emotion Analysis in Mental Health Teleconsultation
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Towards Realistic Human Motion Prediction with Latent Diffusion and Physics-Based Models

1
School of Computer Science and Technology, Dongguan University of Technology, Dongguan 523820, China
2
School of Artificial Intelligence, Dongguan City University, Dongguan 523419, China
3
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
*
Author to whom correspondence should be addressed.
Electronics 2025, 14(3), 605; https://doi.org/10.3390/electronics14030605
Submission received: 25 December 2024 / Revised: 24 January 2025 / Accepted: 1 February 2025 / Published: 4 February 2025

Abstract

:
Many applications benefit from the prediction of 3D human motion based on past observations, e.g., human–computer interactions, autonomous driving. However, while existing methods based on encoding–decoding achieve good performance, prediction in the range of seconds still suffers from errors and motion switching scarcity. In this paper, we propose a Latent Diffusion and Physical Principles Model (LDPM) to achieve accurate human motion prediction. Our framework performs human motion prediction by learning information about the potential space, noise-generated motion, and combining physical control of body motion, where physics principles estimate the next frame through the Euler–Lagrange equation. The framework effectively accomplishes motion switching and reduces the error accumulated over time. The proposed architecture is evaluated on three challenging datasets: Human3.6M (Human 3D Motion Capture Dataset), HumanEva-I (Human Evaluation dataset I), and AMASS (Archive of Motion Capture as Surface Shapes). We experimentally demonstrate the significant superiority of the proposed framework in the prediction range of seconds.

1. Introduction

The task of human motion prediction focuses on predicting future motion sequences by observing past motion, which has many applications, such as autonomous driving [1], intelligent robotics [2], and animation [3]. Traditional animation production often requires animators to draw movements frame by frame, a process that takes a lot of time and effort. Through the use of physical laws, human body motion prediction can generate realistic and reasonable human body motion sequences, which allows animators to work more efficiently and enhances the user’s sense of immersion. When the user interacts virtually, the system determines the movement trend of the human body and provides timely and accurate information, e.g., predicting the patient’s movement trajectory to give appropriate guidance during rehabilitation training. The predicted motions need to be continuous, diverse, and realistic, making this task complex. Traditional methods include Gaussian [4] and restricted Boltzmann machines [5], which combine simple representations of body motion encoded as latent representations, which are then decoded into predictions [6,7,8], but accurate predictions require more complex representations of motion.
Recently, motion data have begun to be used widely, and researchers have made great progress. Researchers have proposed various deep learning models to model motion sequences, such as deep learning models based on recurring neural networks (RNNs) [9,10] and convolutional neural networks (CNNs) [11,12,13]; transformers [14,15], more intuitive than the model shown in Figure 1a, utilize graph convolutional layers to process observation sequences, create motion embeddings, and decode them into future prediction sequences. This is a fully data-driven approach focusing on learning motion patterns from structured input. Variants of generative adversarial networks (GANs) [16,17] and Variational Auto-Encoder (VAE) [18] have been used to model data distributions [19,20]. Among other areas, prior methods with encoding–decoding address the human motion prediction problem, which predicts future motions bases on existing motion frames. Different sampling phases during training result in increased computational effort. While these methods perform well in some cases, such as when distinguishing basic actions, they face challenges in real-world applications that require a wider diversity of motions—such as the transition from walking to sitting. While these methods can capture complex motions, they often ignore the fact that human motion is governed by physical principles and that the human body follows the laws of physics. Therefore, while good prediction results can be achieved, methods that rely on hyperparameter balancing and loss constraints often have difficulty producing consistent, applicable results. This highlights the necessity of carefully designing parameters to balance various loss functions and training requirements. The core argument of this paper is that by incorporating the basic principles of motion diffusion and physical motion into the learning process, prediction accuracy can be significantly improved, especially over short time intervals.
The human body is a physical system with multiple body parts working in concert to form complex motions. Muscle strength and exercise habits vary greatly from person to person. The same person can move differently under different environmental conditions and emotional states. According to the physical point of view, accurately modeling the rotation of joints is not a simple task. Since human movement is a coupling of multiple degrees of freedom, this greatly increases the difficulty of accurate prediction, e.g., the coordination of hands and feet during walking. Depending on the joint angles of the human body, the Euler–Lagrange equation [21] describes the human body motion through generalized positional coordinates along with second-order ordinary differential equations (ODEs), imposing constraints to improve the quality of the output of human body motion prediction [22,23,24], and motion diffusion models provide motion information to the prediction from random noise to improve its performance [25]. Researchers in data-driven estimation have not yet explored this approach of synergizing physics and neural networks in depth. Previous surveillance systems recorded behavior at a given time point, but human motion prediction based on diffusion models can predict future actions. This plays a crucial role in crime prevention and public safety. For example, in airports with high traffic flow, by predicting the movements of suspicious people, measures can be taken in advance to effectively reduce the likelihood of potential dangers occurring. Traditional prediction methods have limitations in dealing with the uncertainty and diversity of human movements. The emergent diffusion model, on the other hand, has significant results in predicting outcomes. It is able to predict the future state of human movement by learning the potential distribution in the data. Therefore, as shown in Figure 1b, our work focuses on fusing physics and deep learning in applying diffusion to predictive models for training in an end-to-end manner without the need for multi-stage training, which avoids balancing multiple lossy hyperparameters.
In recent years, physics-based and diffusion-based deep learning models have received increasing attention in terms of prediction accuracy and training speed [26,27,28,29]. Neural networks can efficiently solve partial differential equations [30]. Inspired by these methods, we do not use encoding–decoding methods but propose a new approach for human motion prediction by diffusion-based and physically driven methods. According to what we know, the diffusion model with denoising procedures can generate motion sequences from random noise, and can easily be pruned to unsmooth results. In concrete terms, we use observation sequences, combine the noisy and denoised mechanisms, use physics-informed deep learning, and incorporate the Euler–Lagrange equations. We demonstrate that the proposed framework (LDPM) can effectively improve the prediction accuracy of existing frameworks and obtain more physically plausible prediction results.
In summary, the main contributions of our work are as follows:
  • To address the problems mentioned above, we propose a new framework. The diffusion-based and physically driven approach has the advantages of long-term prediction, high motion switching capability, and reduced error accumulation. The future motion is predicted by an efficient combination of physical principles and random noise.
  • It is experimentally demonstrated that the proposed method obtains better performance and outperforms existing methods, providing a new perspective for future research.

2. Related Works

This section reviews existing work on human motion prediction and then analyzes potential diffusion-based and physically driven models to improve adjacent human dynamics models.

2.1. Human Motion Prediction

Traditional methods [12,31] try to predict movement in a deterministic way. Previous statistical methods [4] only showed consistent patterns of movements over time. With the widespread use of large amounts of motion data, deep learning has a clear advantage in generating complex motion. Yuan et al. [25] considered physical factors and generated motion trajectories more in line with the real world using a diffusion model based on a physics-guided mechanism, which required high hardware at the time of experimentation. Zhang et al. [26] incorporated physics knowledge into neural networks to learn complex patterns and reduce noise learning for human motion prediction. Karniadakis et al. [27] provided a priori knowledge through physical laws and integrated it into machine learning algorithms to reduce false predictions, but this ran the risk of introducing approximation errors and loss of accuracy. Codec models have been employed for prediction [32,33,34,35,36] using multiple loss constraints [11] to achieve diversity and accuracy of results, learning basis vectors, using sampling strategies, decoupling network parameters and Gaussian distributions, and linear interpolation to control smoothing, which raises training requirements. RNNs and their variations capture temporal information of human movements but fail to capture spatial dependencies; extraction of features by convolutional operations has low computational complexity, but simple splicing or averaging does not yet achieve optimal performance.
To address this problem, some researchers have used CNNs, GCNs [37,38,39], and spatio-temporal graph convolution [31]. Zhong et al. [31] proposed a spatio-temporally gated adjacency graph convolutional network model to focus on channel information, balance spatio-temporal weights, and flexibly adapt to the input situation, but it increased the number of model parameters. Li et al. [38] propose a symbiotic graph neural network that combines action, structural, and bipartite graphs to explain movement changes and to comprehensively understand and utilize inter-joint relationships, but validation of model robustness is lacking. Chen et al. [39] proposed a skeleton segmented graph scattering network to use multi-channel filters to obtain the graph spectral bands, model different body parts, and obtain fine features, but it has high computational cost and requires significant debugging efforts. Recently, Guo et al. [40] showed better performance using multilayer perceptron (MLP), which performs a nonlinear transformation by stacking fully connected layers, where the neurons cooperate with all the previous layers of neurons to capture complex patterns in the input data, but leads to a large number of multiply–add operations, especially when dealing with high-dimensional data. Bouazizi et al. [12] proposed a prediction model based on MLP, which independently applies two different MLPs to solve the complexity as well as the computational difficulties of methods such as RNN, CNN, GCN, etc., but the performance of the model has a certain impact on the processing details and data inputs. Xu et al. [41] utilized geometric isotropy in motion data to improve the accuracy of the model. Even so, these methods are still affected by physical motion and require many stages of training.

2.2. Latent Diffusion and Physically Driven Models

Human dynamics are modeled with equations describing the physical motion, and a lossy diffusion model of human motion can be trained in an end-to-end format to generate enriched actions. Physics-based models can improve on purely data-driven estimation and are widely used for different tasks. This is also true for diffusion models for research tasks such as image/video generation [42,43,44,45], drug discovery [46], and 3D reconstruction [47,48,49,50,51]. Among them, in monocular 3D human reconstruction, some methods estimate parameters in the Euler–Lagrange equation, improve data-driven estimation [23], and simulate realistic motion. Traditional image synthesis is prone to detail loss and image blurring, and the potential diffusion model can perform diffusion operations on the data in the potential space to generate high-resolution images [43], which gives a problem-solving perspective to traditional image synthesis methods. Probabilistic audiovisual diffusion priors [44] play a significant role in the field of audiovisual synthesis. Traditional audiovisual synthesis methods have the problem of generative incoherence. Introducing a priori knowledge and using diffusion models to fuse audio and visual features can improve the realism and naturalness of the synthesis results. Diffusion models can also be used in video processing and generation [45], i.e., modeling inter- and intra-frame relationships in video. Using the geometric diffusion mechanism [46], the complex relationship between atoms in a molecule is dealt with to generate a reasonable molecular conformation.
A 3D shape a priori and text-to-image diffusion model [48] provides a new perspective for generating 3D models directly from text descriptions. Scalable semantic transfer over multi-labeled domains [49] allows for human parsing, which accurately delineates different parts of the human body and semantic information, and provides a more accurate semantic basis for human posture estimation studies. Unlike existing prediction work, we are integrating physical principles into a diffusion model to generate accurate future motion predictions. Cai et al. [22] predict human movement by constructing multiscale hypergraphs specialized for ICH dance videos. Zhang et al. [25] used the Euler–Lagrange equations to infer kinematic forces from observed motions, and then used these forces as a motion prediction model for additional inputs to improve performance. BeLFusion [30] decomposes model training into multiple stages to predict motion through a latent space diffusion model, which can be limited by the codec. In this paper, we propose a new framework that adds a physically driven model to the diffusion model for end-to-end motion prediction.

3. Proposed Framework

Inputting some information such as text or action labels, our goal is to generate a physically plausible human motion x T + 1 : T + N of length N. We propose a Latent Diffusion and Physical Principles Model (LDPM) for human motion generation. The LDPM starts with a noisy motion x 1 : T and models a denoising distribution that denoises motions from a diffusion time step T to N (N < T). Iterative application of the model denoises the motion into a clean motion x 1 : T , which becomes the final output x T + 1 : T + N . An overview of our proposed method LDPM is shown in Figure 2. First, we are able to maintain the continuity of the generated results in time by introducing a noise scheduling strategy and a diffusion process. Meanwhile, a physical data-driven motion prediction model PhysDa is introduced. The model iteratively predicts future body configurations based on input historical motion data. Finally, we explore an LDPM-based training process that effectively mitigates prediction errors accumulated over time.

3.1. Potential Diffusion Model

We take the sequence of observed T o b s coordinate system motions as x 1 : T o b s = ( x 1 , x 2 , , x T o b s ) R T o b s × 3 J , where x T o b s R 3 J are the coordinates of the joints at the coordinate system T o b s and J is the number of joints. Given the observed motions x 1 : T o b s , the goal of the human motion prediction (HMP) problem is to predict the following N motions: x T o b s + 1 : T o b s + N = ( x T o b s + 1 , x T o b s + 2 , , x T o b s + N ) R T o b s × 3 J .
Samples z ∈ latent space V, z can be mapped to the coordinate space. Then, a latent diffusion model is trained to predict z. The generative sequences are formulated as follows:
p ( y | x ) = p ( y , z | x ) = p ( y | z , x ) p ( z | x ) ,
where y represents the conditional probability of generating the output, and z represents the probability distribution of the latent variable.
Latent diffusion models predict the perturbation ε = f ( z t , t , x ) . When trained, the network f can infer z. We choose a parameterization so that z 0 = f ( z t , t , x ) . An approximation of z is predicted in every denoising step z 0 , and we sample the input of the next denoising step z t 1 by diffusing it t 1 times. We use q ( z t 1 z 0 ) to clarify this diffusion process. We use the discrete cosine transform (DCT) and the inverse discrete cosine transform (iDCT) operations. Looking specifically at Algorithm 1, As shown in Figure 2, one linear layer of TransLinear inputs the DCT spectrum with step size t and one linear layer is outputs to map the joint dimensions. At the same time, N TransLinear blocks are stacked in TransLinear, features are extracted through two fully connected layers (TFCs), and the signals are passed and reduced layer by layer, while the iDCT is combined to reduce the action data. The core component prediction process module (PPM) proposed in this paper is shown in Figure 3. We have designed the PPM module, and several PPM modules have been merged to form the prediction processing unit (PPU), to perform human movement prediction analysis based on physical principles and data-driven principles, respectively; “FC” stands for the fully connected layer, “LN” and “ReLU” represent the layer normalization and activation layer. “Transpose” stands for the exchange of the spatial and temporal dimensions. Figure 4 shows the parameters M, C, and G used in the physical principle. “MLP” represents a framework that has the same architecture as “ M L P h ” but is different in the number of blocks of the MLP.
Algorithm 1 DCT and iDCT
Input: sequences: x = [ x 1 , x 2 , , x T ] R T × 3 J .
Output: sequences: x ^ = [ x ^ 1 , x ^ 2 , , x ^ T ] .
  1: DCT transfer operation:
  2: D R ( T + N ) × ( T + N ) ;
  3: y = DCT ( x ) = D x ;
  4: Generate sequences:
  5: y R N × 3 J ;
  6: y t = γ t y + 1 γ t μ , μ N ( 0 , I ) ;
  7: iDCT transfer operation:
  8: x ^ = iDCT ( y ) = D T y ;
  9: return  x ^ .
Having an N-frame motion sequence x T + 1 : T + N R N × 3 J , we put the sequence into the DCT domain through the DCT(·) operation:
y = D C T ( x ) = D x ,
where D R N × N is the predefined DCT basis, and y R N × 3 J is the DCT coefficients. As the DCT operation is completed, we can overturn the motion sequence in the DCT domain through an iDCT(·) operation:
x = i D C T ( y ) = D T y .
In order to achieve smoothness in switching of human motions, we take measures to perform the DCT and iDCT operations selecting the first L rows of D and D T .
We use three different noise scheduling strategies, square root scheduling, cosine scheduling, and sigmoid scheduling. These strategies are used to adjust the level of noise added at different time steps, thus ensuring that smooth and stable predictions are generated during the iteration process. Specifically, we perform the above DCT operation x R N × 3 J for the first E frequency components, obtaining the spectrum y 0 R E × 3 J . The noisy DCT spectrum y t at the time step t can be computed by a reparameterization trick:
y t = γ ¯ t y 0 + 1 γ ¯ t μ ,
where γ ¯ t = i = 1 t γ i , γ i [ 0 , 1 ] are predefined variance parameters, and μ N [ 0 , I ] .

3.2. Physics-Based and Data-Driven Motion Prediction Model

We firstly elaborate the rules of the Euler–Lagrange equations by looking specifically at Algorithm 2. What is more, we introduce the model dynamics process.
Algorithm 2 Training procedure of LDPM
Input: Observation sequences x 1 : T = [ x 1 , x 2 , , x T ] , time steps t + 1 , physic feature H k and geometric feature H s .
Output: Future sequences y ^ T + 1 : T + N .
  1: Extract feature:
  2: Using MLP to obtain two features:
  3: H k = MLP k ( y t 1 T ) ,
  4: H s = MLP s ( y t + 3 T ;
  5: Dynamic modeling:
  6: M T + N = MLP M ( H k H s )
  7: C T + N = MLP C ( H k H s )
  8: G T + N = MLP G ( H k H s )
  9: Data-driven:
10: Using the full connectivity layer:
11: y data = MLP data ( y t = 1 : T ) ;
12: Fusion physic and data:
13: Addition weight to predict:
14: y fusion = ( 1 w t ) y physics + w t y data ;
15: return  y fusion .
The Euler–Lagrange equations are use variables that comprehensively define the configuration of the physical system when applied to modeling human motion, and inspired by the widely used SMPL human model. The SMPL models the body as a 3D mesh. The motion parameters capture joint angles. In contrast, the parameters of shape represent body shape coefficients, governing ingredients of physical aspects such as height and other proportions. Using forward kinematics, the positions of the vertices and body joints in a 3D human model can be determined. The subject’s body shape is assumed to remain constant. Thus, the motion sports routes within a world coordinate system are described by combining the pose parameters with the translation parameters H. Consequently, we define the coordinate as
y = ( δ , H ) .
In the implementation process, δ denotes the Euler angles of the body joints. Our approach takes into account biomechanical constraints. This strategy effectively eliminates redundant-angle solutions and ensures a more accurate representation of joint movements.
Building upon the coordinate system outlined above, we represent the velocity and acceleration at frame t as y ˜ t and y ˘ t , respectively. The dynamics of the human body, governed by the Euler–Lagrange equation, can then be expressed as follows:
M t y ˘ t + C t = G t .
The Euler–Lagrange equations are important for modeling and analyzing human motion using two key processes. Forward dynamics is concerned with solving the equations to predict the body configuration in the subsequent frame based on the physical parameters M t and C t , and the applied forces G t . Starting from a defined system state, we first compute y ˘ t , which is then used to forecast future motion through the application of Euler’s method, represented as follows:
y ˘ t = M t 1 ( G t C t ) ,
y ˜ t + 1 = y ˜ t + y ˘ t T ,
y t + 1 = y t + y ˜ t T ,
where T denotes the time interval between consecutive frames, and M t 1 represents the inverse of the inertia matrix. In contrast, inverse dynamics focuses on estimating the unknown physical parameters based on existing data. In this study, we introduce a novel approach that integrates the Euler–Lagrange equations into the motion prediction framework. By explicitly modeling them, we develop a physics-based motion prediction model, which is detailed below.
The in-depth modeling of human motion is first achieved through a physically based predictive model based on the Euler–Lagrange equation, which is expressed in a generalized coordinate system as
M t y t + C t = G t ,
where M t R 75 × 75 denotes the total inertia matrix, which is determined by the generalized position and parameters such as body mass and inertia; C t R 75 denotes the generalized bias force; and G t R 75 denotes the generalized force. We follow (3) and use processes such as forward and inverse dynamics to model body dynamics. Using geometric features extracted from the first three frames of the Euler–Lagrange equations that fully specify a moment t, we use an MLP to extract the physical features H k and the geometric features H s from the motion history, and in turn estimate these physical parameters:
H k = MLP k ( y t = 1 T ) ,
H s = MLP s ( y T + 1 T + 3 ) .
The generalized inertia matrix M, the bias force C, and the external force G are then predicted by another MLP:
M T + N = MLP M ( H k H s ) ,
C T + N = MLP C ( H k H s ) ,
G T + N = MLP G ( H k H s ) .
Our approach is specifically designed to maximize the expressive capabilities of neural networks while strictly adhering to the underlying physics equations. This ensures that the relationships among the physical parameters are faithfully preserved. Using the physical parameters, the physics-based model employs an ODE solver to compute y p h y s i c s , where forward dynamics is applied to predict future states.
For training of the physics-based model, we utilize
y p h y s i c s = MLP d a t a ( y t = 1 T ) ,
where y T represents the ground truth position.
To forecast future motion, the physics-based motion prediction model is applied iteratively, generating the configuration for the next frame based on prior estimates and historical input motion. However, this approach primarily focuses on the immediate vicinity of the current frame, which can lead to significant error accumulation over longer prediction horizons. To mitigate this challenge, we incorporate a data-driven model and a fusion model, designed to reduce error accumulation, as detailed below.
Current approaches have shown significant progress in leveraging data to model long-term dependencies, enabling the direct prediction of future human motion from the input historical motion sequence y T using neural networks. A data-driven forecasting model is then introduced that utilizes historical motion data to capture long-term temporal dependencies. With a fully connected layer, we generate future data-driven predictions:
y d a t a = MLP d a t a ( y t = 1 T ) .
Based on the prediction y p h y s i c s of the above physical model, the prediction generation for each frame is dynamically adjusted by weighting w t the fusion data driver, and the final output fusion prediction is
y f u s i o n = ( 1 w t ) y p h y s i c s + w t y d a t a .

3.3. Training of the Model

We train the latent diffusion model [24] to learn to sample in this space. The diffusion model is a probabilistic model that learns the data distribution y p y by gradually removing the noise from the random Gaussian noise. Our proposed new model, SLDP, initializes the noise scheduling and prediction modules, and during the diffusion process PhysDa corrects for the introduced noise at each time step, allowing the constraints of the physical model to be reflected in the prediction. The final generated motion data not only has data-driven accuracy, but also conforms to physical plausibility. During the sampling generation process, we perform several iterations through the PhysDa module to generate the predictions step by step, so that the predictions gradually approximate the real motion trajectories. And we use a noise prediction network parameterized as θ to generate the prediction noise μ θ ( y t , t ) , and optimize the parameter θ by the noise prediction loss L:
L = E μ , t μ μ θ ( y t , t ) 2 .

4. Experiments

Datasets. We use the datasets Human3.6M [52], HumanEva-I [53], AMASS [54] to evaluate our model. Human3.6M is a comprehensive 3D human motion capture dataset widely used for motion prediction, pose estimation, and action recognition. It includes 11 participants performing 15 everyday activities, with high-resolution 3D joint data for 17 key joints. The dataset offers both 2D and 3D annotations, making it ideal for 2D-to-3D pose estimation and motion prediction tasks. HumanEva-I features synchronized 2D video frames and 3D motion capture data from multiple subjects performing activities like walking and gesturing. It is suited for multi-view pose estimation and motion prediction, with standardized protocols for reproducibility. AMASS is a large-scale collection of 3D motion capture data, unified in the SMPL format, facilitating motion modeling, 3D pose estimation, and motion prediction. It covers a wide range of activities and provides high-quality 3D meshes and joint data, supporting integration with other datasets like Human3.6M and HumanEva-I for cross-dataset research.
Implementation details. Human3.6M features 7 subjects performing 15 motion types, where five subjects S1, S5, S6, S7, and S8 are used for training and two (S9 and S11) for evaluation. The data are processed at 50 Hz with a 17-joint skeleton excluding the root joint. Our model observes 25 frames, 0.5 s, to predict 100 future frames, 2 s. HumanEva-I includes three subjects performing five actions. It is processed at 60 Hz with a 15-joint skeleton, using 15 observation frames, 0.25 s, to predict 60 future frames, 1 s. For AMASS, we retarget the skeleton to match the Human3.6M format, enabling direct inference using the model trained on Human3.6M. The model is trained using PyTorch on an RTX 4090 GPU with 120 GB of memory. A batch size of 64 is used, with training lasting 500 epochs for the first two datasets and 50 epochs for AMASS. The frame rates are 25 Hz for Human3.6M and 30 Hz for HumanEva-I and AMASS.
The training setup includes an Adam optimizer with a learning rate of 3 × 10−4 and a decay factor of 0.9. A dropout rate of 0.2 is applied. The DCT/iDCT operation uses 20 coefficients and 10 coefficients. The latent dimension is set to 512, with eight self-attention heads. The modulation ratio is set to 1.0 and 0.5. To mitigate error accumulation and improve motion-switching capabilities, the final 20 steps of DCT completion are replaced with simpler denoising steps. Data preprocessing includes applying a Gaussian filter to reduce noise in body translation and joint angles, ensuring smoother predictions.
Evaluation metrics. Based on previous work [28], we report five metrics to measure the prediction quality. They are average pairwise distance (APD), average displacement error (ADE), final displacement error (FDE), multi-modal ADE (MMADE), and multi-modal FDE (MMFDE). APD is the L2 distance all the samples. ADE is defined as the minimum average L2 distance between the actual and predicted motions. FDE is the L2 distance between the predicted result of the last frame and the ground truth. MMADE and MMFDE are multi-modal versions of ADE and FDE, respectively, with ground truth future motions grouped by similar observations.

4.1. Comparison to the State of the Art

Quantitative results. Table 1 shows the quantitative results, highlighting that our method achieves the best ADE, 0.319, and FDE, 0.410, on the Human3.6M dataset, and is competitive on HumanEva-I, with an ADE of 0.234 and FDE of 0.369. These results demonstrate the superior accuracy of our method in both general trajectory prediction (ADE) and final position prediction (FDE), which are critical for real-world motion prediction tasks. The results in Table 2 provide insights into the efficiency of the training process at different stages. While the focus of this section is to compare the performance of the proposed method with the state of the art in terms of motion prediction accuracy (ADE and FDE), these time measurements provide further context about the computational efficiency. Although our method achieves high accuracy, our approach also aims to ensure efficient training at a reasonable computational cost.
Visualization results. As shown in Figure 5, our method performs well on the Human3.6M dataset. Red, black, and gray denote the contextual part of the initial action state, and purple and green denote the predicted action sequences. Over time, our model predicts for a long time based on the physical constraints of the human body, and the predicted values are overall close to the true values. As shown in Figure 6, on the HumanEva-I dataset, our method fuses spatio-temporal information to predict standing, arm swinging, and jumping movements, and the prediction accuracy is significantly improved after 500 epochs of training. In Figure 7, compared with the training error, the value is lower. The training error is always lower than the validation error, indicating that the model performs well on the training set. The validation error and test error show that the model has limited generalization ability on the validation set, but performs well on the test set.

4.2. Ablation Study

LDPM modeling setup. Here, we examine the effects of the denoising length and variance scheduler on the experiments. Table 3 presents the best choice, which is the cosine scheduler for training on the Human3.6M and HumanEva-I datasets. The evaluation indicators include FDE, MMADE, and MMFDE. The cosine scheduler on the Human3.6M dataset achieves a minimum FDE error of 0.410, which is better than linear and sqrt. Compared to the sqrt scheduler, the cosine scheduler’s error is reduced by 44.1%, which shows a significant performance improvement and provides higher prediction accuracy in terms of final position error. On the HumanEva-I dataset, the cosine scheduler achieves a minimum FDE error of 0.229, which is 11.2% and 54.9% lower than linear and sqrt, respectively. The MMADE of the cosine scheduler is 0.369, which is also the lowest among the three schedulers, indicating that it is the most stable in terms of average prediction error.
By approximating the values of the DCT and iDCT to improve the efficiency, the E choices of 5 and 10 have a significantly superior performance. Table 4 and Table 5 show the model’s performance on the Human3.6M and HumanEva-I datasets, respectively. On Human3.6M, E = 5 gives the best results, with the lowest FDE, while on HumanEva-I, E = 10 provides the best overall accuracy and efficiency.
Prediction Networks. The TransLinear layer in the diffusion processing unit (DPU) we proposed performs skip connections. In Table 6, the skip connection results are better and the generated motion is more realistic. Table 7 shows that the best results on Human3.6M occur at eight layers, while on HumanEva-I, the best results are at four layers. Table 8 shows that increasing the noise steps from 100 to 1000 significantly improves performance, with notable reductions in FDE, MMADE, and MMFDE on both datasets.
Moreover, as shown in Figure 8, we use 28.4M and 14.45M parameter, respectively, in Human3.6M and HumanEva-I. Figure 9 compares the performance of various motion prediction methods in two plots. In the left plot, the motion value decreases as the predicted frames increase, where the GT line remains constant and the other lines show a decrease in predicted motion. Our method (represented by the purple line) maintains the lowest motion value in most predicted frames, with better consistency with the GT than other methods such as HumanMAC and MotionDiff. In the right plot, the cumulative error increases as the predicted number of frames increases. Our method again shows superior performance, keeping the cumulative error lower than all other methods, especially in the early predicted frames. As the predicted frames progress, the error of other methods increases faster than our method.
Table 9 shows the performance of our proposed method. A 0 value represents no switching capability or partial body controllability, and 1 represents switching capability or partial body controllability. MotionDiff and Deligan have relatively limited performance, both having 0 values. Our proposed method has the most comprehensive capabilities, supporting switching motions together with HumanMAC and BeLFusion, and can independently control parts of the body. Table 10 compares the performance metrics of various motion prediction models on the Human3.6M dataset. The table evaluates the models based on three criteria: whether they use a single-stage approach (Y for yes and N for no), the loss function used, and the frames per second (FPS) performance. Our model uses a single-stage approach, as indicated by “Y”, while other models such as MotionDiff, BeLFusion, Diversampling, etc., do not. The loss function used by our method is comparable to other models such as HumanMAC, but significantly better than MotionDiff. Compared to other models, our model achieves the highest FPS and performs well in real-time performance.
Table 11 shows that our method has a significantly lower runtime than other methods, with MotionDiff being the closest. Our method has a parameter size of 25.9M, which is larger than HumanMAC but smaller than Gsps. Our method has the lowest ADE metric, outperforming all other methods. Similarly, our method achieves the lowest FDE of 0.410, outperforming other models, including HumanMAC at 0.480. Our method achieves the lowest ADE and FDE values, indicating higher motion prediction accuracy compared to existing methods. As shown in Table 12, the proposed hybrid model outperforms both the data-driven model and the physics-based model in all evaluation metrics on the Human3.6M dataset. Compared with other models, it has a higher APD of 13.158 and performs best in ADE, FDE, MMADE, and MMFDE, with values of 0.319, 0.410, 0.430, and 0.473, respectively. This shows that the hybrid approach effectively combines the advantages of the data-driven model and the physics-based model to achieve more accurate motion prediction.

5. Discussion

Recent advancements in person re-identification (Re-ID) have tackled challenges like occlusion, viewpoint changes, and feature consistency. For instance, [55] shows how addressing depth variations can boost Re-ID performance. Our model, which combines latent diffusion with physical motion prediction, could benefit from similar depth adaptation to handle occlusion better. In [56], the authors explore deep learning methods for handling occlusion, which is a key issue our model addresses using Euler–Lagrange equations for motion prediction. This approach aims to improve prediction reliability in such situations. The 3D Re-ID technique in [57] uses both global guidance and local feature aggregation, which can be applied to our model to improve its prediction accuracy. Additionally, multi-view learning in [58] enhances 3D shape understanding, and we see potential to use this approach to boost long-term motion prediction accuracy. Finally, [59] focuses on feature consistency and contrast enhancement to improve Re-ID. Our model already reduces long-term prediction errors, and adopting these techniques could reduce errors even further.
In summary, our model significantly improves long-term human motion prediction and can be combined with existing Re-ID techniques to address occlusion, depth, and feature consistency, offering new directions for future research in both human motion prediction and pedestrian Re-ID.

6. Conclusions

In this paper, we proposed a model based on latent diffusion modeling and physically driven components, using PPM and PPU modules to address the problems of low long-term prediction accuracy and severe error accumulation. The framework combines motion generated by random noise with physical control derived from the Euler–Lagrange equations to provide accurate predictions for future frames of human motion. Our approach significantly improves the performance over existing methods, achieving error reduction of 15.7% in specific metrics such as ADE. Looking ahead, we plan to further investigate the integration of physical principles with deep learning techniques for a wider range of spatio-temporal tasks, with the potential to improve performance in areas such as human motion prediction.

Author Contributions

Z.R.: Conceptualization, investigation, methodology, validation, visualization, writing—original draft preparation, writing—review and editing. M.J.: Conceptualization, investigation, methodology, validation, visualization. H.N.: Conceptualization, investigation, visualization, writing—review and editing. J.S.: Conceptualization, investigation, methodology, validation, visualization, writing—review and editing. A.D.: Conceptualization, investigation, methodology, validation, visualization, writing—review and editing. Q.Z.: Conceptualization, investigation, methodology, validation, visualization, writing—review and editing. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by Yunnan provincial major science and technology special plan projects under Grant 202302AD08008, Natural Science Foundation of Guangdong Province (Nos. 2022A1515140119, 2023A1515011307), Dongguan Science and Technology Development Project (No. CWSQ20230613782), Characteristic Innovation Project of Ordinary Universities in Guangdong Province (2021KTSCX189).

Data Availability Statement

The datasets analyzed during the current study are available in the https://github.com/jimm77777/LDPM, accessed on 16 December 2024. The datasets used are all public datasets, and this manuscript does not contain any information about the study participants.

Acknowledgments

The authors thank everyone who contributed to this article.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Zheng, W.; Song, R.; Guo, X.; Zhang, C.; Chen, L. Genad: Generative end-to-end autonomous driving. In European Conference on Computer Vision (ECCV); Springer: Cham, Switzerland, 2025; pp. 87–104. [Google Scholar]
  2. Lu, G.; Zhang, S.; Wang, Z.; Liu, C.; Lu, J.; Tang, Y. Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation. In European Conference on Computer Vision (ECCV); Springer: Cham, Switzerland, 2025; pp. 349–366. [Google Scholar]
  3. Hu, L. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 16–22 June 2024; pp. 8153–8163. [Google Scholar]
  4. Kerbl, B.; Kopanas, G.; Leimkühler, T.; Drettakis, G. 3D gaussian splatting for real-time radiance field rendering. Acm Trans. Graph. 2023, 42, 1–14. [Google Scholar] [CrossRef]
  5. Goto, T.; Ohzeki, M. Online calibration scheme for training restricted Boltzmann machines with quantum annealing. arXiv 2023, arXiv:2307.09785. [Google Scholar]
  6. Wei, D.; Sun, H.; Li, B.; Lu, J.; Li, W.; Sun, X.; Hu, S. Human joint kinematics diffusion-refinement for stochastic motion prediction. AAAI Conf. Artif. Intell. 2023, 37, 6110–6118. [Google Scholar] [CrossRef]
  7. Pearce, T.; Rashid, T.; Kanervisto, A.; Bignell, D.; Sun, M.; Georgescu, R.; Macua, S.V.; Tan, S.Z.; Momennejad, I.; Hofmann, K.; et al. Imitating human behaviour with diffusion models. arXiv 2023, arXiv:2301.10677. [Google Scholar]
  8. Adeli, V.; Ehsanpour, M.; Reid, I.; Niebles, J.C.; Savarese, S.; Adeli, E.; Rezatofighi, H. Tripod: Human trajectory and pose dynamics forecasting in the wild. In Proceedings of the International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 11–17 October 2021; pp. 13390–13400. [Google Scholar]
  9. Cai, Z.; Ren, D.; Zeng, A.; Lin, Z.; Yu, T.; Wang, W.; Liu, Z. Humman: Multi-modal 4D human dataset for versatile sensing and modeling. In Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel, 23–27 October 2022; pp. 557–577. [Google Scholar]
  10. Gupta, A.; Johnson, J.; Fei-Fei, L.; Savarese, S.; Alahi, A. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018; pp. 2255–2264. [Google Scholar]
  11. Aliakbarian, S.; Saleh, F.S.; Salzmann, M.; Petersson, L.; Gould, S. A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 5223–5232. [Google Scholar]
  12. Bouazizi, A.; Holzbock, A.; Kressel, U.; Dietmayer, K.; Belagiannis, V. Motionmixer: Mlp-based 3D human body pose forecasting. arXiv 2022, arXiv:2207.00499. [Google Scholar]
  13. Li, B.; Zhao, Y.; Zhelun, S.; Sheng, L. Danceformer: Music conditioned 3D dance generation with parametric motion transformer. AAAI Conf. Artif. Intell. 2022, 36, 1272–1279. [Google Scholar] [CrossRef]
  14. Alexanderson, S.; Nagy, R.; Beskow, J.; Henter, G.E. Listen, denoise, action! audio-driven motion synthesis with diffusion models. Acm Trans. Graph. 2023, 42, 1–20. [Google Scholar] [CrossRef]
  15. Lu, C.; Zhou, Y.; Bao, F.; Chen, J.; Li, C.; Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Adv. Neural Inf. Process. Syst. 2022, 35, 5775–5787. [Google Scholar]
  16. Gurumurthy, S.; Kiran Sarvadevabhatla, R.; Venkatesh Babu, R. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 166–174. [Google Scholar]
  17. Hong, F.; Zhang, M.; Pan, L.; Cai, Z.; Yang, L.; Liu, Z. Avatarclip: Zero-shot text-driven generation and animation of 3D avatars. arXiv 2022, arXiv:2205.08535. [Google Scholar] [CrossRef]
  18. Sun, B.; Yang, Y.; Zhang, L.; Cheng, M.M.; Hou, Q. Corrmatch: Label propagation via correlation matching for semi-supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 16–22 June 2024; pp. 3097–3107. [Google Scholar]
  19. Starke, S.; Mason, I.; Komura, T. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Trans. Graph. 2022, 41, 1–13. [Google Scholar] [CrossRef]
  20. Ju, X.; Zeng, A.; Zhao, C.; Wang, J.; Zhang, L.; Xu, Q. Humansd: A native skeleton-guided diffusion model for human image generation. In Proceedings of the International Conference on Computer Vision (ICCV), Paris, France, 2–6 October 2023; pp. 15988–15998. [Google Scholar]
  21. Tevet, G.; Gordon, B.; Hertz, A.; Bermano, A.H.; Cohen-Or, D.; Liu, Z. Motionclip: Exposing human motion generation to clip space. In Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel, 23–27 October 2022; pp. 358–374. [Google Scholar]
  22. Cai, X.; Cheng, P.; Liu, S.; Zhang, H.; Sun, H. Human Motion Prediction Based on a Multi-Scale Hypergraph for Intangible Cultural Heritage Dance Videos. Electronics 2023, 12, 1–21. [Google Scholar] [CrossRef]
  23. Maeda, T.; Ukita, N. Motionaug: Augmentation with physical correction for human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 6427–6436. [Google Scholar]
  24. Xie, K.; Wang, T.; Iqbal, U.; Guo, Y.; Fidler, S.; Shkurti, F. Physics-based human motion estimation and synthesis from videos. In Proceedings of the International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 11–17 October 2021; pp. 11532–11541. [Google Scholar]
  25. Yuan, Y.; Song, J.; Iqbal, U.; Vahdat, A.; Kautz, J. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the International Conference on Computer Vision (ICCV), Paris, France, 2–6 October 2023; pp. 16010–16021. [Google Scholar]
  26. Zhang, Z.; Zhu, Y.; Rai, R.; Doermann, D. Pimnet: Physics-infused neural network for human motion prediction. IEEE Robot. Autom. Lett. 2022, 7, 8949–8955. [Google Scholar] [CrossRef]
  27. Karniadakis, G.E.; Kevrekidis, I.G.; Lu, L.; Perdikaris, P.; Wang, S.; Yang, L. Physics-informed machine learning. Nat. Rev. Phys. 2021, 3, 422–440. [Google Scholar] [CrossRef]
  28. Chen, L.H.; Zhang, J.; Li, Y.; Pang, Y.; Xia, X.; Liu, T. Humanmac: Masked motion completion for human motion prediction. In Proceedings of the International Conference on Computer Vision (ICCV), Paris, France, 2–6 October 2023; pp. 9544–9555. [Google Scholar]
  29. Zhang, Y.; Kephart, J.O.; Ji, Q. Incorporating physics principles for precise human motion prediction. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 3–8 January 2024; pp. 6164–6174. [Google Scholar]
  30. Barquero, G.; Escalera, S.; Palmero, C. Belfusion: Latent diffusion for behavior-driven human motion prediction. In Proceedings of the International Conference on Computer Vision (ICCV), Paris, France, 2–6 October 2023; pp. 2317–2327. [Google Scholar]
  31. Zhong, C.; Hu, L.; Zhang, Z.; Ye, Y.; Xia, S. Spatio-temporal gating-adjacency gcn for human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 6447–6456. [Google Scholar]
  32. Salzmann, T.; Pavone, M.; Ryll, M. Motron: Multimodal probabilistic human motion forecasting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 6457–6466. [Google Scholar]
  33. Lucas, T.; Baradel, F.; Weinzaepfel, P.; Rogez, G. Posegpt: Quantization-based 3D human motion generation and forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel, 23–27 October 2022; pp. 417–435. [Google Scholar]
  34. Blattmann, A.; Milbich, T.; Dorkenwald, M.; Ommer, B. Behavior-driven synthesis of human dynamics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 12236–12246. [Google Scholar]
  35. Dang, L.; Nie, Y.; Long, C.; Zhang, Q.; Li, G. Diverse human motion prediction via gumbel-softmax sampling from an auxiliary space. In Proceedings of the ACM International Conference on Multimedia (ACM MM), Lisboa, Portugal, 10–14 October 2022; pp. 5162–5171. [Google Scholar]
  36. Xu, S.; Wang, Y.X.; Gui, L.Y. Diverse human motion prediction guided by multi-level spatial-temporal anchors. In Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel, 23–27 October 2022; pp. 251–269. [Google Scholar]
  37. Dang, L.; Nie, Y.; Long, C.; Zhang, Q.; Li, G. Msr-gcn: Multi-scale residual graph convolution networks for human motion prediction. In Proceedings of the International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 11–17 October 2021; pp. 11467–11476. [Google Scholar]
  38. Li, M.; Chen, S.; Chen, X.; Zhang, Y.; Wang, Y.; Tian, Q. Symbiotic graph neural networks for 3D skeleton-based human action recognition and motion prediction. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 44, 3316–3333. [Google Scholar] [CrossRef]
  39. Li, M.; Chen, S.; Zhang, Z.; Xie, L.; Tian, Q.; Zhang, Y. Skeleton-parted graph scattering networks for 3D human motion prediction. In Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel, 23–27 October 2022; pp. 18–36. [Google Scholar]
  40. Guo, W.; Du, Y.; Shen, X.; Lepetit, V.; Alameda-Pineda, X.; Moreno-Noguer, X. Back to mlp: A simple baseline for human motion prediction. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2–7 January 2023; pp. 4809–4819. [Google Scholar]
  41. Xu, C.; Tan, R.T.; Tan, Y.; Chen, S.; Wang, Y.G.; Wang, X.; Wang, Y. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023; pp. 1410–1420. [Google Scholar]
  42. Dhariwal, P.; Nichol, A. Diffusion models beat gans on image synthesis. Adv. Neural Inf. Process. Syst. 2021, 34, 8780–8794. [Google Scholar]
  43. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 10684–10695. [Google Scholar]
  44. Yu, Z.; Yin, Z.; Zhou, D.; Wang, D.; Wong, F.; Wang, B. Talking head generation with probabilistic audio-to-visual diffusion priors. In Proceedings of the International Conference on Computer Vision (ICCV), Paris, France, 2–6 October 2023; pp. 7645–7655. [Google Scholar]
  45. Ho, J.; Gritsenko, A.; Chan, W.; Norouzi, M.; Fleet, D.J. Video diffusion models. Adv. Neural Inf. Process. Syst. 2022, 35, 8633–8646. [Google Scholar]
  46. Xu, M.; Yu, L.; Song, Y.; Shi, C.; Ermon, S.; Tang, J. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv 2022, arXiv:2203.02923. [Google Scholar]
  47. Xu, J.; Wang, X.; Cheng, W.; Cao, Y.P.; Shan, Y.; Qie, X.; Cao, S. Dream3d: Zero-shot text-to-3D synthesis using 3D shape prior and text-to-image diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 18–22 June 2023; pp. 20908–20918. [Google Scholar]
  48. Popov, V.; Vovk, I.; Gogoryan, V.; Sadekova, T.; Kudinov, M. Grad-tts: A diffusion probabilistic model for text-to-speech. In Proceedings of the International Conference on Machine Learning (ICML), Virtal, 18–24 July 2021; pp. 8599–8608. [Google Scholar]
  49. Yang, J.; Wang, C.; Li, Z.; Wang, J.; Zhang, R. Semantic human parsing via scalable semantic transfer over multiple label domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 18–22 June 2023; pp. 19424–19433. [Google Scholar]
  50. Mao, W.; Liu, M.; Salzmann, M. Generating smooth pose sequences for diverse human motion prediction. In Proceedings of the International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 11–17 October 2021; pp. 13309–13318. [Google Scholar]
  51. Zhang, Y.; Black, M.J.; Tang, S. We are more than our joints: Predicting how 3D bodies move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 3372–3382. [Google Scholar]
  52. Ionescu, C.; Papava, D.; Olaru, V.; Sminchisescu, C. Human3. 6m: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 36, 1325–1339. [Google Scholar] [CrossRef]
  53. Sigal, L.; Balan, A.O.; Black, M.J. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int. J. Comput. Vis. 2010, 87, 4–27. [Google Scholar] [CrossRef]
  54. Mahmood, N.; Ghorbani, N.; Troje, N.F.; Pons-Moll, G.; Black, M.J. AMASS: Archive of motion capture as surface shapes. In Proceedings of the International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 5442–5451. [Google Scholar]
  55. Zhang, H.; Ning, X.; Wang, C.; Ning, E.; Li, L. Deformation depth decoupling network for point cloud domain adaptation. Neural Netw. 2024, 180, 106626. [Google Scholar] [CrossRef]
  56. Ning, E.; Wang, C.; Zhang, H.; Ning, X.; Tiwari, P. Occluded person re-identification with deep learning: A survey and perspectives. Expert Syst. Appl. 2024, 239, 122419. [Google Scholar] [CrossRef]
  57. Wang, C.; Ning, X.; Li, W.; Bai, X.; Gao, X. 3D person re-identification based on global semantic guidance and local feature aggregation. IEEE Trans. Circuits Syst. Video Technol. 2023, 34, 4698–4712. [Google Scholar] [CrossRef]
  58. Ning, E.; Wang, C.; Zhang, H.; Ning, X.; Tiwari, P. Pedestrian 3D shape understanding for person re-identification via multi-view learning. IEEE Trans. Circuits Syst. Video Technol. 2024, 34, 5589–5602. [Google Scholar]
  59. Ning, E.; Zhang, C.; Wang, C.; Ning, X.; Chen, H.; Bai, X. Pedestrian Re-ID based on feature consistency and contrast enhancement. Displays 2023, 79, 102467. [Google Scholar] [CrossRef]
Figure 1. Comparison of motion prediction approaches: GCN-based framework and the proposed Latent Diffusion and Physical Principles Model (LDPM). (a) GCN method. (b) The proposed LDPM.
Figure 1. Comparison of motion prediction approaches: GCN-based framework and the proposed Latent Diffusion and Physical Principles Model (LDPM). (a) GCN method. (b) The proposed LDPM.
Electronics 14 00605 g001
Figure 2. Overview of the proposed method LDP; “⊕” denotes the add weight.
Figure 2. Overview of the proposed method LDP; “⊕” denotes the add weight.
Electronics 14 00605 g002
Figure 3. Core components of physics-informed and data-driven motion prediction: (a) Prediction processing unit (PPU) and (b) prediction process module (PPM).
Figure 3. Core components of physics-informed and data-driven motion prediction: (a) Prediction processing unit (PPU) and (b) prediction process module (PPM).
Electronics 14 00605 g003
Figure 4. MLP architecture for physics-based data motion prediction: (a) Parameterization of mass (M), (b) damping coefficient (C), and (c) gravitational constant (G).
Figure 4. MLP architecture for physics-based data motion prediction: (a) Parameterization of mass (M), (b) damping coefficient (C), and (c) gravitational constant (G).
Electronics 14 00605 g004
Figure 5. Visualization of accurate human pose predictions over 2 s using the Human3.6M dataset. (a) presents the predicted poses for shorter durations, 0.25 s, 0.5 s, and 0.75 s, demonstrating strong alignment with the ground truth (GT). (b) extends the prediction timeline to longer durations, 1.0 s, 1.5 s, and 2.0 s, showing consistent performance in capturing the end poses with minimal deviation from the ground truth.
Figure 5. Visualization of accurate human pose predictions over 2 s using the Human3.6M dataset. (a) presents the predicted poses for shorter durations, 0.25 s, 0.5 s, and 0.75 s, demonstrating strong alignment with the ground truth (GT). (b) extends the prediction timeline to longer durations, 1.0 s, 1.5 s, and 2.0 s, showing consistent performance in capturing the end poses with minimal deviation from the ground truth.
Electronics 14 00605 g005
Figure 6. Visualization of accurate 2 s pose predictions on the HumanEva-I dataset. (a) shows the predicted poses at shorter intervals, 0.25 s, 0.5 s, and 0.75 s, with the predicted poses (purple and green) closely aligning with the ground truth (GT) poses (black). (b) extends the prediction timeline to longer intervals, 1.0 s, 1.5 s, and 2.0 s, demonstrating the model’s ability to maintain high accuracy in pose estimation even at extended durations.
Figure 6. Visualization of accurate 2 s pose predictions on the HumanEva-I dataset. (a) shows the predicted poses at shorter intervals, 0.25 s, 0.5 s, and 0.75 s, with the predicted poses (purple and green) closely aligning with the ground truth (GT) poses (black). (b) extends the prediction timeline to longer intervals, 1.0 s, 1.5 s, and 2.0 s, demonstrating the model’s ability to maintain high accuracy in pose estimation even at extended durations.
Electronics 14 00605 g006
Figure 7. Error progression across training, validation, and testing phases over 50 epochs on the AMASS dataset. The training error decreases and stabilizes at a low level, the validation error decreases initially but stabilizes at a higher level, and the testing error remains low and constant throughout 50 epochs, indicating effective training but limited generalization.
Figure 7. Error progression across training, validation, and testing phases over 50 epochs on the AMASS dataset. The training error decreases and stabilizes at a low level, the validation error decreases initially but stabilizes at a higher level, and the testing error remains low and constant throughout 50 epochs, indicating effective training but limited generalization.
Electronics 14 00605 g007
Figure 8. Comparison of model performance on Human3.6M and HumanEva-I datasets with different model parameters: epochs = 5, layers = 6, cosine.
Figure 8. Comparison of model performance on Human3.6M and HumanEva-I datasets with different model parameters: epochs = 5, layers = 6, cosine.
Electronics 14 00605 g008
Figure 9. Comparison of motion prediction methods: average predicted motion vs. cumulative distribution function (CDF) of average absolute motion error on Human3.6M.
Figure 9. Comparison of motion prediction methods: average predicted motion vs. cumulative distribution function (CDF) of average absolute motion error on Human3.6M.
Electronics 14 00605 g009
Table 1. Quantitative results of motion prediction methods on Human3.6M and HumanEva-I datasets.
Table 1. Quantitative results of motion prediction methods on Human3.6M and HumanEva-I datasets.
MethodHuman3.6MHumanEva-I
APDADEFDEMMADEMMFDEAPDADEFDEMMADEMMFDE
MotionDiff [6]15.3530.4110.5090.5360.5365.9310.2320.2360.3520.320
Deligan [16]6.5090.4830.5200.5450.5202.1770.3060.3220.3850.371
HumanMAC [28]6.3010.3690.4800.5090.5456.5540.2090.2230.3420.335
BeLFusion [30]7.6020.4200.4720.4740.5076.1090.2200.2340.3420.316
Diversampling [35]15.3100.3700.4800.4820.5096.1090.2200.2340.3420.316
Gsps [50]14.7570.3890.4960.4760.5255.8250.2330.2400.3440.331
Mojo [51]12.5790.4120.4970.4970.5384.1810.2340.2440.3690.347
Ours13.1580.3190.4100.4300.47310.1190.2340.2290.3690.372
The bold number represents the best results.
Table 2. Time details for different training steps and their corresponding batch and total time measurements for the AMASS dataset.
Table 2. Time details for different training steps and their corresponding batch and total time measurements for the AMASS dataset.
StepBatch Time (s)Total Time (s)
1/25980.251
1001/25980.25770
2001/25980.243138
1/590.0240
1/11410.1030
1001/11410.14925
Table 3. Results of different schedulers on the Human3.6M and HumanEva-I datasets.
Table 3. Results of different schedulers on the Human3.6M and HumanEva-I datasets.
SchedulerHuman3.6MHumanEva-I
FDEMMADEMMFDEFDEMMADEMMFDE
Linear0.5210.5180.5630.2580.4080.422
Sqrt0.7330.6650.7680.5080.5450.612
Cosine0.4100.4300.4730.2290.3690.372
The bold number represents the best results.
Table 4. Experimental results on Human3.6M for varying values of E.
Table 4. Experimental results on Human3.6M for varying values of E.
EHuman3.6M
FDEMMADEMMFDE
5.0000.4100.3900.423
10.0000.5340.5240.572
20.0000.4100.4300.473
The bold number represents the best results.
Table 5. Experimental results on HumanEva-I for varying values of E.
Table 5. Experimental results on HumanEva-I for varying values of E.
EHumanEva-I
FDEMMADEMMFDE
5.0000.2650.4080.438
10.0000.2290.3690.372
20.0000.2480.4010.403
The bold number represents the best results.
Table 6. Results of different network design configurations with and without skip connections on Human3.6M and HumanEva-I datasets.
Table 6. Results of different network design configurations with and without skip connections on Human3.6M and HumanEva-I datasets.
Skip ConnectionHuman3.6MHumanEva-I
APDADEFDEAPDADEFDE
N15.2360.4000.4239.9270.2260.217
Y13.1580.3190.41010.1190.2340.229
The bold number represents the best results.
Table 7. Comparison of model performance when varying the number of layers on the Human3.6M and HumanEva-I datasets.
Table 7. Comparison of model performance when varying the number of layers on the Human3.6M and HumanEva-I datasets.
LayersHuman3.6MHumanEva-I
FDEMMADEMMFDEFDEMMADEMMFDE
20.5800.5460.6090.5080.5450.612
40.5510.5290.5840.2290.3690.372
60.5320.5230.5710.2390.4100.425
80.4100.4300.4730.2550.4170.439
100.5100.5160.5540.2560.4220.450
The bold number represents the best results.
Table 8. Results of different noising steps on Human3.6M and HumanEva-I datasets.
Table 8. Results of different noising steps on Human3.6M and HumanEva-I datasets.
Noising StepsHuman3.6MHumanEva-I
FDEMMADEMMFDEFDEMMADEMMFDE
100.0000.5280.5220.5650.2460.4020.419
1000.0000.4100.4300.4730.2290.3690.372
The bold number represents the best results.
Table 9. The motion editing ability of different methods.
Table 9. The motion editing ability of different methods.
MethodSwitch AbilityIndependent Body Part Control
MotionDiff [6]00
Deligan [16]00
HumanMAC [28]11
BeLFusion [30]10
Diversampling [35]00
Gsps [50]01
Mojo [51]00
Ours11
Table 10. Comparison of performance metrics and design choices across various motion prediction models on the Human3.6M dataset.
Table 10. Comparison of performance metrics and design choices across various motion prediction models on the Human3.6M dataset.
MotionDiff [6]Deligan [16]HumanMAC [28]BeLFusion [30]Diversampling [35]Gsps [50]Ours
One StageNYYNNNY
Loss4114351
FPS15222518201230
Table 11. Comparison of model performance in motion prediction across various methods in terms of runtime, parameter size, and accuracy metrics on the Human3.6M dataset.
Table 11. Comparison of model performance in motion prediction across various methods in terms of runtime, parameter size, and accuracy metrics on the Human3.6M dataset.
MotionDiff [6]HumanMAC [28]BeLFusion [30]Diversampling [35]Gsps [50]Ours
Runtime (ms)15.212.318.114.516.710.8
Parameter Size (M)45.330.135.828.740.225.9
ADE0.4110.3690.4200.3700.3890.319
FDE0.5090.4800.4720.4800.4960.410
The bold number represents the best results.
Table 12. Quantitative comparison of different models across multiple evaluation metrics on the Human3.6M dataset.
Table 12. Quantitative comparison of different models across multiple evaluation metrics on the Human3.6M dataset.
ModelAPDADEFDEMMADEMMFDE
Data-driven12.4060.3950.5170.5120.580
Physics-based11.2380.3290.5340.4870.543
Proposed hybrid13.1580.3190.4100.4300.473
The bold number represents the best results.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Ren, Z.; Jin, M.; Nie, H.; Shen, J.; Dong, A.; Zhang, Q. Towards Realistic Human Motion Prediction with Latent Diffusion and Physics-Based Models. Electronics 2025, 14, 605. https://doi.org/10.3390/electronics14030605

AMA Style

Ren Z, Jin M, Nie H, Shen J, Dong A, Zhang Q. Towards Realistic Human Motion Prediction with Latent Diffusion and Physics-Based Models. Electronics. 2025; 14(3):605. https://doi.org/10.3390/electronics14030605

Chicago/Turabian Style

Ren, Ziliang, Miaomiao Jin, Huabei Nie, Jianqiao Shen, Ani Dong, and Qieshi Zhang. 2025. "Towards Realistic Human Motion Prediction with Latent Diffusion and Physics-Based Models" Electronics 14, no. 3: 605. https://doi.org/10.3390/electronics14030605

APA Style

Ren, Z., Jin, M., Nie, H., Shen, J., Dong, A., & Zhang, Q. (2025). Towards Realistic Human Motion Prediction with Latent Diffusion and Physics-Based Models. Electronics, 14(3), 605. https://doi.org/10.3390/electronics14030605

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop