Towards Realistic Human Motion Prediction with Latent Diffusion and Physics-Based Models

Abstract

Many applications benefit from the prediction of 3D human motion based on past observations, e.g., human–computer interactions, autonomous driving. However, while existing methods based on encoding–decoding achieve good performance, prediction in the range of seconds still suffers from errors and motion switching scarcity. In this paper, we propose a Latent Diffusion and Physical Principles Model (LDPM) to achieve accurate human motion prediction. Our framework performs human motion prediction by learning information about the potential space, noise-generated motion, and combining physical control of body motion, where physics principles estimate the next frame through the Euler–Lagrange equation. The framework effectively accomplishes motion switching and reduces the error accumulated over time. The proposed architecture is evaluated on three challenging datasets: Human3.6M (Human 3D Motion Capture Dataset), HumanEva-I (Human Evaluation dataset I), and AMASS (Archive of Motion Capture as Surface Shapes). We experimentally demonstrate the significant superiority of the proposed framework in the prediction range of seconds.

1. Introduction

The task of human motion prediction focuses on predicting future motion sequences by observing past motion, which has many applications, such as autonomous driving [1], intelligent robotics [2], and animation [3]. Traditional animation production often requires animators to draw movements frame by frame, a process that takes a lot of time and effort. Through the use of physical laws, human body motion prediction can generate realistic and reasonable human body motion sequences, which allows animators to work more efficiently and enhances the user’s sense of immersion. When the user interacts virtually, the system determines the movement trend of the human body and provides timely and accurate information, e.g., predicting the patient’s movement trajectory to give appropriate guidance during rehabilitation training. The predicted motions need to be continuous, diverse, and realistic, making this task complex. Traditional methods include Gaussian [4] and restricted Boltzmann machines [5], which combine simple representations of body motion encoded as latent representations, which are then decoded into predictions [6,7,8], but accurate predictions require more complex representations of motion.

Recently, motion data have begun to be used widely, and researchers have made great progress. Researchers have proposed various deep learning models to model motion sequences, such as deep learning models based on recurring neural networks (RNNs) [9,10] and convolutional neural networks (CNNs) [11,12,13]; transformers [14,15], more intuitive than the model shown in Figure 1a, utilize graph convolutional layers to process observation sequences, create motion embeddings, and decode them into future prediction sequences. This is a fully data-driven approach focusing on learning motion patterns from structured input. Variants of generative adversarial networks (GANs) [16,17] and Variational Auto-Encoder (VAE) [18] have been used to model data distributions [19,20]. Among other areas, prior methods with encoding–decoding address the human motion prediction problem, which predicts future motions bases on existing motion frames. Different sampling phases during training result in increased computational effort. While these methods perform well in some cases, such as when distinguishing basic actions, they face challenges in real-world applications that require a wider diversity of motions—such as the transition from walking to sitting. While these methods can capture complex motions, they often ignore the fact that human motion is governed by physical principles and that the human body follows the laws of physics. Therefore, while good prediction results can be achieved, methods that rely on hyperparameter balancing and loss constraints often have difficulty producing consistent, applicable results. This highlights the necessity of carefully designing parameters to balance various loss functions and training requirements. The core argument of this paper is that by incorporating the basic principles of motion diffusion and physical motion into the learning process, prediction accuracy can be significantly improved, especially over short time intervals.

The human body is a physical system with multiple body parts working in concert to form complex motions. Muscle strength and exercise habits vary greatly from person to person. The same person can move differently under different environmental conditions and emotional states. According to the physical point of view, accurately modeling the rotation of joints is not a simple task. Since human movement is a coupling of multiple degrees of freedom, this greatly increases the difficulty of accurate prediction, e.g., the coordination of hands and feet during walking. Depending on the joint angles of the human body, the Euler–Lagrange equation [21] describes the human body motion through generalized positional coordinates along with second-order ordinary differential equations (ODEs), imposing constraints to improve the quality of the output of human body motion prediction [22,23,24], and motion diffusion models provide motion information to the prediction from random noise to improve its performance [25]. Researchers in data-driven estimation have not yet explored this approach of synergizing physics and neural networks in depth. Previous surveillance systems recorded behavior at a given time point, but human motion prediction based on diffusion models can predict future actions. This plays a crucial role in crime prevention and public safety. For example, in airports with high traffic flow, by predicting the movements of suspicious people, measures can be taken in advance to effectively reduce the likelihood of potential dangers occurring. Traditional prediction methods have limitations in dealing with the uncertainty and diversity of human movements. The emergent diffusion model, on the other hand, has significant results in predicting outcomes. It is able to predict the future state of human movement by learning the potential distribution in the data. Therefore, as shown in Figure 1b, our work focuses on fusing physics and deep learning in applying diffusion to predictive models for training in an end-to-end manner without the need for multi-stage training, which avoids balancing multiple lossy hyperparameters.

In recent years, physics-based and diffusion-based deep learning models have received increasing attention in terms of prediction accuracy and training speed [26,27,28,29]. Neural networks can efficiently solve partial differential equations [30]. Inspired by these methods, we do not use encoding–decoding methods but propose a new approach for human motion prediction by diffusion-based and physically driven methods. According to what we know, the diffusion model with denoising procedures can generate motion sequences from random noise, and can easily be pruned to unsmooth results. In concrete terms, we use observation sequences, combine the noisy and denoised mechanisms, use physics-informed deep learning, and incorporate the Euler–Lagrange equations. We demonstrate that the proposed framework (LDPM) can effectively improve the prediction accuracy of existing frameworks and obtain more physically plausible prediction results.

In summary, the main contributions of our work are as follows:

• To address the problems mentioned above, we propose a new framework. The diffusion-based and physically driven approach has the advantages of long-term prediction, high motion switching capability, and reduced error accumulation. The future motion is predicted by an efficient combination of physical principles and random noise.
• It is experimentally demonstrated that the proposed method obtains better performance and outperforms existing methods, providing a new perspective for future research.

2. Related Works

This section reviews existing work on human motion prediction and then analyzes potential diffusion-based and physically driven models to improve adjacent human dynamics models.

2.1. Human Motion Prediction

Traditional methods [12,31] try to predict movement in a deterministic way. Previous statistical methods [4] only showed consistent patterns of movements over time. With the widespread use of large amounts of motion data, deep learning has a clear advantage in generating complex motion. Yuan et al. [25] considered physical factors and generated motion trajectories more in line with the real world using a diffusion model based on a physics-guided mechanism, which required high hardware at the time of experimentation. Zhang et al. [26] incorporated physics knowledge into neural networks to learn complex patterns and reduce noise learning for human motion prediction. Karniadakis et al. [27] provided a priori knowledge through physical laws and integrated it into machine learning algorithms to reduce false predictions, but this ran the risk of introducing approximation errors and loss of accuracy. Codec models have been employed for prediction [32,33,34,35,36] using multiple loss constraints [11] to achieve diversity and accuracy of results, learning basis vectors, using sampling strategies, decoupling network parameters and Gaussian distributions, and linear interpolation to control smoothing, which raises training requirements. RNNs and their variations capture temporal information of human movements but fail to capture spatial dependencies; extraction of features by convolutional operations has low computational complexity, but simple splicing or averaging does not yet achieve optimal performance.

To address this problem, some researchers have used CNNs, GCNs [37,38,39], and spatio-temporal graph convolution [31]. Zhong et al. [31] proposed a spatio-temporally gated adjacency graph convolutional network model to focus on channel information, balance spatio-temporal weights, and flexibly adapt to the input situation, but it increased the number of model parameters. Li et al. [38] propose a symbiotic graph neural network that combines action, structural, and bipartite graphs to explain movement changes and to comprehensively understand and utilize inter-joint relationships, but validation of model robustness is lacking. Chen et al. [39] proposed a skeleton segmented graph scattering network to use multi-channel filters to obtain the graph spectral bands, model different body parts, and obtain fine features, but it has high computational cost and requires significant debugging efforts. Recently, Guo et al. [40] showed better performance using multilayer perceptron (MLP), which performs a nonlinear transformation by stacking fully connected layers, where the neurons cooperate with all the previous layers of neurons to capture complex patterns in the input data, but leads to a large number of multiply–add operations, especially when dealing with high-dimensional data. Bouazizi et al. [12] proposed a prediction model based on MLP, which independently applies two different MLPs to solve the complexity as well as the computational difficulties of methods such as RNN, CNN, GCN, etc., but the performance of the model has a certain impact on the processing details and data inputs. Xu et al. [41] utilized geometric isotropy in motion data to improve the accuracy of the model. Even so, these methods are still affected by physical motion and require many stages of training.

2.2. Latent Diffusion and Physically Driven Models

Human dynamics are modeled with equations describing the physical motion, and a lossy diffusion model of human motion can be trained in an end-to-end format to generate enriched actions. Physics-based models can improve on purely data-driven estimation and are widely used for different tasks. This is also true for diffusion models for research tasks such as image/video generation [42,43,44,45], drug discovery [46], and 3D reconstruction [47,48,49,50,51]. Among them, in monocular 3D human reconstruction, some methods estimate parameters in the Euler–Lagrange equation, improve data-driven estimation [23], and simulate realistic motion. Traditional image synthesis is prone to detail loss and image blurring, and the potential diffusion model can perform diffusion operations on the data in the potential space to generate high-resolution images [43], which gives a problem-solving perspective to traditional image synthesis methods. Probabilistic audiovisual diffusion priors [44] play a significant role in the field of audiovisual synthesis. Traditional audiovisual synthesis methods have the problem of generative incoherence. Introducing a priori knowledge and using diffusion models to fuse audio and visual features can improve the realism and naturalness of the synthesis results. Diffusion models can also be used in video processing and generation [45], i.e., modeling inter- and intra-frame relationships in video. Using the geometric diffusion mechanism [46], the complex relationship between atoms in a molecule is dealt with to generate a reasonable molecular conformation.

A 3D shape a priori and text-to-image diffusion model [48] provides a new perspective for generating 3D models directly from text descriptions. Scalable semantic transfer over multi-labeled domains [49] allows for human parsing, which accurately delineates different parts of the human body and semantic information, and provides a more accurate semantic basis for human posture estimation studies. Unlike existing prediction work, we are integrating physical principles into a diffusion model to generate accurate future motion predictions. Cai et al. [22] predict human movement by constructing multiscale hypergraphs specialized for ICH dance videos. Zhang et al. [25] used the Euler–Lagrange equations to infer kinematic forces from observed motions, and then used these forces as a motion prediction model for additional inputs to improve performance. BeLFusion [30] decomposes model training into multiple stages to predict motion through a latent space diffusion model, which can be limited by the codec. In this paper, we propose a new framework that adds a physically driven model to the diffusion model for end-to-end motion prediction.

3. Proposed Framework

Inputting some information such as text or action labels, our goal is to generate a physically plausible human motion $x_{T+1:T+N}$ of length N. We propose a Latent Diffusion and Physical Principles Model (LDPM) for human motion generation. The LDPM starts with a noisy motion $x_{1:T}$ and models a denoising distribution that denoises motions from a diffusion time step T to N (N < T). Iterative application of the model denoises the motion into a clean motion $x_{1:T}$, which becomes the final output $x_{T+1:T+N}$. An overview of our proposed method LDPM is shown in Figure 2. First, we are able to maintain the continuity of the generated results in time by introducing a noise scheduling strategy and a diffusion process. Meanwhile, a physical data-driven motion prediction model PhysDa is introduced. The model iteratively predicts future body configurations based on input historical motion data. Finally, we explore an LDPM-based training process that effectively mitigates prediction errors accumulated over time.

3.1. Potential Diffusion Model

We take the sequence of observed $T_{obs}$ coordinate system motions as $x_{1:T_{obs}}=(x_1,x_2,\dots ,x_{T_{obs}})\in R^{T_{obs}\times 3J}$, where $x_{T_{obs}}\in R^{3J}$ are the coordinates of the joints at the coordinate system $T_{obs}$ and J is the number of joints. Given the observed motions $x_{1:T_{obs}}$, the goal of the human motion prediction (HMP) problem is to predict the following N motions: $x_{T_{obs}+1:T_{obs}+N}=(x_{T_{obs}+1},x_{T_{obs}+2},\dots ,x_{T_{obs}+N})\in R^{T_{obs}\times 3J}$.

Samples z ∈ latent space V, z can be mapped to the coordinate space. Then, a latent diffusion model is trained to predict z. The generative sequences are formulated as follows:

$$p\left(y\right|x)=p(y,z\left|x\right)=p\left(y\right|z,x\left)p\right(z\left|x\right),$$

(1)

where y represents the conditional probability of generating the output, and z represents the probability distribution of the latent variable.

Latent diffusion models predict the perturbation $\epsilon =f(z_t,t,x)$. When trained, the network f can infer z. We choose a parameterization so that $z_0$ = $f(z_t,t,x)$. An approximation of z is predicted in every denoising step $z_0$, and we sample the input of the next denoising step $z_{t-1}$ by diffusing it $t-1$ times. We use $q(z_{t-1}\mid z_0)$ to clarify this diffusion process. We use the discrete cosine transform (DCT) and the inverse discrete cosine transform (iDCT) operations. Looking specifically at Algorithm 1, As shown in Figure 2, one linear layer of TransLinear inputs the DCT spectrum with step size t and one linear layer is outputs to map the joint dimensions. At the same time, N TransLinear blocks are stacked in TransLinear, features are extracted through two fully connected layers (TFCs), and the signals are passed and reduced layer by layer, while the iDCT is combined to reduce the action data. The core component prediction process module (PPM) proposed in this paper is shown in Figure 3. We have designed the PPM module, and several PPM modules have been merged to form the prediction processing unit (PPU), to perform human movement prediction analysis based on physical principles and data-driven principles, respectively; “FC” stands for the fully connected layer, “LN” and “ReLU” represent the layer normalization and activation layer. “Transpose” stands for the exchange of the spatial and temporal dimensions. Figure 4 shows the parameters M, C, and G used in the physical principle. “MLP” represents a framework that has the same architecture as “$ML{P}_h$” but is different in the number of blocks of the MLP.

Algorithm 1 DCT and iDCT

Input: sequences: $x=[x_1,x_2,\dots ,x_T]\in {\mathbb{R}}^{T\times 3J}$. Output: sequences: $\widehat{x}=[{\widehat{x}}_1,{\widehat{x}}_2,\dots ,{\widehat{x}}_T]$.   1: DCT transfer operation:   2: $D\in R^{(T+N)\times (T+N)}$;   3: $y=\mathrm{DCT}\left(x\right)=Dx$;   4: Generate sequences:   5: $y\in R^{N\times 3J}$;   6: $y_t=\sqrt{{\gamma }_t}y+\sqrt{1-{\gamma }_t}\mu ,\mu \sim \mathcal{N}(0,I)$;   7: iDCT transfer operation:   8: $\widehat{x}=\mathrm{iDCT}\left(y\right)=D^{T}y$;   9: return $\widehat{x}$.

Having an N-frame motion sequence $x_{T+1:T+N}\in R^{N\times 3J}$, we put the sequence into the DCT domain through the DCT(·) operation:

$$y=DCT\left(x\right)=Dx,$$

(2)

where $D\in R^{N\times N}$ is the predefined DCT basis, and $y\in R^{N\times 3J}$ is the DCT coefficients. As the DCT operation is completed, we can overturn the motion sequence in the DCT domain through an iDCT(·) operation:

$$x=iDCT\left(y\right)=D^{T}y.$$

(3)

In order to achieve smoothness in switching of human motions, we take measures to perform the DCT and iDCT operations selecting the first L rows of D and $D^T$.

We use three different noise scheduling strategies, square root scheduling, cosine scheduling, and sigmoid scheduling. These strategies are used to adjust the level of noise added at different time steps, thus ensuring that smooth and stable predictions are generated during the iteration process. Specifically, we perform the above DCT operation $x\in R^{N\times 3J}$ for the first E frequency components, obtaining the spectrum $y_0\in R^{E\times 3J}$. The noisy DCT spectrum $y_t$ at the time step t can be computed by a reparameterization trick:

$$y_t=\sqrt{{\overline{\gamma }}_t{y}_0}+\sqrt{1-{\overline{\gamma }}_t\mu },$$

(4)

where ${\overline{\gamma }}_t={\prod }_{i=1}^t{\gamma }_i$, ${\gamma }_i\in [0,1]$ are predefined variance parameters, and $\mu \sim N[0,I]$.

3.2. Physics-Based and Data-Driven Motion Prediction Model

We firstly elaborate the rules of the Euler–Lagrange equations by looking specifically at Algorithm 2. What is more, we introduce the model dynamics process.

Algorithm 2 Training procedure of LDPM

Input: Observation sequences $x_{1:T}=[x_1,x_2,\dots ,x_T]$, time steps $t+1$, physic feature $H_k$ and geometric feature $H_s$. Output: Future sequences ${\widehat{y}}_{T+1:T+N}$.   1: Extract feature:   2: Using MLP to obtain two features:   3: $H_k={\mathrm{MLP}}_k\left(y_{t-1}^T\right)$,   4: $H_s={\mathrm{MLP}}_s(y_{t+3}^T$;   5: Dynamic modeling:   6: $M_{T+N}={\mathrm{MLP}}_M(H_k\oplus H_s)$   7: $C_{T+N}={\mathrm{MLP}}_C(H_k\oplus H_s)$   8: $G_{T+N}={\mathrm{MLP}}_G(H_k\oplus H_s)$   9: Data-driven: 10: Using the full connectivity layer: 11: $y_{\mathrm{data}}={\mathrm{MLP}}_{\mathrm{data}}\left(y_{t=1:T}\right)$; 12: Fusion physic and data: 13: Addition weight to predict: 14: $y_{\mathrm{fusion}}=(1-w_t)y_{\mathrm{physics}}+w_t{y}_{\mathrm{data}}$; 15: return $y_{\mathrm{fusion}}$.

The Euler–Lagrange equations are use variables that comprehensively define the configuration of the physical system when applied to modeling human motion, and inspired by the widely used SMPL human model. The SMPL models the body as a 3D mesh. The motion parameters capture joint angles. In contrast, the parameters of shape represent body shape coefficients, governing ingredients of physical aspects such as height and other proportions. Using forward kinematics, the positions of the vertices and body joints in a 3D human model can be determined. The subject’s body shape is assumed to remain constant. Thus, the motion sports routes within a world coordinate system are described by combining the pose parameters with the translation parameters H. Consequently, we define the coordinate as

$$y=(\delta ,H).$$

(5)

In the implementation process, $\delta$ denotes the Euler angles of the body joints. Our approach takes into account biomechanical constraints. This strategy effectively eliminates redundant-angle solutions and ensures a more accurate representation of joint movements.

Building upon the coordinate system outlined above, we represent the velocity and acceleration at frame t as ${\tilde{y}}_t$ and ${\breve{y}}_t$, respectively. The dynamics of the human body, governed by the Euler–Lagrange equation, can then be expressed as follows:

$$M_t{\breve{y}}_t+C_t=G_t.$$

(6)

The Euler–Lagrange equations are important for modeling and analyzing human motion using two key processes. Forward dynamics is concerned with solving the equations to predict the body configuration in the subsequent frame based on the physical parameters $M_t$ and $C_t$, and the applied forces $G_t$. Starting from a defined system state, we first compute ${\breve{y}}_t$, which is then used to forecast future motion through the application of Euler’s method, represented as follows:

$${\breve{y}}_t=M_{t-1}(G_t-C_t),$$

(7)

$${\tilde{y}}_{t+1}={\tilde{y}}_t+{\breve{y}}_t△T,$$

(8)

$$y_{t+1}=y_t+{\tilde{y}}_t△T,$$

(9)

where $△T$ denotes the time interval between consecutive frames, and $M_{t-1}$ represents the inverse of the inertia matrix. In contrast, inverse dynamics focuses on estimating the unknown physical parameters based on existing data. In this study, we introduce a novel approach that integrates the Euler–Lagrange equations into the motion prediction framework. By explicitly modeling them, we develop a physics-based motion prediction model, which is detailed below.

The in-depth modeling of human motion is first achieved through a physically based predictive model based on the Euler–Lagrange equation, which is expressed in a generalized coordinate system as

$$M_t{y}_t+C_t=G_t,$$

where $M_t\in R^{75\times 75}$ denotes the total inertia matrix, which is determined by the generalized position and parameters such as body mass and inertia; $C_t\in R^{75}$ denotes the generalized bias force; and $G_t\in R^{75}$ denotes the generalized force. We follow (3) and use processes such as forward and inverse dynamics to model body dynamics. Using geometric features extracted from the first three frames of the Euler–Lagrange equations that fully specify a moment t, we use an MLP to extract the physical features $H_k$ and the geometric features $H_s$ from the motion history, and in turn estimate these physical parameters:

$$H_k={\mathrm{MLP}}_k\left(y_{t=1}^T\right),$$

(10)

$$H_s={\mathrm{MLP}}_s\left(y_{T+1}^{T+3}\right).$$

(11)

The generalized inertia matrix M, the bias force C, and the external force G are then predicted by another MLP:

$$M_{T+N}={\mathrm{MLP}}_M(H_k\oplus H_s),$$

(12)

$$C_{T+N}={\mathrm{MLP}}_C(H_k\oplus H_s),$$

(13)

$$G_{T+N}={\mathrm{MLP}}_G(H_k\oplus H_s).$$

(14)

Our approach is specifically designed to maximize the expressive capabilities of neural networks while strictly adhering to the underlying physics equations. This ensures that the relationships among the physical parameters are faithfully preserved. Using the physical parameters, the physics-based model employs an ODE solver to compute $y_{physics}$, where forward dynamics is applied to predict future states.

For training of the physics-based model, we utilize

$$y_{physics}={\mathrm{MLP}}_{data}\left(y_{t=1}^T\right),$$

(15)

where $y_T$ represents the ground truth position.

To forecast future motion, the physics-based motion prediction model is applied iteratively, generating the configuration for the next frame based on prior estimates and historical input motion. However, this approach primarily focuses on the immediate vicinity of the current frame, which can lead to significant error accumulation over longer prediction horizons. To mitigate this challenge, we incorporate a data-driven model and a fusion model, designed to reduce error accumulation, as detailed below.

Current approaches have shown significant progress in leveraging data to model long-term dependencies, enabling the direct prediction of future human motion from the input historical motion sequence $y_T$ using neural networks. A data-driven forecasting model is then introduced that utilizes historical motion data to capture long-term temporal dependencies. With a fully connected layer, we generate future data-driven predictions:

$$y_{data}={\mathrm{MLP}}_{data}\left(y_{t=1}^T\right).$$

(16)

Based on the prediction $y_{physics}$ of the above physical model, the prediction generation for each frame is dynamically adjusted by weighting $w_t$ the fusion data driver, and the final output fusion prediction is

$$y_{fusion}=(1-w_t)y_{physics}+w_t{y}_{data}.$$

(17)

3.3. Training of the Model

We train the latent diffusion model [24] to learn to sample in this space. The diffusion model is a probabilistic model that learns the data distribution $y\sim p_y$ by gradually removing the noise from the random Gaussian noise. Our proposed new model, SLDP, initializes the noise scheduling and prediction modules, and during the diffusion process PhysDa corrects for the introduced noise at each time step, allowing the constraints of the physical model to be reflected in the prediction. The final generated motion data not only has data-driven accuracy, but also conforms to physical plausibility. During the sampling generation process, we perform several iterations through the PhysDa module to generate the predictions step by step, so that the predictions gradually approximate the real motion trajectories. And we use a noise prediction network parameterized as $\theta$ to generate the prediction noise ${\mu }_{\theta (y_t,t)}$, and optimize the parameter $\theta$ by the noise prediction loss L:

$$\mathcal{L}={\mathrm{E}}_{\mu ,t}{∥\mu -{\mu }_{\theta }(y_t,t)∥}^2.$$

(18)

4. Experiments

Datasets. We use the datasets Human3.6M [52], HumanEva-I [53], AMASS [54] to evaluate our model. Human3.6M is a comprehensive 3D human motion capture dataset widely used for motion prediction, pose estimation, and action recognition. It includes 11 participants performing 15 everyday activities, with high-resolution 3D joint data for 17 key joints. The dataset offers both 2D and 3D annotations, making it ideal for 2D-to-3D pose estimation and motion prediction tasks. HumanEva-I features synchronized 2D video frames and 3D motion capture data from multiple subjects performing activities like walking and gesturing. It is suited for multi-view pose estimation and motion prediction, with standardized protocols for reproducibility. AMASS is a large-scale collection of 3D motion capture data, unified in the SMPL format, facilitating motion modeling, 3D pose estimation, and motion prediction. It covers a wide range of activities and provides high-quality 3D meshes and joint data, supporting integration with other datasets like Human3.6M and HumanEva-I for cross-dataset research.

Implementation details. Human3.6M features 7 subjects performing 15 motion types, where five subjects S1, S5, S6, S7, and S8 are used for training and two (S9 and S11) for evaluation. The data are processed at 50 Hz with a 17-joint skeleton excluding the root joint. Our model observes 25 frames, 0.5 s, to predict 100 future frames, 2 s. HumanEva-I includes three subjects performing five actions. It is processed at 60 Hz with a 15-joint skeleton, using 15 observation frames, 0.25 s, to predict 60 future frames, 1 s. For AMASS, we retarget the skeleton to match the Human3.6M format, enabling direct inference using the model trained on Human3.6M. The model is trained using PyTorch on an RTX 4090 GPU with 120 GB of memory. A batch size of 64 is used, with training lasting 500 epochs for the first two datasets and 50 epochs for AMASS. The frame rates are 25 Hz for Human3.6M and 30 Hz for HumanEva-I and AMASS.

The training setup includes an Adam optimizer with a learning rate of 3 × 10⁻⁴ and a decay factor of 0.9. A dropout rate of 0.2 is applied. The DCT/iDCT operation uses 20 coefficients and 10 coefficients. The latent dimension is set to 512, with eight self-attention heads. The modulation ratio is set to 1.0 and 0.5. To mitigate error accumulation and improve motion-switching capabilities, the final 20 steps of DCT completion are replaced with simpler denoising steps. Data preprocessing includes applying a Gaussian filter to reduce noise in body translation and joint angles, ensuring smoother predictions.

Evaluation metrics. Based on previous work [28], we report five metrics to measure the prediction quality. They are average pairwise distance (APD), average displacement error (ADE), final displacement error (FDE), multi-modal ADE (MMADE), and multi-modal FDE (MMFDE). APD is the L2 distance all the samples. ADE is defined as the minimum average L2 distance between the actual and predicted motions. FDE is the L2 distance between the predicted result of the last frame and the ground truth. MMADE and MMFDE are multi-modal versions of ADE and FDE, respectively, with ground truth future motions grouped by similar observations.

4.1. Comparison to the State of the Art

Quantitative results.

Table 1. Quantitative results of motion prediction methods on Human3.6M and HumanEva-I datasets.

Method	Human3.6M					HumanEva-I
	APD	ADE	FDE	MMADE	MMFDE	APD	ADE	FDE	MMADE	MMFDE
MotionDiff [6]	15.353	0.411	0.509	0.536	0.536	5.931	0.232	0.236	0.352	0.320
Deligan [16]	6.509	0.483	0.520	0.545	0.520	2.177	0.306	0.322	0.385	0.371
HumanMAC [28]	6.301	0.369	0.480	0.509	0.545	6.554	0.209	0.223	0.342	0.335
BeLFusion [30]	7.602	0.420	0.472	0.474	0.507	6.109	0.220	0.234	0.342	0.316
Diversampling [35]	15.310	0.370	0.480	0.482	0.509	6.109	0.220	0.234	0.342	0.316
Gsps [50]	14.757	0.389	0.496	0.476	0.525	5.825	0.233	0.240	0.344	0.331
Mojo [51]	12.579	0.412	0.497	0.497	0.538	4.181	0.234	0.244	0.369	0.347
Ours	13.158	0.319	0.410	0.430	0.473	10.119	0.234	0.229	0.369	0.372

The bold number represents the best results.

shows the quantitative results, highlighting that our method achieves the best ADE, 0.319, and FDE, 0.410, on the Human3.6M dataset, and is competitive on HumanEva-I, with an ADE of 0.234 and FDE of 0.369. These results demonstrate the superior accuracy of our method in both general trajectory prediction (ADE) and final position prediction (FDE), which are critical for real-world motion prediction tasks. The results in

Table 2. Time details for different training steps and their corresponding batch and total time measurements for the AMASS dataset.

Step	Batch Time (s)	Total Time (s)
1/2598	0.25	1
1001/2598	0.257	70
2001/2598	0.243	138
1/59	0.024	0
1/1141	0.103	0
1001/1141	0.149	25

provide insights into the efficiency of the training process at different stages. While the focus of this section is to compare the performance of the proposed method with the state of the art in terms of motion prediction accuracy (ADE and FDE), these time measurements provide further context about the computational efficiency. Although our method achieves high accuracy, our approach also aims to ensure efficient training at a reasonable computational cost.

Visualization results. As shown in Figure 5, our method performs well on the Human3.6M dataset. Red, black, and gray denote the contextual part of the initial action state, and purple and green denote the predicted action sequences. Over time, our model predicts for a long time based on the physical constraints of the human body, and the predicted values are overall close to the true values. As shown in Figure 6, on the HumanEva-I dataset, our method fuses spatio-temporal information to predict standing, arm swinging, and jumping movements, and the prediction accuracy is significantly improved after 500 epochs of training. In Figure 7, compared with the training error, the value is lower. The training error is always lower than the validation error, indicating that the model performs well on the training set. The validation error and test error show that the model has limited generalization ability on the validation set, but performs well on the test set.

4.2. Ablation Study

LDPM modeling setup. Here, we examine the effects of the denoising length and variance scheduler on the experiments.

Table 3. Results of different schedulers on the Human3.6M and HumanEva-I datasets.

Scheduler	Human3.6M			HumanEva-I
	FDE	MMADE	MMFDE	FDE	MMADE	MMFDE
Linear	0.521	0.518	0.563	0.258	0.408	0.422
Sqrt	0.733	0.665	0.768	0.508	0.545	0.612
Cosine	0.410	0.430	0.473	0.229	0.369	0.372

The bold number represents the best results.

presents the best choice, which is the cosine scheduler for training on the Human3.6M and HumanEva-I datasets. The evaluation indicators include FDE, MMADE, and MMFDE. The cosine scheduler on the Human3.6M dataset achieves a minimum FDE error of 0.410, which is better than linear and sqrt. Compared to the sqrt scheduler, the cosine scheduler’s error is reduced by 44.1%, which shows a significant performance improvement and provides higher prediction accuracy in terms of final position error. On the HumanEva-I dataset, the cosine scheduler achieves a minimum FDE error of 0.229, which is 11.2% and 54.9% lower than linear and sqrt, respectively. The MMADE of the cosine scheduler is 0.369, which is also the lowest among the three schedulers, indicating that it is the most stable in terms of average prediction error.

By approximating the values of the DCT and iDCT to improve the efficiency, the E choices of 5 and 10 have a significantly superior performance.

Table 4. Experimental results on Human3.6M for varying values of E.

E	Human3.6M
	FDE	MMADE	MMFDE
5.000	0.410	0.390	0.423
10.000	0.534	0.524	0.572
20.000	0.410	0.430	0.473

The bold number represents the best results.

and

Table 5. Experimental results on HumanEva-I for varying values of E.

E	HumanEva-I
	FDE	MMADE	MMFDE
5.000	0.265	0.408	0.438
10.000	0.229	0.369	0.372
20.000	0.248	0.401	0.403

The bold number represents the best results.

show the model’s performance on the Human3.6M and HumanEva-I datasets, respectively. On Human3.6M, $E=5$ gives the best results, with the lowest FDE, while on HumanEva-I, $E=10$ provides the best overall accuracy and efficiency.

Prediction Networks. The TransLinear layer in the diffusion processing unit (DPU) we proposed performs skip connections. In

Table 6. Results of different network design configurations with and without skip connections on Human3.6M and HumanEva-I datasets.

Skip Connection	Human3.6M			HumanEva-I
	APD	ADE	FDE	APD	ADE	FDE
N	15.236	0.400	0.423	9.927	0.226	0.217
Y	13.158	0.319	0.410	10.119	0.234	0.229

The bold number represents the best results.

, the skip connection results are better and the generated motion is more realistic.

Table 7. Comparison of model performance when varying the number of layers on the Human3.6M and HumanEva-I datasets.

Layers	Human3.6M			HumanEva-I
	FDE	MMADE	MMFDE	FDE	MMADE	MMFDE
2	0.580	0.546	0.609	0.508	0.545	0.612
4	0.551	0.529	0.584	0.229	0.369	0.372
6	0.532	0.523	0.571	0.239	0.410	0.425
8	0.410	0.430	0.473	0.255	0.417	0.439
10	0.510	0.516	0.554	0.256	0.422	0.450

The bold number represents the best results.

shows that the best results on Human3.6M occur at eight layers, while on HumanEva-I, the best results are at four layers.

Table 8. Results of different noising steps on Human3.6M and HumanEva-I datasets.

Noising Steps	Human3.6M			HumanEva-I
	FDE	MMADE	MMFDE	FDE	MMADE	MMFDE
100.000	0.528	0.522	0.565	0.246	0.402	0.419
1000.000	0.410	0.430	0.473	0.229	0.369	0.372

The bold number represents the best results.

shows that increasing the noise steps from 100 to 1000 significantly improves performance, with notable reductions in FDE, MMADE, and MMFDE on both datasets.

Moreover, as shown in Figure 8, we use 28.4M and 14.45M parameter, respectively, in Human3.6M and HumanEva-I. Figure 9 compares the performance of various motion prediction methods in two plots. In the left plot, the motion value decreases as the predicted frames increase, where the GT line remains constant and the other lines show a decrease in predicted motion. Our method (represented by the purple line) maintains the lowest motion value in most predicted frames, with better consistency with the GT than other methods such as HumanMAC and MotionDiff. In the right plot, the cumulative error increases as the predicted number of frames increases. Our method again shows superior performance, keeping the cumulative error lower than all other methods, especially in the early predicted frames. As the predicted frames progress, the error of other methods increases faster than our method.

Table 9. The motion editing ability of different methods.

Method	Switch Ability	Independent Body Part Control
MotionDiff [6]	0	0
Deligan [16]	0	0
HumanMAC [28]	1	1
BeLFusion [30]	1	0
Diversampling [35]	0	0
Gsps [50]	0	1
Mojo [51]	0	0
Ours	1	1

shows the performance of our proposed method. A 0 value represents no switching capability or partial body controllability, and 1 represents switching capability or partial body controllability. MotionDiff and Deligan have relatively limited performance, both having 0 values. Our proposed method has the most comprehensive capabilities, supporting switching motions together with HumanMAC and BeLFusion, and can independently control parts of the body.

Table 10. Comparison of performance metrics and design choices across various motion prediction models on the Human3.6M dataset.

	MotionDiff [6]	Deligan [16]	HumanMAC [28]	BeLFusion [30]	Diversampling [35]	Gsps [50]	Ours
One Stage	N	Y	Y	N	N	N	Y
Loss	4	1	1	4	3	5	1
FPS	15	22	25	18	20	12	30

compares the performance metrics of various motion prediction models on the Human3.6M dataset. The table evaluates the models based on three criteria: whether they use a single-stage approach (Y for yes and N for no), the loss function used, and the frames per second (FPS) performance. Our model uses a single-stage approach, as indicated by “Y”, while other models such as MotionDiff, BeLFusion, Diversampling, etc., do not. The loss function used by our method is comparable to other models such as HumanMAC, but significantly better than MotionDiff. Compared to other models, our model achieves the highest FPS and performs well in real-time performance.

Table 11. Comparison of model performance in motion prediction across various methods in terms of runtime, parameter size, and accuracy metrics on the Human3.6M dataset.

	MotionDiff [6]	HumanMAC [28]	BeLFusion [30]	Diversampling [35]	Gsps [50]	Ours
Runtime (ms)	15.2	12.3	18.1	14.5	16.7	10.8
Parameter Size (M)	45.3	30.1	35.8	28.7	40.2	25.9
ADE	0.411	0.369	0.420	0.370	0.389	0.319
FDE	0.509	0.480	0.472	0.480	0.496	0.410

The bold number represents the best results.

shows that our method has a significantly lower runtime than other methods, with MotionDiff being the closest. Our method has a parameter size of 25.9M, which is larger than HumanMAC but smaller than Gsps. Our method has the lowest ADE metric, outperforming all other methods. Similarly, our method achieves the lowest FDE of 0.410, outperforming other models, including HumanMAC at 0.480. Our method achieves the lowest ADE and FDE values, indicating higher motion prediction accuracy compared to existing methods. As shown in

Table 12. Quantitative comparison of different models across multiple evaluation metrics on the Human3.6M dataset.

Model	APD	ADE	FDE	MMADE	MMFDE
Data-driven	12.406	0.395	0.517	0.512	0.580
Physics-based	11.238	0.329	0.534	0.487	0.543
Proposed hybrid	13.158	0.319	0.410	0.430	0.473

The bold number represents the best results.

, the proposed hybrid model outperforms both the data-driven model and the physics-based model in all evaluation metrics on the Human3.6M dataset. Compared with other models, it has a higher APD of 13.158 and performs best in ADE, FDE, MMADE, and MMFDE, with values of 0.319, 0.410, 0.430, and 0.473, respectively. This shows that the hybrid approach effectively combines the advantages of the data-driven model and the physics-based model to achieve more accurate motion prediction.

5. Discussion

Recent advancements in person re-identification (Re-ID) have tackled challenges like occlusion, viewpoint changes, and feature consistency. For instance, [55] shows how addressing depth variations can boost Re-ID performance. Our model, which combines latent diffusion with physical motion prediction, could benefit from similar depth adaptation to handle occlusion better. In [56], the authors explore deep learning methods for handling occlusion, which is a key issue our model addresses using Euler–Lagrange equations for motion prediction. This approach aims to improve prediction reliability in such situations. The 3D Re-ID technique in [57] uses both global guidance and local feature aggregation, which can be applied to our model to improve its prediction accuracy. Additionally, multi-view learning in [58] enhances 3D shape understanding, and we see potential to use this approach to boost long-term motion prediction accuracy. Finally, [59] focuses on feature consistency and contrast enhancement to improve Re-ID. Our model already reduces long-term prediction errors, and adopting these techniques could reduce errors even further.

In summary, our model significantly improves long-term human motion prediction and can be combined with existing Re-ID techniques to address occlusion, depth, and feature consistency, offering new directions for future research in both human motion prediction and pedestrian Re-ID.

6. Conclusions

In this paper, we proposed a model based on latent diffusion modeling and physically driven components, using PPM and PPU modules to address the problems of low long-term prediction accuracy and severe error accumulation. The framework combines motion generated by random noise with physical control derived from the Euler–Lagrange equations to provide accurate predictions for future frames of human motion. Our approach significantly improves the performance over existing methods, achieving error reduction of 15.7% in specific metrics such as ADE. Looking ahead, we plan to further investigate the integration of physical principles with deep learning techniques for a wider range of spatio-temporal tasks, with the potential to improve performance in areas such as human motion prediction.