STA-TFM: Spatio-Temporal Aggregation Across Views Transformer for Pose Estimation

Abstract

Monocular 3D human pose estimation (HPE) remains challenging due to depth ambiguity, occlusions, and the need for temporal consistency. While multi-view methods provide superior accuracy over monocular approaches, they often require complex setups. We introduce STA-TFM, a transformer-based architecture that combines spatial and temporal information for multi-view pose estimation. The approach leverages DSTFormer, a monocular feature extractor, to capture long-range pose dependencies within each view. A fusion transformer then aggregates information across views to produce coherent 3D estimates. To address training data scarcity, we use a data generation pipeline that transforms any existing 3D pose dataset into multi-view setups with controllable parameters. Experiments on various datasets demonstrate that STA-TFM outperforms existing camera-parameter-free multi-view methods. STA-TFM achieves 50.9% and 49.5% reductions in mean per joint position error (MPJPE) and mean per joint velocity error (MPJVE) on the DHP19 dataset. Furthermore, it achieves 6.7% and 7.7% respective reductions on HAA4D, and a 15.2% MPJPE reduction on TotalCapture. STA-TFM handles noisy and missing 2D inputs, supporting potential deployment in healthcare monitoring, athletic assessment, and immersive technologies.

1. Introduction

Human pose estimation is a fundamental computational challenge with broad applications spanning healthcare and athletics [1], virtual reality and gaming [2], and beyond [3,4], where developing accurate models holds significant promise.

Vision-based methods are the dominant approach for precise 3D pose estimation.To obtain 3D estimates from videos, the task is often broken into two stages: 2D pose extraction followed by 2D to 3D lifting. While significant advancements have been made in 2D pose extraction from videos, obtaining 3D poses remains a complex problem [4]. Two methods have emerged to address lifting: monocular and multi-view human pose estimation (HPE), each with unique trade-offs between accuracy and accessibility. Monocular methods are more accessible as they rely on a single view, circumventing the need for complex and expensive data acquisition processes [5]. However, the single view results in an ill-posed problem in which a single 2D observation represents multiple 3D locations [6]. Depth ambiguity becomes more challenging in the presence of external and self-occlusions [5]. While monocular pose estimation is typically easier to deploy, it is often at the cost of accuracy. Without accurate 3D estimation, joint angles and movement patterns cannot be relied on to inform motion assessments. These challenges motivate the implementation of multi-view setups where 2D to 3D lifting relies on multiple cameras, making these models robust and better candidates for real-world applications. However, multi-view models often require complex and expensive setups for data acquisition, making them poor candidates for ‘in-the-wild’ scenarios and practical day-to-day monitoring [4,6].

In monocular and multi-view methods, transformer-based architectures have gained traction in recent years for their ability to model long-range spatial and temporal dependencies [7]. Monocular models that harness spatial and temporal information demonstrate that attention mechanisms can improve temporal coherence and pose accuracy [8,9,10,11]. These benefits extend naturally to multi-view settings, where synchronized information from multiple cameras can be leveraged to obtain more robust 3D estimates. Existing methods exemplify this by fusing per-view spatial embeddings with a fusion pose transformer (FPT) to obtain reliable 3D poses from multiple 2D poses [12].

Beyond traditional computer vision, other modalities have been explored for human sensing. WiFi channel state information supports device-free activity and interaction recognition [13,14], offering privacy preservation and resilience to occlusion and poor lighting, though these methods classify activity rather than estimate pose. WiFi-based 3D pose estimation has also been explored [15], but remains at an earlier stage of development, with reported joint localization errors substantially higher than those of multi-view vision methods. Vision-based approaches therefore remain dominant where precise 3D pose is required [16].

Building on this foundation, and specifically on the computer vision transformer-based multi-view methods described above, we introduce STA-TFM (Spatio-Temporal Aggregation across views TransForMer), a multi-view 3D human pose estimation framework that predicts 3D poses from multi-view sequences of 2D poses (see Figure 1). This framework is specifically optimized for applications where capturing accurate joint angles and relative body geometry is the primary objective. For each view, STA-TFM extracts spatio-temporal embeddings using DSTFormer [8], the feature extraction backbone of MotionBERT, originally designed for monocular pose estimation. These representations are then aggregated across views using a fusion pose transformer (FPT) adapted from MPL [12]. This design is motivated by the fact that monocular pose models can leverage abundant single-camera datasets, which are considerably easier to collect and annotate than synchronized multi-view captures. We show that their learned spatio-temporal representations can be effectively reused in a multi-view setting.

Figure 1. Starting with multi-view videos, a 2D pose estimation model (e.g., RTMPose [17]) extracts 2D joint locations from each camera view. These multi-view 2D poses are then processed by STA-TFM through spatio-temporal aggregation to predict the corresponding 3D pose. Only the videos are required; camera calibration parameters are not needed during inference. The example shown is from the BlendMimic dataset [18].

To the best of our knowledge, STA-TFM is the first method to leverage a frozen pretrained monocular feature extractor (DSTFormer) for multi-view 3D HPE. By decoupling per-view feature extraction from cross-view fusion, the proposed architecture simplifies training and allows optimization to focus on multi-view aggregation and 3D pose regression. Combined with the proposed fusion strategy, STA-TFM achieves improved motion consistency and 3D pose accuracy in flexible-layout environments without requiring explicit geometric priors or camera parameters during training or inference. Furthermore, the results suggest that monocular spatio-temporal representations learned by DSTFormer remain effective when transferred to multi-view 3D human pose estimation.

2. Related Works

Multi-view human pose estimation methods typically have higher accuracies as they effectively address the issues of depth ambiguity and occlusions prevalent with monocular HPE [4,12]. These models rely on multiple camera views, decreasing the chance of a fully occluded joint in all images. The information from additional views allows simple techniques, such as triangulation [19], to be applied. Although triangulation is often the baseline for assessing multi-view setups, it is limited to cases where all keypoints are present and accurate [12]. Additionally, accurate camera extrinsic and intrinsic parameters need to be known beforehand.

To address some of the limitations of triangulation, learning-based architectures have emerged. These methods can be broadly divided into camera-parameter methods, which require known camera intrinsics and extrinsic parameters, and camera-parameter-free methods, which do not.

2.1. Camera-Parameter Methods

Camera-parameter methods generally achieve higher accuracy than camera-parameter-free methods due to incorporating important prior knowledge into the network [20,21]. Epipolar Transformers [22] exploit known camera parameters to identify corresponding epipolar lines across views, aggregating features from multiple cameras by attending to spatial locations along the epipolar line in one view using a joint query from another. Similarly, Transfusion [23] encodes 3D positional information into a transformer using an epipolar field, providing an efficient mechanism for encoding cross-view pixel correspondences.

Volumetric methods, such as VoxelPose [24], project 2D joint heatmaps from all views into a shared 3D voxel space, using a Cuboid Proposal Network (CPN) to localize individuals and a Pose Regression Network (PRN) to estimate fine-grained 3D joint locations using 3D convolutional layers. SelfPose3D [25] extends the volumetric paradigm to a self-supervised setting, eliminating the need for 2D or 3D ground truth annotations by adopting a learning-by-projection paradigm, where the model learns 3D outputs by comparing projected 3D joint predictions against their 2D counterparts. While effective, volumetric approaches such as [24,25] incur significant computational cost [26].

MVP [27] addresses some of those inefficiencies by directly regressing multi-person 3D poses from multi-view images without relying on 2D pose priors, representing skeleton joints as learnable query embeddings that progressively attend to multi-view image features via a projective attention mechanism. MVGFormer [28] takes a hybrid approach, combining a learning-free geometric module, which leverages triangulation to constrain the learning task, with a learnable appearance module for estimating 2D poses from images.

TEMPO [26] further extends calibrated methods to the spatio-temporal domain using a recurrent architecture to jointly learn spatial and temporal representations across views, achieving state-of-the-art results and demonstrating the importance of incorporating both time and multi-view cameras.

2.2. Camera-Parameter-Free Methods

Obtaining accurate camera parameters remains a significant hurdle in dynamic, uncontrolled settings [21,29], limiting the practical deployment of camera-parameter methods. Therefore, various methods attempt to remove the need for known camera extrinsics to obtain 3D pose from multi-view set ups.

Multi-view 3D Pose Lifter (MPL) [12] demonstrated strong performance for multi-view fusion, harnessing two transformers, a spatial pose transformer and a fusion pose transformer (FPT). MPL effectively extracts meaningful spatial features from each view and fuses the embeddings to capture spatial features across views. Similarly, EFMK [30] combines Local–Global Pose Embedding with a Spatial-View Joint Transformer that integrates kinematic and geometric priors into attention computation. EFMK enforces bone length symmetry through a Bone-wise Alternating Optimization mechanism, and optionally refines predictions using reprojection-based multi-view aggregation when camera intrinsics are available. However, neither MPL nor EFMK incorporates temporal information, which has been shown to improve the accuracy and coherence of predicted 3D poses in both the monocular [8] and multi-view setting [26].

MTF-Transformer [21] addresses this by employing a relative-attention mechanism to measure implicit relationships between camera views using a Multi-view Fusing Transformer (MFT). This is followed by a Temporal Fusing Transformer (TFT) that aggregates features across the full sequence for final 3D pose prediction. However, this sequential spatial-then-temporal fusion restricts cross-domain communication between the two representations [20].

ESMformer [31] also leverages relative attention and performs hierarchical fusion across spatial, temporal, cross-view, and cross-level pose features. Given multi-view 2D pose sequences, it extracts multi-level spatial features per-view using a relative-attention-based encoder, performs multi-view intra-level fusion across views and time, and then applies cross-level fusion before regressing the final 3D pose. While its error-aware self-supervised training reduces reliance on 3D annotations by using triangulation and reprojection losses, these geometric constraints still require known camera parameters during training.

Other methods such as [20,29,32] also incorporate spatio-temporal features for multi-view 3D human pose estimation. FusionFormer [20] encodes 2D HPE results into pose features, fuses them using a transformer encoder into a global spatio-temporal representation, and employs a transformer decoder that takes both global and view-specific features to estimate the 3D pose. FLEX [29] takes multi-view 2D poses as input and uses a deep convolutional network to learn 3D rotations and bone lengths rather than joint locations directly, incorporating cross-view attention and temporal information. SGraFormer [32] employs a deep semantic graph transformer encoder to enrich spatial feature information by mining structural and skeletal edge knowledge of joints and their correlations. This is followed by a multi-view spatio-temporal fusion framework to mitigate joint depth uncertainty.

Recent advancements have introduced an intermediate category that internally estimates camera geometry, eliminating the need for explicitly provided external camera parameters during inference. EasyRet3D [33] reconstructs human meshes independently from each view and uses Perspective-n-Point with RANSAC to initialize camera extrinsics from 3D-to-2D correspondences. Camera, body, and ground-plane parameters are then jointly refined. Qin et al. [34] instead predict camera intrinsics and extrinsics directly from multi-view features using a transformer-based architecture and use the estimated parameters for differentiable triangulation. Additional algebraic and temporal constraints are incorporated to improve pose quality. While both methods eliminate the need for externally supplied camera calibration during inference, they still rely on a camera-estimation stage with sufficient cross-view correspondence. Furthermore, EasyRet3D relies on heavy sequence-level test-time optimization that limits real-time utility.

In contrast, STA-TFM directly regresses 3D poses from multi-view features without an intermediate camera-parameter estimation step. STA-TFM takes multi-view 2D keypoints as input and feeds them to a frozen DSTFormer [8] to extract per-view spatio-temporal features. These embeddings are then passed to a fusion network (FPT) inspired by MPL [12] to fuse cross-view features before a regression head produces the final 3D pose estimate.

In contrast to existing methods like MTF-Transformer and FusionFormer that train their entire spatio-temporal and multi-view fusion components end-to-end, STA-TFM leverages a pretrained, frozen DSTFormer feature extractor from monocular HPE. By keeping the DSTFormer frozen, we inherit strong temporal modeling capabilities learnt from training on large-scale monocular datasets while focusing optimization exclusively on the fusion network and regression head for cross-view aggregation.

3. Materials and Methods

3.1. Architecture

The lifting architecture involves two key steps to transform 2D poses into 3D representations: DSTFormer and a fusion pose transformer (FPT). As shown in Figure 2, the keypoints obtained from a generic 2D pose estimator (e.g., RTMPose [17]) are first passed to DSTFormer to generate a motion representation, and a bottleneck layer is applied to compress the embedding before concatenating the embeddings from each view. The concatenated embedding is passed to an FPT to generate a fused embedding before the 3D pose is predicted by the final regression head.

Figure 2. STA-TFM architecture overview. Starting with multi-view 2D poses generated by an off-the-shelf pose estimator, the framework processes them through parallel DSTFormer blocks to generate spatio-temporal embeddings for each view. These embeddings are compressed via bottleneck layers and concatenated before being passed to the fusion pose transformer, which aggregates cross-view information using learnable positional embeddings. The final 3D pose is predicted through a lightweight regression head. The example shown is from the TotalCapture dataset [35].

DSTFormer effectively captures the spatial and temporal information of the 2D skeleton within each view independently [8]. This dual-stream backbone was chosen over single-order variants because it processes each pose sequence in both spatial-then-temporal and temporal-then-spatial orders, to complementarily model within-frame joints and across-frame motion. With DSTFormer, an input skeleton, $\mathrm{x}\in {\mathbb{R}}^{T\times J\times C_{\mathrm{in}}}$, with sequence length T, J joints, and input channel size, $C_{\mathrm{in}}=3$, (x, y, $conf$), is first projected to a high dimensional feature space ${\mathrm{F}}^0\in {\mathbb{R}}^{T\times J\times C_{\mathrm{f}}}$ and two learnable spatial and temporal positional encodings are added. Here, $C_{\mathrm{f}}$ is the number of high dimensional features. DSTFormer takes ${\mathrm{F}}^0$ and passes it to the primary backbone. This backbone includes sequential blocks consisting of a dual stream of alternating temporal and spatial multi-head self-attention (MHSA) with skip connections. Spatial MHSA attends to the joint relationships within the skeleton, while temporal MHSA models the relationships of single joints over time. The streams are subsequently fused using adaptive weighting. The DSTFormer backbone of dual stream alternating MHSA is repeated N times ($N=5$, obtained through hyperparameter tuning) to get ${\mathrm{F}}^N\in {\mathbb{R}}^{T\times J\times C_{\mathrm{f}}}$. A shared DSTFormer backbone is applied independently to each camera view, producing one spatio-temporal embedding per view. These embeddings are compressed with a bottleneck layer before stacking the views $\mathrm{E}\in {\mathbb{R}}^{T\times J\times V\times C_{\mathrm{b}}}$ for the FPT, where V is the number of views (default = 4), and $C_{\mathrm{b}}$ is the bottleneck dimension.

The FPT architecture [12] aggregates multiple per-view pose features into a single latent representation capturing structural and temporal cues across cameras. Given the stacked embeddings of DSTFormer, the joints and channels are flattened to obtain a compatible input $\mathrm{E}\in {\mathbb{R}}^{T\times V\times (J\ast C_b)}$, for the FPT. Learnable embeddings are added to the input to distinguish the cameras and provide positional context for each view. The fused tokens are then passed to a series of transformer blocks, each with MHSA that operates on the view dimension to learn inter-view relationships.

The fusion transformer incorporates mask-aware attention to selectively attenuate contributions from missing or low-confidence camera inputs during cross-view information aggregation. By applying view masks to the attention weights ${\alpha }_{ij}\in {\mathbb{R}}^{V\times V}$ during cross-view aggregation, the model prevents invalid tokens from influencing attention computation. In combination with stable attention normalization, the model preserves numerical stability and performance under heterogeneous or incomplete multi-camera inputs.

Following the transformer blocks, the resulting features are normalized before applying a convolutional weighted mean across the view axis to obtain a unified embedding, $\mathrm{E}\in {\mathbb{R}}^{T\times (J\ast C_b)}$. The embedding is reshaped, $\mathrm{E}\in {\mathbb{R}}^{T\times J\times C_b}$, before passing to a lightweight regression head to obtain the final output ${\mathrm{X}}^{\mathrm{pred}}\in {\mathbb{R}}^{T\times J\times 3}$.

3.2. Datasets

The DHP19 [36], HAA4D [37], BlendMimic3D [18], CMUPanoptic [38], and TotalCapture [35] datasets are selected to represent human-centric motion across diverse environments. Table 1 outlines the different datasets and their specific use in this work.

Table 1. Summary of datasets used for training and evaluation. (†) indicates datasets transformed into synthetic multi-view scenarios for both training and evaluation, as described in Section 3.3.

Dataset	Type	Environment	Use
HAA4D †	Monocular	In-the-wild	Generating multi-view dataset + testing STA-TFM
DHP19 †	Multi-view	Laboratory	Generating multi-view dataset + testing STA-TFM
BlendMimic3D	Multi-view	Animated	Testing generalization to unseen camera setups
CMUPanoptic	Multi-view	Laboratory	Benchmarking against camera-parameter methods
TotalCapture	Multi-view	Laboratory	Benchmarking against camera-parameter-free methods

DHP19 and HAA4D are used to generate the training set for STA-TFM as described in Section 3.3. DHP19 depicts standard human motion in controlled laboratory environments. It consists of 17 subjects performing 33 motions recorded using 4 orthogonal cameras with Vicon ground truth for 13 joints. Conversely, HAA4D [37] includes 3390 samples and 300 action classes captured in uncontrolled environments. The videos are hand-labeled and processed using EvoSkeleton [39] to obtain 17 3D joint positions, providing diverse and dynamic motion scenarios.

Additional datasets are also included to assess the peformance of STA-TFM and benchmark it against other methods. BlendMimic3D [18] is a synthetic animated dataset generated using Blender [40], consisting of three subjects performing various actions in different scenes. Each scenario is captured from a four-camera convergent setup, with evaluation in this work restricted to scenario S1. CMUPanoptic [38] is a multi-view dataset providing synchronized video from five cameras that capture multiple subjects in dense, multi-person scenarios. Finally, TotalCapture [35] offers synchronized multi-view video, IMU data, and ground truth 3D poses captured by a Vicon motion capture system. In total, this dataset consists of 1.9M frames spanning multiple subjects and activities, designed for 3D pose estimation evaluation. Figure 3 illustrates the diverse camera configurations across these datasets.

Figure 3. Overhead (top) and 3D (bottom) camera placement layouts for BlendMimic3D, CMUPanoptic, and TotalCapture datasets. Triangles and arrows indicate camera position and viewing direction; axes are in metres.

While Human3.6M [16] is commonly used for benchmarking, licensing constraints prevented its inclusion. Instead, our diverse dataset selection enables a comprehensive assessment of generalization and performance across synthetic, controlled, and in-the-wild scenarios.

3.3. Training Data Generation

To address the challenges of collecting labeled multi-view human pose data, we use a data generation pipeline that converts any 3D pose collection into synthetic multi-view environments, as shown in Figure 4. This approach is inspired by established strategies in the 3D human pose estimation and pose-lifting literature, such as MPL [12] and Chen et al. [41], which generate synthetic 2D pose observations by projecting known 3D poses through simulated camera viewpoints.

Figure 4. Training data generation framework. (a) Camera placement with four cameras arranged radially around the subject. Yellow regions indicate approximate camera locations simulated by adding noise to camera extrinsics. (b) 3D poses from DHP19 and HAA4D datasets are globally aligned and scaled. (c) Aligned poses are transformed according to noisy camera extrinsics to simulate realistic setups with approximate camera placement. (d) 2D poses are obtained by applying perspective projection with camera intrinsics. (e) Clean generated 2D keypoints with zero noise. (f) Noisy generated 2D keypoints under realistic noise conditions.

In this work, we utilize 3D poses from DHP19 and HAA4D as our base data. Given potential skeletal topology differences across datasets, we first establish a consistent joint representation. Let $\mathcal{X}=X_j\in {\mathbb{R}}^3\mid j=1,2,\dots ,J$ denote the set of 3D joint positions for a skeleton with J joints. We interpolate specific anatomical landmarks to ensure consistency. The root joint position $X_{\mathrm{root}}$ is defined as follows:

$$X_{\mathrm{root}}={\displaystyle \frac{1}{2}}(X_{\mathrm{LHip}}+X_{\mathrm{RHip}}),$$

(1)

where $X_{\mathrm{LHip}}$ and $X_{\mathrm{RHip}}$ are the locations of the left and right hip, respectively.

Processing involves three key steps: (1) global alignment, (2) scaling, and (3) perspective projection.

3.3.1. Global Alignment

The 3D skeleton keypoints from the datasets, $X_j$, are first globally centered at the location of the root, $X_{\mathrm{root}}$, to ensure the subject remains within the synthetic image space, as shown in Equation (2). Since multiple datasets are employed, this mapping aims to create a consistent coordinate system to reduce inter-dataset differences.

$$X_j^{\mathrm{aligned}}=X_j-X_{\mathrm{root}}$$

(2)

3.3.2. Scaling

Next, we apply scale normalization to standardize skeleton dimensions. For each frame t in a sequence, we compute a scaling factor $s_t$ based on the maximum spatial extent across all coordinate axes, as defined in Equation (3):

$$\begin{array}{cc}\hfill s_t& =\underset{c\in \left\{\mathrm{x},\mathrm{y},\mathrm{z}\right\}}{\max }\left(\underset{j=1\dots J}{\max }\left(X_{j,t}^{\left(c\right)}\right)-\underset{j=1\dots J}{\min }\left(X_{j,t}^{\left(c\right)}\right)\right)\hfill \\ \hfill X_{j,t}^{\mathrm{scaled}}& =X_{j,t}^{\mathrm{aligned}}·{\displaystyle \frac{1}{s_t}},\forall j\in \{1,2,\dots ,J\}\hfill \end{array},$$

(3)

where $X_{j,t}$ is the 3D key point for joint j at time t. J is the total number of joints. c is the coordinate axis (x, y, or z).

3.3.3. Perspective Projection

The final step involves generating 2D keypoints for multiple views to simulate the outputs of a 2D pose estimator applied across cameras. An imaging environment is simulated, with four cameras arranged radially around the origin. To simulate different setups, we add random noise to the extrinsic parameters, E (Equation (4)) defining the camera position and orientation, specifically to the rotation matrix $\mathcal{R}\in {\mathbb{R}}^{3\times 3}$ and to the translation vector $\overrightarrow{T}\in {\mathbb{R}}^{3\times 1}$ of the camera with respect to the global coordinate system.

$$E=\left[\mathcal{R}\right|\overrightarrow{T}]$$

(4)

The rotation matrix, $\mathcal{R}$ is composed as shown in Equation (5):

$$\mathcal{R}=R_z(\gamma +{\delta }_{\gamma })R_y(\beta +{\delta }_{\beta })R_x(\alpha +{\delta }_{\alpha }),$$

(5)

where the matrices $R_z(.)$, $R_y(.)$, and $R_x(.)$ are the elementary rotation matrices around the z-, y-, and x-axis, respectively, and ${\delta }_{\gamma },{\delta }_{\beta },{\delta }_{\alpha }\sim \mathcal{U}[-{ϵ}_{\mathrm{angle}},{ϵ}_{\mathrm{angle}}]$ are the corresponding camera angle noise terms.

The translation vector, $\overrightarrow{T}$ is composed as shown in Equation (6):

$$\begin{array}{cc}\hfill \overrightarrow{T}& =-\mathcal{R}c\hfill \\ \hfill c& =-{\mathcal{R}}^T\left[\begin{array}{c}0\\ 0\\ d+{\delta }_d\end{array}\right],\hfill \end{array}$$

(6)

where d is the desired camera distance and ${\delta }_d\sim \mathcal{U}[-{ϵ}_{\mathrm{dist}},{ϵ}_{\mathrm{dist}}]$ is the corresponding distance noise term.

The intrinsic parameters, represented by matrix $\mathcal{K}$, are based on a standard iPhone camera, with random noise sampled uniformly from the interval $[-{ϵ}_{\mathrm{int}},{ϵ}_{\mathrm{int}}]$ added to the focal length ($f_x$, $f_y$) and principal point ($c_x$, $c_y$). Equation (7) describes the intrinsics matrix:

$$\mathcal{K}=\left[\begin{array}{ccc}{f}_x+{\delta }_f& 0& c_x+{\delta }_c\\ 0& f_y+{\delta }_f& c_y+{\delta }_c\\ 0& 0& 1\end{array}\right].$$

(7)

where ${\delta }_f,{\delta }_c\sim \mathcal{U}[-{ϵ}_{\mathrm{int}},{ϵ}_{\mathrm{int}}]$ are the corresponding intrinsics noise terms.

Intrinsic and extrinsic parameters are held constant across all frames in each view and sampled independently for each sequence. As shown in Equation (8), the 3D joint is first mapped into the camera frame and then into homogeneous image coordinates, and the resulting homogeneous vector is normalized by its third component so that the projection is expressed in pixel units:

$$\begin{array}{cc}\hfill {\tilde{x}}_j& =\left[\begin{array}{c}{\tilde{u}}_j\\ {\tilde{v}}_j\\ w_j\end{array}\right]=\mathcal{K}\left(\mathcal{R}{X}_j^{\mathrm{scaled}}+\overrightarrow{T}\right),w_j>0,\hfill \\ \hfill x_j& =\left[\begin{array}{c}{u}_j\\ v_j\end{array}\right]={\displaystyle \frac{1}{w_j}}\left[\begin{array}{c}{\tilde{u}}_j\\ {\tilde{v}}_j\end{array}\right],\hfill \end{array}$$

(8)

where $w_j$ is the depth of joint j along the optical axis. This projection accounts for the camera intrinsic and extrinsic properties to simulate realistic 2D poses from any sample 3D pose.

To mimic noisy predictions from a 2D HPE model, Gaussian noise proportional to body height is added to each keypoint in view v:

$$x_{j,v}^{\mathrm{noisy}}=x_{j,v}+{\eta }_{j,v},{\eta }_{j,v}\sim \mathcal{N}(0,{\sigma }_{j,v}^2),$$

(9)

where ${\sigma }_{j,v}={ϵ}_{\mathrm{joint},v}·h_v$ is the noise standard deviation, $h_v$ is the body height in pixels for view v, and ${ϵ}_{\mathrm{joint},v}$ is the noise scale parameter that varies across views.

The combined noise in camera intrinsic and extrinsic parameters simulates realistic deployments where cameras are radially distributed, positioned at varying distances, and subjected to different pitch and roll angles. While this training strategy improves STA-TFM’s generalization to diverse camera placements without requiring camera parameters to be specified, all setups draw from the same distribution of parameters, rather than fundamentally different layout families. Additionally, since the subject moves freely throughout each sequence, there is no fixed notion of a ‘front’ or ‘back’ camera, preventing the network from learning view-specific anatomical shortcuts. In practice, arranging cameras in an ordered sequence around the subject remains important for optimal performance.

3.4. Model Training

The synthetic dataset described in Section 3.3 is partitioned strictly at the level of continuous recordings rather than individual frames. Specifically, DHP19 is split by subject, ensuring the test set contains unseen subjects, while HAA4D is split by action class, ensuring the test set contains entirely unseen actions. The union of the datasets is proportionally divided into 70% for training and validation and 30% for testing, where the training subset is further split with a 20% validation ratio. The normalized samples are windowed to a length of 36 frames with a stride of 12, ensuring overlap within a sequence. Since this windowing operation is applied exclusively within these pre-assigned recordings, any stride-induced overlap is fully confined to a single video, preventing cross-subset frame or window leakage.

During training, we implement two loss metrics: a standard spatial loss and a composite spatial–temporal loss. Mean Per Joint Position Error (MPJPE) is the primary loss metric and measures the average Euclidean distance between the predicted and ground truth joints. The Mean Per Joint Velocity Error (MPJVE) is a supplemental loss metric, and accounts for temporal differences in joints across frames. The composite loss, ${\mathcal{L}}_{\mathrm{composite}}$, as shown in Equation (10) is used to account for both the spatial and temporal features of the model during training, where $\lambda \in [0,1]$ controls the strength of the MPJVE component. Through hyperparameter tuning, we found $\lambda =1$ yields the best results.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{composite}}& ={\mathcal{L}}_{\mathrm{MPJPE}}+\lambda {\mathcal{L}}_{\mathrm{MPJVE}}\hfill \\ \hfill {\mathcal{L}}_{\mathrm{MPJPE}}& ={\displaystyle \frac{1}{T}}{\displaystyle \frac{1}{J}}{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{j=1}^J}{∥X_{j,t}^{\mathrm{pred}}-X_{j,t}^{\mathrm{scaled}}∥}_2\hfill \\ \hfill {\mathcal{L}}_{\mathrm{MPJVE}}& ={\displaystyle \frac{1}{T-1}}{\displaystyle \frac{1}{J}}{\displaystyle \sum _{t=1}^{T-1}}{\displaystyle \sum _{j=1}^J}{∥V_{j,t}^{\mathrm{pred}}-V_{j,t}^{\mathrm{scaled}}∥}_2\hfill \end{array}$$

(10)

Here, the velocity $V_{j,t}$ is as defined in Equation (11), T is the video sequence length, and J is the number of joints supported in the dataset.

$$V_{j,t}=X_{j,t+1}-X_{j,t},t\in \{1,\dots ,T-1\}$$

(11)

We train the model on a single NVIDIA Tesla V100-SXM2-16GB GPU with 4 physical (8 logical) CPU cores (NVIDIA, Santa Clara, CA, USA). Training employs automated mixed precision and gradient clipping ($\left|\right|\Delta \le 1\left|\right|$). The AdamW [42] optimizer is used with an initial learning rate of $1\times {10}^{-4}$ and cosine annealing for weight decay. Each run uses a batch size of 16 and is trained for 5 epochs. Training was limited to 5 epochs to ensure fair comparison with MPL under identical conditions, despite training curves indicating potential for further improvement. STA-TFM’s architecture contains 51M total parameters, with 35M trainable parameters. The complete training process takes approximately 2.5 h on 73,590 training samples with 4-camera, 36-frame sequences. Inference on the same hardware achieves 37.73 ms per instance (36-frame sequence, 4 cameras).

As there is a temporal embedding mismatch between DSTFormer (window length = 243) and STA-TFM (window length = 36), linear interpolation is required to reduce the temporal embedding length. The DSTFormer pretrained backbone is initialized from pretrained weights from [8] and could optionally be fine-tuned, while the fusion pose transformer and 3D regression head weights are learned.

To better generalize to occlusions, camera flicker, and view dropout, stochastic masking is applied during training. A dropout rate of 0.25 is applied to each view to simulate environments with missing cameras. If all cameras are dropped, a single camera is randomly re-initialized to ensure at least one view remains. To simulate camera flicker, a per-frame dropout rate of 0.01 is also applied.

3.5. Evaluation Metrics

To evaluate the predicted 3D keypoints in comparison to ground truth (GT) data, we use commonly reported metrics for human pose estimation outlined in [5,24,43]: Mean Per Joint Position Error (MPJPE), Percentage of Correct Keypoints (PCK), and Mean Per Joint Velocity Error (MPJVE). Additionally, Average Precision (${\mathrm{AP}}_K$) is computed for comparison with other camera-parameter and camera-parameter-free methods.

The MPJPE calculates the average Euclidean distance between predicted and ground truth joint locations across all frames. Smaller MPJPE values indicate better performance. Formally, it is defined as follows:

$$\mathrm{MPJPE}={\displaystyle \frac{1}{T}}{\displaystyle \frac{1}{J}}{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{j=1}^J}{\parallel {\widehat{X}}_{j,t}-X_{j,t}\parallel }_2$$

(12)

where T is the number of frames, J is the number of joints, ${\widehat{X}}_{j,t}$ is the predicted position of joint j at time t, and $X_{j,t}$ is the corresponding ground truth position.

PCK assesses accuracy by checking if predicted keypoints are within a certain distance threshold from the real joints, with the threshold set to 20% of the torso length. This is calculated as follows:

$$\mathrm{PCK}={\displaystyle \frac{1}{T}}{\displaystyle \frac{1}{J}}{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{j=1}^J}\mathbb{1}\left(\parallel {\widehat{X}}_{j,t}-X_{j,t}{\parallel }_2\le 0.2·l_{\mathrm{torso}}\right)\times 100\%$$

(13)

where $l_{\mathrm{torso}}$ is the torso length and $\mathbb{1}(·)$ is the indicator function that returns 1 if the condition is true and 0 otherwise.

To evaluate temporal consistency and the smoothness of predictions over time, the MPJVE measures the average error of the first derivative of 3D pose sequences. It is defined as follows:

$$\mathrm{MPJVE}={\displaystyle \frac{1}{T-1}}{\displaystyle \frac{1}{J}}{\displaystyle \sum _{t=1}^{T-1}}{\displaystyle \sum _{j=1}^J}{\parallel {\widehat{V}}_{j,t}-V_{j,t}\parallel }_2$$

(14)

where $V_{j,t}=X_{j,t+1}-X_{j,t}$ represents the velocity of joint j at time t, and ${\widehat{V}}_{j,t}$ and $V_{j,t}$ are the predicted and ground truth velocities, respectively.

Lastly, ${\mathrm{AP}}_K$ [24] measures 3D pose estimation accuracy by computing precision–recall curves at a distance threshold set to K mm. Each predicted pose is matched to the closest ground truth by MPJPE. A prediction is a true positive if its MPJPE < K mm and the matched ground truth has not been previously detected; otherwise, it is a false positive. AP is computed as the area under the precision–recall curve, evaluated at thresholds $K\in 25,50,100,150$ mm, with the mean reported as mAP.

STA-TFM is trained to predict zero-centered 3D skeletons centered at the root. Since the predicted skeletons are normalized and root-centered, they exist in a different coordinate frame than the ground truth, making direct comparison without alignment uninformative. Procrustes alignment is therefore applied to account for variations in translation, rotation, and uniform scaling between the predicted and ground truth skeletons. Importantly, Procrustes alignment does not modify the relative positions between joints or the overall skeleton shape, preserving inter-joint relationships and ensuring that the alignment step does not artificially inflate performance. This yields the primary evaluation metrics: P-MPJPE, P-PCK, and P-MPJVE. This approach is appropriate for applications such as movement assessment, joint angle estimation, and range of motion analysis, where accurate inter-joint relationships and body geometry are the primary objectives rather than absolute location in space.

4. Results

We evaluate STA-TFM across five datasets to assess its performance under different conditions. To clarify the contribution of our specific design choices spanning architectural components, loss functions, and training data, we structure these evaluations as a series of targeted ablations and comparative experiments. In Section 4.1 and Section 4.2, we evaluate the 2D-to-3D lifting capabilities of STA-TFM, examining the impact of temporal incorporation and composite loss function on datasets derived from DHP19 and HAA4D. Section 4.3 then assesses generalizability to novel datasets using BlendMimic3D, which was not seen during training. We then benchmark STA-TFM against both camera-parameter-free and camera-parameter methods on two widely used benchmarks, CMUPanoptic and TotalCapture, in Section 4.4 and Section 4.5. To simulate real-world constraints, we further analyze the model’s performance under varying camera counts and conduct a systematic quantitative robustness analysis under controlled 2D keypoint noise and joint dropout in Section 4.7 and Section 4.8. We then examine the impact of temporal sequence length in Section 4.9, concluding with a comparative analysis against leading monocular baselines in Section 4.10. Finally, we present a qualitative assessment of STA-TFM to demonstrate its multi-view 3D pose reconstruction capabilities in Section 4.11. A consolidated summary attributing STA-TFM’s overall performance gains to individual architectural components, drawing on all of the above experiments, is provided in Appendix A.1.2.

4.1. Impact of Temporal Modeling

To quantify the performance gains of incorporating temporal information into pose embeddings, STA-TFM is compared against vanilla MPL [12], which lacks temporal information. Table 2 presents the performance comparison between these methods. To ensure a fair comparison, both methods are trained on the same dataset generated from HAA4D and DHP19, for the same number of epochs. For all experiments, predictions utilize four 2D skeletons sequences captured at approximately 90-degree intervals with varying distances from the subject. Each 2D skeleton is normalized and zero-centered around the root before processing. Since MPL lacks temporal integration, relying solely on spatial features, STA-TFM’s improvement over the vanilla model is expected. Therefore, to establish a fair baseline, a naive Savitzky–Golay temporal filter is applied post hoc to MPL’s predictions.

Table 2. Performance comparison on DHP19, HAA4D, and BlendMimic3D datasets. All models are trained on a dataset generated from a union of HAA4D and DHP19 datasets. Lower ↓ indicates better performance; higher ↑ indicates better accuracy. Best results are bolded. w indicates the window size for the Savitzky–Golay filter that yields the best results.

Dataset	Model	Loss Function	P-MPJPE (mm) ↓	P-PCK (%) ↑	P-MPJVE (mm/Frame) ↓
DHP19	STA-TFM	Composite Loss	17.56	99.96	1.66
	STA-TFM	MPJPE Loss	18.71	99.93	2.82
	MPL	Default	35.78	99.49	3.29
	MPL + SG (w = 11)	Default	35.72	99.49	1.28
HAA4D	STA-TFM	Composite Loss	34.77	98.70	8.55
	STA-TFM	MPJPE Loss	36.48	98.42	8.73
	MPL	Default	37.27	98.44	9.26
	MPL + SG (w = 5)	Default	37.88	98.23	10.63
BlendMimic3D †	STA-TFM	Composite Loss	61.97	85.92	2.00
	MPL	Default	212.25	10.02	2.99
	MPL + SG (w = 5)	Default	212.25	10.02	3.00

(†) indicates novel dataset, unseen during training. Here, Blendmimic3D 2D keypoints are generated by data generation process described in Section 3.3.

Table 2 shows that STA-TFM significantly outperforms vanilla MPL on both DHP19 and HAA4D datasets. Even when a Savitzky–Golay temporal filter is applied post hoc to MPL’s predictions, improvements remain marginal and inconsistent across datasets, suggesting that a learned approach to temporal modeling is more effective than naive post-processing.

On DHP19, STA-TFM achieves 50.9% and 49.5% reductions in P-MPJPE and P-MPJVE, respectively, compared to MPL, with similar improvements of 6.7% and 7.7% on HAA4D. These gains in spatial accuracy and temporal coherence are directly attributed to incorporating temporal information, which provides an inherent denoising effect by considering multiple time points.

The performance variation between datasets can be attributed to several key factors. First, the training data distribution contains a natural imbalance, with DHP19 comprising approximately 80% of the training frames. Second, the datasets employ different splitting strategies: DHP19 uses subject-based splits that maintain action diversity during training, while HAA4D employs action-based splits that require generalization to unseen motion patterns. HAA4D also presents additional challenges through its more diverse and dynamic actions and unconstrained environments.

Despite these challenging conditions in HAA4D, the temporal modeling in STA-TFM still provides improvements over MPL, demonstrating the value of integrating temporal information even for novel actions.

4.2. Impact of Loss Function

MPJPE is the primary loss function typically used in 3D HPE model training. However, this loss function does not penalize 3D HPE predictions with low temporal consistency (jittery predictions). Therefore, STA-TFM improves temporal consistency by utilizing a composite loss, ${\mathcal{L}}_{\mathrm{composite}}$, as described in Section 3.4. To assess the impact of the composite loss, two versions of STA-TFM are trained: one with a standard loss based solely on MPJPE and another with ${\mathcal{L}}_{\mathrm{composite}}$. Both models are trained on DHP19 + HAA4D, and results are reported on each dataset separately in Table 2.

The results in Table 2 demonstrate a clear improvement when using ${\mathcal{L}}_{\mathrm{composite}}$ on both DHP19 and HAA4D datasets. Specifically, STA-TFM achieves 41.0% and 2.3% reductions in P-MPJVE on DHP19 and HAA4D, respectively. An improvement in P-MPJPE is also observed, with reductions of 6.2% and 5.0% on DHP19 and HAA4D, respectively. This spatial accuracy improvement can be attributed to ${\mathcal{L}}_{\mathrm{composite}}$ encouraging temporally consistent poses, which reduces frame-to-frame jitter and consequently improves per-frame predictions. The varying impact of ${\mathcal{L}}_{\mathrm{composite}}$ on P-MPJVE between datasets reflects their inherent differences, as described in Section 4.1.

4.3. Generalization to Unseen Datasets

To assess generalizability to unseen datasets, STA-TFM and MPL are trained on DHP19 and HAA4D datasets and evaluated on BlendMimic3D, which features new motion patterns and subjects. The 2D keypoints for BlendMimic3D are generated using the data generation process described in Section 3.3, producing synthetic 2D keypoints with added noise. The camera extrinsic parameters follow the same distribution as the training set (DHP19, HAA4D), with perturbations applied to both extrinsic and intrinsic parameters.

As shown in Table 2, STA-TFM substantially outperforms MPL across all metrics, achieving 70.8% and 41.6% reductions in P-MPJPE and P-MPJVE, respectively. These improvements stem from STA-TFM’s ability to use pretrained DSTFormer backbone, which provides transferable spatio-temporal representations that enhance generalization to unseen data. Since DSTFormer is adapted from monocular human pose estimation, it benefits from training on monocular datasets that are more abundant than multi-view datasets, contributing to improved generalization across diverse scenarios.

4.4. Performance Against Camera-Parameter-Free Methods

To evaluate STA-TFM against existing camera-parameter-free methods, we use the TotalCapture dataset following the training and testing split of [21]. The evaluation protocol tests generalization across four scenarios defined by combinations of seen and unseen cameras and subjects. In all scenarios, the test samples are novel and were not included in the training set. Specifically: (1) seen cameras and seen subjects, where both the camera setup and subjects appeared in the training set but are performing different actions; (2) seen cameras and unseen subjects, where the camera setup was seen during training but the subjects are novel; (3) unseen cameras and seen subjects, where the subjects appeared in training but the camera configuration is novel; and (4) unseen cameras and unseen subjects, where both the camera setup and subjects are novel to the model.

While various comparable camera-parameter-free methods report results on TotalCapture [12,20,21,30,32], many do not release their code bases, preventing direct comparison. We therefore limit our comparison to methods with publicly accessible code bases. Among these, none provide TotalCapture-specific model weights or evaluation code, requiring us to retrain all baselines on identical dataset splits using the hyperparameters specified in their original papers. The training set consists of 2D keypoints obtained from RTMPose [17] paired with their corresponding 3D ground truth skeletons, introducing realistic 2D keypoint predictor noise into the training and evaluation. Results are reported in Table 3. P-MPJPE is reported as the primary evaluation metric for all methods, ensuring that differences in alignment protocols are accounted for to provide a balanced comparison. Under this standard, STA-TFM achieves highly competitive results when looking at the mean of both seen and unseen subjects and camera setups, achieving 15.2%, 35.7%, 40.8%, 50.2%, and 55.22% reductions in P-MPJPE compared to MPL [12], SGraFormer [32], MTF-Transformer [21], EFMK [30], and ESMFormer [31], respectively. As expected, all methods experience a performance drop when evaluated on unseen subjects or cameras.

Table 3. Performance comparison on the TotalCapture dataset. Values show P-MPJPE in millimeters. All models use identical training data with 2D keypoints from RTMPose and hyperparameters from the original papers. (†) denotes modified hyperparameters for improved performance. Action types: W (walking), FS (free style), A (acting). Best results are bolded.

Method	Seen Subjects (S1, S2, S3)			Unseen Subjects (S4, S5)			Mean
	W2	FS3	A3	W2	FS3	A3
Seen Cameras (1,3,5,7)
EFMK [30]	48.94	89.24	55.61	55.03	112.62	66.45	71.31
MTF-Transformer [21]	32.99	80.59	57.50	56.89	105.58	76.26	68.30
MTF-Transformer [21] (†)	30.43	74.97	56.20	50.53	102.37	73.00	64.50
ESMFormer [31]	44.65	72.49	55.11	53.36	93.16	60.95	63.29
SGraFormer [32]	15.46	41.55	24.98	27.93	73.99	40.61	37.42
MPL [12]	18.19	42.18	26.21	26.76	68.80	39.43	36.93
STA-TFM	13.15	40.64	22.65	25.98	64.17	37.65	34.04
Unseen Cameras (2,4,6,8)
EFMK [30]	104.90	139.13	104.90	99.01	138.10	99.37	114.24
MTF-Transformer [21]	69.78	101.49	75.67	81.30	112.45	88.38	88.17
MTF-Transformer [21] (†)	59.12	94.90	66.50	63.76	108.47	81.66	79.06
ESMFormer [31]	103.31	142.76	110.87	115.32	157.65	128.39	126.38
SGraFormer [32]	82.54	101.87	91.29	89.86	111.19	91.32	94.67
MPL [12]	51.01	73.78	56.91	56.27	85.47	56.14	63.26
STA-TFM	32.21	61.18	40.87	42.04	77.90	51.10	50.90

4.5. Performance Against Camera-Parameter Methods

CMUPanoptic is a popular benchmark for evaluating camera-parameter methods. While these baseline models provide a valuable performance ceiling across the broader literature, they are not directly comparable to camera-parameter-free approaches. This is because they operate in a different problem space by relying on known camera parameters as geometric priors, allowing them to operate in absolute world space. On the other hand, STA-TFM functions as a camera-parameter-free method, focusing on capturing accurate body geometry in normalized space. Consequently, the evaluation metrics differ: calibrated models report absolute MPJPE, while normalized outputs require Procrustes aligned MPJPE (P-MPJPE).

To contextualize the performance of STA-TFM against these models, both STA-TFM and MPL are trained on CMUPanoptic following the training and testing split of VoxelPose [24]. We include MPL, the most architecturally comparable camera-parameter-free method to STA-TFM, as a reference point for comparison. Training is performed using 2D keypoints from RTMPose [17] paired with their corresponding 3D ground truth skeletons.

CMUPanoptic contains multi-person scenarios, but STA-TFM like all other comparable camera-parameter-free methods [12,21,30,32] supports only single-person inference. To handle this, we implemented a pre-processing step where 2D keypoints from all camera views were cross-associated to identify and group keypoints belonging to the same person, isolating individual subjects before passing them to the models for inference.

Table 4 presents STA-TFM and MPL against several camera-parameter methods. Despite the advantage of using Procrustes alignment, as expected, camera-parameter methods achieve higher accuracy than camera-parameter-free methods due to incorporating geometric priors [20,21]. Nevertheless, STA-TFM outperforms MPL (↓ 16.5% P-MPJPE, ↑ 1.12% PCK, ↓ 41.65% MPJVE). To further benchmark STA-TFM against traditional non-learning camera-parameter methods, the framework is also evaluated against a classical linear triangulation baseline, with full methodological details and comparative results provided in Appendix A.1.1.

Table 4. Comparison of camera-parameter and camera-parameter-free methods on the CMUPanoptic dataset. MPL and STA-TFM use RTMPose 2D keypoints as input. (†) denotes model performance using noiseless ground truth 2D keypoints as input. ${\mathrm{AP}}_K$ is computed from precision–recall curves at distance threshold K mm, where a prediction is a true positive if its MPJPE $<K$ mm. For camera-parameter-free methods, reported metrics are computed after applying Procrustes alignment between the predicted and ground truth poses.

Method	PCK	MPJVE	AP25	AP50	AP100	AP150	Recall @500	MPJPE (mm)
Multi-view methods with camera parameters
VoxelPose [24]	-	-	83.60	98.30	99.80	99.90	98.80	17.70
MVP [27]	-	-	92.30	96.60	97.50	97.70	98.20	15.80
TEMPO [26]	-	-	89.01	99.08	99.76	99.93	-	14.68
MVGFormer [28]	-	-	92.30	-	-	-	-	16.00
SelfPose3D [25]	-	-	55.10	96.40	98.50	99.00	99.60	24.50
Multi-view methods without camera parameters
MPL [12]	95.78	5.81	5.90	63.96	98.73	99.79	100	36.15
STA-TFM	96.86	3.39	19.70	76.70	99.40	99.90	99.90	30.17
STA-TFM (†)	96.76	3.08	25.00	68.00	99.20	99.70	99.90	28.83

4.6. Impact of Fine-Tuning DSTFormer

To assess whether fine-tuning DSTFormer yields additional performance gains for STA-TFM, we compare frozen and unfrozen variants trained for 50 epochs on CMUPanoptic with identical hyperparameters. Table 5 shows that unfrozen training yields negligible improvements (MPJPE: $30.17\to 30.26$ mm with RTMPose keypoints, $28.83\to 28.29$ mm with GT keypoints), demonstrating that monocular spatio-temporal features transfer effectively to multi-view 3D HPE without the need for task-specific fine-tuning.

Table 5. Ablation study comparing frozen and unfrozen DSTFormer variants of STA-TFM on CMUPanoptic. Both variants are trained for 50 epochs with identical hyperparameters, initialized from pretrained monocular DSTFormer weights. For all methods, reported metrics are computed after applying Procrustes alignment between the predicted and ground truth poses.

DSTFormer	2D Keypoints	PCK	MPJVE	AP25	AP50	AP100	AP150	MPJPE (mm)
Frozen	RTMPose	96.86	3.39	19.70	76.70	99.40	99.90	30.17
Frozen	GT	96.76	3.08	25.00	68.00	99.20	99.70	28.83
Unfrozen	RTMPose	96.79	3.35	20.00	77.30	99.40	99.90	30.26
Unfrozen	GT	96.88	2.92	32.40	68.40	99.40	99.90	28.29

4.7. Impact of Number of Cameras in Multi-View Methods

STA-TFM requires a minimum of two distinct camera views to predict 3D keypoints from a given set of 2D keypoints. Ideally, these cameras should be radially positioned for optimal performance (see Figure 5a). To assess the impact of the number of views on 3D keypoint prediction accuracy, Table 6 reports STA-TFM performance relative to the number of cameras. Starting from a four-camera setup, we simulate reduced coverage by removing cameras using two strategies: consecutive dropout, where the last m consecutive cameras (∼90° apart) are removed, and random dropout, where m cameras are randomly dropped regardless of their position. Figure 5b demonstrates both dropout strategies.

Figure 5. (a) Ideal camera placement. (b) Camera dropout strategies. Starting from four cameras positioned at 0°, 90°, 180°, and 270°, we evaluate two removal patterns: (Top) consecutive dropout removes the last m cameras in order (e.g., 270°, then 180°). (Bottom) random dropout removes m cameras at random positions (e.g., 90°, then 270°). Red cross indicates cameras that are dropped.

Table 6. Performance comparison on DHP19 and HAA4D datasets for different numbers of cameras and camera dropout strategies. Lower ↓ indicates better performance; higher ↑ indicates better accuracy. Best results are bolded.

Dataset	Dropout Strategy	No. Cameras (No. Dropped)	P-MPJPE (mm) ↓	P-PCK (%) ↑	P-MPJVE (mm/Frame) ↓
DHP19	No dropout	4 (0)	17.56	99.96	1.66
	Ordered	3 (1)	37.51	97.83	1.94
		2 (2)	74.45	86.16	2.35
	Random	3 (1)	43.87	96.83	2.13
		2 (2)	120.66	69.88	5.30
HAA4D	No dropout	4 (0)	34.77	98.70	8.55
	Ordered	3 (1)	68.26	86.62	10.66
		2 (2)	109.44	60.73	13.27
	Random	3 (1)	72.71	83.98	10.80
		2 (2)	155.77	46.44	19.30

Table 6 shows that performance decreases with fewer cameras, with the four-camera setup providing the lowest error rates. The results also reveal that camera order significantly impacts accuracy: consecutive camera dropout yields better performance than random dropout. This can be attributed to the learnable positional embeddings in the fusion transformer, which encode the spatial relationships between views. When cameras are dropped consecutively, the remaining views maintain their expected positional context. However, random dropout disrupts these learned spatial relationships, leading to degraded performance.

4.8. Impact of 2D Pose Estimation Noise on 3D Prediction

To investigate individual impacts of Gaussian noise and keypoint dropout, both were systematically injected into the DHP19 testing dataset. Table 7 presents a performance analysis across varying levels of additive Gaussian noise and keypoint dropout, simulating the effects of a noisy 2D keypoint detector and occluded joints. Noise level is expressed as a percentage of subject height, where Gaussian noise with a standard deviation of $\sigma =\mathrm{height}\times p$ is added to each keypoint, with p denoting the noise percentage. Keypoint dropout is simulated by randomly removing between $[n_{\min },n_{\max }]$ keypoints per view per frame, where $(n_{\min },n_{\max })\in (0,1),(0,3),(0,5)$. Figure 6 and Figure 7 illustrate the noise levels experimented with alongside the corresponding predicted and ground truth 3D poses.

Table 7. Robustness analysis on the DHP19 dataset evaluating the effect of additive Gaussian noise and keypoint dropout on P-MPJPE, P-PCK, and P-MPJVE. Bold text indicates better performance. ↑ indicates that higher is better, and ↓ indicates that lower is better.

# Missing KP	% Noise	P-MPJPE (mm) ↓	P-PCK (%) ↑	P-MPJVE (mm/Frame) ↓
0	0	17.56 ± 4.99	99.96 ± 0.22	1.66 ± 0.46
0	1	18.28 ± 4.92	99.96 ± 0.21	3.63 ± 0.98
0	3	30.76 ± 8.14	99.79 ± 0.57	22.24 ± 8.68
0	5	54.88 ± 10.96	95.96 ± 4.22	52.18 ± 13.64
0–1	0	18.26 ± 4.81	99.95 ± 0.23	4.03 ± 0.79
0–3	0	20.42 ± 4.58	99.66 ± 0.37	6.88 ± 1.41
0–5	0	23.83 ± 4.61	97.77 ± 0.69	9.14 ± 1.87

Figure 6. Effect of input corruption on 2D keypoint observations and 3D pose predictions. Rows show increasing noise levels: Gaussian perturbation at 1%, 3%, and 5%, and keypoint removal of 0–1, 0–3, and 0–5 joints. Columns show the four camera views. The rightmost column shows the predicted 3D pose. Gray underlays indicate ground truth. Results shown for DHP19 sequence S1_2_3.

Figure 7. Effect of input corruption on 2D keypoint observations and 3D pose predictions. Rows show increasing noise levels: Gaussian perturbation at 1%, 3%, and 5%, and keypoint removal of 0–1, 0–3, and 0–5 joints. Columns show the four camera views; the rightmost column shows the predicted 3D pose. Gray underlays indicate ground truth. Results shown for DHP19 sequence S4_3_1.

The model demonstrates reasonable tolerance to keypoint dropout, maintaining stable performance when up to five keypoints are randomly occluded per frame (P-MPJPE increases from 17.56 → 23.83 mm), which covers typical occlusion scenarios in real-world capture. For Gaussian noise, the model shows minimal degradation at 1% of subject height (P-MPJPE: 18.28 mm), a noise level representative of modern 2D pose detectors like RTMPose. At 5% noise, performance degrades substantially (P-MPJPE: 54.88 mm, P-MPJVE: 52.18 mm/frame), indicating that the model is sensitive to the larger positional uncertainties characteristic of low-quality 2D detections. This behavior is expected for camera-parameter-free methods, which lack the geometric constraints available to calibrated approaches and must rely more heavily on input quality.

4.9. Impact of Sequence Length

As DSTFormer [8] was originally pretrained using 243-frame windows, an ablation was performed to evaluate the effects of temporal window length when applied to datasets with variable sequence lengths. HAA4D contains short clips with an average length of 64 frames, while DHP19 contains substantially longer recordings, averaging 2110 frames. To accommodate shorter sequences, STA-TFM was trained with reduced window lengths, 81, 54, and 36 frames, while window strides remained a third of the sequence length. Table 8 shows that DHP19 performance remains almost similar across window lengths, with a decrease in P-MPJPE (↓ 3.78 mm) but an increase in P-MPJVE (↑ 0.09 mm/frame) when reducing the window length from 81 frames to 36 frames. This indicates that the model can effectively extract motion cues from temporal segments of multiple lengths.

Table 8. Comparison of P-MPJPE, P-PCK, and P-MPJVE across sequence lengths on DHP19 and HAA4D. ↑ indicates that higher is better, and ↓ indicates that lower is better. Bold indicates better performance.

Dataset	Seq. Length	P-MPJPE (mm) ↓	P-PCK (%) ↑	P-MPJVE (mm/Frame) ↓
DHP19	81	21.34 ± 7.13	99.80 ± 0.60	1.57 ± 0.47
	54	22.61 ± 5.87	99.90 ± 0.50	1.61 ± 0.43
	36	17.56 ± 4.99	99.96 ± 0.22	1.66 ± 0.46
HAA4D	81	46.74 ± 7.13	95.40 ± 8.40	11.05 ± 11.50
	54	43.47 ± 13.71	97.00 ± 6.30	11.09 ± 11.69
	36	34.77 ± 11.45	98.70 ± 4.21	8.55 ± 8.23

The primary performance benefit was seen with the short HAA4D videos as shown in Table 8. When windowed at 36 frames, the P-MPJPE (↓ 11.97mm) and P-MPJVE (↓ 2.5 mm/frame) decreased significantly in comparison to a temporal length of 81. This improvement arises as shorter windows better match HAA4D’s intrinsic clip duration and reduce the influence of padding. These findings demonstrate that compact temporal windows can effectively capture motion in brief videos without compromising performance in long-duration videos.

4.10. Multi-View vs. Single-View Analysis

To contextualize STA-TFM performance relative to monocular approaches, we compare against popular single-camera 3D HPE models: MotionBERT [8], RTMW3D [44], VideoPose3D [43], BlazePose (MediaPipe) [45], and MotionAGFormer [9]. Multi-view methods inherently benefit from reduced depth ambiguity and improved occlusion handling compared to monocular approaches. We quantify this advantage in this analysis. These models are selected because they incorporate temporal information across multiple frames to predict 3D poses, ensuring a fair comparison. Since STA-TFM utilizes the DSTFormer block from MotionBERT to generate spatially and temporally aware 2D pose embeddings, the comparison with MotionBERT is particularly relevant, as any performance gains can be directly attributed to multi-view fusion via the FPT block. Comparisons with other models are provided for completeness.

We use the model trained on the union of the DHP19 and HAA4D datasets. To ensure fairness, all models are provided with 2D keypoints computed by running RTMPose [17] on videos from the four camera views in BlendMimic3D. Unlike Section 4.3, where 2D keypoints are back-projected using the data generation pipeline with camera placements matching the training distribution, here the 2D keypoints are extracted from real cameras with substantially different placements. This represents a completely unseen camera configuration for STA-TFM.

Table 9 shows that STA-TFM achieves substantially better performance compared to the best monocular model (VideoPose3D): P-MPJPE ($\downarrow 11.6\%$), P-PCK ($+44.72\%$), and P-MPJVE ($\downarrow 57.1\%$). Similar improvements are observed compared to MotionBERT: P-MPJPE ($\downarrow 20.9\%$), P-PCK ($+47.12\%$), and P-MPJVE ($\downarrow 61.4\%$). As expected, this performance advantage decreases with fewer cameras, as shown in the three-camera and two-camera configurations in Table 9. Notably, when constrained to a two-camera configuration, STA-TFM does not outperform the leading monocular model in terms of P-MPJPE; however, it consistently outperforms monocular models in P-PCK and P-MPJVE. This suggests that even limited cross-view aggregation provides a regularizing effect on temporal jitter, enabling smoother and more anatomically consistent 3D motions than single-camera architectures can produce. These results demonstrate that multi-view information provides more accurate 3D poses compared to monocular approaches, which suffer from depth ambiguity and occlusions.

Table 9. Performance comparison on BlendMimic3D for multi-view and monocular 3D HPE models. Lower ↓ is better; higher ↑ is better. Best results are bolded. 2D keypoints provided to STA-TFM are computed by running RTMPose on each video.

No. Cameras	Model	P-MPJPE (mm) ↓	P-PCK (%) ↑	P-MPJVE (mm/Frame) ↓
4	STA-TFM	94.17 ± 17.38	70.07 ± 10.11	4.77 ± 2.46
3	STA-TFM	98.39 ± 17.23	67.61 ± 10.25	4.86 ± 2.41
2	STA-TFM	108.02 ± 13.86	60.39 ± 9.06	4.45 ± 2.00
1	MotionBert [8]	119.04 ± 39.25	22.95 ± 22.38	12.84 ± 5.92
1	RTMW3D [44]	125.81 ± 34.15	20.79 ± 20.05	32.24 ± 18.89
1	VideoPose3D [43]	106.49 ± 30.68	25.35 ± 23.86	11.49 ± 5.29
1	BlazePose [45]	216.80 ± 27.56	6.49 ± 6.08	16.85 ± 8.63
1	MotionAGFormer [9]	116.80 ± 28.44	38.59 ± 9.80	7.90 ± 4.57

4.11. Qualitative Assessment

Qualitative results of STA-TFM under challenging conditions are presented in Figure 8. The associated full video sequences are provided at the Zenodo repository https://zenodo.org/records/22832620 (accessed on 20 October 2026) showing multi-view 3D pose reconstruction for subjects S1, S2, and S3 performing acting and freestyle sequences on the TotalCapture dataset.

Figure 8. Qualitative assessment of STA-TFM under challenging conditions; blue poses represent predictions and red poses represent ground truth. (a) Noisy input 2D keypoints when subjects are partially visible or completely absent from camera views. (b) 3D pose reconstruction from remaining camera views when one or more views are missing or contain incomplete 2D keypoint detections. (c) Noise in predicted 3D keypoints at the extremities, particularly at wrists and arms, compared to stable core body keypoints. (d) Noise in predicted 3D keypoints during extreme movements in freestyle scenarios where rapid motion challenges temporal modeling.

Notably, when subjects move out of view and the 2D detector hallucinates non-human artifacts as keypoints, STA-TFM maintains reasonable 3D pose reconstruction (Figure 8a). Furthermore, the model demonstrates tolerance to occluded joints in single views and successfully reconstructs poses from remaining cameras when one or more views contain missing keypoints (Figure 8b). This behaviour is attributed to the integration of keypoint confidence scores into the DSTFormer feature embeddings, which down-weights unreliable detections. Additionally, the model’s predictions are less sensitive to scale and translation, as training on root-centered and height-normalized skeletons removes these dependencies.

Despite these strengths, an increase in noise is observed in the extremities, particularly at the wrists and elbows. As these distal joints typically exhibit higher velocities and larger ranges of motion, they challenge temporal consistency, while core body keypoints remain comparatively stable (Figure 8c). Beyond these localized inaccuracies, the model exhibits broader failure cases in extreme scenarios. In the event the majority of camera views lack valid 2D keypoints, the model performance degrades significantly. Complex self-occlusions, particularly in the TotalCapture freestyle scenario, where participants perform rapid atypical movements such as tucking into a ball and jumping, challenge the model’s accuracy (Figure 8d). These cases highlight the limitations of temporal modeling under highly dynamic motion patterns.

5. Discussion

This paper presents STA-TFM, a multi-view spatio-temporal transformer for 3D human pose estimation that integrates DSTFormer for temporal modeling with a fusion transformer for cross-view aggregation. Unlike traditional approaches that depend on known camera calibration parameters, STA-TFM learns to aggregate multi-view 2D poses to directly predict 3D body poses. This makes STA-TFM more practical for real-world deployments where calibration data may be unavailable or unreliable.

Extensive experiments show that STA-TFM outperforms comparable camera-parameter-free methods. On DHP19, STA-TFM achieves 50.9% and 49.5% reductions in P-MPJPE and P-MPJVE compared to MPL, with improvements of 6.7% and 7.7% on HAA4D. On TotalCapture, STA-TFM outperforms all compared camera-parameter-free methods, achieving 15.2%, 35.7%, 40.8%, 50.2%, and 55.22% reductions in P-MPJPE compared to MPL [12], SGraFormer [32], MTF-Transformer [21], EFMK [30], and ESMFormer [31], respectively. The composite loss function ${\mathcal{L}}_{\mathrm{composite}}$ is shown to enhance temporal consistency, yielding 41.0% and 2.3% additional P-MPJVE reductions on DHP19 and HAA4D, respectively, compared to a standard loss function solely based on MPJPE. Qualitative and quantitative results demonstrate tolerance to missing views and noisy keypoints. A summary attributing these overall performance gains to individual architectural components is provided in Appendix A.1.2.

To the best of our knowledge, STA-TFM is the first method to leverage frozen pretrained monocular features (DSTFormer) for multi-view 3D HPE. The limited gains observed from fine-tuning DSTFormer (Section 4.6) suggest that spatio-temporal representations learned for monocular pose estimation data remain effective when transferred to multi-view 3D human pose estimation. To address training data scarcity, a data generation pipeline is utilized that simulates n radially distributed cameras with controllable noise parameters, enabling training without expensive multi-camera setups.

5.1. Limitations

While STA-TFM eliminates the need for camera calibration, it introduces several operational constraints. First, the model requires frame synchronization across all camera streams, which is a practical constraint during data acquisition. Furthermore, computational cost scales with the number of views, which may limit deployment in resource-constrained settings. Additionally, while STA-TFM operates without explicit geometric priors at inference, the model learns from a specific radial distribution during training; consequently, it operates optimally when maintaining a sufficient view count arranged in an approximately radial and ordered setup.

While the data generation pipeline addresses cases where multi-view 2D-3D data is scarce, we acknowledge that our data generation pipeline does not fully replicate real multi-camera conditions. The pipeline does not model correlated joint occlusions across views. Relying on an idealized pinhole camera model and bypassing image formation entirely, the pipeline does not directly capture lens distortion, rolling shutter, motion blur, or illumination effects. These factors are instead approximated through added keypoint-level and camera intrinsics noise.

Additional limitations relate to the model’s prediction scope. Like other camera-parameter-free methods, STA-TFM supports only single-person inference. In multi-person scenarios, it requires per-person multi-view keypoints, necessitating keypoint association across views before inference, adding an extra step. Additionally, STA-TFM is currently trained to output only major body landmarks. Dense keypoints such as hand, foot, and facial landmarks are not predicted, representing a key direction for future extension. Lastly, STA-TFM outputs normalized, root-centered 3D poses, and Procrustes alignment with ground truth is needed to obtain poses in the world frame of reference. This means that, while the predicted poses maintain accurate relative joint positions and can be used to measure body angles and skeletal proportions, they cannot be used to measure real-world distances between joints or track the subject’s overall travel distance.

5.2. Practical Implications of STA-TFM and Future Work

The practical implications of STA-TFM extend across domains where flexible, low-cost motion capture is needed. By only requiring radially distributed cameras with no calibration, STA-TFM enables deployment in environments where traditional motion capture is impractical. In clinical settings, clinicians could capture high-quality 3D pose estimates of a single patient using synchronized, low-cost smartphones on tripods for quantitative assessment of neuromotor tasks, rehabilitation exercises, and movement disorders. This capability opens the door to generating large, matched datasets of videos and 3D poses for patients performing clinical assessments, data that is currently scarce and expensive to collect. Similar setups could support athletic performance analysis and sports medicine, making 3D pose estimation accessible to practitioners in gyms, on fields, and in rehabilitation facilities for assessing relative joint kinematics.

While a performance gap remains relative to camera-parameter methods that leverage geometric priors, STA-TFM narrows this gap without requiring camera calibration. Closing the remaining gap would benefit from exploring world-frame pose representations to eliminate the need for Procrustes alignment, extending the fusion mechanism to multi-person scenarios, and optimizing for real-time inference. Although DSTFormer was selected because its dual-stream adaptive fusion outperforms sequential spatio-temporal pipelines, evaluating feature extractors from alternative monocular pose estimators such as MotionAGFormer [9], MixSTE [10], and STGFormer [11] could explore further performance gains. Extending STA-TFM to finer-grained joint prediction, such as full 3D hand reconstruction [46], represents a natural and valuable direction for future work, enabling downstream applications including hand gesture recognition that are highly relevant to healthcare monitoring and human–computer interaction. Finally, establishing standardized benchmarks for camera-parameter-free multi-view HPE would also accelerate progress by enabling fair cross-method comparison across the field.

Appendix A.1

Appendix A.1.1. Performance Against Triangulation Baseline

To assess the impact of using a 2D keypoint detector on the 2D-to-3D lifting process, STA-TFM is evaluated on videos from the BlendMimic3D dataset using 2D keypoints predicted by RTMPose [17]. The model was trained on data generated from the HAA4D and DHP19 datasets using the data generation pipeline described in Section 3.3. This setup reflects a realistic inference scenario in which the 2D inputs contain inherent detection noise from a 2D HPE model. The resulting 3D poses are compared against linear triangulation [19], which estimates 3D keypoints from multi-view 2D observations and corresponding camera parameters. Experiments are conducted across two-, three-, and four-view configurations, as summarized in Table A1.

Triangulation provides a meaningful baseline only when 2D poses are obtained from a keypoint detector rather than ground truth annotations. Under synthetic conditions with perfect camera parameters, triangulation gains an unrealistic advantage because ground truth 2D keypoints together with exact camera parameters allow near-deterministic recovery of the original 3D pose when subjects are fully visible and unoccluded across views [19]. Although triangulation still benefits from the ideal camera parameters provided in the BlendMimic3D dataset, noise introduced by 2D keypoint detection results in a more realistic and informative performance benchmark.

As shown in Table A1, the P-MPJPE obtained with ground truth 2D poses is 34.2% lower than that obtained using the RTMPose-predicted 2D keypoints. The increase in P-MPJPE can be attributed to the propagation of errors from the 2D pose detector to the subsequent 3D lifting process. Despite this degradation, STA-TFM consistently outperforms triangulation across all camera configurations in terms of P-PCK and P-MPJVE. Notably, STA-TFM with three cameras achieves lower MPJPE than triangulation with four, reducing setup cost and complexity. STA-TFM also enforces greater temporal coherence and reduces jitter, as demonstrated by lower P-MPJVE values. This highlights STA-TFM’s robustness and generalization when relying solely on noisy 2D detections without requiring camera calibration.

Table A1. Performance comparison of STA-TFM and triangulation using both ground truth and RTMPose 2D keypoints on the BlendMimic3D dataset. Lower ↓ is better; higher ↑ is better. Best results are bolded.

No. Cameras	Model	2D Keypoints Source	P-MPJPE (mm) ↓	P-PCK (%) ↑	P-MPJVE (mm/Frame) ↓
4	STA-TFM	Data Generation Pipeline	61.97	85.92	2.00
	STA-TFM	RTMPose	94.17	70.07	4.77
	Triangulation		99.33	49.10	8.62
3	STA-TFM	RTMPose	98.39	67.61	4.86
	Triangulation		99.59	48.98	9.46
2	STA-TFM	RTMPose	108.02	60.39	4.45
	Triangulation		101.09	47.84	10.78

Table A1. Performance comparison of STA-TFM and triangulation using both ground truth and RTMPose 2D keypoints on the BlendMimic3D dataset. Lower ↓ is better; higher ↑ is better. Best results are bolded.

No. Cameras	Model	2D Keypoints Source	P-MPJPE (mm) ↓	P-PCK (%) ↑	P-MPJVE (mm/Frame) ↓
4	STA-TFM	Data Generation Pipeline	61.97	85.92	2.00
	STA-TFM	RTMPose	94.17	70.07	4.77
	Triangulation		99.33	49.10	8.62
3	STA-TFM	RTMPose	98.39	67.61	4.86
	Triangulation		99.59	48.98	9.46
2	STA-TFM	RTMPose	108.02	60.39	4.45
	Triangulation		101.09	47.84	10.78

Appendix A.1.2. Component Attribution Summary

To clarify the individual contributions of DSTFormer, multi-view fusion, and temporal modeling, we summarize the specific impact of each architectural component below. Fully isolating the temporal attention mechanism from scratch is omitted, as studies in monocular lifting, such as MotionBERT, MotionAGFormer, MixSTE, and STGFormer [8,9,10,11], have consistently demonstrated the importance of temporal modeling over purely spatial methods. However, the following ablations collectively demonstrate the quantitative benefits of our chosen modules:

• Per-view spatio-temporal encoder (DSTFormer vs. spatial pose transformer):holding the fusion pose transformer (FPT), sequence length, dataset splits, and training identical, substituting the spatial-only pose transformer in MPL with DSTFormer yields a 50.9% (DHP19), 6.7% (HAA4D), and 70.8% (BlendMimic3D) reduction in P-MPJPE.
• Multi-view fusion: sweeping camera coverage from two to four views demonstrates a direct improvement in P-MPJPE, as shown in Table 6 by a decrease in P-MPJPE from 74.45 mm to 17.56 mm, confirming the scaling benefit of the fusion module. Furthermore, we contrast STA-TFM with other multi-view methods such as MTF-Transformer [21], which represents an architectural contrast: MTF-Transformer fuses views before temporal aggregation, whereas STA-TFM extracts spatio-temporal features per view prior to cross-view fusion.
• Temporal supervision: isolating the velocity penalty (${\mathcal{L}}_{\mathrm{MPJVE}}$) within the loss function reduces temporal jitter by 41.0% (P-MPJVE) and simultaneously refines per-frame spatial accuracy by 6.2% (P-MPJPE) on DHP19.
• Temporal receptive field: sweeping the temporal window length from 81 down to 36 frames, as shown in Table 8, demonstrates that optimizing the temporal context yields significant performance improvements.
• Backbone fine-tuning: freezing versus fine-tuning DSTFormer in Table 5 confirms that pretrained monocular temporal representations transfer directly to multi-view fusion without retraining the 16M-parameter backbone.