2D to 3D Human Skeleton Estimation Based on the Brown Camera Distortion Model and Constrained Optimization

Abstract

In the rapidly evolving field of computer vision and machine learning, 3D skeleton estimation is critical for applications such as motion analysis and human–computer interaction. While stereo cameras are commonly used to acquire 3D skeletal data, monocular RGB systems attract attention due to benefits including cost-effectiveness and simple deployment. However, persistent challenges remain in accurately inferring depth from 2D images and reconstructing 3D structures using monocular approaches. The current 2D to 3D skeleton estimation methods overly rely on deep training of datasets, while neglecting the importance of human intrinsic structure and the principles of camera imaging. To address this, this paper introduces an innovative 2D to 3D gait skeleton estimation method that leverages the Brown camera distortion model and constrained optimization. Utilizing the Azure Kinect depth camera for capturing gait video, the Azure Kinect Body Tracking SDK was employed to effectively extract 2D and 3D joint positions. The camera’s distortion properties were analyzed, using the Brown camera distortion model which is suitable for this scenario, and iterative methods to compensate the distortion of 2D skeleton joints. By integrating the geometric constraints of the human skeleton, an optimization algorithm was analyzed to achieve precise 3D joint estimations. Finally, the framework was validated through comparisons between the estimated 3D joint coordinates and corresponding measurements captured by depth sensors. Experimental evaluations confirmed that this training-free approach achieved superior precision and stability compared to conventional methods.

1. Introduction

Advances in computer vision and machine learning technologies have positioned human pose analysis as an indispensable tool across various domains, including sports biomechanics [1,2], human–computer interaction [3], and surveillance systems [4], among diverse applications. The use of 2D pose estimation algorithms based on a monocular camera and 3D pose estimation algorithms based on a depth camera or binocular camera has made great progress and they are widely used in action recognition, motion analysis, and other fields. The reconstruction of 3D human pose in 2D video sequences is also of concern as a related issue.

Estimating 3D joint positions from monocular RGB inputs (images/video sequences containing human subjects) remains an active focus in computer vision research. By contrast with 2D counterparts, 3D human pose estimation (HPE) delivers enhanced spatial representations and enables more comprehensive use cases across application domains. The method to obtain 3D coordinates of joints is a key supporting technology in many fields, such as human body reconstruction, behavior recognition, virtual reality, game modeling, and has been applied to many products. Estimating 3D poses from 2D video sequences using a monocular camera [5,6] not only enhances the comprehension of the inner principles of human motion but also plays a significant role in evaluating movement ability and assisting rehabilitation training.

Using a single camera to collect data has the advantages of low cost and easy deployment, but it has inherent problems: a single image may correspond to countless scenes in the physical world, so it is difficult to estimate the depth from the image and achieve 3D reconstruction by using a monocular vision method [7]. Traditional pose estimation methods [8] from 2D to 3D mainly rely on deep learning models, which often require a lot of labeled data for training and are sensitive to image noise, such as camera distortion. Moreover, utilizing advanced motion capture systems frequently presents significant hurdles or proves to be impractical for acquiring 3D labels in actual environments.

As a result, unsupervised methods that operate without the need for labeled datasets have attracted significant interest. In the real world, images are inevitably distorted due to camera imaging principles, which creates challenges for pose estimation from 2D to 3D. Therefore, it is essential not only to devise novel algorithms for creating 3D pose estimation models from 2D data but also to enhance their adaptability to image distortion. At the same time, combined with the practical application scenarios, it is necessary to expand the application field of pose estimation, and provide technical support for research in related fields.

The method is inspired by the principle that if a line intersects a sphere, then the line has two points of intersection with the sphere. The human torso is parallel to the imaging plane, and the Z-axis coordinates of the torso can be calculated according to the imaging principle, so all 3D coordinates of the torso part can be obtained. Assuming that the 3D coordinates of a joint i are known and the bone lengths of joint i and joint j are also known, then according to the 2D pixel coordinates of joint j, the projection coordinates of joint j on the plane formed by the torso part can be obtained according to the distortion model and imaging principle, and the actual coordinates of this point are located on the line connected between the projection point of the torso plane and the optical center. Because the bone length is known, the 3D coordinate point of joint j is also located on the sphere with joint i as the center and the length as the radius.

This point is located in both the sphere and the line, so the 3D coordinate is one of the intersection points of the sphere and the line, from which the coordinates of the joint j can be obtained, and the coordinates of other joints connected to the joint j can be derived. It is well known that the sphere intersecting a straight line has two points of intersection, and a series of connections can obtain several 3D possible poses; however, the actual 3D pose is only one of many possibilities.

This principle provides inspiration for 3D estimation from a 2D pose with an optimized model in this paper, which shows that it is feasible to obtain a 3D pose by using bone lengths as constraints when the human torso is parallel to the imaging plane. The process involves the transformation from pixel coordinates to actual coordinates, so the undistorted coordinates should first be acquired. Due to the particularity of the human body structure and the characteristics of gait, this paper also provides some symmetry restrictions, which are helpful to eliminate some wrong poses when estimating a 3D pose with optimization. These constraint conditions are similar to choosing the one that is more consistent with the human body structure and gait law among the two intersection points of a straight line and a sphere.

From the concept of algorithm construction used in this paper, it can be seen that the algorithm in this paper can be easily extended to other scenarios.

A 2D to 3D gait skeleton estimation method based on a camera distortion model and constraint optimization is proposed. In summary, the key contributions of this paper include the following:

• This paper uses Azure Kinect Body Tracking SDK to extract 2D and 3D key points of the human body. First, the Azure Kinect depth sensor captures continuous depth images and color images. These image data are processed by the SDK, which can output the 2D and 3D skeleton of the human body at the same time.
• The camera distortion model is used to eliminate the distortion of 2D skeleton joints: By analyzing the camera imaging principle, the distortion model applicable to Azure Kinect camera is established, and the camera distortion is reduced by an iterative method, thus reducing the impact of distortion on subsequent 3D gait skeleton estimation.
• A constraint optimization algorithm to estimate the 3D skeleton is developed. In the process of 3D skeleton gait estimation, a series of constraints are set to ensure that the estimation results conform to biomechanical principles and actual human gait laws. The objective of the optimization model is to minimize the distance between the estimated 3D skeleton joint projection and the undistorted 2D skeleton joint, and to keep the relationship between the symmetric limb and the torso of the body. The constraints of the optimization model include consistent bone length, bilateral symmetry, linear structure of the specific joint, etc. By solving the optimization problem, the estimated 3D skeleton of a single frame can be obtained.

The structure of the proposed method is shown in Figure 1. Through these steps, the stable 3D skeleton sequence can be obtained, and  the 3D skeleton joints estimated in this paper can be compared with those directly obtained by the depth camera to verify the effectiveness of the proposed method. This study provides an efficient and robust solution for the field of skeleton estimation, and promotes the progress and application of related technologies. The algorithm proposed in this paper provides the possibility of calculating human gait parameters from video and evaluating human gait balance. The commonly used 2D estimation algorithm based on monocular RGB cameras obtains pixel coordinates, which are based on the pixel coordinate system. As is well known, the pixel coordinate system of the human skeleton is large when it is close to the camera and small when it is far away from the camera, so the method proposed in this paper provides the possibility of normalization of the skeleton sequence. The normalized skeleton makes it easy to calculate various parameters of human motion, such as speed, acceleration and angle.

2. Related Work

2.1. 3D Human Pose Estimation

With recent advancements in computer vision and machine learning technologies [9], the task of 3D human pose estimation has gained significant attention within motion capture and virtual reality applications. Research paradigms in 3D human pose estimation are typically divided, based on supervision levels, into three principal categories: fully supervised, weakly supervised, and self-supervised approaches.

Fully supervised 3D frameworks [10,11,12,13] utilize precise ground-truth data acquired through labor-intensive annotation processes. In response to these demands, multiple strategies [14,15] have emerged, particularly neural network-based techniques [10,16,17]. These architectures automatically extract discriminative features and establish reconstruction models through large-scale training, demonstrating strong generalizability. Notable implementations incorporating convolutional neural networks [17] and LSTM architectures [16] exhibit enhanced precision in skeletal joint prediction. While achieving state-of-the-art performance, such approaches have two fundamental constraints: substantial dependence on meticulously annotated 3D datasets and intensive computational requirements. Furthermore, their adaptability to complex scenarios involving dynamic motions remains suboptimal compared to traditional methods.

To mitigate the expense of acquiring 3D motion data, weakly supervised frameworks [18,19] have been developed, necessitating minimal or unaligned 3D labels. Self-supervised strategies [20,21,22], differing fundamentally from preceding techniques, eliminate reliance on 3D annotations entirely. These autonomous learning paradigms primarily comprise multi-camera systems and use of a monocular camera.

Multi-camera systems [20,21] implement self-supervisory mechanisms as alternatives to 3D labeling. View-invariant consistency serves as the foundational principle for numerous techniques [21,22], effectively resolving depth uncertainties during model training. This paradigm operates under the core premise that reconstructed 3D poses must maintain invariance across varying viewpoints. Consequently, multi-view 3D reconstruction prioritizes precise camera rotation estimation to achieve view-consistent results.

Unlike multi-view systems, monocular-based 3D reconstruction under self-supervision presents greater complexity. Pioneering works [23,24] have addressed this challenge through cyclic consistency mechanisms, circumventing 3D annotation requirements in monocular settings. The framework [23] develops a 2D-to-3D lifting architecture that deduces skeletal structures from 2D keypoints. Stochastic projections of reconstructed 3D skeletons are evaluated by a 2D pose validator, creating a self-correcting loop through geometric verification. This cyclic process involves reprojecting synthesized 3D poses into multiple 2D planes, enabling autonomous supervision via spatial coherence constraints. Complementing this, [24] introduces stochastic projection modules that map 3D predictions back to 2D space. Their adversarial framework trains discriminators to distinguish between synthesized poses and authentic 2D pose distributions, enhancing reconstruction fidelity through competitive learning.

Numerous studies have explored diverse constraint-based approaches for skeletal estimation and refinement [7,25,26]. Kinematic restrictions are frequently incorporated as supplementary loss functions to guide neural networks in generating anatomically plausible 3D pose predictions. Two predominant constraint formulations involve limb dimension preservation [27] and bilateral symmetry regularization [28]. In particular, the study [25] demonstrates the effectiveness of integrating biological skeletal length preservation principles, bilateral symmetry priors, and temporal continuity constraints to improve anatomical plausibility and motion fluidity in dynamic human pose estimation. Alternative methodologies employ spatial reasoning frameworks [7], where gait analysis systems are developed through parametric geometric representations synchronized with visual data analysis. While these constraint-driven implementations typically achieve computationally lightweight implementations, their predictive accuracy remains contingent upon the fidelity of geometric parameterization.

The constrained optimization model framework proposed in this paper takes into account the characteristics of gait and is highly targeted. In addition to the bone length and symmetry constraints commonly used in previous unsupervised methods, linear constraints on the parts of bones are also added. The construction of objective functions involves the characteristics of gait limb swing, which provides a theoretical basis for research in mathematical modeling.

2.2. Imaging Principle of Pinhole Camera Model

This study considers unsupervised 3D pose estimation for a monocular camera, which uses a constraint optimization model to estimate the 3D pose for a monocular camera in a specific scene. It involves the lens imaging principle. Nowadays, imaging devices, including professional camera systems, mobile imaging units, and surveillance apparatus, predominantly adhere to the optical imaging mechanisms defined by the Pinhole Camera Model [29]. This geometric formulation facilitates the mathematical mapping of three-dimensional spatial coordinates onto two-dimensional pixel matrices through perspective projection and rigid transformation processes. The complete imaging architecture comprises four principal coordinate domains: a World Coordinate System, a Camera Coordinate System, an Image Plane Coordinate System, and a Pixel Coordinate System. Essential to this paradigm is the sequential coordinate transformation pipeline that systematically converts physical world measurements into digital image representations through precisely defined geometrical operations.

Due to lens curvature error, the stacking sequence of the lens group and system error, lens distortion occurs during the imaging process [30]. In particular, the distortion is more obvious in the use of wide-angle lenses and fisheye lenses of non-measuring cameras. Optical distortion affects the geometry and size of the image, thus causing problems in image measurement and analysis. Therefore, in practical use, it is necessary to eliminate the distortion error of the lens, which is of great significance for photogrammetry.

The mathematical characterization of optical aberrations traces its origins to Conrady’s 1919 pioneering work on decenter distortion modeling [31]. Building upon this foundation, Brown’s seminal 1960s framework integrated both radial and decenter distortion components [32]. Radial aberrations originate from lens curvature imperfections, whereas decenter distortions stem from misaligned optical axis configurations. Subsequent investigations [33,34,35,36] proposed incremental refinements to these distortion parameterizations through modified coefficient formulations. Nevertheless, practical implementation of these enhanced models remains constrained by two critical limitations: excessive computational complexity resulting in ill-posed estimation challenges, and insufficient generalizability across diverse imaging hardware configurations.

Researchers have extensively investigated camera distortion models [33,37,38], and proposed a variety of distortion correction methods [39], such as radial distortion, tangential distortion, and use of the thin lens model. These methods are designed to mitigate the adverse effects of camera distortions on image processing. Although the camera distortion model has been continuously developed and refined, their application in 3D skeleton gait estimation remains limited. In this paper, a 2D to 3D gait skeleton estimation method based on the Brown camera distortion model is proposed.

The imaging process of the pinhole camera model involves mapping from three-dimensional space coordinates to the image plane. The 3D coordinate reconstruction process is the process of restoring the geometric structure and topological relationship of three-dimensional objects from a series of two-dimensional images or point cloud data captured under known angles or viewpoints.

To sum up, many researchers have achieved fruitful research results in deep learning in the field of 2D to 3D skeleton estimation. However, there is little research on 2D to 3D gait skeleton estimation based on a camera distortion model and constrained optimization, and there are still some challenges in this field, such as how to fully utilize a camera distortion model to compensate the distortion of skeleton joints and to estimate 3D gait with full consideration of human gait characteristics.

In order to solve this problem, the Brown camera distortion model is fully used in this paper. Then, a constraint optimization algorithm is proposed for 3D gait skeleton reconstruction, which can fully consider the human skeleton structure and gait characteristics, improving the accuracy of the estimation results.

3. Methods

3.1. Skeleton from Azure Kinect

The framework is based on the skeleton captured from the Azure Kinect Body Tracking SDK using the Azure Kinect camera. The joint position is estimated relative to the global depth sensor frame of reference. The position is specified in millimeters. All joint coordinate systems are absolute coordinate systems in the depth camera 3D coordinate system. A skeleton includes 32 joints with the joint hierarchy flowing from the center of the body to the extremities. As demonstrated in Figure 2, this structural mapping accurately replicates the joint locations and connections relative to the human body.

The participant was instructed to walk in a straight path, with the depth camera positioned at their front for this study. The user’s 2D and 3D skeleton gaits were obtained by Azure Kinect Body Tracking SDK.

3.2. Camera Transformation Pipeline

The parameters for the Brown distortion model [38] in Azure Kinect can be determined through a camera calibration process or obtained from the recorded file. Several studies [34,38] have indicated that estimating just the initial few parameters of the distortion model is sufficient to achieve high precision in modeling distortion.

A series of transformations for coordinate changes is established through multiple stages [38]. There is a shift from the global (world) coordinate system to the camera coordinate system:

Transition from reference (world) coordinates to camera coordinates:

$$\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=R\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]+t$$

(1)

where R denotes the rotation matrix while t represents the displacement vector defining the geometric mapping between the global reference and camera spatial coordinates. Both the coordinate systems are orthogonal Cartesian coordinate systems.

The mathematical representation of perspective projection involves the coordinate transformation process from the camera’s three-dimensional coordinate system to its two-dimensional projection plane, where the normalized image plane is defined at $\left(z^{\prime }=1\right)$. This fundamental conversion establishes the relationship between spatial positions observed through the imaging device and their corresponding planar projections within the camera’s optical framework.

$$x^{\prime }={\displaystyle \frac{x}{z}},y^{\prime }={\displaystyle \frac{y}{z}},z^{\prime }=1$$

(2)

Distortion, expressed as the transition from undistorted to distorted coordinates:

$$\left\{\begin{array}{c}{x}^{″}=x^{\prime }\left({\displaystyle \frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}}\right)+\left[2p_1{x}^{\prime }{y}^{\prime }+p_2\left(r^2+2{x^{\prime }}^2\right)\right]\hfill \\ y^{″}=y^{\prime }\left({\displaystyle \frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}}\right)+\left[p_1(r^2+2{y^{\prime }}^2)+2p_2{x}^{\prime }{y}^{\prime }\right]\hfill \end{array}\right.$$

(3)

where $r^2=x^{\prime 2}+y^{\prime 2}$ and $k_1,k_2,\dots k_6$ are the distortion parameters, while $p_1,p_2$ are the decentering parameters.

Transition from distorted coordinates to pixel coordinates:

$$u=f_x{x}^{″}+c_x,v=f_y{y}^{″}+c_y$$

(4)

where $\left(c_x,c_y\right)$ denote the principal point coordinates in the image plane, corresponding to the intersection position between the optical axis and the imaging projection surface, with their 2D positions referenced to the pixel coordinate framework. The parameter $f_x$ represents the normalized focal length (calculated as the ratio of the lens focal length to the physical width) of individual photosensitive units. Similarly, $f_y$ indicates the vertical normalized focal length (calculated as the ratio of the lens focal length to the physical height) in the vertical dimension.

The Azure Kinect Body Tracking SDK was utilized to obtain both 2D and 3D joints of the human body. The 2D key joints are represented as pixel coordinates that have been distorted, necessitating the correction of these joint positions to achieve undistorted data.

3.3. Obtaining the Undistorted Joints

The process to obtain the undistorted joints [40,41] is actually the process of reprojection, that is, the process going from pixel coordinates to physical coordinates and to pixel coordinates. Compensation for image distortion can be performed in two ways, which are called forward correction and reverse correction. Forward correction involves calculating the standard coordinates from the distortion coordinates, and reverse correction involves calculating the distortion coordinates from the standard coordinates. Either forward correction or reverse correction must first transition to physical coordinates, and then convert to pixel coordinates. The 2D skeleton joint coordinates used in this paper are distorted pixel coordinates, so the joints need to be corrected forward. The UndistortPoints Algorithm 1 used to obtain the undistorted joints in our method refers to the process of UndistortPoints in OpenCV4.7.0. Following OpenCV is an iterative process.

Algorithm 1 UndistortPoints Algorithm

Input: Distorted pixel coordinates: $\left(u,v\right)$, Camera parameters: $f_x, f_y, c_x, c_y, k_1, k_2, k_3, k_4,$ $k_5, k_6, p_1, p_2$ Output: Undistort coordinates $\left(x^{\prime },y^{\prime }\right)$   1: Iterations initialization: $count\Leftarrow 0$   2: Reprojection error initialization: $error\Leftarrow 1$   3: Calculate distorted coordinates: $x^{″}\Leftarrow {\displaystyle \frac{u-c_x}{f_x}},y^{″}\Leftarrow {\displaystyle \frac{v-c_y}{f_y}}$   4: $x_0^{\prime }\Leftarrow x^{″}$   5: $y_0^{\prime }\Leftarrow y^{″}$   6: while  $count<5$do   7:     if $error<0.08$ then   8:         break   9:     else   10:         $r^2\Leftarrow x^{\prime 2}+y^{\prime 2}$   11:         $icdist\Leftarrow \left({\displaystyle \frac{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}}\right)$   12:         $deltaX\Leftarrow 2p_1{x}^{\prime }{y}^{\prime }+p_2\left(r^2+2{x^{\prime }}^2\right)$   13:         $deltaY\Leftarrow p_1(r^2+2y^{\prime 2})+2p_2{x}^{\prime }{y}^{\prime }$   14:         $x^{\prime }\Leftarrow \left({x^{\prime }}_0-deltaX\right)·icdist$   15:         $y^{\prime }\Leftarrow \left({y^{\prime }}_0-deltaY\right)·icdist$   16:         $count\Leftarrow count+1$   17:         Calculate the reprojection pixel $\left(u_{proj},v_{proj}\right)$ coordinates according to $\left(x^{\prime },y^{\prime }\right)$:   18:         $r^2\Leftarrow x^{\prime 2}+y^{\prime 2}$   19:         $icdist2\Leftarrow \left({\displaystyle \frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}}\right)$   20:         $deltaX2\Leftarrow 2p_1{x}^{\prime }{y}^{\prime }+p_2\left(r^2+2{x^{\prime }}^2\right)$   21:         $deltaY2\Leftarrow p_1(r^2+2y^{\prime 2})+2p_2{x}^{\prime }{y}^{\prime }$   22:         $x^{″}\Leftarrow x^{\prime }·icdist2+deltaX2$, $y^{″}=y^{\prime }·icdist2+deltaY2$   23:         $u_{proj}\Leftarrow f_x{x}^{″}+c_x$   24:         $v_{proj}\Leftarrow f_y{y}^{″}+c_y$   25:         $error\Leftarrow \sqrt{{\left(u_{proj}-u\right)}^2+{\left(v_{proj}-v\right)}^2}$   26:     end if   27: end while

In the iterative calculation, the currently distorted point is taken as the initial value. In step 5, the loop iteration is used to remove the distortion. The iteration error was set to 0.08 and the maximum number of iterations is 5. In steps 17–25, the undistorted point is substituted into the distortion model to obtain the pixel coordinates of reprojection, and the deviation between the pixel coordinates of reprojection and the original pixel coordinates is obtained. The point without distortion is iteratively calculated until the condition of iteration error or maximum number of iterations is met. Through the iteration algorithm, undistorted skeletal joints are obtained.

3.4. Joint Depth Initialization

The model presupposes that the length of bones is known and the length of symmetric bones is consistent. In the subsequent modeling process, if only the length of bones is constrained, the estimated skeleton will show a distorted state due to the absence of a depth benchmark, so a depth value of one of the skeleton joints needs to be assigned. The joint we selected is joint 1, the center of the skeleton. The depth value constraint coupled with the constraint of the bone structure basically determines the position and pose of the skeleton. This section is the depth initialization of joint 1.

As is well known, in the process of walking in front of the camera, the torso part of the body is almost parallel to the imaging plane, and each bone of the torso is also parallel to the imaging plane, so the depth value of the two points connecting the bone is the same. Therefore, the relationship of depth to bone length and the coordinates of these two points can be determined. The process is as follows:

After we obtain the undistorted joints, the coordinates of point i and point j in the imaging plane are $(x_{i,t}^{\prime },y_{i,t}^{\prime },1)$, $(x_{j,t}^{\prime },y_{j,t}^{\prime },1)$.

If the z coordinates of the 3D skeleton joints are $z_{i,t}$, $z_{j,t}$, respectively, the 3D coordinates can be represented as $(x_{i,t}^{\prime }{z}_{i,t},y_{i,t}^{\prime }{z}_{i,t},z_{i,t})$, $(x_{j,t}^{\prime }{z}_{j,t},y_{j,t}^{\prime }{z}_{j,t},z_{j,t})$.

The bone length is

$$l_{i,j}=∥p_{i,t}-p_{j,t}∥=\sqrt{{\left({x^{\prime }}_{i,t}{z}_{i,t}-{x^{\prime }}_{j,t}{z}_{j,t}\right)}^2+{\left({y^{\prime }}_{i,t}{z}_{i,t}-{y^{\prime }}_{j,t}{z}_{j,t}\right)}^2+{\left(z_{i,t}-z_{j,t}\right)}^2}$$

(5)

If the line between points i and j is parallel to the imaging plane, then $z_{i,t}=z_{j,t}$.

Therefore,

$$z_{i,t}=z_{j,t}={\displaystyle \frac{l_{i,j}}{\sqrt{{\left({x^{\prime }}_{i,t}-{x^{\prime }}_{j,t}\right)}^2+{\left({y^{\prime }}_{i,t}-{y^{\prime }}_{j,t}\right)}^2}}}$$

(6)

In the process of walking directly towards the camera, the joint combinations (1, 2, 3, 4, 5, 11, 12, 18, 22) in Figure 2 are parallel to the camera and almost in the same plane, so the mean length of these combinations is used as the initial estimate of joint 1.

3.5. 3D Pose Estimated Based on Skeleton Constraints

To obtain the coordinates of the 3D skeleton, the skeleton intrinsic structure and initialization depth of joint 1 are used as constraints. A function is defined that relates the distance squared between the estimated 3D skeleton joint projection and the undistorted 2D skeleton joint and it also relates the relationship between the symmetric limb and the torso of the body, ensuring that the function is minimized.

The generated coordinate positions can reflect the 3D skeleton representation of human gait well. Therefore, accurate reconstruction of the gait skeleton can be obtained.

The physical coordinates of the undistorted joints are $\left[x^{\prime },y^{\prime }\right]$, and the estimated 3D coordinates (based on the camera coordinate system) are $\left[\widehat{x},\widehat{y},\widehat{z}\right]$.

According to Formula (2), $x^{\prime }={\displaystyle \frac{\widehat{x}}{\widehat{z}}},y^{\prime }={\displaystyle \frac{\widehat{y}}{\widehat{z}}},z^{\prime }=1$.

Therefore, the distance between $\left[x^{\prime },y^{\prime },1\right]$ and $\left[{\displaystyle \frac{\widehat{x}}{\widehat{z}}},{\displaystyle \frac{\widehat{y}}{\widehat{z}}},1\right]$ is minimized.

For convenience, it was converted into the first term of the objective function. $S_{mp}$ represents the set of points in the gait with symmetric relations, $S_{sp}$ represents the set of all bones, and $S_{lp}$ represents the set of skeleton joins with a linear structure.

$$\begin{array}{c}\min \sum _{i=1}^M\left[{\left({x^{\prime }}_{i,t}{\widehat{z}}_{i,t}-{\widehat{x}}_{i,t}\right)}^2+{\left({y^{\prime }}_{i,t}{\widehat{z}}_{i,t}-{\widehat{y}}_{i,t}\right)}^2\right]+\sum _{(i,j,m)\in S_{mp}}{\left[\left({\widehat{z}}_{i,t}+{\widehat{z}}_{j,t}\right)-2z_{m,t}\right]}^2\end{array}$$

(7)

$$\begin{array}{c}s.t.l_{i,j}=\sqrt{{\left({\widehat{x}}_{i,t}-{\widehat{x}}_{j,t}\right)}^2+{\left({\widehat{y}}_{i,t}-{\widehat{y}}_{j,t}\right)}^2+{\left({\widehat{z}}_{i,t}-{\widehat{z}}_{j,t}\right)}^2},\forall \left(i,j\right)\in S_{sp}\end{array}$$

(8)

$$\begin{array}{cc}\hfill & {\widehat{z}}_{1,t}=z_{average,t}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & \left\{\begin{array}{c}\left({\widehat{x}}_{j,t}-{\widehat{x}}_{i,t}\right)\left({\widehat{y}}_{k,t}-{\widehat{y}}_{i,t}\right)-\left({\widehat{y}}_{j,t}-{\widehat{y}}_{i,t}\right)\left({\widehat{x}}_{k,t}-{\widehat{x}}_{i,t}\right)=0,\hfill \\ \left({\widehat{x}}_{j,t}-{\widehat{x}}_{i,t}\right)\left({\widehat{z}}_{k,t}-{\widehat{z}}_{i,t}\right)-\left({\widehat{z}}_{j,t}-{\widehat{z}}_{i,t}\right)\left({\widehat{x}}_{k,t}-{\widehat{x}}_{i,t}\right)=0,\hfill & \hfill \forall \left(i,j,k\right)\in S_{lp}.\\ \left({\widehat{y}}_{j,t}-{\widehat{y}}_{i,t}\right)\left({\widehat{z}}_{k,t}-{\widehat{z}}_{i,t}\right)-\left({\widehat{z}}_{j,t}-{\widehat{z}}_{i,t}\right)\left({\widehat{y}}_{k,t}-{\widehat{y}}_{i,t}\right)=0,\hfill \end{array}\right.\hfill \end{array}$$

(10)

The objective function of the model has two terms. The first term minimizes the distance between the estimated 3D skeleton joint projection and the undistorted 2D skeleton joint. The second item ensures a symmetrical position relationship between the limbs and the torso.

There are three constraints. The first constraint guarantees the consistency of the distance between skeleton joints; the second constraint gives the depth information of the skeleton center; and the third constraint ensures the linear characteristics of the spine and hip joints. Through these constraints, a 3D skeleton that is consistent with the human body structure can be obtained.

The bone length obtained by the sensor is inconsistent, but we set the bone length in advance. There will be deviation when the depth information is initialized by the 2D bone length setting. In order to reduce the impact of such error, a second optimization process is used to obtain a more accurate skeleton estimate. The optimization algorithm of this paper is shown in Algorithm 2. In the second optimization process, the depth information of joint 1 with the depth information of other joints of the torso part are compared, and the second constraint is updated to obtain a more accurate 3D skeleton. In the second step, the optimization algorithm dynamically adjusts the constraints according to the estimation results to improve the robustness of the estimation.

Algorithm 2 2D to 3D human skeleton estimation algorithm

Input: Distorted pixel coordinates of skeleton:$\left\{\left(u_i,v_i\right)\left|i\in \left[0,31\right]\right.\right\}$ Bone lengths: $\left\{l_{i,j}\left|\left(i,j\right)\in S_{sp}\right.\right\}$ Camera parameters: $f_x, f_y, c_x, c_y, k_1, k_2, k_3, k_4, k_5, k_6, p_1, p_2$ Output: 3D coordinates of skeleton: $\left\{\left({\widehat{x}}_i,{\widehat{y}}_i,{\widehat{z}}_i\right)\left|i\in \left[0,31\right]\right.\right\}$   1: Get the undistort coordinates from Algorithm 1 (input: $\left\{\left(u_i,v_i\right)\left|i\in \left[0,31\right]\right.\right\}$): $\left\{\left({x^{\prime }}_i,{y^{\prime }}_i\right)\left|i\in \left[0,31\right]\right.\right\}$   2: The Set of child joints of bones in torso: $S=\left\{j\left|j\in \left[1,2,3,4,5,11,12,18,22\right]\right.\right\}$   3: $z_j\Leftarrow {\displaystyle \frac{l_{i,j}}{\sqrt{{\left({x^{\prime }}_i-{x^{\prime }}_j\right)}^2+{\left({y^{\prime }}_i-{y^{\prime }}_j\right)}^2}}},j\in S$   4: $z_{average}\Leftarrow \sum _{j\in S}{z}_j/9$   5: Set the second constraint: $z_1=z_{average}$   6: Get the 3D coordinates $\left\{\left({\widehat{x}}_i,{\widehat{y}}_i,{\widehat{z}}_i\right)\left|i\in \left[0,31\right]\right.\right\}$ by solving unconstrained optimization model   7: $count\Leftarrow 0,error\Leftarrow$ 10,000   8: while  $count<5$do   9:     if $error<50$ then   10:         break   11:     else   12:         Update the second constraint $z_1=z_{average}\Leftarrow {\displaystyle \sum _{j\in S\setminus 1}}{\widehat{z}}_i/8$   13:         Get the 3D coordinates $\left\{\left({\widehat{x}}_i,{\widehat{y}}_i,{\widehat{z}}_i\right)\left|i\in \left[0,31\right]\right.\right\}$ by solving unconstrained optimization model   14:         $error\Leftarrow \sqrt{{\left(z_{average}-{\displaystyle \sum _{j\in S\setminus 1}}{\widehat{z}}_i/8\right)}^2}$   15:         $count\Leftarrow count+1$   16:     end if   17: end while

4. Experiments and Results

4.1. Implementation Details

The computational framework presented in this paper was developed using the Python 3.9 programming language combined with Azure Kinect Body Tracking SDK 1.0.1 on the Windows 10 OS. The experiments are performed on real human motions captured by an Azure Kinect camera from Microsoft, a company in Washington, DC, USA. All the experimental tests are run on a PC with an Intel Core i5 CPU i5-12400 at 2.50 GHz and 16.0 GB RAM.

4.2. The Visualization of Estimated 3D Skeleton

The skeleton processed by this approach is shown in Figure 3, where red, green, and blue represent the skeletons of Azure Kinect, and the skeletons of the proposed method in step1 and step2. As shown in the figure, the proposed method is consistent with the generated skeleton of Azure Kinect.

The skeletons obtained with or without linear constraints are shown in Figure 4. As shown in the figure, without linear restraint, the bones of the pelvic and spine become distorted and do not conform to the structure of the human body. Therefore, linear constraints are very necessary.

4.3. Comparison of the 2D and 3D Skeleton with and Without UndistortPoints Algorithm

The 2D skeleton with and without the UndistortPoints Algorithm is shown in Figure 5, where yellow and blue are the 2D skeleton without and with the UndistortPoints Algorithm.

The 3D skeleton with and without use of the UndistortPoints Algorithm processed by this approach is shown in Figure 6, where red, blue and yellow represent the 3D skeletons of Azure Kinect, and the estimated 3D skeletons of the proposed method with and without the UndistortPoints Algorithm.

As shown in Figure 5, the 2D skeletons with and without the UndistortPoints Algorithm are slightly inconsistent, but the 3D skeleton with the UndistortPoints Algorithm is closer to the 3D skeleton of Azure Kinect. This shows that the UndistortPoints Algorithm plays an important role in our estimation algorithm. It is necessary to introduce the Brown camera model into our optimization method.

4.4. The Statistics of Bone Length

This experimental evaluation was designed to quantitatively assess the skeletal dimension accuracy enhancement achieved by the novel algorithm introduced in this research. Figure 7, Figure 8 and Figure 9 demonstrate comparative analyses of bone length fluctuations across distinct anatomical regions during ambulatory motion, where the red, green, and blue curves represent the bone lengths of Azure Kinect, the preliminary phase results from our methodology, and the results of the secondary phase, respectively. Reference values captured during the initialization procedures appear as yellow horizontal baselines.

As can be seen from Figure 7, Figure 8 and Figure 9, the symmetrical bone length changes obtained by Azure Kinect are consistent, but the bone length is not consistent at the time level. Conversely, the developed methodology demonstrates superior performance, achieving both bilateral symmetry and temporal stability in skeletal dimension monitoring across all evaluated motion sequences. Similar results were obtained in experiments with different subjects.

4.5. The Statistics of Joint Position

The coordinates of the three axes of joint 0 are shown in Figure 10. Chromatically differentiated trajectories in the visualization denote distinct data sources: red plots represent Azure Kinect measurements, green plots represent preliminary phase outputs from our methodology, and blue plots indicate refined results from the subsequent processing stage. The graphical data demonstrate spatial congruence between coordinates generated in the optimization phase of our approach and those acquired by Azure Kinect, substantiating the essential role of the refinement procedure. This secondary processing phase effectively mitigates residual errors persisting from the initial estimation phase.

5. Application

From the idea of algorithm construction in this paper, it can be seen that the algorithm in this paper can be easily extended to other scenarios.

The algorithm proposed in this paper provides the possibility of calculating human gait parameters from video and evaluating human gait balance. The commonly used 2D estimation algorithm based on monocular RGB cameras obtains pixel coordinates, which are based on a pixel coordinate system. As is well known, the pixel coordinate system of the human skeleton is large when it is close to the camera and small when it is far away from the camera. The method proposed in this paper provides the possibility of normalization of the skeleton sequence. The normalized skeleton makes it easy to calculate various parameters of human motion, such as speed, acceleration and angle.

In addition, the Azure Kinect camera contains a single RGB sensor and a depth sensor. The RGB image information can be separated from the recorded files. In order to expand the application scenario of this study, this paper uses the human pose estimation algorithm HRnet [42] to estimate the 2D skeleton sequence for the separated RGB video. Because the HRnet skeleton is relatively simple, some unnecessary constraints are removed. Instead of joint 1 in this article, we use the center coordinates of both shoulders from HRnet to initialize the depth.

Analysis of experimental outcomes indicates that the developed method demonstrates robust capabilities in reconstructing 3D skeleton structures. A representative output from a single frame in Figure 11 is illustrated in Figure 12. These findings confirm that the methodology presented in this research possesses broad practical utility across different 2D human pose estimation methods.

6. Discussion

Using a single camera has the advantages of low cost and easy deployment, but there are some problems: a single image may correspond to countless real physical world scenes, so it is difficult to use a monocular vision method to estimate the depth from the image and achieve 3D reconstruction. In this study, a camera distortion model and constraint optimization methods are introduced to reconstruct the 3D skeleton with a single camera.

The application of a camera distortion model can effectively correct the errors caused by camera lens distortion. The distortion model provides a more reliable reference for the subsequent 3D skeleton estimation. The integration of dimensional stability restrictions and bilateral symmetry regulations ensures temporal coherence within the articulated framework. By modeling anatomical joints as dynamic spatial coordinates, their displacement trajectories must obey fundamental principles governing rigid body dynamics in three-dimensional space. Consequently, this dual constraint mechanism maintains invariant interosseous distances throughout motion sequences. Furthermore, depth regularization imposed on the geometric centroid of the human morphological configuration establishes stable positional referents for the skeletal system. In the second step of optimization, this constraint value is updated according to the results obtained in the first step. These constraints not only ensure the stability of the bone structure, but also help eliminate estimation errors caused by data noise and occlusion.

The experimental results show that the 2D to 3D gait skeleton estimation method based on a camera distortion model and constraint optimization achieves remarkable results. Compared to deep learning methods, this method can obtain accurate bone positions while maintaining the stability of the skeleton structure, and does not require prior training.

However, the method proposed in this paper also has some limitations. The gait assessed in this study is based on walking in front of the camera lens, which is mainly based on the consideration that the upper part of the human body is basically parallel to the camera imaging plane, so as to simplify the model solving process. However, this method still has some limitations, which constrain its application in different scenarios. In addition, this research method relies heavily on the preset bone length information, and errors in the preset bone length may significantly affect the estimated results.

In the future, we will further study how to explore more effective constraints and algorithms, enhance the robustness of the model, and further improve the estimation accuracy and real-time performance to meet the needs of more practical application scenarios.

7. Conclusions

In this paper, a method of estimating a human 3D skeleton from 2D is proposed. Optical distortion is a significant problem in many computer vision applications and needs to be compensated. In the 3D gait skeleton estimation technique presented within this paper, the integration of a distortion model markedly enhanced the precision of 2D skeleton joints. This advancement consequently established a robust foundation, which is pivotal for the subsequent 3D skeleton estimation process. Skeleton estimation is based on an optimization framework on frames, which aims to minimize the distance between projection of the estimated 3D skeleton joints and the 2D joints after compensation for optical distortion, as well as the positional relationship between the symmetrical limbs and the body trunk, and takes the human structure characteristics, such as bone length, bilateral symmetry, and linear structures at specific joint points, as constraints. Through comparative experiments, we found that the optimized algorithm proposed in this paper can effectively estimate the positions of 3D skeleton joints while capturing the inherent structure of the human body. The comparison of the 2D and 3D skeleton with and without the UndistortPoints algorithm shows that it is necessary to introduce the Brown camera model into the optimization method. From the application of the algorithm with HRnet, it can be seen that the methodology presented in this research possesses broad practical utility across different 2D human pose estimated methods. Compared with traditional methods, this method does not rely on training on the dataset and provides a new perspective for the estimation of 3D skeleton joints. In summary, the 2D to 3D gait skeleton estimation method based on the camera distortion model and constrained optimization proposed in this paper has significant advantages in terms of accuracy and robustness. In practical applications, this method has broad prospects and is expected to provide support in fields such as gait analysis and human–computer interaction.