Expectation–Maximization Method for RGB-D Camera Calibration with Motion Capture System

Abstract

Camera calibration is an essential research direction in photonics and computer vision. It achieves the standardization of camera data by using intrinsic and extrinsic parameters. Recently, RGB-D cameras have been an important device by supplementing deep information, and they are commonly divided into three kinds of mechanisms: binocular, structured light, and Time of Flight (ToF). However, the different mechanisms cause calibration methods to be complex and hardly uniform. Lens distortion, parameter loss, and sensor degradation et al. even fail calibration. To address the issues, we propose a camera calibration method based on the Expectation–Maximization (EM) algorithm. A unified model of latent variables is established for the different kinds of cameras. In the EM algorithm, the E-step estimates the hidden intrinsic parameters of cameras, while the M-step learns the distortion parameters of the lens. In addition, the depth values are calculated by the spatial geometric method, and they are calibrated using the least squares method under an optical motion capture system. Experimental results demonstrate that our method can be directly employed in the calibration of monocular and binocular RGB-D cameras, reducing image calibration errors between 0.6 and 1.2% less than least squares, Levenberg–Marquardt, Direct Linear Transform, and Trust Region Reflection. The deep error is reduced by 16 to 19.3 mm. Therefore, our method can effectively improve the performance of different RGB-D cameras.

1. Introduction

RGB-D cameras have attracted increasing attention and have been widely applied in computer vision tasks, such as object detection [1,2,3], depth estimation [4,5,6], image processing [7,8,9], Structure from Motion (SfM) [10,11], Simultaneous Localization and Mapping (SLAM) [12,13], and so on. As an essential step in standardizing the data quality of RGB-D cameras, camera calibration makes it possible to acquire more accurate images and depth information. RGB-D cameras are developed in various ways, such as monocular depth sensors, and binocular stereo vision technology. However, the variety of camera types results in many calibration methods [14], and these methods are often limited to specific types of cameras in application, which restricts their widespread use. Therefore, RGB-D camera calibration methods are rarely unified. In addition, RGB-D cameras face challenges similar to those of conventional RGB cameras, such as lens distortion, camera intrinsic parameter loss, and degeneration of electronic components, which result in failures of image data standardization.

RGB-D cameras are primarily implemented through binocular cameras, structured light, and ToF technologies [15]. Binocular cameras reconstruct scene depth by capturing paired RGB images with dual lenses and estimating disparity through stereo matching, typically selecting one view as the RGB image. This approach requires accurate calibration of the dual lenses, and the disparity matching depends upon the texture information in the images. Due to the complex mechanism of binocular camera systems, unified calibration is difficult to achieve. To compensate for the lack of texture information, structured light technology is often integrated to augment texture details missing from the original. Depth information can be estimated by projecting structured lights and analyzing their deformations [16]. However, structured lights can lead to data loss due to specular reflections on smooth surfaces. ToF cameras calculate depth distance by measuring the round-trip time or phase delay of light signals [17,18]. RGB information is usually acquired by an additional monocular camera to form RGB-D data. Despite the advantages of ToF, such as high accuracy and fast response, the technology suffers from poor anti-interference capability, making it prone to losing information at long distances [19]. Based on the above analysis, RGB-D cameras can be divided into two independent parts: depth data and RGB data. The depth data collects the straight-line distance between the lens center and the scene point, while the calibration of RGB image data lacks an effective unified model.

RGB-D cameras face issues similar to those of RGB cameras, where intrinsic parameters are uncertain due to lens distortion, loss of parameters, and degeneration of sensor components [20,21]. To calibrate intrinsic and extrinsic parameters separately, Wu and Zhu [17] proposed a calibration approach that bridges traditional calibration and self-calibration, addressing recalibration after camera movement. To improve the accuracy of binocular camera calibration, Yin et al. [22] proposed to use a single calibration image and a coded planar target, through multi-constraint optimization on the condition of missing camera information. The related research above primarily focuses on fitting parameters based on the pinhole model. The parameters are mainly divided into two independent parts: hardware-related parameters such as focal length, and lens distortion correction parameters. Both are mutually dependent in the calibration process.

From the above analysis, it can be seen that the RGB-D camera can be calibrated in a two-stage model; one stage is for the RGB camera, and the other is for the depth camera. In this way, we proposed a general RGB-D camera calibration method based on the Expectation–Maximization (EM) and depth correction algorithms, which can be applied consistently across different representative RGB-D cameras without introducing camera-specific modeling or parameter tuning. This design helps alleviate inconsistencies in model parameters across different types of depth cameras. We utilized the EM algorithm to estimate the hardware and distortion parameters separately, achieving the first-stage calibration of the RGB image. In the second stage, the obtained parameters were employed to convert the depth information of the scene, and the depth was calibrated using least squares regression under a motion capture system. This method is suitable for motion capture–based applications such as human pose estimation and behavior analysis. The EM algorithm is adopted because its probabilistic optimization naturally handles incomplete or noisy observations, enabling robust parameter estimation even under real-world noise and minor outliers. The main contributions of this paper include the following:

• A unified RGB-D camera calibration framework has been established to solve the calibration problem of RGB-D cameras. It can calibrate different types of RGB-D cameras, such as monocular and binocular cameras, in a unified way.
• A camera calibration method is proposed based on the EM algorithm, which simultaneously calculates hardware parameters and lens distortion parameters. This method can efficiently improve the accuracy of camera calibration.
• The depth data is calibrated under a motion capture system, transferring the lens’s straight-line depth to spatial depth, which provides a new solution for calibration of depth data.

The overview of this paper is organized as follows: Section 2 reviews related work on traditional, learning-based, and RGB-D calibration methods; Section 3 presents equipment, data, and the proposed EM-based RGB calibration and depth correction framework; Section 4 describes the experimental setup and comparative evaluation results; and Section 5 concludes the study and outlines future work.

2. Related Work

2.1. Traditional Universal Camera Calibration Methods

Traditional universal camera calibration methods aim to develop flexible imaging models. Early research [23,24,25,26] focused on the pinhole model [27]. Zhang’s groundbreaking method [28], which is based on the observation of planar patterns, combines closed-form solutions with nonlinear optimization techniques grounded in maximum likelihood criteria, thereby greatly simplifying the camera calibration process. Besides the pinhole model, imaging models also include fisheye models, catadioptric models, and others, leading to calibration methods for highly distorted conditions [23,24,29]. The rational function model [30] has been adopted by OpenCV due to its efficiency and versatility in extreme distortion environments such as fisheye and catadioptric cameras. Sturm and Ramalingam [24] integrated multiple imaging models and proposed a universal calibration method suitable for central, non-central, and axial cameras. Additionally, some studies returned to the essence of optical imaging. Non-parametric methods were directly used to calibrate image distortions. These methods established mapping relationships between positions and pixels, and simplified parameter calculation and calibration steps [31,32]. Grossberg and Nayar [23] introduced the concept of a raxel, a virtual mapping from light rays to pixels, and constructed an ideal optical model. Bergamasco et al. [33] proposed a general unconstrained nonlinear model method. A unified calibration center was enforced for non-central cameras, and the method was achieved by constraining light rays through the optical center. These methods provide flexible and general approaches for image distortion correction but have drawbacks, including high computational cost, limited generalization, and increased memory requirements.

2.2. Learning-Based Camera Calibration Methods

With the development of deep learning, learning-based camera calibration methods have become popular. There are two mainstream approaches: one is incorporating image models into loss functions or network modules to estimate distortion parameters [34,35,36,37,38]. Rong et al. [37] used CNNs to train synthetic image data for radial distortion calibration, proving that distorted images from pinhole camera models can be used to train a CNN to estimate wide-angle lens distortion. The other is learning universal distortion calibration maps or training distortion calibration networks [36]. However, there are also some limitations. For example, image distortions make the problem difficult to represent and describe accurately, and they affect the accuracy of feature extraction and training. Liao et al. [39] addressed this issue by proposing an ordinal distortion estimation method, which improves the model’s ability to represent and learn distortion features by modeling distortion information. Due to the scarcity of distortion annotation data, synthetic images generated by algorithms are often used as substitutes, but these images can only simulate distortions and cannot reflect real distortion characteristics [40].

2.3. RGB-D Camera Calibration

The calibration of RGB-D cameras needs simultaneous consideration of both RGB lens calibration and depth information alignment. For binocular cameras, it is essential to calibrate and match the two camera lenses while accounting for the disparity. Yang et al. [41] proposed a universal calibration algorithm for two uncalibrated images. The Epipolar geometry constraints were utilized to obtain more accurate disparity information. Zhou et al. [42] used chessboards to calibrate under auxiliary infrared lighting. They calculated the intrinsic and extrinsic parameters of the depth camera by Zhang’s method, and directly calibrated the distortion of depth maps based on depth correction. This approach is more intuitive and faster than these distortion correction models. However, these methods are tailored to specific types of RGB-D cameras, lacking universal applicability. Some research has explored methods for unifying the calibration of various RGB-D cameras. Basso et al. [43] proposed a two-part error model that unified the error sources of RGB-D pairs. The model can be used for structured light 3D cameras and ToF cameras, which are widely used in robotic scenarios. Ramírez-Hernández et al. [44] used the least squares method to model errors by converting pixel points into angular information to find accurate 3D points. However, these methods have limited error correction capabilities for depth information and fail to accurately recover depth information.

3. Method

3.1. Equipment and Data

Two types of common RGB-D cameras and an optical motion capture system were employed for evaluating our calibration algorithm. The RGB-D cameras were designed and manufactured in different modes; one is the binocular camera with structured light, and the other is the ToF camera. Their common intrinsic camera parameters are presented in

Table 1. Common intrinsic parameters of the two cameras.

Parameter	Intel RealSense D455 (Cam 1)	Orbbec Femto Bolt (Cam 2)
$f_x/f_y$	387.3/388.7	1121.29/1120.45
$c_x/c_y$	321.2/243.7	950.894/547.579
Width/Height	640/480	1920/1080
$k_1/k_2/k_3$	0.0063/−0.0040/0.0028	0.0790/−0.0111/0.0480
Depth range	0.6–6 m	0.25–5.46 m

. The optical motion capture system is used to collect the ground truth of position in a 3D space.

Camera 1 (Cam 1): Intel RealSense D455 (Figure 1), employs binocular vision technology combined with active stereo vision to capture depth information. The paired images are obtained by infrared lenses located on the two sides of the camera, and the depth map is calculated corresponding to each pixel of the RGB images. Its depth measurement range spans from 0.6 m to 6 m. The RGB and depth output resolutions were set to 640 × 480, with a capture frame rate of 15 FPS.

Camera 2 (Cam 2): The Orbbec Femto Bolt (Figure 2) utilizes Time-of-Flight (ToF) sensing technology and integrates multiple sensing modules, including a multi-mode depth camera and an RGB video camera. The RGB and depth resolutions were set to 1920 × 1080, with a capture frame rate of 15 FPS.

Motion Capture System: The Optitrack motion capture system, can provide positional information of position points within the camera’s field of view, thereby providing the necessary 3D reference space for the calibration of depth cameras. The system employs an infrared light-sensitive camera group consisting of 10 cameras, each with a lens resolution of up to 4.2 million pixels. It offers precise measurements within a spatial area of 5 m × 3.2 m × 2.5 m, with a capture accuracy of 0.1 mm. The system is equipped with advanced software for real-time acquisition of positional data. Figure 3 shows the structure schematic diagram of our system.

Dataset: Each camera collected a dataset. The dataset consists of a training set with 27 position points and a testing set with 81 points (Figure 4). The dataset also includes the cameras’ extrinsic parameters, used to transform points from their original coordinates to the camera coordinate system. In the training set, the positions of points are distributed at various distances (far, medium, near), heights (top, middle, bottom), and locations (left, middle, right), basically covering the entire acquisition 3D space. In the testing set, points are randomly distributed. During the collection, a motion capture marker was attached to a camera stand. The height and location of the marker were changed by adjusting the camera stand. When collecting training set data, the marker was moved in a similar interval to cover the entire data collection space. For test set data, we randomly change the positions of the marker. When the position of the marker was determined by the motion capture system, the corresponding RGB image and depth map were captured by our collection software. These data can be disclosed via correspondence email.

In Figure 5, Rows 1 and 3 display RGB images of three instance points (near, medium, and far) captured by two cameras at the same height (low height) and the same position (middle position), respectively. Row 2 and Row 4 then present the corresponding depth maps for these RGB images. Cam 1, equipped with binocular infrared cameras, has a far measurement range but relatively low precision. It can capture markers and the camera stand, but objects at a greater distance, such as walls, appear blurred depth-wise, failing to exhibit abrupt changes in depth information at object edges. Cam 2, using a Time-of-Flight (ToF) sensor, has a limited measurement range, assigning a depth of 0 to areas beyond this range. However, it boasts high precision, with objects in the depth map differentiated by distinct boundaries.

3.2. EM Algorithm for RGB Camera Calibration

The EM algorithm, as an iterative method for estimating parameters, is primarily used to address the maximum likelihood estimation of parameters in probabilistic models with hidden variables. Ultimately, the iterative process converges both parameter sets to the optimal solution [45]. The EM algorithm consists of an E-step and an M-step. In our study, the Expectation step (E-step) transforms the intrinsic camera parameters, such as focal length and image sensor size, into hidden variables ${\theta }_x$ and ${\theta }_y$ and computes them. The Maximization step (M-step) re-estimates the distortion parameters K using the hidden variables calculated from the E-step.

To place the above procedure into a probabilistic EM framework, a likelihood model is induced by the camera imaging function. For a 3D point $P_c^i$ observed as a 2D pixel $P_p^i$, the imaging model provides the predicted pixel location.

$${\widehat{P}}_p^i=f(P_c^i;\theta ,K),$$

(1)

where $\theta =({\theta }_x,{\theta }_y)$ denotes the latent variables related to the intrinsic imaging properties, and $K=(k_1,k_2,k_3,k_4)$ denotes the distortion parameters.

Based on the reprojection consistency principle, the conditional likelihood of observing $P_p^i$ given $P_c^i$, $\theta$, and K is defined as a monotonic decreasing function of the reprojection error:

$$p(P_p^i\mid P_c^i,\theta ,K)\propto exp\left(-{∥P_p^i-f(P_c^i;\theta ,K)∥}^2\right).$$

(2)

Let $\mathcal{D}={\left\{(P_c^i,P_p^i)\right\}}_{i=1}^N$ denote the observed dataset. By augmenting the observed data with the latent variables $\theta$, the complete-data likelihood can be written as Equation (3).

$$p(\mathcal{D},\theta \mid K)=p\left(\theta \right)\prod _{i=1}^{N}p(P_p^i\mid P_c^i,\theta ,K),$$

(3)

In Equation (3), a non-informative prior is assumed for $\theta$. Under this formulation, maximizing the complete-data log-likelihood is equivalent to minimizing the overall reprojection error used in this paper.

The E-step consists of estimating the latent variables $\theta$ by maximizing the expected complete-data log-likelihood under the current distortion parameters $K^{\left(t\right)}$. With the likelihood defined by the reprojection error and a non-informative prior on $\theta$, this Expectation–Maximization reduces to a deterministic optimization problem. Specifically, the E-step update can be written as

$${\theta }^{(t+1)}=arg\underset{\theta }{min}\sum _{i=1}^N{∥P_p^i-f(P_c^i;\theta ,K^{\left(t\right)})∥}^2,$$

(4)

which is exactly the objective used for updating ${\theta }_x$ and ${\theta }_y$ in the following E-step derivation.

In our method, the principal point is fixed at (w/2, h/2) because, for engineering stability and parameter identifiability, joint optimization with other parameters offers little accuracy gain and may reduce numerical stability.

3.2.1. Expectation Step (E-Step)

The camera projection can be described as a linear perspective projection. For the 3D position points and 2D pixel points $i=1,2,3,\dots ,n$, the projection can be shown as Equation (5):

$$\left(\begin{array}{c}{x}_u\\ y_u\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{x_c}{z_c}}{f}_x+{\displaystyle \frac{w}{2}}\\ {\displaystyle \frac{y_c}{z_c}}{f}_y+{\displaystyle \frac{h}{2}}\end{array}\right)$$

(5)

Camera intrinsic parameters include lens focal lengths ${\mathrm{f}}_x$, ${\mathrm{f}}_y$, sensor size parameters ${\mathrm{s}}_x$, ${\mathrm{s}}_y$, image width and height $\mathrm{w}$, $\mathrm{h}$. However, when the camera intrinsic parameters are unknown, it is hard to employ Equation (5). To solve this issue and enable calculation between distortion parameters and intrinsic camera parameters, we further transformed ${\mathrm{f}}_x$, ${\mathrm{f}}_y$, ${\mathrm{s}}_x$, ${\mathrm{s}}_y$, $\mathrm{w}$, and $\mathrm{h}$ into a set of hidden variables ${\theta }_x$ and ${\theta }_y$. Thus, the transformation between ${\mathrm{P}}_{\mathrm{ui}}$ and ${\mathrm{P}}_{\mathrm{ci}}$ can be revised as Equation (6):

$$\left(\begin{array}{c}{x}_{ui}\\ y_{ui}\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{x_{ci}}{z_{ci}}}{\theta }_x+{\displaystyle \frac{w}{2}}\\ {\displaystyle \frac{y_{ci}}{z_{ci}}}{\theta }_y+{\displaystyle \frac{h}{2}}\end{array}\right)$$

(6)

For lens distortion, there exists a deviation between the ideal pixel point ${\mathrm{P}}_{\mathrm{ui}}$ and the actual pixel point ${\mathrm{P}}_{\mathrm{pi}}$. The transformation between ${\mathrm{P}}_{\mathrm{ui}}$ and ${\mathrm{P}}_{\mathrm{pi}}$ is described by Equation (7):

$$\left(\begin{array}{c}{x}_{pi}\\ y_{pi}\end{array}\right)=\left(\begin{array}{c}\left(k_1+k_3{r}_i^2\right)x_{ui}\\ \left(k_2+k_4{r}_i^2\right)y_{ui}\end{array}\right)$$

(7)

Here, ${\mathrm{k}}_1$, ${\mathrm{k}}_2$, ${\mathrm{k}}_3$, and ${\mathrm{k}}_4$ are the radial distortion parameters. ${\mathrm{r}}_i$ denotes the center radial distance, defined as the Euclidean distance from the pixel point ${\mathrm{P}}_{\mathrm{ui}}$ to the distortion center $(\mathrm{w}/2,\mathrm{h}/2)$. This model differs from the conventional one as it uses a low-order, pixel-coordinate parameterization to reduce parameters and improve optimization stability, but the standard normalized-plane model is also applicable and yields similar calibration accuracy in our scenario. Substituting Equation (6) into Equation (7). yields the transformation relationship between the 3D position points ${\mathrm{P}}_{\mathrm{ci}}$ and the real pixel points ${\mathrm{P}}_{\mathrm{pi}}$, as shown in Equation (8):

$$\left(\begin{array}{c}{x}_{pi}\\ y_{pi}\end{array}\right)=\left(\begin{array}{c}\left(k_1+k_3{r}_i^2\right)\left({\displaystyle \frac{x_{ci}}{z_{ci}}}{\theta }_x+{\displaystyle \frac{w}{2}}\right)\\ \left(k_2+k_4{r}_i^2\right)\left({\displaystyle \frac{y_{ci}}{z_{ci}}}{\theta }_y+{\displaystyle \frac{h}{2}}\right)\end{array}\right)$$

(8)

The parameters ${\theta }_x$ and ${\theta }_y$ can be calculated using n points, as shown in Equation (9):

$$\left(\begin{array}{c}{\theta }_x\\ {\theta }_y\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{1}{n}\sum _{i=1}^n\left(\frac{x_{pi}}{k_1+k_3{r}_i^2}-\frac{w}{2}\right)\frac{z_{ci}}{x_{ci}}}\\ {\displaystyle \frac{1}{n}\sum _{i=1}^n\left(\frac{y_{pi}}{k_2+k_4{r}_i^2}-\frac{h}{2}\right)\frac{z_{ci}}{y_{ci}}}\end{array}\right)$$

(9)

Since the radial distortion parameters are unknown, the initial values of the latent variables ${\theta }_x$ and ${\theta }_y$ can be obtained through random initialization or an approximate solution, as shown in Equation (10):

$$\left(\begin{array}{c}{\theta }_x\\ {\theta }_y\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{1}{n}\sum _{i=1}^n\left(x_{pi}-\frac{w}{2}\right)\frac{z_{ci}}{x_{ci}}}\\ {\displaystyle \frac{1}{n}\sum _{i=1}^n\left(y_{pi}-\frac{h}{2}\right)\frac{z_{ci}}{y_{ci}}}\end{array}\right)$$

(10)

The complete process of the E-step algorithm is presented in

Table 2. Algorithm for the expectation step in intrinsic parameter estimation.

Algorithm: Expectation Step for Intrinsic Parameter Estimation
Input: 3D points ${\mathbf{P}}_{ci}=(x_{ci},y_{ci},z_{ci})$, 2D pixel points ${\mathbf{P}}_{pi}=(x_{pi},y_{pi})$, distortion parameters $k_1,k_2,k_3,k_4$, image width w and height h.
Output: Intrinsic parameters $\theta =({\theta }_x,{\theta }_y)$.
Step 1: Initialization. The intrinsic parameters are initialized as $$\left(\begin{array}{c}{\theta }_x\\ {\theta }_y\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{1}{n}\sum _{i=1}^n\left(x_{pi}-\frac{w}{2}\right)\frac{z_{ci}}{x_{ci}}}\\ {\displaystyle \frac{1}{n}\sum _{i=1}^n\left(y_{pi}-\frac{h}{2}\right)\frac{z_{ci}}{y_{ci}}}\end{array}\right).$$ Step 2: Radial distance computation. For each correspondence $i=1,\dots ,n$, the radial distance is computed as $$r_i=\sqrt{{\left(x_{pi}-{\displaystyle \frac{w}{2}}\right)}^2+{\left(y_{pi}-{\displaystyle \frac{h}{2}}\right)}^2}.$$ Step 3: Parameter update. The intrinsic parameters are updated according to $$\left(\begin{array}{c}{\theta }_x\\ {\theta }_y\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{1}{n}\sum _{i=1}^n\left(\frac{x_{pi}}{k_1+k_3{r}_i^2}-\frac{w}{2}\right)\frac{z_{ci}}{x_{ci}}}\\ {\displaystyle \frac{1}{n}\sum _{i=1}^n\left(\frac{y_{pi}}{k_2+k_4{r}_i^2}-\frac{h}{2}\right)\frac{z_{ci}}{y_{ci}}}\end{array}\right).$$ Return: ${\theta }_x,{\theta }_y$.

.

3.2.2. Maximization Step (M-Step)

Based on the calculation of the E-step, we can obtain the intrinsic latent variable parameters. Then, the distortion parameters are iteratively updated in the M-step. The radial distortion parameters include ${\mathrm{k}}_x({\mathrm{k}}_1,{\mathrm{k}}_3)$ and ${\mathrm{k}}_y({\mathrm{k}}_2,{\mathrm{k}}_4)$, which quantify the degree of distortion along the $\mathrm{x}$-axis and $\mathrm{y}$-axis directions of the image, respectively. To improve computational efficiency, the M-step introduces matrices ${\mathrm{A}}_x$, ${\mathrm{A}}_y$ and vectors ${\mathrm{p}}_x$, ${\mathrm{p}}_y$ to estimate the parameters, as shown in Equation (11):

$${\mathbf{A}}_x=\left[\begin{array}{cc}{x}_{c1}& r_1\\ x_{c2}& r_2\\ ⋮& ⋮\\ x_{cn}& r_n\end{array}\right],{\mathbf{A}}_y=\left[\begin{array}{cc}{y}_{c1}& r_1\\ y_{c2}& r_2\\ ⋮& ⋮\\ y_{cn}& r_n\end{array}\right],{\mathbf{p}}_x=\left[\begin{array}{c}{x}_{p1}\\ x_{p2}\\ ⋮\\ x_{pn}\end{array}\right],{\mathbf{p}}_y=\left[\begin{array}{c}{y}_{p1}\\ y_{p2}\\ ⋮\\ y_{pn}\end{array}\right]$$

(11)

The matrices ${\mathrm{A}}_x$ and ${\mathrm{A}}_y$ are constructed based on the radial distances ${\mathrm{r}}_i$ and the position points ${\mathrm{P}}_{\mathrm{ci}}$, while ${\mathrm{p}}_x$ and ${\mathrm{p}}_y$ represent the vectors composed of the $\mathrm{x}$ and $\mathrm{y}$ coordinates from n pixel points. The distortion parameters ${\mathrm{k}}_x$ and ${\mathrm{k}}_y$ are calculated using ${\theta }_x$, ${\theta }_y$, ${\mathrm{A}}_x^{-1}$, ${\mathrm{A}}_y^{-1}$, ${\mathrm{p}}_x$, ${\mathrm{p}}_y$, $\mathrm{w}$, and $\mathrm{h}$, as shown in Equation (12):

$$\left(\begin{array}{c}{\mathbf{k}}_x\\ {\mathbf{k}}_y\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{{\mathbf{A}}_x^{-1}\left({\mathbf{p}}_x-{\displaystyle \frac{w}{2}}\right)}{{\theta }_{x}w}}\\ {\displaystyle \frac{{\mathbf{A}}_y^{-1}\left({\mathbf{p}}_y-{\displaystyle \frac{h}{2}}\right)}{{\theta }_{y}h}}\end{array}\right)$$

(12)

The matrices ${\mathrm{A}}_x^{-1}$ and ${\mathrm{A}}_y^{-1}$ denote the pseudo-inverses of ${\mathrm{A}}_x$ and ${\mathrm{A}}_y$, whose dimensions are $2\times n$, while the vectors ${\mathrm{p}}_x$ and ${\mathrm{p}}_y$ are n-dimensional. The complete process of the Maximization step algorithm is presented in

Table 3. Algorithm for the maximization step in distortion parameter estimation.

Algorithm: Maximization Step for Distortion Parameter Estimation
Input: 3D points ${\mathbf{P}}_{ci}=(x_{ci},y_{ci},z_{ci})$, 2D pixel points ${\mathbf{P}}_{pi}=(x_{pi},y_{pi})$, intrinsic parameters $\theta =({\theta }_x,{\theta }_y)$, image width w and height h.
Output: Distortion parameters ${\mathbf{k}}_x$ and ${\mathbf{k}}_y$.
Step 1: Radial distance computation. For each correspondence $i=1,\dots ,n$, the radial distance is computed as $$r_i=\sqrt{{\left(x_{pi}-{\displaystyle \frac{w}{2}}\right)}^2+{\left(y_{pi}-{\displaystyle \frac{h}{2}}\right)}^2}.$$ Step 2: Matrix construction. The matrices ${\mathbf{A}}_x$ and ${\mathbf{A}}_y$ are constructed as $${\mathbf{A}}_x=\left[\begin{array}{cc}{x}_{c1}& r_1\\ x_{c2}& r_2\\ ⋮& ⋮\\ x_{cn}& r_n\end{array}\right],{\mathbf{A}}_y=\left[\begin{array}{cc}{y}_{c1}& r_1\\ y_{c2}& r_2\\ ⋮& ⋮\\ y_{cn}& r_n\end{array}\right].$$ Step 3: Distortion parameter estimation. The distortion parameters are estimated as $$\left(\begin{array}{c}{\mathbf{k}}_x\\ {\mathbf{k}}_y\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{{\mathbf{A}}_x^{-1}\left({\mathbf{p}}_x-\frac{w}{2}\right)}{{\theta }_{x}w}}\\ {\displaystyle \frac{{\mathbf{A}}_y^{-1}\left({\mathbf{p}}_y-\frac{h}{2}\right)}{{\theta }_{y}h}}\end{array}\right).$$ Return: ${\mathbf{k}}_x,{\mathbf{k}}_y$.

.

The computational complexity of the proposed EM procedure is linear in the number of observations. In the Expectation step, the dominant cost arises from iteratively computing radial distances and updating distortion parameters for all points, resulting in a time complexity of $\mathcal{O}(T·n)$ with a space complexity of $\mathcal{O}\left(n\right)$. The Maximization step constructs a small linear system from the same set of observations, leading to a time complexity of $\mathcal{O}\left(n\right)$ and a space requirement of $\mathcal{O}\left(n\right)$. Consequently, the overall EM calibration process exhibits efficient linear scalability while maintaining modest memory usage.

3.3. Depth Correction

The depth of the position point captured by RGB-D cameras is the straight-line distance between the depth sensor and these position points. The relationship can be represented by Euclidean distance, as shown in Equation (13):

$$d=\sqrt{{\left(x-x_0\right)}^2+{\left(y-y_0\right)}^2+{\left(z-z_0\right)}^2}$$

(13)

where $(\mathrm{x},\mathrm{y},\mathrm{z})$ represents the position point in the motion capture system, and $({\mathrm{x}}_0,{\mathrm{y}}_0,{\mathrm{z}}_0)$ represents the position of the camera, which is recorded by the motion capture system as the extrinsic parameters. The variable d denotes the straight-line distance (depth) from the camera to the position point.

The coordinates $(\mathrm{x},\mathrm{y})$ can be calibrated by utilizing the parameters obtained from the EM algorithm, as shown in Equation (14):

$$\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}\left({\displaystyle \frac{x_{ci}}{k_1+k_3{r}^2}}-{\displaystyle \frac{w}{2}}\right){\displaystyle \frac{z_{ci}}{{\theta }_x}}\\ \left({\displaystyle \frac{y_{ci}}{k_2+k_4{r}^2}}-{\displaystyle \frac{h}{2}}\right){\displaystyle \frac{z_{ci}}{{\theta }_y}}\end{array}\right)$$

(14)

The remaining value $\mathrm{z}$ in Equation (13) can be calculated using Equation (15). Since the camera is placed in the forward direction, the depth value $\mathrm{z}$ within the camera’s field of view should be smaller than ${\mathrm{z}}_0$. Therefore, the negative sign of the square root should be selected.

$$z=-\sqrt{d^2-{\left(x-x_0\right)}^2-{\left(y-y_0\right)}^2}+z_0$$

(15)

After transforming the straight-line distances into depth coordinates, we further employ the least-squares method to calibrate the depth values in the motion capture system, as shown in Equation (16):

$$\left(\begin{array}{c}X\\ Y\\ Z\end{array}\right)=\left(\begin{array}{c}{w}_{x}x+b_x\\ w_{y}y+b_y\\ w_{z}z+b_z\end{array}\right)$$

(16)

where $(\mathrm{x},\mathrm{y},\mathrm{z})$ represent the back-projected positions of the points obtained from Equations (14) and (15), while $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ denote the original positions in the motion capture system. The parameters ${\mathrm{w}}_x$, ${\mathrm{w}}_y$, ${\mathrm{w}}_z$, ${\mathrm{b}}_x$, ${\mathrm{b}}_y$, and ${\mathrm{b}}_z$ represent the weight parameters.

Figure 6 shows our method’s overall calibration workflow. The calibration process can be divided into two parts. The first part involves applying the EM algorithm to calibrate the 2D-pixel points. The second section focuses on transforming and calibrating depth information.

4. Results and Discussion

The experiment was conducted in two parts: (1) The pixel calibration in RGB images—the EM algorithm was compared with other similar methods to assess its feasibility and accuracy. In addition, we studied the impact of different initialization methods on the results of the EM algorithm. (2) The depth value calibration: The depth values were transformed and calibrated corresponding to the pixel points.

4.1. Evaluation Criteria

The residue between the calibrated pixel points and the real pixel points is used as a metric for evaluating calibration accuracy. Two equations are employed to evaluate the performance. For qualitative analysis, we calculate the relative pixel points $P_{\mathrm{relative}}$ by subtracting the original pixel points from the calibrated ones, as shown in Equation (17). For quantitative analysis, the Root Mean Square Error (RMSE) is adopted to compute the residue, as shown in Equation (18).

$${\mathbf{p}}_{\mathrm{relative}}={\mathbf{p}}_{\mathrm{inference}}-{\mathbf{p}}_{\mathrm{real}}$$

(17)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{m}}\sum _{i=1}^m{∥{\mathbf{p}}_{\mathrm{inference},i}-{\mathbf{p}}_{\mathrm{real},i}∥}^2}$$

(18)

Here, ${\mathbf{P}}_{\mathrm{real}}$ represents the original pixel points, and ${\mathbf{P}}_{\mathrm{inference}}$ represents the calibrated pixel points.

For depth error measurement, the average absolute error is used to evaluate the calibration accuracy, as shown in Equation (19):

$${\mathrm{Error}}_{\mathrm{depth}}=\left|D_{\mathrm{gt}}-D\right|$$

(19)

Here, ${\mathrm{D}}_{\mathrm{gt}}$ denotes the ground-truth depth, and $\mathrm{D}$ represents the estimated depth.

4.2. Pixel Point Calibration for RGB Image

To validate the effectiveness of the Expectation–Maximization (EM) algorithm in the two cameras, we compared it with Direct Linear Transformation (DLT) [46], Linear Least Squares (LS) [47], Levenberg–Marquardt algorithm (LM) [48], Trust Region Reflective algorithm (TRF) [49], and our Expectation–Maximization (EM) algorithm. Among these methods, the Levenberg–Marquardt (LM) algorithm is commonly used in bundle adjustment (BA) [50] and related optimization-based calibration approaches. In our experiments, the triangulation step in BA is omitted, and the optimization is performed directly using the 3D marker positions provided by the motion capture system. The dataset is divided into 27 training points and 81 testing points, yielding a roughly 1:3 ratio. This setting is adopted to more clearly distinguish the performance differences among various methods under a limited number of markers. In practical engineering applications, a larger number of markers is typically used.

To ensure fair comparison, all methods were evaluated under identical experimental settings. For the EM algorithm, the iteration limit was set to 10 with a convergence tolerance of $1\times {10}^{-6}$, using standard pinhole estimates for intrinsic initialization and zero initialization for distortion parameters. The optimization-based baselines (LS, LM, TRF) adopted the same initialization, each with a maximum of 20 iterations. All experiments used the same data split, identical image resolution and preprocessing, and independent evaluation on the two cameras. Runtime was measured under single-thread CPU execution for consistency. The results are shown in Figure 7. In the dataset, All the pixel points were displayed in plots.

In Figure 7, from top to bottom, the methods are DLT (blue), LS (pink), LM (orange), TRF (purple), and our EM algorithm (red). In these methods, LS, LM, TRF, and our EM algorithm belong to iterative algorithms. In the training set (row 1 and row 2), results in DLT, LS, and EM are similar, and the calibrated pixel points are close to real pixel points. LM and TRF methods are relatively inferior, with significant residues between the calibrated points and the actual points. In the test set (row 3 and row 4), DLT and EM continue to maintain a small residue. The performance of LS begins to decline, and the residues of LM and TRF further increase.

In Figure 8, it can be seen that the scatter points closer to the center $(0,0)$ indicate smaller errors. The distance between each adjacent concentric circle is 1 pixel. By comparison, pixels of our EM algorithm (red) were observed to be closer to the center compared to the other methods. Their errors in the x-axis direction are smaller compared to those in the y-axis direction. The results obtained by the DLT and LS methods are similar, but both are inferior to our method. Their pixel points surround those obtained by our method, with an error of approximately one pixel. The pixel points of LM and TRF are relatively poor, with some points falling outside the range of our comparison. We counted the number of points with errors < 2 pixels, 4 pixels, and 6 pixels for different methods, as shown in

Table 4. Pixel error statistics for Cam 1 and Cam 2 under different methods.

Method	Cam 1 (Pixel)				Cam 2 (Pixel)
	<2	<4	<6	>6	<2	<4	<6	>6
DLT	12	26	27	0	3	15	22	5
LS	8	23	26	1	1	13	20	7
LM	4	21	24	3	0	12	16	11
TRF	9	22	26	1	1	13	17	10
EM	24	27	27	0	6	22	26	1

.

The error reported in Table 4 is the pixel reprojection error, defined as the Euclidean distance between the corrected pixel locations and the ground-truth image observations, as formulated in Equation (18). It should be noted that RMSE is a pixel-domain metric and is sensitive to image resolution. Therefore, the absolute RMSE values are not strictly comparable across cameras with different resolutions. In Cam 1, all 27 pixel points by EM exhibit errors < 4 pixels, outperforming DLT (26), LS (23), LM (21), TRF (22). In Cam 2, due to its higher resolution, its average error is also higher compared to Cam 1. Out of the 27 points, 26 points have errors < 6 pixels, better than DLT (22), LS (20), LM (16), TRF (17). Under identical experimental conditions, the error of the points obtained through our method is significantly smaller than that of other methods. Moreover, in our experiments, linear methods such as LS outperform nonlinear ones because, although the RGB images were uncorrected, factory calibration of the RGB-D cameras ensures that the images have minimal distortion, making the imaging model approximately linear and allowing LS to perform well under these conditions.

Table 5. Comparison of RMSE and speed among various methods on data from two cameras.

Method	RMSE (Pixel)		Speed (s)
	Cam 1	Cam 2	Cam 1	Cam 2
DLT	1.26	4.02	0.001	0.001
LS	1.47	4.73	1.494	0.083
LM	1.70	5.02	1.802	1.871
TRF	1.50	4.72	1.011	0.370
EM (Ours)	0.89	3.22	0.007	0.007

presents the overall error of each method on two cameras calculated by Equation (18). The RMSE of the EM algorithm on the datasets from Cam 1 and Cam 2 are 1.26 pixels and 3.22 pixels, respectively. Within the same camera, these errors are significantly better than those of other methods. The results for LS are 1.47 (in Cam 1) and 4.73 (in Cam 2), and for DLT are 1.26 (in Cam 1) and 4.02 (in Cam 2). Theoretically, since DLT does not take distortion information into account, its accuracy should be inferior to other methods. However, the results indicate that its performance is comparable to LS, and even surpasses LS in Cam 1. The RMSE of LM and TRF algorithms exhibit substantial discrepancies compared to our method, with differences exceeding an amplitude of 10 pixels. The runtime of the EM algorithm is 0.007 s in 10 iterations, outperforming the LM (1.802 s, 1.871 s), LS (1.494 s, 0.083 s), and TRF (1.011 s, 0.370 s) methods. Our method’s running speed is the fastest among iterative methods (LS, LM, and TRF), second only to the non-iterative DLT algorithm (0.001 s, 0.001 s). In general, the experimental results indicate that the EM algorithm exhibits the best overall performance among the compared methods for both cameras.

4.3. Initialization of Latent Variable Parameters

The EM algorithm is an iterative method, and highly sensitive to initial parameters. We compared two initialization methods to validate the effectiveness of these initialization methods for the EM algorithm: (1) Random initialization. This method utilized 10 random numbers within the range from −10 to 10. (2) Initialization based on prior information. This method used data in the training set to calculate parameter ${\theta }_x$, ${\theta }_y$ by Equation (10).

Figure 9 shows the process of two iteration methods from training data in Cam 1 (a) and Cam 2 (b), respectively. In method 1 (blue), 10 random numbers ranging from −10 to 10 ultimately converged to −0.032 (in Cam 1) and −0.064 (in Cam 2) after 20 epochs. In method 2 (red), the initial values converged to −0.031 (in Cam 1) and −0.062 (in Cam 2). The experimental data indicates that regardless of the initial value setting, both methods ultimately converge to highly similar results, with an error within 0.1%. However, under the same epochs of iterations, the initialization method based on Method 2 converges faster.

4.4. Depth Calibration

The straight-line distance depth value of the RGB camera was converted by Equation (15). Compared to the original depth, the calibrated depth values are closer to the ground truth obtained from the motion capture system. The calibration results are illustrated in Figure 10 and Figure 11.

Figure 10 exhibits the original depth (green), calibrated depth (blue), and ground truth depth of all the 27 position points in the training set for Cam 1 and Cam 2. Rows 1, 2, and 3 respectively display the depth calibration results for points at close, medium, and long distances. In the near distance (ranging from 1300 to 1800 mm in Cam 1), our methods are accurate in both Cam 1 and 2. All the calibrated points are close to the ground truth with errors less than 8 mm. In the medium distance (ranging from 2000 mm to 2700 mm in Cam 1 and 3000 mm to 3700 mm in Cam 2), the results are also accurate with errors less than 14 to 20 mm. For far distances (ranging from 3300 mm to 3800 mm in Cam 1 and 4500 mm to 5000 mm in Cam 2), some points are over-calibrated, but most of them are effectively calibrated. The results in the training set indicate that the depths of most points are closer to the ground truth values after calibration. For the test set, we represented the results of all 81 points using a box plot to analyze its scale and distribution, as shown in Figure 11.

In Figure 11, before calibration, the average depth error for Cam 1 ranges from −9.8 mm to 80.3 mm (differing by 90.1 mm). After calibration, the error ranges from −22.5 to 34.3 (differing by 56.8 mm). For Cam 2, the depth error before calibration ranges from −1.5 mm to 108.3 mm (differing by 109.8 mm). After calibration, the error ranges from −22.5 to 51.3 (differing by 73.8 mm). For Cam 1, the average absolute error was dropped from 27.0 mm to 11.0 mm. For Cam 2, it dropped from 41.0 mm to 21.7 mm.

Our method uses least squares to model the linear relationship between calibrated and original depths of the position points. The resulting parameters are applied to transform the original depth maps accordingly. Depth information is visualized using a rainbow colormap. Pixel colors transition from red to blue, representing depths from minimum to maximum. The result is shown in Figure 12.

Figure 12 shows the calibration results of Cam 1 and Cam 2. (a) presents the original RGB images of both cameras, (b) displays the original depth maps as well as their overlays with RGB channels, and (c) shows the depth maps after depth calibration, also overlaid with their RGB channels. The results indicate that the depth accuracy of both cameras has been improved through depth-to-distance conversion and least squares regression, with Cam 1 experiencing a more significant enhancement.

Figure 13 shows a comparison of local details in the depth maps of Cam 1 before and after calibration. We selected a section of the depth maps and zoomed in for comparison. In the original depth map, we observed that there was no clear edge between the background wall and the object (an air conditioner visible in the overlay image). However, after calibration, a more distinct depth discontinuity at the edge became apparent. Furthermore, in the original depth, the wall appeared yellow in Cam 1’s image but green in Cam 2’s. After calibration, the results from both cameras became more consistent (the depth of the wall in Cam 1’s image changed to green). These observations confirm that our calibrated depth data reflect spatial depth better.

4.5. Discussion

In this study, the proposed EM-based calibration method is compared with DLT, LS, LM, and TRF. The results show that the proposed method converges more stably when high-precision three-dimensional ground-truth marker points are available. It also achieves lower RMSE and depth error than the compared methods. These results indicate that incorporating camera parameters and three-dimensional point information from the motion capture system into a unified probabilistic model is effective. Optimization within the EM framework enables reliable parameter estimation in multi-type camera environments.

In addition, although some methods are not included in the experimental comparison, they are worth discussion, such as three-dimensional measurement methods based on simple calibration targets and calibration approaches based on structured light [16,17,22]. These methods are generally easy to operate, technically mature, and specifically tailored to particular types of cameras. In contrast, the proposed method is more suitable for experimental environments that inherently employ optical motion capture systems, such as three-dimensional human pose estimation or multi-view motion analysis. In such scenarios, the motion capture system can directly provide high-precision three-dimensional marker data, allowing camera calibration to be performed without additional calibration objects. Moreover, the present work is inspired by several studies that also leverage motion capture systems, in which the motion capture framework contributes to improved calibration or measurement accuracy [51,52].

5. Conclusions

The research on the application of RGB-D cameras is significant for advancing computer vision technology. It involves the calibration issues of RGB cameras and the depth information. In response to this issue, we have proposed a calibration method suitable for multiple common RGB-D camera configurations. By introducing the Expectation–Maximization algorithm and utilizing a motion capture system, spatial depth is obtained. During the calibration process, the EM method employs a learning model to swiftly determine the camera’s intrinsic parameters. The depth correction can significantly improve depth measurements at an average of 16.0 to 19.3 mm, enabling a rapid conversion to Euclidean distance measurements for depth. However, the adopted linear depth fitting model is mainly suitable for motion capture–based scenarios with controlled conditions, while outdoor or unconstrained environments may introduce additional interference. Despite this, we have proposed a rapid and efficient RGB-D depth camera calibration method that performs well on currently mainstream commercial depth cameras. Next, future work will focus on extending the applicability of the proposed method to broader scenarios and improving the generalization of depth calibration.