A Novel and Simplified Extrinsic Calibration of 2D Laser Rangefinder and Depth Camera

Abstract

It is too difficult to directly obtain the correspondence features between the two-dimensional (2D) laser-range-finder (LRF) scan point and camera depth point cloud, which leads to a cumbersome calibration process and low calibration accuracy. To address the problem, we propose a calibration method to construct point-line constraint relations between 2D LRF and depth camera observational features by using a specific calibration board. Through the observation of two different poses, we construct the hyperstatic equations group based on point-line constraints and solve the coordinate transformation parameters of 2D LRF and depth camera by the least square (LSQ) method. According to the calibration error and threshold, the number of observation and the observation pose are adjusted adaptively. After experimental verification and comparison with existing methods, the method proposed in this paper easily and efficiently solves the problem of the joint calibration of the 2D LRF and depth camera, and well meets the application requirements of multi-sensor fusion for mobile robots.

1. Introduction

With the rapid development of sensor technology and computer vision technology, laser-range-finder (LRF) and cameras have become indispensable sensors for autonomous driving, mobile robots and other fields [1]. Two-dimensional (2D) LRF is commonly used to measure depth information in a single plane due to its high precision, light weight and low power consumption. The camera acquires rich information, such as color and texture, but it is sensitive to light and weather, resulting in its poor stability. On the other hand, it is difficult for the camera to measure depth directly over long distances. Therefore, laser vision fusion plays an important role in robot self-localization [2,3], environmental perception [4], target tracking [5], and path planning [6].

To integrate data information from 2D LRF and depth cameras, the relative positional relationship between the two sensors needs to be precisely known [7]. This is a classical extrinsic calibration problem, where the objective is to determine the conversion relationship between two coordinate systems. In contrast to 3D LRF, which identifies different features, 2D LRF only measures depth information in a single plane, and it is difficult for the camera to see the plane scanned by 2D LRF, which makes extrinsic calibration for 2D LRF and cameras more challenging. Therefore, additional constraints must be used to find the correspondence between the 2D LRF and the camera.

There has been a large amount of research work on the extrinsic calibration of 2D LRF and cameras, which is divided into two categories: target-based calibration and non-target calibration. References [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24] are target-based calibration. Zhang and Pless [8] proposed a method by using point constraints on a plane, but only two degrees of freedom are constrained in a single observation, which required a large number of different observations to ensure accuracy. Vasconcelos et al. [9] proposed to solve the problem in [8] by forming a perspective three-point (P3P) problem. Zhou [10] further proposed an algebraic method for extrinsic calibration. Both methods in [9,10] required multiple observations and suffered from multi-solution problems. Kaiser et al. [11] proposed a calibration algorithm such that the problem of rigid displacement estimation between two sensors was reduced to the registration of plane and line. Li [12], Kwak et al. [13] used isosceles triangles and foliate panels as calibration targets, and used point-line constraint to calibrate the camera and LRF. Dong et al. [14] proposed a special V-shaped calibration target method with the checkerboard, which was used for both camera and LRF calibration by a single observation, but it needed a cumbersome solution process. Itami et al. [15,16] proposed an improved method for the checkerboard calibration of the camera and LRF, which directly obtained the point-to-point correspondence between the LRF and camera. Huang et al. [23] proposed a method to calibrate the 2D LRF and camera using a one-side transparent hollow calibration board, but it required the scanning surface of the LRF to form a certain angle with the hollow calibration board. Tu et al. [24] proposed an accuracy criterion for directional synthesis to eliminate the large error data during observation, but their laser and visual observation points were less constrained and still required multiple observations. Although there are many extrinsic parameters calibration methods for the 2D LRF and camera, the problems of a complex calibration process, high requirements for calibration board production, many calibration times and a limited calibration environment still exist.

References [25,26,27,28,29,30,31] are the calibration without target, which are further divided into feature-based extrinsic calibration and motion-based extrinsic calibration. Levison et al. [25] proposed a self-calibration method based on edge feature matching. Zhao et al. [26] proposed an extrinsic calibration framework based on motion LRF and visible light camera. Yang et al. [27] built on previous methods to achieve matching of images and 3D LRF point clouds by methods such as keyframing and motion recovery structures. However, since different sensors acquired data on different principles, the transformation of each sensor was determined based on the sensor that acquired data at the lowest frequency. Schneider et al. [28] proposed to apply deep learning to LRF and the extrinsic calibration of the visible light camera, constructing loss functions by photometric loss and point cloud distance loss, and using unsupervised learning methods for training. However, such methods rely on image feature points and radar point cloud data that are often difficult to obtain in natural scenes and have harsh usage conditions. In order to estimate the LiDAR to stereo camera extrinsic parameters for driving platforms, applying 3D mesh reconstruction-based point cloud registration, a photometric error function was built [31]. In addition to directly obtaining the extrinsic parameter of calibration, CFNet proposed by Wang [32] was utilized to predict the calibration flow based on convolutional neural networks. Recently, other authors proposed to optimize the external parameter calibration with additional sensors [33]. The targetless extrinsic parameter calibration has a common feature, where lidar collects 3D point cloud data with rich feature information. So they are too difficult to be applied to the calibration of 2D LRF.

In order to solve the calibration problems mentioned above, in this paper, we propose a method to constrain the correspondence between the depth camera and 2D LRF within a special calibration plate. The point-line features constraint of the 2D LRF and the depth camera on the calibration plate is used to realized the joint calibration of the 2D LRF and depth camera. For the extrinsic calibration of the 2D laser rangefinder and depth camera, the main contributions of the article are as follows:

• Provide a novel specific calibration board which is simple to manufacture for the 2D LRF and camera calibration to construct three observation feature point-line constraints between two sensors.
• Through the method in this paper, the joint calibration of 2D LRF and depth camera is completed by only two observations with an oversimplified operation.
• By setting the calibration threshold, the joint calibration of the 2D LRF and depth camera placed on a movable device adjusts the number of observations autonomously.

The layout of the article is as follows: Section 2 describes the calibration principles of the 2D LRF and depth cameras. Section 3 describes the calibration methods and algorithms for the 2D LRF and depth cameras. Section 4 verifies the proposed method by experiments, and we draw conclusions in Section 5.

2. The Calibration Basis of 2D LRF and Depth Camera

In the process of mobile robot localization and mapping, it is often necessary to unify the environmental perception data of each sensor to the world coordinate system called base frame to describe. The joint calibration of 2D LRF and depth camera is to determine the coordinate transformation relationship between the coordinate system of the depth camera, 2D LRF and the world. As shown in Figure 1, the calibration involves four coordinate systems, which are world coordinate system ${\mathit{O}}_{\mathit{w}}-{\mathit{x}}_{\mathit{w}}{\mathit{y}}_{\mathit{w}}{\mathit{z}}_{\mathit{w}}$, LRF coordinate system ${\mathit{O}}_{\mathit{l}}-{\mathit{x}}_{\mathit{l}}{\mathit{y}}_{\mathit{l}}{\mathit{z}}_l$ called laser frame, depth camera coordinate system ${\mathit{O}}_c-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$ called camera frame, and pixel coordinate system $\mathit{O}-\mathit{u}\mathit{v}$. The pixel coordinate system is the reference coordinate system of the camera observation data, which usually needs to be transferred to the camera coordinate system. With the determined relative position among ${\mathit{O}}_w-{\mathit{x}}_w{\mathit{y}}_w{\mathit{z}}_w$, ${\mathit{O}}_l-{\mathit{x}}_l{\mathit{y}}_l{\mathit{z}}_l$ and ${\mathit{O}}_c-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$, the observation point coordinates are able to be transformed among the coordinate systems. Depending on the application scenarios and needs, the observation data from the depth camera can be expressed under the LRF coordinate system and then the data can be projected to describe in the world coordinate system. The scanned data of the LRF can be also expressed under the camera coordinate system, and then the data are transformed from the camera coordinate system to the world coordinate system.

The 2D LRF is often mounted horizontally in mobile robot applications. However, due to installation error and uncertainty, the mounting will have some deflection angles. The camera also has some angles relative to the world coordinate system. According to the rigid body coordinate transformation relationship, the position data collected by the 2D LRF and depth camera in their respective coordinate systems correspond to the position of the observation point in the world coordinate system, as shown in Equation (1).

$$\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\end{array}\right]={\mathbf{R}}_i^w\left[\begin{array}{c}{x}_i\\ y_i\\ z_i\end{array}\right]+{\mathbf{T}}_i^w,$$

(1)

In the formula, $x_i$ is the coordinate value in i coordinate system. $i=l,c$, l is the radar coordinate system, c is the camera coordinate system. ${\mathbf{R}}_i^w$ is the rotation matrix from the coordinate system to the world coordinate system, and ${\mathbf{T}}_i^w$ is the translation matrix of the i coordinate system with respect to the world coordinate system. The camera used in this paper is the ZED depth camera, and its imaging principle is shown in Figure 2. The figure contains two coordinate systems: the camera coordinate system ${\mathit{O}}_c-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$, and the pixel coordinate system $\mathit{O}-\mathit{u}\mathit{v}$. The two cameras of the depth camera are located in the same plane, the optical axes of the left and right cameras are parallel, and the focal length parameters f are the same. The coordinates of the observation point in the camera coordinate system are assumed to be $\mathbf{P}(x_c,y_c,z_c)$.

According to the camera’s small-aperture imaging principle and triangle similarity, we have

$$\begin{array}{c}\frac{z_c}{f}=\frac{x_c}{u_l}\\ \frac{z_c}{f}=\frac{x_c-b}{u_r}\\ \frac{z_c}{f}=\frac{y_c}{v_l}=\frac{y_c}{v_r}\end{array},$$

(2)

where $x_c,y_c,z_c$ are the coordinates of ${\mathit{O}}_c-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$; $u_l$ and $v_l$ are the coordinates in the left camera pixel coordinate system; $u_r$ and $v_r$ are the coordinates in the right camera pixel coordinate system; f is the focal length of the camera; and b isthe distance which is called the base line between the binocular cameras. Using the camera coordinate system of the left camera as the camera coordinate system of the depth camera, the relationship between the pixel coordinates and the camera coordinates of the depth camera is obtained as

$$\begin{array}{c}{x}_c=\frac{u_l\ast z_c}{f}\\ y_c=\frac{v_l\ast z_c}{f}\\ z_c=\frac{f\ast b}{u_l-u_r}\end{array},$$

(3)

By Equation (3),

$$\left[\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right]=\left[\begin{array}{ccc}\frac{b}{d}& 0& 0\\ 0& \frac{b}{d}& 0\\ 0& 0& \frac{b\ast f}{d}\end{array}\right]\left[\begin{array}{c}{u}_l\\ v_l\\ 1\end{array}\right],$$

(4)

where $d=u_l-u_r$, d is called the parallax of the two cameras. According to the depth data acquisition principle of the depth camera, the point cloud data of the observed object is gathered. For the 2D LRF, it directly obtains the distance and angle information of the obstacle. In practical application, the points of the actual object scanned by the LRF have a unique point corresponding to them in the depth camera. The polar coordinate data of the LRF are converted into Cartesian coordinate data, then the Cartesian coordinates are converted to be expressed under the coordinate system of ${\mathit{O}}_c-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$ by Equation (5).

$$\left[\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right]={\mathbf{R}}_l^c\left[\begin{array}{c}{x}_l\\ y_l\\ z_l\end{array}\right]+{\mathbf{T}}_l^c,$$

(5)

where ${\mathbf{R}}_l^c=\left[\begin{array}{ccc}{r}_{11}& {\mathrm{r}}_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]$,${\mathbf{T}}_l^c=\left[\begin{array}{c}{t}_1\\ t_2\\ t_3\end{array}\right]$. ${\mathbf{R}}_l^c$ denotes the 3 × 3 rotation matrix from ${\mathit{O}}_l-{\mathit{x}}_l{\mathit{y}}_l{\mathit{z}}_l$ to ${\mathit{O}}_c-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$, and ${\mathbf{T}}_l^c$ denotes the translation matrix from ${\mathit{O}}_l-{\mathit{x}}_l{\mathit{y}}_l{\mathit{z}}_l$ to ${\mathit{O}}_c-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$.

Because the geometric relationships do not vary with the coordinate system, the coordinate system transformation does not affect the geometric constraint relationships. The data points acquired by the 2D LRF are transformed to be expressed under the depth camera coordinate system, and the equations are constructed using the constraints of the LRF scanned points on the depth camera observation line. The set of equations constructed using multiple observations is solved by linear least squares for the rotation matrix ${\mathbf{R}}_l^c$ and translation matrix ${\mathbf{T}}_l^c$.

3. Calibration Methods

3.1. Feature Extraction

Although the scanned points of 2D LRF are not visible, it accurately obtains the contour of the obstacle. Based on the characteristics of 2D LRF, a specific calibration plate is used in this paper, as shown in Figure 3. The special feature of the calibration plate is its clever shape, which makes the laser scan points form three uncorrelated straight lines and makes the camera’s 3D point cloud form three uncorrelated planes. The calibration plate consists of two rectangular planes and two triangular planes. There is no limit to the angle between the various planes in the production of the calibration plate. The only thing that needs to be ensured is that any three planes are uncorrelated and obtain as many observation points as possible on the calibration plate. In addition, the calibration plate eliminates the limitations of the environment and installation relationship during the calibration process, just ensuring that the LRF and camera observe the calibration plate at the same time. By the specific calibration plate, the characteristic information of 2D LRF scanning is obtained. At the same time, the 3D point cloud data of the depth camera observation on calibration plate are obtained too.

As shown in Figure 4, the calibration plate is placed at the position where the 2D LRF and depth camera observe simultaneously, and the observation data of 2D LRF and depth camera are collected. The 2D LRF data are observed based on the LRF coordinate system $O-{\mathit{x}}_l{\mathit{y}}_l{\mathit{z}}_l$, and the observation data of depth camera are observed based on the left camera of depth camera coordinate system $O-{\mathit{x}}_c{\mathit{y}}_c{\mathit{z}}_c$.

On the calibration plate, the scanned points of LRF will form a folded shape of $EFGGHI$, as shown in Figure 4. By the RANSAC (random sample consensus) method, the scanned points on the rectangular plane $PBDQ$ are fitted to the straight line $\overline{EI}$, the points on the plane $ABC$ are fitted to the straight line $\overline{FG}$, and the points on the plane $ACD$ are fitted to the straight line $\overline{GH}$. Any two of the three lines are respectively associated to find the intersection points $\mathbf{F}(x_1^l,y_1^l,0)$, $\mathbf{G}(x_2^l,y_2^l,0)$, $\mathbf{H}(x_3^l,y_3^l,0)$.

The depth data from the depth camera are transformed into the 3D point cloud, and the point cloud data of plane $PBDQ$, plane $ABC$, and plane $ACD$ under the observation of the depth camera are extracted. By RANSAC (random sample consensus) method, the 3D point clouds on each plane are fitted to the plane equations of the corresponding planes. The spatial linear equations of straight line $\overline{AB}$, straight line $\overline{AC}$ and straight line $\overline{AD}$ are obtained by associating any two plane equations of plane $PBDQ$, plane $ABC$ and plane $ACD$. Suppose the plane equations of the plane $PBDQ$, plane $ABC$, and plane $ACD$ are

$$\begin{array}{c}{A}_1{x}^c+B_1{y}^c+C_1{z}^c+D_1=0\\ A_2{x}^c+B_2{y}^c+C_2{z}^c+D_2=0\\ A_3{x}^c+B_3{y}^c+C_3{z}^c+D_3=0\end{array},$$

(6)

Then the spatial linear equations of the line $\overline{AB}$, line $\overline{AC}$ and line $\overline{AD}$ are

$$\begin{array}{c}\begin{array}{c}\begin{array}{c}\begin{array}{c}{A}_1{x}^c+B_1{y}^c+C_1{z}^c+D_1=0\\ A_2{x}^c+B_2{y}^c+C_2{z}^c+D_2=0\\ A_2{x}^c+B_2{y}^c+C_2{z}^c+D_2=0\end{array}\\ A_3{x}^c+B_3{y}^c+C_3{z}^c+D_3=0\end{array}\\ A_3{x}^c+B_3{y}^c+C_3{z}^c+D_3=0\end{array}\\ A_1{x}^c+B_1{y}^c+C_1{z}^c+D_1=0\end{array},$$

(7)

where $A_1$, $B_1$, $C_1$, $A_2$, $B_2$, $C_2$, $A_3$, $B_3$, $C_3$ are known. By extracting and fitting 2D LRF data and depth camera point cloud data, the linear equations of three feature points based on the LiDAR coordinate system and three spatial straight lines based on the depth camera coordinate system are obtained.

3.2. Parameter Fitting

The projection of the three feature points of 2D LRF into the depth camera coordinate system by Equation (5) has

$$\left[\begin{array}{c}{x}_{li}^c\\ y_{li}^c\\ z_{li}^c\end{array}\right]=\left[\begin{array}{c}{r}_{11}{x}_i^l+r_{12}{y}_i^l+t_1\\ r_{21}{x}_i^l+r_{22}{y}_i^l+t_2\\ r_{31}{x}_i^l+r_{32}{y}_i^l+t_3\end{array}\right],$$

(8)

where l denotes the 2D LRF coordinate system, i denotes the i-th intersection point under one 2D LRF observation, and i = 1, 2, 3. From the projection results, it is known that only nine unknowns need to be solved to obtain the rotation and translation matrices for the 2D LRF and depth camera coordinate system conversions, so only nine mutually independent sets of equations need to be coupled. Since the transformation of the coordinate system of points does not change the geometric relationship, the points under the 2D LRF coordinate system should be on the straight line $\overline{AB}$, the straight line $\overline{AC}$ and the straight line $\overline{AD}$, respectively, after the points are transformed to the camera coordinate system, i.e.,

$$\begin{array}{c}\begin{array}{c}\begin{array}{c}{A}_1{r}_{11}{x}_1^l+A_1{r}_{12}{y}_1^l+A_1{t}_1+B_1{r}_{21}{x}_1^l+B_1{r}_{22}{y}_1^l+B_1{t}_2+C_1{r}_{31}{x}_1^l+C_1{r}_{32}{y}_1^l+C_1{t}_3+D_1=0\\ A_2{r}_{11}{x}_1^l+A_2{r}_{12}{y}_1^l+A_2{t}_1+B_2{r}_{21}{x}_1^l+B_2{r}_{22}{y}_1^l+B_2{t}_2+C_2{r}_{31}{x}_1^l+C_2{r}_{32}{y}_1^l+C_2{t}_3+D_2=0\\ A_2{r}_{11}{x}_2^l+A_2{r}_{12}{y}_2^l+A_2{t}_1+B_2{r}_{21}{x}_2^l+B_2{r}_{22}{y}_2^l+B_2{t}_2+C_2{r}_{31}{x}_2^l+C_2{r}_{32}{y}_2^l+C_2{t}_3+D_2=0\end{array}\\ A_3{r}_{11}{x}_2^l+A_3{r}_{12}{y}_2^l+A_3{t}_1+B_3{r}_{21}{x}_2^l+B_3{r}_{22}{y}_2^l+B_3{t}_2+C_3{r}_{31}{x}_2^l+C_3{r}_{32}{y}_2^l+C_3{t}_3+D_3=0\end{array}\\ \begin{array}{c}{A}_3{r}_{11}{x}_3^l+A_3{r}_{12}{y}_3^l+A_3{t}_1+B_3{r}_{21}{x}_3^l+B_3{r}_{22}{y}_3^l+B_3{t}_2+C_3{r}_{31}{x}_3^l+C_3{r}_{32}{y}_3^l+C_3{t}_3+D_3=0\\ A_1{r}_{11}{x}_3^l+A_1{r}_{12}{y}_3^l+A_1{t}_1+B_1{r}_{21}{x}_3^l+B_1{r}_{22}{y}_3^l+B_1{t}_2+C_1{r}_{31}{x}_3^l+C_1{r}_{32}{y}_3^l+C_1{t}_3+D_1=0\end{array}\end{array},$$

(9)

Collated by

$$\left[\begin{array}{c}\begin{array}{c}\begin{array}{c}\begin{array}{cccc}\begin{array}{cccc}\begin{array}{ccc}{A}_1{x}_1^l& A_1{y}_1^l& A_1\end{array}& B_1{x}_1^l& B_1{y}_1^l& B_1\end{array}& C_1{x}_1^l& C_1{y}_1^l& C_1\end{array}\\ \begin{array}{cccc}\begin{array}{cccc}\begin{array}{ccc}{A}_2{x}_1^l& A_2{y}_1^l& A_2\end{array}& B_2{x}_1^l& B_2{y}_1^l& B_2\end{array}& C_2{x}_1^l& C_2{y}_1^l& C_2\end{array}\\ \begin{array}{cccc}\begin{array}{cccc}\begin{array}{ccc}{A}_2{x}_2^l& A_2{y}_2^l& A_2\end{array}& B_2{x}_2^l& B_2{y}_2^l& B_2\end{array}& C_2{x}_2^l& C_2{y}_2^l& C_2\end{array}\end{array}\\ \begin{array}{cccc}\begin{array}{cccc}\begin{array}{ccc}{A}_3{x}_2^l& A_3{y}_2^l& A_3\end{array}& B_3{x}_2^l& B_3{y}_2^l& B_3\end{array}& C_3{x}_2^l& C_3{y}_2^l& C_3\end{array}\\ \begin{array}{cccc}\begin{array}{cccc}\begin{array}{ccc}{A}_3{x}_3^l& A_3{y}_3^l& A_3\end{array}& B_3{x}_3^l& B_3{y}_3^l& B_3\end{array}& C_3{x}_3^l& C_3{y}_3^l& C_3\end{array}\end{array}\\ \begin{array}{cccc}\begin{array}{cccc}\begin{array}{ccc}{A}_1{x}_3^l& A_1{y}_3^l& A_1\end{array}& B_1{x}_3^l& B_1{y}_3^l& B_1\end{array}& C_1{x}_3^l& C_1{y}_3^l& C_1\end{array}\end{array}\right]\left[\begin{array}{c}\begin{array}{c}\begin{array}{c}{r}_{11}\\ r_{12}\\ t_1\end{array}\\ r_{21}\\ r_{22}\\ t_2\end{array}\\ r_{31}\\ r_{32}\\ t_3\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}-D_1\\ -D_2\\ -D_2\end{array}\\ -D_3\\ -D_3\\ -D_1\end{array}\right],$$

(10)

The six equations in Equation (9) are independent of each other. By changing the calibration model or the position and attitude of the calibration plate, the six constraint equations are obtained again by the same procedure. Since there are only nine unknowns in the calibration parameters, the set of equations from the last observation is combined to form the super-stationary set of equations in Equation (11), where n is the number of observations. The parameters of the rotation and translation matrices are solved using linear least squares to determine the coordinate transformation of the 2D LRF and depth camera.

$${\left[\begin{array}{ccccccccc}{A}_{11}{x}_{11}^l& A_{11}{y}_{11}^l& A_{11}& B_{11}{x}_{11}^l& B_{11}{y}_{11}^l& B_{11}& C_{11}{x}_{11}^l& C_{11}{y}_{11}^l& C_{11}\\ A_{12}{x}_{11}^l& A_{12}{y}_{11}^l& A_{12}& B_{12}{x}_{11}^l& B_{12}{y}_{11}^l& B_{12}& C_{12}{x}_{11}^l& C_{12}{y}_{11}^l& C_{12}\\ A_{12}{x}_{12}^l& A_{12}{y}_{12}^l& A_{12}& B_{12}{x}_{12}^l& B_{12}{y}_{12}^l& B_{12}& C_{12}{x}_{12}^l& C_{12}{y}_{12}^l& C_{12}\\ A_{13}{x}_{12}^l& A_{13}{y}_{12}^l& A_{13}& B_{13}{x}_{12}^l& B_{13}{y}_{12}^l& B_{13}& C_{13}{x}_{12}^l& C_{13}{y}_{12}^l& C_{13}\\ A_{13}{x}_{13}^l& A_{13}{y}_{13}^l& A_{13}& B_{13}{x}_{13}^l& B_{13}{y}_{13}^l& B_{13}& C_{13}{x}_{13}^l& C_{13}{y}_{13}^l& C_{13}\\ A_{11}{x}_{13}^l& A_{11}{y}_{13}^l& A_{11}& B_{11}{x}_{13}^l& B_{11}{y}_{13}^l& B_{11}& C_{11}{x}_{13}^l& C_{11}{y}_{13}^l& C_{11}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ A_{n1}{x}_{n1}^l& A_{n1}{y}_{n1}^l& A_{n1}& B_{n1}{x}_{n1}^l& B_{n1}{y}_{n1}^l& B_{n1}& C_{n1}{x}_{n1}^l& C_{n1}{y}_{n1}^l& C_{n1}\\ A_{n2}{x}_{n1}^l& A_{n2}{y}_{n1}^l& A_{n2}& B_{n2}{x}_{n1}^l& B_{n2}{y}_{n1}^l& B_{n2}& C_{n2}{x}_{n1}^l& C_{n2}{y}_{n1}^l& C_{n2}\\ A_{n2}{x}_{n2}^l& A_{n2}{y}_{n2}^l& A_{n2}& B_{n2}{x}_{n2}^l& B_{n2}{y}_{n2}^l& B_{n2}& C_{n2}{x}_{n2}^l& C_{n2}{y}_{n2}^l& C_{n2}\\ A_{n3}{x}_{n2}^l& A_{n3}{y}_{n2}^l& A_{n3}& B_{n3}{x}_{n2}^l& B_{n3}{y}_{n2}^l& B_{n3}& C_{n3}{x}_{n2}^l& C_{n3}{y}_{n2}^l& C_{n3}\\ A_{n3}{x}_{n3}^l& A_{n3}{y}_{n3}^l& A_{n3}& B_{n3}{x}_{n3}^l& B_{n3}{y}_{n3}^l& B_{n3}& C_{n3}{x}_{n3}^l& C_{n3}{y}_{n3}^l& C_{n3}\\ A_{n1}{x}_{n3}^l& A_{n1}{y}_{n3}^l& A_{n1}& B_{n1}{x}_{n3}^l& B_{n1}{y}_{n3}^l& B_{n1}& C_{n1}{x}_{n3}^l& C_{n1}{y}_{n3}^l& C_{n1}\end{array}\right]}_{6n\times 9}{\left[\begin{array}{c}\begin{array}{c}\begin{array}{c}{r}_{11}\\ r_{12}\\ t_1\end{array}\\ r_{21}\\ r_{22}\\ t_2\end{array}\\ r_{31}\\ r_{32}\\ t_3\end{array}\right]}_{9\times 1}={\left[\begin{array}{c}\begin{array}{c}\begin{array}{c}-D_{11}\\ -D_{12}\\ -D_{12}\end{array}\\ -D_{13}\\ -D_{13}\\ -D_{11}\end{array}\\ ⋮\\ \begin{array}{c}\begin{array}{c}-D_{n1}\\ -D_{n2}\\ -D_{n2}\end{array}\\ -D_{n3}\\ -D_{n3}\\ -D_{n1}\end{array}\end{array}\right]}_{6n\times 1},$$

(11)

3.3. Calibration Algorithm

For the joint calibration of the 2D LRF and depth camera, the method in this paper aims to determine the coordinate transformation relationship between the depth camera, LRF and the world. Before the calibration work, the calibrated depth camera had completed internal parameter calibration. To evaluate the calibration accuracy of the method in this paper, the three feature points under the LRF observation are projected under the point cloud reference coordinate system of the depth camera by solving the obtained calibration parameters, and the projection error of the laser points is calculated using the distance from the formula point to the straight line. The calibration plate or calibration model’s position is changed, the experiment is repeated several times, and the average calibration accuracy is calculated by multiple sets of data.

$$err=\frac{1}{3N}\sum _{i=1}^N\sum _{j=1}^3\frac{|A_j{x}_j+B_j{y}_j+C_j{z}_j+D_j|}{\sqrt{{A_j}^2+{B_j}^2+{C_j}^2}},$$

(12)

where N is the number of test, $(x_j,y_j,z_j)$ is the coordinate of the point projected to the depth camera coordinate system. $A_j$,$B_j$,$C_j$,$D_j$ is the linear equation coefficient of the corresponding line of the projected point. $err$ is the calibration accuracy. The smaller the $err$ is, the higher the calibration accuracy.

The algorithm framework of the calibration method in this paper is shown in Algorithm 1, where $\mathbf{R}$, $\mathbf{T}$ and $ObservNum$ are defined as the final extrinsic parameter calibration result. Datasets $\mathit{I}$ and $\mathit{S}$ are the point cloud of the depth camera and 2D LRF. Dataset $\mathit{P}$ is the pose sequence of AGV. Additionally, $\mathcal{E}$ is the error threshold, and ${\mathbf{R}}_{}gt$ and ${\mathbf{T}}_{}gt$ are the ground truth of the rotation and translation matrices. After two observations, the algorithm first solves the calibration error and compares it with the calibration threshold. When the calibration error is less than the target threshold, the rotation matrix $\mathbf{R}$, the translation matrix $\mathbf{T}$ and the number of observations are output to complete the extrinsic parameter calibration of LRF and depth camera.

Algorithm 1 LRF and Depth Camera Extrinsic Parameter Calibration.

Input: Image set $\mathit{I}$, point cloud set $\mathit{S}$, calibration position of AGV set $\mathit{P}$, and error threshold $\mathcal{E}$, ground truth ${\mathbf{R}}_{}gt$, ground truth ${\mathbf{T}}_{}gt$ Output: Calibration result $\mathbf{R}$, $\mathbf{T}$ and $Observ$   1: $\mathbf{R}\leftarrow 0,\mathbf{T}\leftarrow 0,Observ\leftarrow 0$   2: $\mathbf{A}\leftarrow 0,\mathbf{b}\leftarrow 0,err\leftarrow 100$   3: $AngularError\leftarrow 0$   4: $DistanceError\leftarrow 0$   5: $\mathbf{X}\leftarrow {\left[\begin{array}{ccccccccc}{r}_{11}\hfill & r_{12}\hfill & t_1\hfill & r_{21}\hfill & r_{22}\hfill & t_2\hfill & r_{31}\hfill & r_{32}\hfill & t_3\hfill \end{array}\right]}^{\mathrm{T}}$   6: Set firt calibration position of AGV from $\mathit{P}$   7: $Observ\leftarrow 1$   8: while$err>\mathcal{E}$do   9:     Get set $\mathit{I}$ and $\mathit{S}$  10:     Extracted 3D point cloud of calibration plate features  11:     Calculate $\mathbf{M}\leftarrow \left[\begin{array}{cccc}{A}_1\hfill & B_1\hfill & C_1\hfill & D_1\hfill \\ A_2\hfill & B_2\hfill & C_2\hfill & D_2\hfill \\ A_3\hfill & B_3\hfill & C_3\hfill & D_3\hfill \end{array}\right]$  12:     Extracted 2D point cloud of calibration plate shape features  13:     Calculate $\mathbf{L}\leftarrow \left[\begin{array}{cc}{x}_1^l\hfill & y_1^l\hfill \\ x_2^l\hfill & y_2^l\hfill \\ x_3^l\hfill & y_3^l\hfill \end{array}\right]$  14:     Calculate $\mathbf{C}\leftarrow \left[\begin{array}{cc}{x}_1^c\hfill & y_1^c\hfill \\ x_2^c\hfill & y_2^c\hfill \\ x_3^C\hfill & y_3^c\hfill \end{array}\right]$  15:      $\mathbf{A}1\leftarrow \left[\begin{array}{ccccccccc}{A}_1{x}_1^l& A_1{y}_1^l& A_1& B_1{x}_1^l& B_1{y}_1^l& B_1& C_1{x}_1^l& C_1{y}_1^l& C_1\\ A_2{x}_1^l& A_2{y}_1^l& A_2& B_2{x}_1^l& B_2{y}_1^l& B_2& C_2{x}_1^l& C_2{y}_1^l& C_2\\ A_2{x}_2^l& A_2{y}_2^l& A_2& B_2{x}_2^l& B_2{y}_2^l& B_2& C_2{x}_2^l& C_2{y}_2^l& C_2\\ A_3{x}_2^l& A_3{y}_2^l& A_3& B_3{x}_2^l& B_3{y}_2^l& B_3& C_3{x}_2^l& C_3{y}_2^l& C_3\\ A_3{x}_3^l& A_3{y}_3^l& A_3& B_3{x}_3^l& B_3{y}_3^l& B_3& C_3{x}_3^l& C_3{y}_3^l& C_3\\ A_1{x}_3^l& A_1{y}_3^l& A_1& B_1{x}_3^l& B_1{y}_3^l& B_1& C_1{x}_3^l& C_1{y}_3^l& C_1\end{array}\right]$  16:     $\mathbf{b}1\leftarrow {\left[\begin{array}{cccccc}-D_1\hfill & -D_2\hfill & -D_2\hfill & -D_3\hfill & -D_3\hfill & -D_1\hfill \end{array}\right]}^{\mathrm{T}}$  17:     if $Observ$ > 2 then  18:         $err\leftarrow \frac{1}{3}{\displaystyle \sum _{j=1}^3}\frac{\left|A_j{x}_j^c+B_j{y}_j^c+C_j{z}_j^c+D_j\right|}{\sqrt{A_j^2+B_j^2+C_j^2}}$  19:         if $err<\mathcal{E}$ then  20:            Calculate $\mathbf{R}\leftarrow \left[\begin{array}{ccc}{r}_{11}\hfill & r_{12}\hfill & r_{13}\hfill \\ r_{21}\hfill & r_{22}\hfill & r_{23}\hfill \\ r_{31}\hfill & r_{32}\hfill & r_{33}\hfill \end{array}\right]$  21:            Calculate $\mathbf{T}\leftarrow {\left[\begin{array}{ccc}{t}_1\hfill & t_2\hfill & t_3\hfill \end{array}\right]}^{\mathrm{T}}$  22:            $AngularError\leftarrow {cos}^{-1}\left(\frac{trace\left({\mathbf{R}}_{gt}^{-1}\mathbf{R}\right)-1}{2}\right)$  23:            $DistanceError\leftarrow {∥\mathbf{T}-{\mathbf{T}}_{gt}∥}_2$  24:         else  25:            Add $\mathbf{A}1$ into $\mathbf{A}$  26:            Add $\mathbf{b}1$ into $\mathbf{b}$  27:            Calculate $\mathbf{X}$ by $\mathbf{A}\mathbf{X}=\mathbf{b}$  28:            Set another calibration of AGV position from $\mathit{P}$  29:            $Observ\leftarrow Observ+1$  30:         end if  31:     else  32:         Add $\mathbf{A}1$ into $\mathbf{A}$  33:         Add $\mathbf{b}1$ into $\mathbf{b}$  34:         if $Observ=2$ then  35:            Calculate $\mathbf{X}$ by $\mathbf{A}\mathbf{X}=\mathbf{b}$  36:         end if  37:         Set another calibration board position from $\mathit{P}$  38:         $Observ\leftarrow Observ+1$  39:     end if  40: end while  41: return$\mathbf{R}$, $\mathbf{T}$ and $Observ$

4. Calibration Experiments and Analysis of Results

4.1. Experimental Equipment and Environment

The experiments in this paper are based on a ROS (robot operating system) under the Linux environment. As shown in Figure 5a, Pepperl+Fuchs R2000 2D LRF and a stereolabs ZED2i stereo binocular camera are used to collect the 2D point cloud data from the LRF and the 3D point cloud data from the depth camera, respectively. The detailed parameters of depth camera and LRF are shown in

Table 1. 2D LRF basic parameters.

Range/m	Rate/Hz	Resolution/°	Accuracy/mm	Angle/°
0.1–30	10–50	0.042	±25	360

and

Table 2. Depth camera basic parameters.

Depth Range/m	Depth FPS/Hz	Resolution	Aperture	Field/°
0.2–20	15–100	3840 × 1080	f/1.8	110H × 70V × 120D

. In Experiment, both the depth camera and LRF select a 30 Hz sampling rate.

The calibration experiments are performed on a homemade automated guided vehicles (AGV) vehicle. The installation relationship of the experimental equipment is shown in Figure 5. The 2D LRF is installed behind and above the depth camera. It is basically installed horizontally. The camera is installed horizontally in front of the 2D LRF, and the relative position of the depth camera and the LRF is kept constant during the experiment. The calibration plate is placed at the location where the 2D LRF and the depth camera observe simultaneously. Through the calibration algorithm proposed in this paper, experimental data under different observation positions, which are adjusted by controlling the AGV, are collected to complete the autonomous joint calibration of the 2D LRF and the depth camera.

4.2. Experimental Steps

The experiments in this paper are performed in a ROS environment. After running the function package of 2D LRF and depth camera, a node and two topics are established to receive respectively the scan topic of 2D LRF and the depth image topic of the depth camera. Then the depth data of the depth camera are converted into 3D point cloud data. The calibration of the 2D LRF and depth camera are completed by executing the following steps in the calibration program.

• Identify and extract the point cloud data gathered by 2D LRF on the calibration plate by line and corner feature detection algorithms; split the point cloud data into three parts; fit the point cloud of each part into a straight line; and solve the intersection point of any two straight lines. The feature extraction process is shown in Figure 6.
• Project the intersection points found in the previous step under the depth camera coordinate system by Equation (8).
• Identify and extract the point cloud collected by the depth camera on the calibration plate by edge and corner detection algorithms. Segment the three planes of the calibration plate; obtain the equation of the plane by fitting the point cloud on the plane; and find the equation of the intersection line between two planes in the three planes. The feature extraction and fitting process are shown in Figure 7.
• Using the point on the line as a constraint, the coordinates of the projected point are substituted into the intersection equation to obtain six equations.
• Move the AGV to adjust the observation position, and complete the data collection and extraction again.
• Solve the rotation and translation matrices of the depth camera and 2D LRF coordinate transformation according to Equation (11).
• Take multiple experiments, calculate the average of the calibration results.

Figure 6. Features of LRF extraction and fitting. (a) Laser observation position. (b) Laser observation data. (c) 2D point cloud of calibration plate shape features. (d) Feature data of calibration plate fitting.

Figure 6. Features of LRF extraction and fitting. (a) Laser observation position. (b) Laser observation data. (c) 2D point cloud of calibration plate shape features. (d) Feature data of calibration plate fitting.

Figure 7. Features of depth camera extraction and fitting. (a) Depth camera observation position. (b) Depth camera point cloud. (c) 3D point cloud of calibration plate features. (d) 3D point cloud plane fitting of calibration plate.

Figure 7. Features of depth camera extraction and fitting. (a) Depth camera observation position. (b) Depth camera point cloud. (c) 3D point cloud of calibration plate features. (d) 3D point cloud plane fitting of calibration plate.

4.3. Experimental Results and Analysis

In this paper, the baseline of ZED2i camera is used as the ground truth. The LRF is first calibrated with the left camera frame to obtain $\mathbf{R}{}_L^{C_l}$ and $\mathbf{T}{}_L^{C_l}$, and calibrated with right camera frame to obtain $\mathbf{R}{}_L^{C_r}$ and $\mathbf{T}{}_L^{C_r}$. We then compute the relative pose (baseline) between the binocular cameras and compare it with the ground truth $\mathbf{R}{}_{C_r}^{C_l}$ =$\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$ and $\mathbf{T}{}_{C_r}^{C_l}$ =${\left[\begin{array}{ccc}0& 120& 0\end{array}\right]}^{\mathrm{T}}$ from ZED2i Camera parameters. In order to verify the calibration accuracy and calibration efficiency of the calibration method proposed in this paper, the number of observation is used as the independent variable. For each independent variable, 10 repeated experiments were conducted to obtain the mean values of rotation and translation errors and standard deviation. Based on the experiments, we obtain the relationship between the result of the mean values and standard deviation, and the number of observations as shown in Figure 8.

As in Figure 8, the mean values of the rotation and translation errors and the standard deviations of the calibration results gradually decrease as the number of observations increases. After the number of observations is greater than 6, the average value of the rotation error is less than 1${}^{\circ }$. With the increasing number of observations, the average value of the error gradually tends to be flat. The mean value of the translation error is obtained as 3.74${}^{\circ }$ of the rotation error and 28.31 mm of translation error under only two observation experiments. The experimental data show that the calibration method of this paper is feasible and achieves high calibration accuracy.

In order to verify the feasibility of this paper’s method, the calibration accuracy of our method is compared with the more representative methods of current 2D LRF and camera calibration work. For the methods of Refs. [13,23] and the method of this paper, each method divides the experiments into three groups of 2 observations, 10 observations, and 20 observations, and each group of experiments is conducted 10 times. For each method, the LRF is first calibrated for both left and right cameras to obtain the corresponding rotation and translation matrix. Then, the coordinate transformation matrix between the two cameras is calculated and compared with the ground truths $\mathbf{R}{}_{C_r}^{C_l}$ and $\mathbf{T}{}_{C_r}^{C_l}$. The average of the 10 experimental results is used as the calibration accuracy. The calibration comparison results are shown in Figure 9. From the experimental results, it is seen that under only two observations, the method in this paper obtains smaller rotation and translation errors than the other two methods. Under multiple observation experiments, the method in this paper achieves the average value of rotation error of 0.68${}^{\circ }$ and the average value of translation error of 6.67 mm, which is better than the other two methods. Finally, the calibration results are shown in

Table 3. Calibration results.

Pose	X/mm	Y/mm	Z/mm	Yaw/${}^{\circ }$	Pitch/${}^{\circ }$	Roll/${}^{\circ }$
${}_{C_r}^{C_l}{\mathbf{P}}_{gt}$	0	120	0	0	0	0
${}_{C_r}^{C_l}{\mathbf{P}}_{calib}$	1.27	125.75	3.13	0	0	0.11
${}_L^{C_l}{\mathbf{P}}_{des}$	128	60	10	0	0	0
${}_L^{C_l}{\mathbf{P}}_{calib}$	162.35	72.51	32.58	0	0	0.57

, where ${}_{C_r}^{C_l}{\mathbf{P}}_{gt}$ is the ground truth of the relative position of the left and right cameras, and ${}_{C_r}^{C_l}{\mathbf{P}}_{calib}$ is the calibration result of the relative position of the left and right cameras. ${}_L^{C_l}{\mathbf{P}}_{des}$ is the designed installation position of the LRF and depth camera, and ${}_L^{C_l}{\mathbf{P}}_{calib}$ is the calibration result of the relative position of the LRF and camera.

Based on the calibration results, the LRF data are projected into the depth color point cloud map of the depth camera, and the effects before and after calibration are compared visually. The effect before calibration is shown in Figure 10a,b. Before calibration, the 2D lidar point cloud is obscured by the depth point cloud of the camera, and the position of the laser point cloud is lower than the actual observation position. The effect after calibration is shown in Figure 10c,d. After calibration, the point cloud data of the depth camera and the 2D LRF point cloud data basically overlap in the given dimension, which basically meets the application requirements. Due to the occlusion of the vehicle body model, the tf coordinates of the reference system in ROS are shown additionally in Figure 11.

The calibration method in this paper is experimentally verified to be simple and easy to implement, and the joint calibration of the 2D LRF and depth camera is completed with high accuracy and efficiency without excessive position and angle observations. Compared with Ref. [23], there is no complicated feature point extraction and display operation in the calibration process, and no need to adjust the calibration plate position manually. The algorithm is designed to adaptively adjust the AGV observation poses. The calibration method has no restriction on the relative installation position of 2D LRF and depth camera, and the calibration plate is easy and convenient to make. It greatly simplifies the calibration process of the 2D LRF and depth camera and improves the calibration efficiency. Compared with Refs. [13,23], the calibration method in this paper has obvious advantages in calibration accuracy, efficiency and operability.

5. Conclusions

In this paper, we present a novel and simplified calibration method to construct point-line constraints for 2D LRF and depth camera observation data by using a specific calibration plate, which effectively achieves the fusion of 2D LRF point cloud data and depth camera 3D point cloud data. The specific calibration plate proposed in this paper eliminates the influence of environment, calibration plate fabrication limitation and sensor installation limitation on calibration. Compared with previous methods, we greatly simplify the calibration process of depth camera and 2D LRF by automatically adjusting the number of observations with a defined error threshold. A series of experiments verify that our method achieves higher accuracy than the compared methods in two observations. Our method is also extended to the case of multiple observations to reduce noise. The calibration accuracy, efficiency and operability of the calibration method meet the practical requirements of mobile robots.