On-Orbit Calibration for Spaceborne Line Array Camera and LiDAR

Abstract

For a multi-mode Earth observation satellite carrying a line array camera and a multi-beam line array LiDAR, the relative installation attitude of the two sensors is of great significance. In this paper, we propose an on-orbit calibration method for the relative installation attitude of the camera and the LiDAR with no need for the calibration field and additional satellite attitude maneuvers. Firstly, the on-orbit joint calibration model of the relative installation attitude of the two sensors is established. However, there may exist a multi-solution problem in the solving of the above model constrained by non-ground control points. Thus, an alternate iterative method by solving the pseudo-absolute attitude matrix of each sensor in turn is proposed. The numerical validation and simulation experiments results show that the relative positioning error of the line array camera and the LiDAR in the horizontal direction of the ground can be limited to 0.8 m after correction by the method in this paper.

1. Introduction

Light Detection and Ranging (LiDAR) has been more and more used in aerial remote sensing in recent years because of its ability to obtain accurate ranging information [1,2]. After the launch of the LiDAR remote sensing satellite Ice, Cloud, and land Elevation Satellite-2 (ICESat-2), the acquired data have become an important tool in the research of lake water levels changes, sea ice monitoring, and vegetation canopy height measurement [3,4,5]. In particular, the elevation data and optical remote sensing data are coordinated and complemented, and more high-precision results have been obtained in the above-mentioned fields [6,7]. On this basis, researches on estimating water volumes of lakes, shallow water depths, and other related fields have been carried out [8,9].

The existing researches are based on data collected by multiple satellites in different periods. The LiDAR data are relatively sparse and unevenly distributed in the research area. The asynchronism and the unbalanced distribution of optical images and LiDAR data lead to difficulties in data fusion applications. Remote sensing satellites equipped with both optical cameras and LiDARs can greatly ensure the synchronization of data acquisition and balance of the spatial distribution of data. This paper focuses on the calibration method for a sensing satellite equipped with a line array camera and multi-beam line array LiDAR. A common problem for remote sensing satellites is that, during launching and during orbiting, the attitude of the camera and LiDAR in the satellite coordinate system will be slightly changed by various factors [10], which results in a large error when the observation data are transferred to the ground coordinate system by using the installation attitude obtained by calibration before launching. Hence, the relative installation parameters of the two sensors need to be calibrated on orbit [11].

For optical remote sensing satellites, ground feature areas or control points are often used as reference sources for calibration. In the research of [11], the author studied the error source of the spaceborne optical camera, and established a geometric calibration model and proposed a method for solving the model parameters. In the research of [12], the author introduced a generalized offset matrix to eliminate external errors such as attitude, orbit measurement error, GPS error, etc., established internal and external calibration models, respectively, and adopted a partial solution method that solved the external parameters and then solved the internal parameters firstly. However, these methods rely on ground control points and require a calibration field to be designed on the ground, which will bring greater additional costs. Additionally, the camera can only be calibrated when the satellite passes over the calibration field using these methods, so the calibration time flexibility is low. Pi et al. [13] introduced the constraint of overlapping images to prevent the calibration image from matching dense ground control points in the ground reference data and proposed a calibration method that only requires sparse control points. In the research of [14], the author studied the fieldless calibration of the agile imaging satellite cross image and realized the adjustment model with additional digital elevation model (DEM) constraints. This method has high accuracy and gets rid of the dependence on ground control points, but it is only suitable for agile imaging satellites with strong attitude adjustment capabilities. For conventional remote sensing satellites, Wang et al. [15] used equivalent frame photo (EFP) beam adjustment to calibrate the line-matrix charge-coupled device (LMCCD) stereo mapping camera equipped on the TH-1 optical remote sensing satellite in orbit. As the digital elevation map of the calibration area is also needed to assist, the flexibility of calibration is limited. Yang et al. [16] established the relative constraints between the images of each camera and the intersection constraints of stereo images of the ZY-3 satellite by matching the uniformly distributed corresponding image points in the two pairs of three linear camera images. In these constraints, the authors conducted a self-calibration method without additional reference data to reduce the costs and improve the flexibility of time and space for on-orbit calibration.

The calibration method of LiDARs is different from that of optical cameras. Luthcke et al. [17] used the satellite attitude maneuver method to perform Bayesian least square estimation of the sea level ranging residual to calibrate the LiDAR pointing, but this method is dependent on the satellite’s attitude adjustment ability and accuracy. In the research of [18], the author derived a pointing angle system error on-orbit correction model based on pointing angle residuals and used the on-orbit verification method of footprint detection for the on-orbit correction of the system error. By extracting the position of the corner cube retro-reflectors (CCR) in the returned photon signal and associating the natural ground with the laser footprint, Guo et al. [19] performed on-orbit calibration of spaceborne single-photon laser altimeter. Compared with the previous methods, the accuracy of the methods has been improved, but energy detectors or the CCRs need to be arranged in the ground calibration field to calculate the laser footprint center, which leads to higher calibration costs and lower time flexibility. Yi et al. [20] proposed a spaceborne laser altimeter pointing on-orbit calibration method based on the analysis of natural surface ranging residuals, which avoids satellite attitude maneuvering but requires the digital elevation map of the calibration field as reference data. Tang et al. [21] proposed a method for spaceborne laser altimeter calibrating by using published digital terrain data as reference data to match with stripes of a point cloud obtained by the geolocation model of the laser altimeter. After calibration, the elevation error is less than 3 m, and the efficiency of the method is 10 times higher than the previous algorithm. Those methods do not need the satellite attitude maneuvering or ground optical components, however, the digital elevation map of the calibration field is required as reference data.

Since there are no satellites equipped with optical cameras and LiDAR, the research on the calibration of the relative installation parameters of the two types of sensors is mostly based on vehicles, unmanned aerial vehicles, or robotic platforms. Pusztai et al. [22] extracted each visible surface from the LiDAR point cloud by imaging ordinary boxes, and after matching with the camera image, the relative pose of the camera and LiDAR can be obtained through efficient perspective-n-point (EPnP) algorithm. Zhou et al. [23] proposed a new method of camera–LiDAR relative extrinsic parameter calibration based on the line and plane correspondences of the checkerboard, which obtained more accurate results by fewer numbers of checkerboard poses. Verma et al. [24] took the checkerboard as the reference object and established the correspondence between the center point and the normal vector of the checkerboard through the method of automatically extracting the features of the image and point cloud, by which the relative pose between the camera and the LiDAR can be obtained. Tóth et al. [25] took the sphere as the reference object and extracted its center from the LiDAR point cloud and the camera image respectively, and the relative poses were calculated by several pairs of the coordinates of the sphere’s center. As these methods require the assistance of checkerboards or reference objects with distinctive features, it is difficult to achieve calibration for spaceborne camera–LiDAR using them. By projecting the laser point cloud onto the image plane and matching the edge points of the projected image with the edge points of the camera image, Wang et al. [26] proposed a vehicle-mounted camera–LiDAR online calibration method based on daily road information, but the mismatch rate and the resulting misunderstanding rate are slightly high. In the research of [27], the author proposed a method to solve the relative pose of the camera and LiDAR by generating point clouds from image sequences and using object-based methods to match them with LiDAR point clouds. However, it is difficult to obtain continuous image sequences for spaceborne line array cameras. In summary, those methods of relative pose between camera and LiDAR calibration at short range are hard to be applied to remote sensing satellites.

In this paper, we propose a method for line array camera and multi-beam line array LiDAR relative pose on-orbit calibration that does not rely on the calibration field and does not require frequent satellite attitude adjustment and propose its solution method. First, the joint calibration model is established by combining the line array camera imaging model and LiDAR observation model. Second, point pairs are generated by matching the optical image and LiDAR data projection image. Then the scale factor is calculated under the assumption that the on-orbit shifting angle is close to 0. Finally, the relative extrinsic parameters are innovatively calculated by solving pseudo-poses of the camera and LiDAR through an alternating iterative method.

2. Calibration Model and the Method of Calculation

In this section, a joint calibration method without control points is adopted, and a joint calibration model is established based on the imaging model of the line array camera and the observation model of the LiDAR.

2.1. Line Array Camera Imaging Model

2.1.1. Definition of Line Array Camera Coordinate System

The origin of the linear array camera coordinate system is the optical center of the line array camera. The Z-axis is the vertical line of the line array direction on the plane determined by the CCD array and the optical center, and the direction to the ground is specified as the positive direction. The linear array direction of the line array camera is defined as the Y-axis. The X-axis is determined according to the right-hand system rules. The positive direction of the X-axis and Y-axis select the group with the smaller angle between the positive direction of the X-axis and the satellite flight direction, as shown in Figure 1.

By using this rule to define the coordinate system of the line array camera, the deflection angle of the line array camera along the direction of the line array can be absorbed into the rotation parameters around the Y-axis to be calculated together to avoid singular problems caused by the coupling of the two parameters.

2.1.2. Line Array Camera Imaging Model

The imaging model of the line array camera is related to the imaging time. For the point P with coordinates $(x_a,y_a)$ in the camera image coordinate system imaged by the line array camera at time t, the coordinates in the line array camera coordinate system is $\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right]$, where $y_c^0$ is the coordinates of principal point in the image, ${\lambda }_{ccd}^c$ is the length of the CCD, and $f_c$ is the focal distance of the camera, which are all known constants.

The relationship between the coordinate in the camera image coordinate system and the ground coordinate system of the point P is shown in Equation (1):

$$\mathbf{P}={\mathbf{P}}_s^t+{\mathbf{R}}_s^t{\mathbf{T}}_c+k_c{\mathbf{R}}_s^t{\mathbf{R}}_c^u{\mathbf{R}}_c\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right],$$

(1)

where the rotation matrix of the installation angle relative to the satellite body obtained by the line array camera calibration on the ground is ${\mathbf{R}}_c$, the rotation matrix corresponding to the on-orbit shifting angle is ${\mathbf{R}}_c^u$, the translation vector of the line array camera coordinate system relative to the satellite coordinate system is ${\mathbf{T}}_c$, the rotation matrix of the satellite relative to the ground coordinate system at time t is ${\mathbf{R}}_s^t$, the translation vector relative to the ground coordinate system is ${\mathbf{P}}_s^t$, and the imaging scale factor of the camera is $k_c$. Since the on-orbit shifting of ${\mathbf{T}}_c$ has little effect on the ground positioning accuracy, its on-orbit changes are not considered in the imaging model.

The geometric meaning of the above matrices is shown in Figure 2.

2.2. LiDAR Observation Model

The LiDAR discussed in this article is a multi-beam LiDAR. A single laser light source emits a single laser beam, which is converted into several coplanar laser beams by optical components such as grating. The laser beam plane is approximately perpendicular to the satellite’s flight direction.

2.2.1. Definition of LiDAR Coordinate System

The origin of the LiDAR coordinate system is set at the laser beam emission point, with the middle laser beam pointing to the Z-axis, and the side with the smaller angle between the plane normal of the laser beam and the satellite flying direction as the positive X-axis, and the right-hand system to determine the Y-axis, as shown in Figure 3.

2.2.2. LiDAR Observation Model

If the LiDAR scans to a certain point P on the ground at a certain time t, the angle between the laser beam scanned to this point and the positive Z-axis of the LiDAR coordinate system is $\beta$, and the rotation matrix corresponding to $\beta$ is ${\mathbf{R}}_{\beta }$. The distance between the LiDAR and point P measured by the laser beam is $\rho$. ${\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right]$ is called the coordinate of the point P in the LiDAR coordinate system, as shown in Figure 4.

The coordinate of point P in the ground coordinate system satisfies Equation (2):

$$\mathbf{P}={\mathbf{P}}_s^t+{\mathbf{R}}_s^t{\mathbf{T}}_l+{\mathbf{R}}_s^t{\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right],$$

(2)

where the rotation matrix of the installation angle of the LiDAR relative to the satellite body obtained by the ground calibration is ${\mathbf{R}}_l$, the rotation matrix corresponding to the on-orbit shifting angle of the LiDAR is ${\mathbf{R}}_l^u$, the translation vector of the LiDAR coordinate system relative to the satellite coordinate system is ${\mathbf{T}}_l$, regardless of its on-orbit shifting, the rotation matrix of the satellite relative to the ground coordinate system at time t is ${\mathbf{R}}_s^t$, and the translation vector relative to the ground coordinate system is ${\mathbf{P}}_s^t$.

The geometric meaning of the above matrices is shown in Figure 5.

2.3. Line Array Camera and LiDAR Joint Calibration Model

Due to the relative installation angle, the times when the camera and the LiDAR detect the same point on the ground are different, as shown in Figure 6.

For a certain ground point, set the time when the line camera observes it as $t_i$, and the time when the LiDAR observes it as $t_j$, the ground coordinates can be eliminated by combining Equations (1) and (2):

$${\mathbf{P}}_s^{t_i}+{\mathbf{R}}_s^{t_i}{\mathbf{T}}_c+k_c{\mathbf{R}}_s^{t_i}{\mathbf{R}}_c^u{\mathbf{R}}_c\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right]={\mathbf{P}}_s^{t_j}+{\mathbf{R}}_s^{t_j}{\mathbf{T}}_l+{\mathbf{R}}_s^{t_j}{\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right].$$

(3)

Equation (3) is the joint calibration model, which can calibrate the extrinsic parameters of the two sensors on-orbit in real time without ground control point constraints. Because there is no ground control point constraint, the line array camera extrinsic parameter on-orbit shifting matrix ${\mathbf{R}}_c^u$ obtained by the above model solution may have the same direction as the LiDAR extrinsic parameter on-orbit shifting matrix ${\mathbf{R}}_l^u$. The deviation of the relative on-orbit deviation can be obtained by making the difference between the Euler angles corresponding to the two matrices.

2.4. Method of Calculation

Losing the constraint of the ground coordinates of the control points, the calculation of the joint calibration model is a rank deficient problem, and the solution accuracy of the calibration equation (Equation (3)) is strongly dependent on the solution accuracy of the imaging scale factor $k_c$. This paper adopts the method of solving the scale factor first, and then iteratively solving the on-orbit shifting matrices ${\mathbf{R}}_c^u$ and ${\mathbf{R}}_l^u$.

During the imaging process of each scene, the satellite attitude will remain stable, that is, ${\mathbf{R}}_s^{t_i}\equiv {\mathbf{R}}_s^{t_j},\forall i,j.$ The satellite attitude matrix in the entire observation is recorded as ${\mathbf{R}}_s$, then the calibration equation (Equation (3)) can be simplified to

$${\mathbf{P}}_s^{t_i}+{\mathbf{R}}_s{\mathbf{T}}_c+k_c{\mathbf{R}}_s{\mathbf{R}}_c^u{\mathbf{R}}_c\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right]={\mathbf{P}}_s^{t_j}+{\mathbf{R}}_s{\mathbf{T}}_l+{\mathbf{R}}_s{\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right].$$

(4)

2.4.1. Solving the Imaging Scale Factor

For different ground points, the imaging scale factor $k_c$ is also slightly different, and its exact solution depends on the value of the coefficient to be calibrated, so an approximate solution method is adopted here. Equation (4) is processed as follows:

$$k_c{\mathbf{R}}_s{\mathbf{R}}_c^u{\mathbf{R}}_c\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right]={\mathbf{P}}_s^{t_j}+{\mathbf{R}}_s{\mathbf{T}}_l-{\mathbf{P}}_s^{t_i}-{\mathbf{R}}_s{\mathbf{T}}_c+{\mathbf{R}}_s{\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right],$$

(5)

$$k_c{∥{\mathbf{R}}_s{\mathbf{R}}_c^u{\mathbf{R}}_c\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right]∥}_2={∥{\mathbf{P}}_s^{t_j}+{\mathbf{R}}_s{\mathbf{T}}_l-{\mathbf{P}}_s^{t_i}-{\mathbf{R}}_s{\mathbf{T}}_c+{\mathbf{R}}_s{\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right]∥}_2.$$

(6)

Since the LiDAR shifting angle in the orbit is small, it is approximately

$${∥{\mathbf{P}}_s^{t_j}+{\mathbf{R}}_s{\mathbf{T}}_l-{\mathbf{P}}_s^{t_i}-{\mathbf{R}}_s{\mathbf{T}}_c+{\mathbf{R}}_s{\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right]∥}_2\approx {∥{\mathbf{P}}_s^{t_j}+{\mathbf{R}}_s{\mathbf{T}}_l-{\mathbf{P}}_s^{t_i}-{\mathbf{R}}_s{\mathbf{T}}_c+{\mathbf{R}}_s{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right]∥}_2.$$

(7)

$k_c$ can be calculated by (6) and (7):

$$k_c\approx \frac{{∥{\mathbf{P}}_s^{t_j}+{\mathbf{R}}_s{\mathbf{T}}_l-{\mathbf{P}}_s^{t_i}-{\mathbf{R}}_s{\mathbf{T}}_c+{\mathbf{R}}_s{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right]∥}_2}{{∥\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right]∥}_2}.$$

(8)

2.4.2. Solving the Relative Extrinsic Parameters

It has been pointed out in Section 2.3 that due to the lack of ground control point constraints, the parameters to be calibrated ${\mathbf{R}}_c^u$ and ${\mathbf{R}}_l^u$ cannot be accurately calculated, but the coordinate constraints of the point pairs can ensure that the relative extrinsic parameters of the two sensors are solved accurately. An alternate iterative solution method is proposed in this paper.

Since the on-orbit shifting angles are close to 0, that is, the rotation matrixes ${\mathbf{R}}_c^u$ and ${\mathbf{R}}_l^u$ are approximately equal to identity matrix $\mathbf{I}$. In the first step of the iteration method, to approximately calculate the rotation matrix ${\mathbf{R}}_l^u$, ${\mathbf{R}}_c^u$ is substituted by $\mathbf{I}$, and after multiplying ${\mathbf{R}}_s^{-1}$ on both sides, Equation (4) is transformed to

$${\mathbf{R}}_s^{-1}{\mathbf{P}}_s^{t_i}+{\mathbf{T}}_c+k_c{\mathbf{R}}_c^u{\mathbf{R}}_c\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right]-{\mathbf{R}}_s^{-1}{\mathbf{P}}_s^{t_j}-{\mathbf{T}}_l={\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right].$$

(9)

In Formula (9), except for the matrix ${\mathbf{R}}_l^u$ to be obtained, all other items are known. Equation (9) holds for each point, and for all points, it can be combined into

$$\mathbf{Q}={\mathbf{R}}_l^u{\mathbf{Q}}_l,$$

(10)

where the i-th column of $\mathbf{Q}$ is the value on the left side of Equation (9) for the i-th point, and the i-th column of ${\mathbf{Q}}_l$ is the value on the right side of Equation (9) for the i-th point.

According to reference [28], and in view of the situation in this article, Equation (10) is solved using the following method. Mark $\mathbf{M}={\mathbf{Q}}_l{\mathbf{Q}}^T$, $\mathbf{S}=\mathbf{M}+{\mathbf{M}}^T$, $\beta ={∥{\mathbf{Q}}_l∥}^2+{∥\mathbf{Q}∥}^2$, $\gamma =tr\left(\mathbf{M}\right)$, $\Delta ={(n_{32}-m_{23},m_{13}-m_{31},m_{21}-m_{12})}^T$, where $\Delta$ is the element of the matrix $\mathbf{M}$, $tr\left(\right)$ is the trace of the matrix. Then define the symmetric matrix $\mathbf{H}=\left[\begin{array}{cc}\beta -\gamma & {\Delta }^T\\ \Delta & (\beta +\gamma )\mathbf{I}-\mathbf{S}\end{array}\right]$. The eigenvector corresponding to the minimum eigenvalue of $\mathbf{H}$ is the quaternion vector corresponding to the rotation matrix to be obtained, and the rotation matrix ${\mathbf{R}}_l^u$ can be obtained.

In the second step of the iteration, the calculation result of ${\mathbf{R}}_l^u$ in the previous step is used as its value. For calculating the matrix ${\mathbf{R}}_l^u$, Equation (4) is transformed to

$$\frac{\left({\mathbf{R}}_s^{-1}{\mathbf{P}}_s^{t_j}+{\mathbf{T}}_l+{\mathbf{R}}_l^u{\mathbf{R}}_l{\mathbf{R}}_{\beta }\left[\begin{array}{c}0\\ 0\\ \rho \end{array}\right]-{\mathbf{R}}_s^{-1}{\mathbf{P}}_s^{t_i}-{\mathbf{T}}_c\right)}{k_c}={\mathbf{R}}_c^u{\mathbf{R}}_c\left[\begin{array}{c}0\\ (y_a-y_c^0){\lambda }_{ccd}^c\\ f_c\end{array}\right].$$

(11)

In Formula (11), except for the matrix ${\mathbf{R}}_c^u$ to be obtained, all other items are known, and Formula (11) can be simply written as

$$\mathbf{Q}={\mathbf{R}}_c^u{\mathbf{Q}}_c.$$

(12)

Using the same method as solving Equation (10), the matrix ${\mathbf{R}}_c^u$ in Equation (12) can be solved.

In the next step of the iteration, the calculation result of ${\mathbf{R}}_c^u$ in the previous step is used as its value, and the updated value of ${\mathbf{R}}_c^u$ can be calculated using the same method as the first iteration step.

The alternate iteration is repeated until the result is stable.

3. Simulation Results and Analysis

Since the relative extrinsic parameters are embodied by the pseudo-absolute extrinsic parameters, the difference between the pseudo-absolute extrinsic parameters and the real extrinsic parameters makes the accuracy evaluation by arcseconds lose a certain value. The goal of joint calibration is to enable the observation data of the two sensors to be fused with the smallest error. Therefore, the accuracy evaluation method adopted in this paper is to use the calibration results and the coordinates of the points in the two images to calculate the corresponding ground coordinates using Equations (1) and (2), respectively, and measure the calibration accuracy by the ground horizontal distance between calculation results of the two sensors.

3.1. Numerical Validation

The simulation solution is performed by generating points to verify the accuracy of the proposed calibration algorithm with MATLAB R2021a (9.10.0.1851785 Update 6, from MathWorks, in Guangzhou, China). Two sets of point-pair coordinates are generated from the preset line array camera and LiDAR extrinsic parameters. One set is processed with the proposed method for calibration calculation, and the other set is used as the true value for subsequent error calculation. The flowchart is shown in Figure 7.

Different aspects including observation error, on-orbit shifting angle of sensors, number of LiDAR beams, and number of point pairs, which are most likely to affect the calibration accuracy, are analyzed in the following subsections.

3.1.1. Influence of Observation Error

The observation error is a kind of inevitable and common error that affects the calculation accuracy of the proposed method. In this section, the three direct observation errors from coordinates of point pairs in images of camera and LiDAR, and laser ranging are considered, respectively.

In real operation, coordinate errors of point pairs in images are usually caused by pixel extraction or the feature matching process, and such errors are generally less than 0.5 pixels. Errors of laser ranging are directly caused by the laser sensors or its correction algorithm of atmospheric refraction and are generally less than 10 m. In this section, we set the coordinate errors in the range of 0 to 1 pixel, and the laser ranging errors in the range of 0 to 50 m.

The errors whose distribution is similar to that of the set for calibration are added to each observation to test their influence on calibration accuracy in this part. The fixed parameters are set as follows:

• The fixed number of point pairs is 100;
• The fixed number of LiDAR beams is 127;
• The on-orbit attitude shifting Euler angle of the line array camera is $[{-0.05}^{\circ }, {0.03}^{\circ },$ ${-0.04}^{\circ }{]}^T$;
• The on-orbit attitude shifting Euler angle of the LiDAR is $[{0.01}^{\circ }, {-0.03}^{\circ }, {0.01}^{\circ }{]}^T$.

The errors added and their corresponding solution errors are shown in

Table 1. Observation error and calibration solution result.

	Error	X Direction			Y Direction
		Min	Max	Mean	Min	Max	Mean
CC	0.2
LC	0	0.0009	1.4169	0.3335	0.0030	1.3077	0.3727
LR	0
CC	0.5
LC	0	0.0025	3.5425	0.8338	0.0005	3.0639	0.8698
LR	0
CC	0
LC	0.2	0.0124	1.0598	0.3120	0.0037	0.3106	0.1432
LR	0
CC	0
LC	0.5	0.0307	2.6498	0.7801	0.0041	0.3103	0.1431
LR	0
CC	0
LC	0	0.0059	1.7430	0.5113	0.0048	0.3106	0.1433
LR	10
CC	0
LC	0	0.0284	8.7158	2.5567	0.0044	0.3092	0.1437
LR	50
CC	0.1
LC	0.1	0.0156	1.8940	0.5843	0.0014	0.7228	0.2249
LR	10
CC	0.2
LC	0.2	0.0035	2.0974	0.7265	0.0033	1.3083	0.3728
LR	10
CC	0.5
LC	0.5	0.0201	3.9419	1.3341	0.0009	3.0648	0.8698
LR	10
CC	1
LC	1	0.0170	10.4874	3.6323	0.0706	5.9933	1.7229
LR	50

CC: camera image coordinates (error unit: pixel). LC: LiDAR image coordinates (error unit: pixel). LR: laser ranging (error unit: meter).

. It can be seen that the mean error of horizontal positioning is less than 0.8 m at the average error level of feather points extracting and laser ranging, where the camera image coordinate error and LiDAR image coordinate error are both less than 0.2 pixels and the laser ranging error is less than 10 m.

3.1.2. Influence of On-Orbit Shifting Angles of Sensors

The on-orbit shifting angles of sensors are the objective variables in the proposed calibration model. In the step of the calculation, the assumption of the on-orbit shifting angles are close to 0 is grounded, so the values of on-orbit shifting angles are important factors of the proposed method. The sources of on-orbit shifting angles of sensors have been presented in Section 1, and therefore, the angles are generally less than 0.1 degrees. To verify possible situations for the two sensors on the satellite platform, the absolute values of angles are set to range from 0.01 degrees to 2 degrees in the numerical validation in this part.

Multiple groups of on-orbit shifting Euler angles of sensors are simulated and solved as shown in

Table 2. On-orbit shifting angle of sensors and calibration results.

	On-Orbit Shifting Angle (Degree)			X Direction			Y Direction
	x	y	z	Min	Max	Mean	Min	Max	Mean
Camera	−0.0500	0.0300	−0.0400	0.0035	2.0974	0.7265	0.0033	1.3083	0.3728
LiDAR	0.0100	−0.0300	0.0100
Camera	0.0700	0.0500	−0.0400	0.0071	2.1185	0.7328	0.0189	1.3376	0.3915
LiDAR	−0.0200	0.0500	0.0300
Camera	0.1000	0.3000	−0.2000	0.0037	2.0867	0.7222	0.0250	2.6545	0.9250
LiDAR	−0.3000	0.2000	0.1000
Camera	1.0000	1.0000	−2.0000	0.0196	2.1164	0.7293	0.0086	2.3571	0.7361
LiDAR	−1.0000	1.0000	1.0000

. In this part, the fixed parameters are set as follows:

• The fixed number of point pairs is 100;
• The fixed number of LiDAR beams is 127;
• The camera image coordinate and LiDAR image coordinate errors are both normal distribution errors with a standard deviation of 0.2 pixels;
• The laser ranging errors are normal distribution errors with a standard deviation of 10 m.

The results show that the mean error of horizontal positioning is less than 0.8 m when the on-orbit shifting angle of the camera and LiDAR on a normal scale, that is, are both less than 1 degree.

3.1.3. Influence of the Number of LiDAR Beams

The number of LiDAR beams is changeable according to the model of the LiDAR used. The beams of multi-beam LiDAR are usually generated by a single beam through a grating, hence the number subject to $2^n$ or $2^n-1$ is considered in this section. The parameter possibly affects the calibration accuracy, while the number of LiDAR beams is limited by the power of the laser source, therefore, different numbers of LiDAR beams are simulated and solved as shown in

Table 3. Number of LiDAR beams and calibration results.

Number of LiDAR Beams	X Direction			Y Direction
	Min	Max	Mean	Min	Max	Mean
4	0.0123	2.1072	0.6310	0.0129	5.8226	2.0472
7	0.0114	2.0402	0.7025	0.0346	4.4871	1.9170
15	0.0017	2.0794	0.7235	0.0025	2.9571	1.2076
31	0.0028	2.0951	0.7263	0.0012	1.8963	0.6721
63	0.0033	2.0974	0.7265	0.0010	1.4485	0.4510
127	0.0035	2.0974	0.7265	0.0033	1.3083	0.3728
255	0.0037	2.0971	0.7264	0.0018	1.2387	0.3502

. In this part, the fixed parameters are set as follows:

• The fixed on-orbit shifting Euler angle of the line array camera is $[{-0.05}^{\circ }, {0.03}^{\circ }, {-0.04}^{\circ }{]}^T$;
• The fixed on-orbit shifting Euler angle of the LiDAR is $[{0.01}^{\circ }, {-0.03}^{\circ }, {0.01}^{\circ }{]}^T$;
• The fixed number of points is 100;
• The camera image coordinate and LiDAR image coordinate errors are both normal distribution errors with a standard deviation of 0.2 pixels;
• The laser ranging errors are normal distribution errors with a standard deviation of 10 m.

The result in Table 3 shows that the positioning error in the X direction is little affected by the number of LiDAR beams while the positioning error in the Y direction strongly relies on the number of LiDAR beams, and when the LiDAR beams are more than 31, the geo-positioning error is less than 0.8 m.

3.1.4. Influence of the Number of Point Pairs

The number of points deployed on the ground is a key factor of the traditional method. Additionally, the calibration model established in Section 2 is also a point-based equation, therefore, the number of point pairs may affect the efficiency and accuracy of the proposed method.

The number of point pairs is a controllable parameter that can be set as flexible. In order to determine the effect of the number of point pairs on calibration, different numbers of point pairs were simultaneously chosen from both sets for comparison in this simulation. The fixed parameters are set as follows:

• The fixed number of LiDAR beams is 127;
• The fixed on-orbit shifting Euler angle of the line array camera is $[{-0.05}^{\circ }, {0.03}^{\circ }, {-0.04}^{\circ }{]}^T$;
• The fixed on-orbit shifting Euler angle of the LiDAR is $[{0.01}^{\circ }, {-0.03}^{\circ }, {0.01}^{\circ }{]}^T$;
• The camera image coordinate errors and LiDAR image coordinate errors are both normal distribution errors with a standard deviation of 0.2 pixels;
• The laser ranging errors are normal distribution errors with a standard deviation of 10 m.

The corresponding results shown in

Table 4. Number of points and calibration results.

Number of Points	X Direction			Y Direction
	Min	Max	Mean	Min	Max	Mean
10	0.0124	1.5103	0.5351	0.0144	1.0556	0.4617
50	0.0175	2.4454	0.7416	0.0052	1.0953	0.4035
100	0.0035	2.0974	0.7265	0.0033	1.3083	0.3728
1000	0.0003	2.8153	0.7196	0.0007	1.5610	0.3382
10,000	0.0005	3.7995	0.6994	0.0007	1.6772	0.3411

proved that, although the calculation error decreases slightly with the increase of point pairs number, the calculation error is little affected by the number of point pairs overall, and the mean horizontal positioning mean error is less than 0.8 m when the point pairs are more than 10.

From the above simulation results, the proposed method shows good performance in error compatibility, dependence on the number of points, and solution accuracy in the case of small shifting angles, which is suitable for the multi-beam LiDAR calibration situation with low resolution in the vertical scanning direction. The data after calibration can be applied for fusion and subsequent research, despite the fact that some measurement errors like the pose error of satellite are not considered yet.

3.2. Simulation Experiment

Considering that there are few definite satellites equipped with both a line array camera and a multi-beam line array LiDAR currently, this paper only discusses the results of indoor simulation experiments, to verify the feasibility of the proposed method.

3.2.1. Scheme of Hardware-in-Loop Simulation Experiment

The flowchart is shown in Figure 8. During the experiment, a flat plane is selected as the imaging area, several marker blocks are pasted to simulate ground buildings. Moreover, there is a calibration board and some diagonal markers scattering in the field of view for ground coordinate system establishment. The above parts build the whole scene, as shown in Figure 9.

In order to simulate the actual imaging process in orbit, an area array camera and a time-of-flight (ToF) camera are fixed on the same base placed on the guide rail for simulating the satellite orbit, as shown in Figure 10. Limited by the hardware and experiment site, the telephoto camera is hard to use in the experiment, so a camera with an equivalent focal length of about 1500 is selected instead, and the ToF camera is used for simulating the LiDAR, which also can measure distances. The base is pushed along the guide rail and an image will be taken every 5 cm. A sequence of images and a sequence of 3D point clouds were obtained in the push broom step. The image sequence is stitched to obtain a simulated line array image. Different from the former, the point cloud sequence obtained by the ToF camera is not only stitched but also down-sampled, to simulate the LiDAR point cloud data with 127 laser beams. After stitching and down-sampling, the LiDAR data are a set of dense point clouds with higher resolution in the X direction and lower resolution in the Y direction, where the definition of the direction corresponds to Section 2.2.1. To extract the same points both in the camera image and LiDAR data, the point cloud data are binarized to obtain the LiDAR image. In the point cloud, the distance of each point satisfies a clear statistical distribution and the majority of points are on the ground. On this basis, a distance threshold is set to generate the corresponding binary image where long-distance points are set to 0 and the rest are set to 255. The simulated line array image and LiDAR image are shown in Figure 11a,b, respectively.

3.2.2. Simulation Experiment Results

Due to the large difference in sampling rate, it is difficult to directly extract point pairs from optical images and LiDAR images. We segmented the simulated ground buildings from the optical image [29,30], and after binarization, the two images were matched using the mutual information method to generate points pairs. According to the translation and scaling relationships between the two images obtained by matching, 100 pairs of points are generated randomly for calibration and calibration accuracy verification, respectively. The distribution of the point pairs for calibration in the two images is shown in Figure 12, and the distribution of the point pairs for the accuracy verification in the two images is shown in Figure 13.

The proposed approach, in which is no need to maneuver the satellite, coordinates of points in the calibration field, or other reference data, is different from other calibration methods for spaceborne sensors. Meanwhile, the data for our method are from two different sensors, one line array camera, and one LiDAR, with different sampling rates and imaging properties, therefore, the existing relative pose calibration methods are hard to apply to the situation we are concerned with. However, the camera image and LiDAR in this section are both simulated by area array sensors, the calibration methods for area array cameras can be used to confirm the effect of the proposed approach. A traditional calibration method for area array cameras, efficient perspective-n-point (EPnP) method [31], is presented using the original area array image and point cloud data before stitching, where six points for calibration and three points for accuracy verification are selected manually.

The solution results are shown in

Table 5. Horizontal positioning error before and after calibration (m).

	X Direction			Y Direction
	Min	Max	Mean	Min	Max	Mean
Before Calibration	0.0155	0.0236	0.0197	0.0470	0.0529	0.0495
Proposed Method	0.0000	0.0052	0.0021	0.0000	0.0029	0.0013
EPnP Method	0.0001	0.0188	0.0091	0.0052	0.1039	0.0440

.

As shown in the above results, the line array camera and LiDAR have a higher joint ground positioning accuracy after the correction through the joint calibration algorithm proposed in this paper. The average error of total geo-positioning is about 0.0024 m according to Table 5, which represents that the calculation error using the proposed method is about 48 m in real satellites situation. The error is affected by hardware accuracy, for example, the measurement error of the ToF camera is several kilometers after proportional conversion to the situation of the real satellite.

Compared with EPnP, the proposed method has higher geo-positioning accuracy. The reasons are as follows:

• The proposed method has a higher accuracy of matching between the camera image and LiDAR data compared to manual matching.
• The proposed method has stronger fault tolerance to the error of distance measurements of points on the ground.
• The EPnP method is based on the coordinates of points, and its error will decrease as the number of points increases. However, in the hardware-in-loop experiment, it is hard to increase the number of points for EPnP because the 3D coordinates of the points are required in the method, and the measurement of the 3D coordinates is less efficient, which is different from the proposed method.

The simulation results show that the proposed method is efficient and the accuracy is about level with traditional methods, which is promising for the satellite equipped with the line array camera and LiDAR.

4. Discussion

A novel on-orbit relative pose calibration method for the spaceborne line array camera and LiDAR is proposed in this paper, including a joint calibration model and the calculation method for the deficient-rank model. Compared with other methods mentioned in Section 1, the proposed method has a higher temporal and spatial flexibility and application value. However, there are few studies on the case we are interested in, therefore, some relevant parameters are not considered in the modeling process and several limitations are exposed in the numerical validation and simulation experiment. The advantages and limitations of the presented method are discussed in this section.

• The relative installation parameters of the line array camera and LiDAR on the satellite are calibrated with ground features in this paper. Thus, the proposed method does not need any additional control points on the ground.
• During the calibration procedure, there is no need to maneuver the satellite, which simplified the calibration steps.
• The feature-point searching from the multi-source images is avoided due to the mutual information matching method for the camera image and the LiDAR image.
• The difficulty in solving the deficient-rank equation is ingeniously overcome by the alternating iterative method. Moreover, the convergence rate of the alternating iterative method is fast and the constraint that the on-orbit shifting angles are all small angles is introduced in the calculation step.
• The results of the numerical validation and hardware-in-loop experiment show that the proposed joint calibration method is effective for the spaceborne line array camera and LiDAR in the cases considered in Section 3.1 and in the indoor simulated scene.
• The EPnP method is one of the most popular pose estimation methods for area array cameras with representative results. Due to the differences between the proposed method and other calibration methods for spaceborne sensors, the comparison is difficult to perform. Instead, the EPnP method is used for the data from the hardware-in-loop experiment and its results are compared with the results of our approach in Section 3.2.2. The comparison result shows that the proposed method has higher geo-positioning accuracy compared with the EPnP method in the indoor simulated scene.
• The influence of the satellite attitude error is not considered in the calibration model, which is a key factor in the calibration effect and our next research focus.
• The accuracy of distance $\rho$ in Equations (2) and (3) is affected by many factors, such as atmospheric refraction and the reflectivity of ground, which affects the calibration accuracy. The detail was not talked about in the paper yet, but it is the focus of our next study.
• The mean error of horizontal positioning of the camera and LiDAR is used as the criterion for the accuracy of the proposed method because it is an important factor in optical image and LiDAR data fusion. Meanwhile, the vertical positioning error calculated by the proposed method is inconsistent with the actual situation, because the camera and LiDAR vertical positioning are both mainly determined by the laser ranging data and the vertical positioning error is always small in this way.
• The results of numerical validation show that the error of horizontal positioning will be less than 0.8 m when the parameters are well set and the measurement errors are in the reasonable range. However, the result of the hardware-in-loop experiment shows that the calculation error is about 48 m in real satellite situations. Two reasons are listed subsequently:

  (1)

  To focus on the calibration performance of the proposed method on the relative pose of the camera and LiDAR, the numerical validation is performed in the ideal situation, that is, measurement errors of parameters and observations are not considered except for the aspects which are listed in Section 3.1. Therefore, the calibration errors of numerical validation are smaller than in the real situation.

  (2)

  The hardware-in-loop simulation is a scaled-down experiment. When the solution is zoomed to a normal scale, the measurement errors of parameters and observations and their effects are amplified to unreasonable ranges. The calculation errors are much larger than in the real situation in this way.

• Since there is almost no operational satellite simultaneously equipped with the line array camera and LiDAR and relevant actual data, the proposed method is verified by numerical validation and simulation experiments rather than real remote sensing data. After the satellite is in operation, the data will be used to further verify the accuracy and reliability of the proposed method, and the method will be modified based on the real data.

5. Conclusions

In this paper, we established a novel joint calibration model with a spaceborne line array camera and a LiDAR. The results of numerical validation and simulation experiments show that the proposed method is reliable and effective. When the operation of satellites and measuring equipment are under normal conditions, the horizontal positioning mean error of the two sensors to the ground is less than 0.8 m after correction in the numerical validation, and the accuracy is better than the accuracy of the calibration method for spaceborne optical footprint camera with about 2 m ground positioning error [32]. Meanwhile, we explored the limits and scalability of the proposed approach in the hardware-in-loop simulation.