Extrinsic Parameter Calibration Method for a Visual/Inertial Integrated System with a Predefined Mechanical Interface

Abstract

For a visual/inertial integrated system, the calibration of extrinsic parameters plays a crucial role in ensuring accurate navigation and measurement. In this work, a novel extrinsic parameter calibration method is developed based on the geometrical constraints in the object space and is implemented by manual swing. The camera and IMU frames are aligned to the system body frame, which is predefined by the mechanical interface. With a swinging motion, the fixed checkerboard provides constraints for calibrating the extrinsic parameters of the camera, whereas angular velocity and acceleration provides constraints for calibrating the extrinsic parameters of the IMU. We exploit the complementary nature of both the camera and IMU, of which the latter assists in the checkerboard corner detection and correction while the former suppresses the effects of IMU drift. The results of the calibration experiment reveal that the extrinsic parameter accuracy reaches 0.04° for each Euler angle and 0.15 mm for each position vector component (1σ).

1. Introduction

Visual/inertial integrated systems have been used across many contexts, including indoor [1,2,3], underwater [4], space environments [5], and taking some measurement tasks [6,7]. To achieve accurate navigation and measurement, camera and inertial sensor frames should be aligned to the carrier frame in a process often called extrinsic parameter calibration or alignment. In this process, a coordinate transforming relationship is established between the sensor frame and the system body frame. Without calibrating the extrinsic parameters, the navigation errors will be coupled with the misalignment errors [8].

Previous studies have developed theoretical models of the camera [9] and the inertial measurement unit (IMU) [10,11,12], based on which the extrinsic parameters of visual/inertial integrated systems have been calibrated. These parameters are generally calibrated in three ways. First, these parameters are calibrated based on the corresponding rotation differences. Given that both the camera and IMU can evaluate their own rotations, a calibration method that minimizes the overall matching error of the rotation matrix between the visual and inertial coordinate systems in multiple rotations has been proposed in [13], whereas You et al. [14] developed a calibration method that relies on the angular velocity differences between the IMU and camera in the subsequent rotations. Second, these parameters are calibrated based on the vertical direction constraint. Given that the gravity and vertical line are measured by an IMU and camera in the same direction (i.e., vertical direction), a calibration method that uses this direction as reference has been proposed in [15,16,17]. Third, these parameters are calibrated based on filtering or optimization. In vision-aided inertial navigation, the coordinate transforming relation between the IMU and camera is estimated. The methods developed by Mourikis et al. [18] and Kelly and Sukhatme [19] estimate the extrinsic parameters based on the extended Kalman filter (EKF) and the unscented Kalman filter, respectively. In addition, Kaminer et al. [20] proposed a method based on nonlinear, globally stable filters, while Yang and Shen [21] proposed an optimization-based calibration method.

Most of these methods align the camera frame to the IMU frame. However, in some cases where a high-precision operation is required, the visual/inertial system should be aligned to the system body frame and not to the IMU frame. In addition, when a carrier, such as a spacecraft, is too heavy to conduct a calibration operation, the before mentioned methods cannot fully solve the problem. Previous studies have attempted to address this problem by aligning the sensors frame to the system body frame. For instance, Wendel and Underwood [22] aligned the line scanning cameras to the ground vehicle frame and achieved an accuracy of 0.06 m in translation and 1.05° in rotation, while Shi et al. [23] aligned three cameras to the body system frame and achieved 0.6 mm and 0.1° translation and rotation accuracies, respectively. Only the camera frame is aligned to the body frame and the accuracy can be further improved. Given that self-integrated and some commercial IMUs lack a mechanical interface. Foxlin and Naimark [24] aligned the IMU to the system body frame defined by a mechanical interface by rolling on a flat surface and then aligned the camera to the system body frame by using a calibration device. However, the accuracy of the extrinsic parameters is not clearly shown. Pittelkau [25] proposed a method for aligning an inertial sensor assembly of three fiber-optic gyros, two star trackers, and a camera to a spacecraft by using an alignment Kalman filter, but this method has only been validated via simulation.

We present an extrinsic parameter calibration method that uses IMU and camera measurements to align the IMU and camera frames to the system body frame defined by the mechanical interface. This method offers two advantages. First, the sensor frames are aligned to the carrier frame through the mechanical interface without requiring a complex calibration process. Second, this method can be seen as a standard calibration step for factory production and user operation. The operation process of calibration is not complex. Fix a checkerboard on the wall within the field of view of the camera, fix the integrated system on a two-axis turntable, and then manually control this turntable to swing around its two axes. The checkerboard is in the camera’s field of view all the time.

Our novelties are that,

• In this study, we exploit the complementary nature of IMU and camera to improve the calibration accuracy. The camera measurements can suppress the inertial propagation drift, whereas the motion parameters evaluated by IMU can accurately extract the feature points under a smearing effect. We exploit the complementary nature of these two components to improve the calibration accuracy.
• The method for IMU aided checkerboard corner correction under motion blur is introduced, which is the tip step to improve accuracy of camera calibration. We use the IMU to evaluate the motion parameters, and use these motion parameters to eliminate the effects of motion blur in image. It is described in Section 2.3. The extrinsic parameters are evaluated based on EKF, and the rotation angle is evaluated using camera measurement to suppress IMU’s drift.
• The calibration process can be a standard calibration step for factory production and user operation. The turntable is the standard equipment for IMU calibration, and the checkerboard is the standard device for camera calibration. So, the cost is low, and the calibration process is simple and convenient. Our experiment results indicate that our proposed method is valid and achieves a fair level of accuracy. Our method can also align the camera frame to the IMU frame.

The rest of this paper is organized as follows. Section 2 introduces the mathematical model and the proposed calibration method. Section 3 presents the simulation and the real-world experiment. Section 4 concludes the paper.

2. Calibration Method

The main idea of the proposed calibration method is shown in Figure 1, which can help understand the part of the calibration. There are four step of this calibration. Firstly, “Preparation”, the following preparations should be performed before the system calibration:

• The checkerboard should be fixed in an appropriate location that can be observed by the cameras while the system rotates along with the turntable;
• The intrinsic parameters of the camera and IMU should be calibrated;
• The visual/inertial system should be fixed on the turntable.

Secondly, “data acquisition”, the turntable is controlled manually to rotate around two orthogonal axes, such as the x and y axes, for the checkerboard image acquisition by camera and the inertial data acquisition by IMU, respectively.

Thirdly, “blur correction”, the motion blur exits in the checkerboard image. The checkerboard corners are detected with the aid of IMU to achieve higher accuracy.

Finally, “estimator”, we estimate the extrinsic parameters based on EKF.

The main idea of this calibration method is that establish the equations that provide the extrinsic parameters and solve these equations. To better understand the method, the basic measurement models of the IMU and camera are described in the Section 2.1. Because the calibration operation process is swinging motion, the detailed measurement equations of IMU and camera under swinging motion are described in Section 2.2. We exploit the complementary nature of IMU and camera to improve the calibration accuracy, the method of inertial aided checkerboard corner extraction under motion blur is described in Section 2.3. And the extrinsic parameters are evaluated based on EKF, which is shown in Section 2.4. The rotation angle of turntable is evaluated by camera measurement to suppress the IMU’s drift.

2.1. Measurement Model

The measurement model for IMU is formulated as follows (The superscripts and subscripts F, B, I, and V represent the frame of checkerboard, system body, IMU, and camera, respectively, while F₀, B₀, I₀, and V₀ represent the frame fixed on the earth and coincident to the frame of checkerboard, system body, IMU, and camera in its initial pose, respectively).

$$\begin{array}{l}{f}_{meas}^I={\mathit{a}}_{true}^I-{\mathit{g}}_{}^I+{\mathit{b}}_a+{\mathit{n}}_a\\ {\mathit{\omega }}_{meas}^I={\mathit{\omega }}_{true}^I+{\mathit{b}}_g+{\mathit{n}}_g\end{array},$$

(1)

where ${\mathit{a}}_{true}^I$ is the true specific force excluding gravity, ${\mathit{\omega }}_{true}^I$ is the true angular velocity, ${\mathit{g}}_{}^I$ is the gravity expressed in the IMU frame, ${\mathit{b}}_a$ and ${\mathit{b}}_g$ are the bias errors of the accelerometer and gyro, respectively, and ${\mathit{n}}_a$ and ${\mathit{n}}_g$ are the process white Gaussian noises of the accelerometer and gyro, respectively. Both ${\mathit{b}}_a$ and ${\mathit{b}}_g$ show minimal changes within a short period.

We use the ideal pinhole model as the measurement model for the camera that maps a 3D point ${\mathit{P}}^F\left(x^F,y^F,z^F\right)$ to a 2D image $p\left(u,v\right)$ as [26]

$$z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{cccc}{\alpha }_x& \gamma & u_0& 0\\ 0& {\alpha }_y& v_0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}{\mathit{C}}_F^V& {\mathit{t}}_{FV}^V\\ 0& 1\end{array}\right]\left[\begin{array}{c}{\mathit{P}}_{}^F\\ 1\end{array}\right]=\left[\begin{array}{cc}M& 0\end{array}\right]\left[\begin{array}{cc}{\mathit{C}}_F^V& {\mathit{t}}_{FV}^V\\ 0& 1\end{array}\right]\left[\begin{array}{c}{\mathit{P}}_{}^F\\ 1\end{array}\right],$$

(2)

where $z_c$ is the camera frame optic axis coordinate of point $P$, ${\alpha }_x$, and ${\alpha }_y$ are the scale factors of the u and v axes of the image plane, respectively, $\gamma$ is the non-orthogonal factor of the image plane axes, $\left(u_0^{},v_0^{}\right)$ is the pixel coordinate of the camera principal point, ${\mathit{C}}_F^V$ is the 3 × 3 rotation matrix, ${\mathit{t}}_{FV}^V$ is the 3D translation vector, and $M$ is the intrinsic parameter matrix. In the actual situation, the principal point, focal length deviation, distortion, and other error factors should be considered based on the ideal camera model. These parameters can be acquired via an intrinsic calibration, such as by employing Zhang’s method [27].

2.2. Extrinsic Parameter Calibration Method for a Visual/Inertial Integrated System

The physical quantities that should be calibrated include the following:

• The translation matrices ${\mathit{C}}_B^V$ and ${\mathit{C}}_B^I$, which occur between the system body frame and the camera frame as well as between the system body frame and the IMU frame, respectively.
• The position vectors ${\mathit{t}}_{BV}^B$ and ${\mathit{t}}_{BI}^B$, which are derived from the camera and IMU principal points to the origin of the system body frame, respectively.

Given that IMU can measure angular velocity and acceleration vectors during rotation, the constraint equations of inertial vectors and the extrinsic parameters of IMU can be established. When the turntable rotates around by one axis, the position vector and Euler angle along the rotation axis cannot be computed. However, when the turntable rotates by around two orthogonal axes, extrinsic parameters of IMU can be computed. Fortunately, IMU calibration is generally performed by using a turntable with two or three orthogonal axes.

Camera calibration generally involves the use of a checkerboard. In the calibration process, both the intrinsic parameters and the translation matrix and position vector between the camera and checkerboard frames can be computed. By turning the turntable, the constraint equation can be established based on the relationship between the extrinsic parameters of camera and the change in the camera position and attitude. The constraint equations of the visual/inertial system are described as follows.

In general, when the Coriolis acceleration is ignored during rotation, the accelerometer measurement is

$$\begin{array}{c}{\mathit{f}}_{}^I={\mathit{a}}_n^I+{\mathit{a}}_{\tau }^I+{\mathit{g}}_{}^I+{\mathit{b}}_a^{}\\ ={\mathit{C}}_B^I\left(a_n^B+a_{\tau }^B+g_{}^B\right)+{\mathit{b}}_a^{}\end{array},$$

(3)

where ${\mathit{a}}_n^B={\left[{\Omega }_{}^B\right]}^2{\mathit{t}}_{BI}^B$ is the radial acceleration, ${\mathit{a}}_{\tau }^B={\mathit{E}}_{}^B{\mathit{t}}_{BI}^B$ is the tangential acceleration, ${\mathit{g}}_{}^I$ is the gravitational acceleration expressed in the IMU frame, and ${\mathit{C}}_B^I$ is the translation matrix from the system body frame to the IMU frame, which can be calculated by the Euler angle ${\mathit{\psi }}_B^I$.

${\mathit{\Omega }}_{}^B$ and ${\mathit{E}}_{}^B$ are skew-symmetric matrices defined as

$${\mathit{\Omega }}_{}^B=\left[{\mathit{\omega }}_{}^B\times \right]=\left[\begin{array}{ccc}0& -{\omega }_z^{}& {\omega }_y^{}\\ {\omega }_z^{}& 0& -{\omega }_x^{}\\ -{\omega }_y^{}& {\omega }_x^{}& 0\end{array}\right], {\mathit{E}}_{}^B=\left[{\mathit{\epsilon }}_{}^B\times \right]=\left[\begin{array}{ccc}0& -{\epsilon }_z^{}& {\epsilon }_y^{}\\ {\epsilon }_z^{}& 0& -{\epsilon }_x^{}\\ -{\epsilon }_y^{}& {\epsilon }_x^{}& 0\end{array}\right],$$

(4)

where ${\mathit{\omega }}_{}^B={\left[\begin{array}{ccc}{\omega }_x& {\omega }_y& {\omega }_z\end{array}\right]}^T$ is the angular velocity vector, and ${\mathit{\epsilon }}_{}^B={\left[\begin{array}{ccc}{\epsilon }_x& {\epsilon }_y& {\epsilon }_z\end{array}\right]}^T$ is the angular acceleration vector.

Therefore, the accelerometer measurement can be rewritten as

$$\begin{array}{ll}\hfill {\mathit{f}}^I& ={\mathit{C}}_B^I\left(a_n^B+a_{\tau }^B+g_{}^B\right)+{\mathit{b}}_a^{}\\ & ={\mathit{C}}_B^I{\mathit{C}}_{B_0}^B\left[\left({\left({\Omega }_{}^{B_0}\right)}^2+{\mathit{E}}_{}^{B_0}\right)t_{B_0{I}_0}^{B_0}+C_{I_0}^{B_0}{g}_{}^{I_0}\right]+{\mathit{b}}_a\hfill \\ & ={\mathit{C}}_B^I{\mathit{C}}_{B_0}^B\left[\left({\left({\Omega }_{}^{B_0}\right)}^2+{\mathit{E}}_{}^{B_0}\right)t_{BI}^B+C_I^B{g}_{}^{I_0}\right]+{\mathit{b}}_a\end{array},$$

(5)

where ${\mathit{g}}_{}^{I_0}$ is the gravity accelerometer expressed in the {$I_0$} frame and can be written as

$${\mathit{g}}_{}^{I_0}={\mathit{f}}_{}^{I_0}-{\mathit{b}}_a^{},$$

(6)

where ${\mathit{f}}_{}^{I_0}$ is the accelerometer measurement in the initial pose. Therefore, ${\mathit{b}}_a^{}$ must be initially evaluated to determine ${\mathit{g}}_{}^{I_0}$. The gyro measurement is formulated as

$${\mathit{\omega }}_{}^I={\mathit{C}}_B^I{\mathit{\omega }}_{}^B={\mathit{C}}_B^I{\mathit{C}}_{B_0}^B{\mathit{\omega }}_{}^{B_0}.$$

(7)

An integrated system rotates with a two-axis turntable. When turntable rotates around the x axis, we have

$${\mathit{C}}_{B_0}^B=\left[\begin{array}{ccc}1& & \\ & \cos {\beta }_x& \sin {\beta }_x\\ & -\sin {\beta }_x& \cos {\beta }_x\end{array}\right], {\mathit{f}}_x^I={\mathit{f}}_{}^I, {\mathit{\omega }}_x^I={\mathit{\omega }}_{}^I,$$

(8)

where ${\beta }_x$ is the rotation angle of the turntable when rotating around the x axis.

However, when rotating around the y axis, we have

$${\mathit{C}}_{B_0}^B=\left[\begin{array}{ccc}\cos {\beta }_y& & \sin {\beta }_y\\ & 1& \\ -\sin {\beta }_y& & \cos {\beta }_y\end{array}\right], {\mathit{f}}_y^I={\mathit{f}}_{}^I, {\mathit{\omega }}_y^I={\mathit{\omega }}_{}^I,$$

(9)

where ${\beta }_y$ is the rotation angle of the turntable when rotating around the y axis.

${\beta }_k^{}$, $k=x,y$, can be computed by integrating the gyro measurement, but an accumulation error is observed. This error is then evaluated based on the camera measurement in the Kalman process.

The camera measurement can be formulated as

$$\begin{array}{ll}\hfill z& ={\left[\begin{array}{cccc}{u}_x^{}& v_x^{}& u_y^{}& v_y^{}\end{array}\right]}^T\\ \left[\begin{array}{ll}\hfill u_x^{}& v_x^{}\end{array}\right]& =\left[\begin{array}{llllll}\hfill u_{x,1}& \cdots & u_{x,n}& v_{x,1}& \cdots & v_{x,n}\end{array}\right]\\ \left[\begin{array}{cc}{u}_y^{}& v_y^{}\end{array}\right]& =\left[\begin{array}{cccccc}{u}_{y,1}& \cdots & u_{y,n}& v_{y,1}& \cdots & v_{y,n}\end{array}\right]\end{array},$$

(10)

where subscripts x and y indicate that the turntable rotates around the x and y axes, respectively, $\left(u_i^{},v_i^{}\right)$ is the pixel coordinate of each checkerboard corner, and n is the number of checkerboard corners.

The value of $\left(u_i^{},v_i^{}\right)$ can be computed as

$$\begin{array}{l}\left[\begin{array}{c}{u}_i^{}\\ v_i^{}\end{array}\right]=z_i^{}\left[\begin{array}{c}{x}_i^{}\\ y_i^{}\end{array}\right], \mathrm{where}\left[\begin{array}{c}{x}_i^{}\\ y_i^{}\\ z_i^{}\end{array}\right]=M\left[\begin{array}{c}{x}_i^V\\ y_i^V\\ z_i^V\end{array}\right],\left[\begin{array}{c}{x}_i^V\\ y_i^V\\ z_i^V\end{array}\right]={\mathit{P}}_i^V={\mathit{C}}_{V_0^{}}^{V_{}^{}}{\mathit{P}}_i^{V_0^{}}-{\mathit{t}}_{V_0^{}{V}_{}^{}}^{V_{}^{}}\end{array},$$

(11)

$\left(u_i^V,v_i^V\right)$ can be defined as

$$\left[\begin{array}{c}{u}_i^V\\ v_i^V\end{array}\right]=z_i^V\left[\begin{array}{c}{x}_i^V\\ y_i^V\end{array}\right],$$

(12)

and ${\mathit{P}}_i^{V_0^{}}$ and ${\mathit{t}}_{V_0^{}V}^V$ in Equation (11) can be written as

$$\begin{array}{ll}\hfill {\mathit{P}}_i^{V_0^{}}& ={\mathit{C}}_F^{V_0^{}}{\mathit{P}}_i^F+{\mathit{t}}_{V_0^{}F}^{V_0^{}}\\ {\mathit{t}}_{V_0^{}{V}_{}^{}}^{V_{}^{}}& ={\mathit{C}}_B^V\left(t_{BV}^B-C_{B_0}^B{t}_{BV}^B\right)\end{array},$$

(13)

where ${\mathit{P}}_i^F$ is the 3D coordinate of the ith checkerboard corner in the checkerboard frame. In Equations (8) and (9), ${\mathit{C}}_{B_0}^B$ can be computed based on the rotational angle of the turntable and correspond to the rotation of the turntable around the x and y axes, respectively.

Meanwhile, ${\mathit{P}}_i^F$ is determined by the user, and $\left(u_i^{},v_i^{}\right)$ denotes the pixel coordinate of the checkerboard corner in an image. If the number of checkerboard corners is >8, then ${\mathit{C}}_F^{V_0^{}}$ and ${\mathit{t}}_{V_0^{}F}^{V_0^{}}$ can be evaluated by using least squares.

2.3. Checkerboard Corner Extraction Method under Motion Blur

During the working process of a digital camera, the shutter needs to open for a moment to project light onto the photographic material. This brief moment is called the exposure time. Under highly dynamic conditions, the relative pose between the camera and the object changes evidently during the exposure time, thereby blurring or stretching the generated image [28]. The calibration of the extrinsic parameters requires a rotation, especially for IMU. Therefore, for the camera measurements, we must extract the checkerboard corners during rotation, but a motion blur may be generated in the process. Conventional checkerboard corner detection methods compute the local optimum value [29] with static images. Therefore, when noise and motion blur are present in an image, the errors in the extraction results evidently increase.

We utilize inertial data to eliminate the smearing effect and then extract the checkerboard corners with the aid of the IMU data. The corner detection algorithm is developed as follows. The Lucy–Richardson method is used to rectify the blurred image in step A, and the conventional corner detection method is used to find the corners’ pixel coordinates in step B. Given that checkerboard corners are constrained on a few lines, we add a linear constraint to refine the checkerboard corner location. Through the linear constraint, the corners’ pixel coordinates are refined in step C. Finally, IMU measurement is used to correct the position of corner detected for movement in step D.

• A. Pretreatment and smearing effect elimination.

We deblur the image using the Lucy–Richardson method [30,31,32]. The point–spread function (PSF) is a 2D Gaussian model $N\left(0,0,{\sigma }_x^2,{\sigma }_y^2\right)$, and ${\sigma }_x^{}$ and ${\sigma }_y^{}$ denote the standard deviations in the x and y axes, respectively. If the velocity of the checkerboard expressed in the camera frame coincides with the x axis, then a motion blur is only observed in the x axis. Therefore, the PSF can be written as $N\left(0,0,{\sigma }_{}^2,0\right)$, and the corresponding covariance matrix is

$$C=\left[\begin{array}{cc}{\sigma }^2& \\ & 0\end{array}\right],$$

(14)

where ${\sigma }^2=\eta {\omega }_{}^B$, $\eta$ is the scale factor, and ${\omega }_{}^B$ is the angular velocity of rotation.

When the cross angle between the velocity of the checkerboard is expressed in the camera frame and the x axis of the camera frame is $\theta$, the covariance matrix of PSF can be written as

$${\mathit{C}}_{\theta }^{}=R\left(\theta \right)CR{\left(\theta \right)}^T,$$

(15)

where $R\left(\theta \right)=\left[\begin{array}{cc}\cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{array}\right]$. $\theta$ can be evaluated by camera data.

Figure 2 compares the original and deblurred images and shows that the motion blur has been effectively eliminated.

• B. Corner detection.

This step is the same as that in the conventional method. The corners in the image, including the complete checkerboard, are examined. $I_x$ and $I_y$ denote the x (horizontal) and y (vertical) components of the 2D numerical gradient, and the first and second derivatives are computed as

$$\begin{array}{c}{I}_x=\frac{\partial I}{\partial x}, I_y=\frac{\partial I}{\partial y}, I_{xy}=\frac{\partial I}{\partial x\partial y}\\ I_{45}=I_x\cos \left(\frac{\pi }{4}\right)+I_y\sin \left(\frac{\pi }{4}\right), I_{-45}=I_x\cos \left(-\frac{\pi }{4}\right)+I_y\sin \left(-\frac{\pi }{4}\right)\\ I_{45,x}=\frac{\partial I_{45}}{\partial x}, I_{45,y}=\frac{\partial I_{45}}{\partial y}, I_{45,45}=I_{45,x}\cos \left(-\frac{\pi }{4}\right)+I_{45,y}\sin \left(-\frac{\pi }{4}\right)\end{array}.$$

(16)

The gradient direction is evaluated by

$$C_{xy}=\left|I_{xy}\right|-\mu \left(\left|I_{45}\right|+\left|I_{-45}\right|\right), C_{45}=\left|I_{45,45}\right|-\mu \left(\left|I_x\right|+\left|I_y\right|\right),$$

(17)

where $\mu$ is the weight coefficient.

$C_{xy}$ and $C_{45}$ are computed for each pixel in the picture. Afterward, $C_{xy}$ and $C_{45}$ are determined—one of which is valid—by minimizing the energy function proposed in [29]. Based on the threshold set in advance and non-maxima suppression, the detection accuracy reaches the pixel level. The results are presented in Figure 3a.

• C. Linear constraint refinement.

We can evaluate the lines of checkerboard based on the corners by using least squares. We assume that the line equation is $a_{i}x+b_{j}y+c_{ij}=0$ and that i and j represent the row and column numbers, respectively.

The cross points of these lines are then computed for the global optimization of the checkerboard corners (Figure 3b; refer to the yellow ”+”). Figure 3c compares the results before and after refinement, respectively. The error of some corners detected before the refinement has remarkably increased due to motion blur.

• D. IMU-aided checkerboard corner modification

At time t, camera and IMU start collecting data simultaneously. The corners extracted have some delay due to the motion blur in the image, with real data at time $t+\tau /2$. $\tau$ is the exposure time of camera. We can modify the corner coordinates with the inertial data. The details are presented as follows.

We consider the motion of camera in $\Delta t$, and point P in the camera frame can be written as

$${\mathit{P}}_t^{V_t}={\mathit{C}}_F^{V_t}\left({\mathit{P}}_t^F+{\mathit{t}}_{F{V}_t}^F\right) \mathrm{and}$$

(18)

$$\begin{array}{ll}\hfill {\mathit{P}}_{t+\Delta t}^{V_{t+\Delta t}}& ={\mathit{C}}_F^{V_{t+\Delta t}}\left(P_t^F+t_{F{V}_{t+\Delta t}}^F\right)\\ & ={\mathit{C}}_{V_t}^{V_{t+\Delta t}}{\mathit{C}}_F^{V_t}\left(P_t^F+t_{F{V}_t}^F+t_{V_t{V}_{t+\Delta t}}^F\right)\\ & ={\mathit{C}}_{V_t}^{V_{t+\Delta t}}{\mathit{P}}_t^{V_t}+{\mathit{C}}_{V_t}^{V_{t+\Delta t}}{\mathit{t}}_{V_t{V}_{t+\Delta t}}^{V_t}\end{array},$$

(19)

where

$$\begin{array}{l}{\mathit{C}}_{V_t}^{V_{t+\Delta t}}=I-{\mathit{\Omega }}_t^B\Delta t=\left[\begin{array}{ccc}1& {\omega }_z^t\Delta t& -{\omega }_y^t\Delta t\\ -{\omega }_z^t\Delta t& 1& {\omega }_x^t\Delta t\\ {\omega }_y^t\Delta t& -{\omega }_y^t\Delta t& 1\end{array}\right]\\ {\mathit{t}}_{V_t{V}_{t+\Delta t}}^{V_t}={\mathit{t}}_{V_t{R}_t}^{V_t}-{\mathit{C}}_{V_{t+\Delta t}}^{V_t}{\mathit{t}}_{V_{t+\Delta t}{R}_t}^{V_{t+\Delta t}}=\left(I-C_{V_{t+\Delta t}}^{V_t}\right){\mathit{t}}_{V_t{R}_t}^{V_t}=-{\mathit{\Omega }}_t^B{\mathit{t}}_{V_t{R}_t}^{V_t}\Delta t=-{\mathit{\Omega }}_t^B{\mathit{t}}_{VR}^V\Delta t\end{array},$$

(20)

where ${\mathit{\Omega }}_t^B$ is ${\mathit{\Omega }}_{}^B=\left[{\mathit{\omega }}_{}^B\times \right]$ at time t.

Equation (19) can be written as

$$\begin{array}{ll}\hfill {\mathit{P}}_{t+\Delta t}^{V_t}& =\left(I-{\Omega }_t^B\Delta t\right){\mathit{P}}_t^{V_t}-\left(I-{\Omega }_t^B\Delta t\right){\mathit{\Omega }}_t^B{\mathit{t}}_{V_t{R}_t}^{V_t}\Delta t\\ & \approx \left(I-{\Omega }_t^B\Delta t\right){\mathit{P}}_t^{V_t}-{\mathit{\Omega }}_t^B{\mathit{t}}_{VR}^V\Delta t\end{array}.$$

(21)

${\mathit{t}}_{VR}^V={\left[\begin{array}{ccc}{t}_x^{}& t_y^{}& t_z^{}\end{array}\right]}^T$ is defined, and Equation (21) is expanded as

$${\mathit{P}}_{t+\Delta t}^{V_t}=\left[\begin{array}{ccc}1& {\omega }_z^t\Delta t& -{\omega }_y^t\Delta t\\ -{\omega }_z^t\Delta t& 1& {\omega }_x^t\Delta t\\ {\omega }_y^t\Delta t& -{\omega }_x^t\Delta t& 1\end{array}\right]\left[\begin{array}{c}{x}_t^{V_t}\\ y_t^{V_t}\\ z_t^{V_t}\end{array}\right]+\left[\begin{array}{ccc}0& {\omega }_z^t\Delta t& -{\omega }_y^t\Delta t\\ -{\omega }_z^t\Delta t& 0& {\omega }_x^t\Delta t\\ {\omega }_y^t\Delta t& -{\omega }_x^t\Delta t& 0\end{array}\right]\left[\begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\right].$$

(22)

By substituting Equation (22) into Equation (12), we have

$$\begin{array}{ll}\hfill \left[\begin{array}{c}{u}_{t+\Delta t}^V\\ v_{t+\Delta t}^V\end{array}\right]& =\left[\begin{array}{c}\frac{x_t^{V_t}+y_t^{V_t}{\omega }_z^t\Delta t-z_t^{V_t}{\omega }_y^t\Delta t+t_y{\omega }_z^t\Delta t-{\omega }_y^t{t}_z\Delta t}{x_t^{V_t}{\omega }_y^t\Delta t-y_t^{V_t}{\omega }_x^t\Delta t+z_t^{V_t}+t_x{\omega }_y^t\Delta t-t_y{\omega }_x^t\Delta t}\\ \frac{-x_t^{V_t}{\omega }_z^t\Delta t+y_t^{V_t}+z_t^{V_t}{\omega }_x^t\Delta t-t_x{\omega }_z^t\Delta t+t_z{\omega }_x^t\Delta t}{x_t^{V_t}{\omega }_y^t\Delta t-y_t^{V_t}{\omega }_x^t\Delta t+z_t^{V_t}+t_x{\omega }_y^t\Delta t-t_y{\omega }_x^t\Delta t}\end{array}\right]\\ & =\left[\begin{array}{c}\frac{u_t^V+v_t^V{\omega }_z^t\Delta t-{\omega }_y^t\Delta t+\left(t_y{\omega }_z^t\Delta t-{\omega }_y^t{t}_z\Delta t\right)/z_t^{V_t}}{u_t^V{\omega }_y^t\Delta t-v_t^V{\omega }_x^t\Delta t+\left(t_x{\omega }_y^t\Delta t-t_y{\omega }_x^t\Delta t\right)/z_t^{V_t}+1}\\ \frac{-u_t^V{\omega }_z^t\Delta t+v_t^V+{\omega }_x^t\Delta t+\left(-t_x{\omega }_z^t\Delta t+t_z{\omega }_x^t\Delta t\right)/z_t^{V_t}}{u_t^V{\omega }_y^t\Delta t-v_t^V{\omega }_x^t\Delta t+\left(t_x{\omega }_y^t\Delta t-t_y{\omega }_x^t\Delta t\right)/z_t^{V_t}+1}\end{array}\right]\end{array}.$$

(23)

Given that $u_t^V{\omega }_y^t\Delta t-v_t^V{\omega }_x^t\Delta t+\left(t_x{\omega }_y^t\Delta t-t_y{\omega }_x^t\Delta t\right)/z_t^{V_t}\ll 1$ in this study, we have:

$$\begin{array}{ll}\hfill \left[\begin{array}{c}{u}_{t+\Delta t}^V\\ v_{t+\Delta t}^V\end{array}\right]& \approx \left[\begin{array}{c}{u}_t^V+v_t^V{\omega }_z^t\Delta t-{\omega }_y^t\Delta t+\left(t_y{\omega }_z^t\Delta t-{\omega }_y^t{t}_z\Delta t\right)/z_t^{V_t}\\ -u_t^V{\omega }_z^t\Delta t+v_t^V+{\omega }_x^t\Delta t+\left(-t_x{\omega }_z^t\Delta t+t_z{\omega }_x^t\Delta t\right)/z_t^{V_t}\end{array}\right]\\ & =\left[\begin{array}{c}{u}_t^V\\ v_t^V\end{array}\right]+\left[\begin{array}{c}{v}_t^V{\omega }_z^t-{\omega }_y^t+\left(t_y{\omega }_z^t-{\omega }_y^t{t}_z\right)/z_t^{V_t}\\ -u_t^V{\omega }_z^t+{\omega }_x^t+\left(-t_x{\omega }_z^t+t_z{\omega }_x^t\right)/z_t^{V_t}\end{array}\right]\Delta t\\ & =\left[\begin{array}{c}{u}_t^V\\ v_t^V\end{array}\right]+\left[\begin{array}{c}\lambda \\ \xi \end{array}\right]\Delta t\end{array}.$$

(24)

Moreover, given that $M=\left[\begin{array}{ccc}{\alpha }_x& & u_0\\ & {\alpha }_y& v_0\\ & & 1\end{array}\right]$, we have

$$\left[\begin{array}{c}{u}_t\\ v_t\\ 1\end{array}\right]=\left[\begin{array}{ccc}{\alpha }_x& & u_0\\ & {\alpha }_y& v_0\\ & & 1\end{array}\right]\left[\begin{array}{c}{u}_t^V\\ v_t^V\\ 1\end{array}\right]=\left[\begin{array}{c}{\alpha }_x{u}_t^V+u_0\\ {\alpha }_y{v}_t^V+v_0\\ 1\end{array}\right] \mathrm{and}$$

(25)

$$\left[\begin{array}{c}{u}_{t+\Delta t}\\ v_{t+\Delta t}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{\alpha }_x& & u_0\\ & {\alpha }_y& v_0\\ & & 1\end{array}\right]\left[\begin{array}{c}{u}_{t+\Delta t}^V\\ v_{t+\Delta t}^V\\ 1\end{array}\right]=\left[\begin{array}{c}{\alpha }_x{u}_{t+\Delta t}^V+u_0\\ {\alpha }_y{v}_{t+\Delta t}^V+v_0\\ 1\end{array}\right]=\left[\begin{array}{c}{\alpha }_x{u}_t^V+u_0+\lambda {\alpha }_x\Delta t\\ {\alpha }_y{v}_t^V+v_0+\xi {\alpha }_y\Delta t\\ 1\end{array}\right]=\left[\begin{array}{c}{u}_t\\ v_t\\ 1\end{array}\right]+\left[\begin{array}{c}\lambda {\alpha }_x\Delta t\\ \xi {\alpha }_y\Delta t\\ 1\end{array}\right].$$

(26)

When $\Delta t=-\tau /2$, we can modify the coordinates of the checkerboard corners, thereby eliminating the smearing effect.

The image process algorithm is summarized as

Table 1. Checkerboard Corner Detection under Motion Blur.

Inputs: Camera and IMU measurements

Output: Coordinates of checkerboard corners

Algorithm:

1. Deblur the image with the aid of inertial data.

2. Roughly extract the checkerboard corners based on the equation of the second gradient.

3. Refine the corners through the linear constraint.

4. Modify the checkerboard corners with IMU-aided.

.

The corner extraction accuracy can be significantly enhanced by using the above checkerboard corner detection algorithm, especially when a smearing effect exists.

2.4. Description of the Estimator

There are many methods that can solve the equations that provide the extrinsic parameters, such as EKF [33,34], genetic algorithm [35], and so on. Here, we utilize an EKF for calibrating the extrinsic parameters. The EKF algorithm is briefly introduced in

Table 2. EKF updating process.

1. State Equation

$\mathit{x}=\left[\begin{array}{c}{x}_I\\ x_V\end{array}\right]$, $x_I={\left[\begin{array}{cccc}{t}_{BI}^I& {\psi }_B^I& b_a& b_g\end{array}\right]}^T$, $x_V={\left[\begin{array}{ccccc}{t}_{BV}^V& {\psi }_B^V& t_{F{V}_0}^{V_0}& {\psi }_F^{V_0}& \delta {\beta }_R\end{array}\right]}^T$

2. Measurement Model

$\mathit{z}=\left[\begin{array}{c}{z}_I\\ z_V\end{array}\right]$, $z_I={\left[\begin{array}{cccc}{f}_x^I& {\omega }_x^I& f_y^I& {\omega }_y^I\end{array}\right]}^T$, $\mathit{z}={\left[\begin{array}{cccc}{u}_x^{}& v_x^{}& u_y^{}& v_y^{}\end{array}\right]}^T$

3. Updating

$$\begin{gathered} {\widehat{x}}_j^{-}={\Phi }_{j-1}^{}{\widehat{x}}_j^{+}, P_j^{-}={\Phi }_{j-1}^{}{P}_{j-1}^{+}{\Phi }_{j-1}^T+Q_{j-1}^{}, \mathit{\Phi }=\left[\begin{array}{cc}{\Phi }_I& 0_{12,14}\\ 0_{14,12}& {\Phi }_V\end{array}\right], \mathit{Q}=\left[\begin{array}{cc}{Q}_I& 0_{12,14}\\ 0_{14,12}& Q_V\end{array}\right], \\ {K}_j^{}=P_j^{-}{H}_j^{}{\left(H_j^{}{P}_j^{-}{H}_j^T+R_j^{}\right)}^{-1}, {\widehat{x}}_j^{+}=x_j^{-}+K_j^{}\delta z_j^{}, P_j^{+}=\left(I-K_j^{}{H}_j^{}\right)P_j^{-}, \mathit{P}=\left[\begin{array}{cc}{P}_I& P_{IV}\\ P_{IV}^T& P_V\end{array}\right] \end{gathered}$$

. The state vector $x$ includes the extrinsic parameters (translation vectors ${\mathit{t}}_{BI}^I$, ${\mathit{t}}_{BV}^V$ and Euler angle ${\mathit{\psi }}_B^I$, ${\mathit{\psi }}_B^V$), the IMU bias (${\mathit{b}}_a$ and ${\mathit{b}}_g$), the camera initial parameter (${\mathit{t}}_{F{V}_0}^{V_0}$ and ${\mathit{\psi }}_F^{V_0}$), and the rotation angle ($\delta {\mathit{\beta }}_R$). Meanwhile, the measurement vector includes the accelerometer (${\mathit{f}}_x^I$ and ${\mathit{f}}_y^I$) and gyro (${\mathit{\omega }}_x^I$ and ${\mathit{\omega }}_y^I$) measurements as well as the extracted checkerboard corners (${\mathit{u}}_x^{}$,${\mathit{v}}_x^{}$, ${\mathit{u}}_y^{}$, and ${\mathit{v}}_y^{}$).

3. Simulation and Real-World Experiment

3.1. Simulation

A simulation test is designed to validate the performance of the extrinsic parameter calibration method. During the simulation, the turntable is assumed to demonstrate swinging motions around the x and y axes. The swinging rule is ${\omega }_k=A_k\sin \left(2\pi f_k+{\phi }_k\right)+{\omega }_{k0}$, where $k=x,y$, ${\omega }_k$ denotes the angular velocity, $A_k$ and $f_k^{}$ denote the swinging amplitude and frequency, respectively, and ${\phi }_k$ and ${\omega }_{k0}$ denote the initial phase and swinging center, respectively. The swinging parameters of the simulation are defined in

Table 3. Swinging parameters.

Rotation Axis	X Axis	Y Axis
Amplitude (deg/s)	50	10
Frequency (Hz)	2	3
Initial phase (°)	90	90
Swinging center (deg/s)	0	0

.

The true values for simulation data of the visual/inertial integrated system are defined in

Table 4. The true values for simulation data.

Parameters	Value
Euler angle ${\psi }_B^I$ (°)	[−0.57, 0.57, 0.57]
Translation $t_{BI}^B$ (m)	[0.05, −0.05, 0.05]
Euler angle ${\psi }_B^V$ (°)	[0.57, −0.57, 0.57]
Translation $t_{BV}^B$ (m)	[0.10, 0.10, 0.10]
Euler angle ${\psi }_F^{V_0}$ (°)	[5.73, 5.73, 5.73]
Translation $t_F^{V_0}$ (m)	[0.10, 0.10, 1.10]
Gravity accelerometer $g_{}^{I_0}$ (m/s2)	[0.10, 9.70, 0.10]

.

Given the parameters in Table 3 and Table 4, the true measurement of the gyroscope and accelerometer can be simulated by using the dynamics equation. The true measurement of the camera (pixel coordinates of the checkerboard corners) can be simulated by using the image model. When the errors in

Table 5. Sensor errors.

	Gyro Noise (deg/h)		Accelerometer Noise (µg)
	Constant	Random	Constant	Random
x axis	0	20	0	200
y axis	0	20	0	200
z axis	0	20	0	200
	Camera Noise (Pixel)
	Constant		Random
u axis	0		1
v axis	0		1

are added into the ideal measurement data, real inertial sensor and camera outputs can be generated. The sampling rates of the IMU and camera are 100 Hz and 10 Hz, respectively.

The alignment errors are shown in Figure 4, whereas the alignment error statistics are listed in

Table 6. Results and deviation of simulation.

Rotation Error 1				Position Error
Matrix	Euler Angle	Mean (°)	Std 2 (°)	Vector	Component	Mean (mm)	Std (mm)
$C_B^I$	${\varphi }_I$	0.0011	0.0236	$t_{BI}^B$	$t_{BI,x}^B$	−0.0462	0.1048
	${\theta }_I$	0.0010	0.0213		$t_{BI,y}^B$	−0.0108	0.0907
	${\psi }_I$	−0.0003	0.0223		$t_{BI,z}^B$	0.0219	0.0922
$C_B^V$	${\varphi }_V$	−0.0022	0.0256	$t_{BV}^B$	$t_{BV,x}^B$	0.0183	0.0993
	${\theta }_V$	0.0008	0.0253		$t_{BV,y}^B$	−0.1221	0.0956
	${\psi }_V$	0.0016	0.0253		$t_{BV,z}^B$	0.0244	0.0690

¹ The error is the different between the simulation results and the true value. ² Std is the standard deviation.

. The calibration method can evaluate the extrinsic parameters correctly. The attitude error is <0.03° for each Euler angle, and the position error is <0.10 mm for each position vector component.

3.2. Real-World Experiment

3.2.1. Experiment Setting

A calibration experiment is conducted to confirm the validity of the proposed method and to evaluate the accuracy of the system. Figure 5 shows the experiment architecture, whereas

Table 7. Major devices involved in the experiment.

Device	Manufacturer	Model	Main Parameters
Camera	Dalsa Image, Canada	FA-21-1M120	Resolution: 1024 × 1024 Refresh rate: 10 frames/s Lens respective scale factor: 1600; FOV ~45°
MIMU	SBG Systems, France	Ellipse-A2-G4	Accelerometer: 8 g full scale, 20 µg in-run bias Instability; Gyro: 450 deg/s full scale, 8 deg/h in-run bias instability
Turntable	Aircraft Industry Precision Engineering Institute, China	902-1	Accuracy of angular position: 8′’ in both axes

presents the main devices. The system body frame coincides with the turntable frame for the sake of simplicity because the mechanical interface of the turntable frame is clearly defined.

The experiment is designed as follows. First, frames {I₀}, {V₀}, and {B₀} are defined to coincide at the initialization time. The system body, camera, and MIMU coordinates are fixed with the turntable, camera, and MIMU, respectively. Second, the turntable is manually controlled to rotate around its axes, whereas the visual/inertial integrated system moves along with the turntable.

The intrinsic parameters obtained through Zhang’s method [27] are shown in

Table 8. Calibration result of intrinsic parameters of camera.

Parameter	${\mathit{\alpha }}_{\mathit{x}}$	${\mathit{\alpha }}_{\mathit{y}}$	${\mathit{u}}_0$	${\mathit{v}}_0$	$\mathit{\gamma }$	k1	k2	P1	P21
Calibration result	1123.0114	1123.9004	527.7581	527.8911	0	−0.1027	0.1738	0.0012	0.0014
error	0.4994	0.4916	0.5249	0.6548	0	0.0008	0.0035	0.0001	0.0001

¹k₁ and k₂ are the radial distortion of the lens, while P₁ and P₂ are the tangential distortion.

. The calibration achieved an accuracy of 0.08 pixel based on 18 images.

3.2.2. Experiment Results and Discussion

The test results are presented in this section. The extrinsic calibration results before and after the checkerboard corner modification are nearly similar. The standard uncertainty of the modified method (0.15 mm) (Table 10) is lower than that of the unmodified method (0.18 mm) (

Table 9. Results and deviation of the experiment based on the unmodified corner detection method.

Rotation Matrix				Position Vector
Matrix	Euler Angle	Calibration Result (°)	Error 1 1σ (°)	Vector	Component	Calibration Result (mm)	Error 1σ (mm)
$C_B^I$	${\varphi }_I$	−0.537	0.0166	$t_{BI}^B$	$t_{BI,x}^B$	−102.781	0.133
	${\theta }_I$	−2.183	0.0168		$t_{BI,y}^B$	−246.875	0.145
	${\psi }_I$	0.480	0.0185		$t_{BI,z}^B$	−1.362	0.108
$C_B^V$	${\varphi }_V$	−1.176	0.0372	$t_{BV}^B$	$t_{BV,x}^B$	−77.434	0.147
	${\theta }_V$	0.085	0.0488		$t_{BV,y}^B$	−229.920	0.113
	${\psi }_V$	−0.175	0.0178		$t_{BV,z}^B$	39.264	0.184

¹ The error is evaluated via the standard deviation (the same below).

). The camera measurement residuals of the two methods are different as shown in Figure 6. The measurement residuals’ 3σ bound of the method based on a linear constraint is approximately 0.193 pixels, whereas the measurement residuals’ 3σ bound of the unmodified method is approximately 0.220 pixels. Thus, the motion blur correction described in Section 2.3 is effective.

The difference of IMU calibration parameters before and after modification is not significant. The camera measurements can suppress the angle integral error due to IMU’s drift, and increase the accuracy of IMU extrinsic parameters calibration results. Thus, more accuracy camera measurements will lead to more accuracy IMU extrinsic parameters calibration results. But the accuracy of IMU extrinsic parameters calibration results is closed before and after checkerboard corner modification, through compare the results in Table 9 and

Table 10. Results and deviation of the experiment based on the modified corner detection method.

Rotation Matrix				Position Vector
Matrix	Euler Angle	Calibration Result (°)	Error 1σ (°)	Vector	Component	Calibration Result (mm)	Error 1σ (mm)
$C_B^I$	${\varphi }_I$	−0.536	0.0163	$t_{BI}^B$	$t_{BI,x}^B$	−102.752	0.128
	${\theta }_I$	−2.183	0.0165		$t_{BI,y}^B$	−246.966	0.146
	${\psi }_I$	0.480	0.0186		$t_{BI,z}^B$	−1.321	0.107
$C_B^V$	${\varphi }_V$	−1.197	0.0320	$t_{BV}^B$	$t_{BV,x}^B$	−77.091	0.117
	${\theta }_V$	0.085	0.0376		$t_{BV,y}^B$	−229.742	0.103
	${\psi }_V$	−0.187	0.0205		$t_{BV,z}^B$	38.294	0.128

. There may be three reasons explain it.

• For camera measurements, the rotation angle is computed based on all the checkerboard corners’ pixel coordinates. The effects of motion blur are eliminate by involve all corners into computation process.
• The calibration process is not long, so the effect of accumulation error is not significant.
• There are system errors exits, such as the non-orthogonal of rotation axes, and the time delay of data acquisition, which also influence the error level.

Table 10 and Figure 7 summarize the experiment results. The origin of the MIMU frame is discussed in the sbg-IMU user manual, while that of the camera is the optical center of the lens. Therefore, the translation vector between the MIMU and camera can be roughly evaluated. The results that are evaluated based on the mechanical structure coincide with those that are evaluated by using EKF.

The experiment results, the existing problems and possible reasons, the strategies for improving the results, and some directions for future work are presented below:

• IMU and camera frames are aligned to the system body frame. The standard deviations of the three-axis position error are (0.13, 0.15, 0.11) mm and (0.12, 0.10, 0.13) mm for the MIMU and camera, respectively. Meanwhile, the standard deviations of the three-axis Euler angle error are (0.02°, 0.02°, 0.02°) and (0.03°, 0.04°, 0.02°) for the MIMU and camera, respectively. Compare the difference between the method with corner correction and without corner correction. We find the camera extrinsic parameters’ accuracy of former is higher than the latter (Table 9 and Table 10), and the camera measurement residuals of former is lower than the latter (Figure 6). It indicates the corner correction described in Section 2.3 is effective. The reasons why the difference of IMU calibration parameters before and after modification is not significant have been briefly discussed.
• There are three errors affecting the calibration accuracy. Firstly, the calibration errors of IMU and camera intrinsic parameters, which affect the measurements’ accuracy. At present, a reasonable choice of camera calibration method ensures that imaging accuracy reaches the sub-pixel level, and IMU is factory calibrated. Secondly, the time delay between the IMU and camera data acquisition, which affects the calibration accuracy and stability of filter. We align the data by the time label (both IMU and camera data are marked on the time label, respectively), and don’t evaluate the time delay exactly. Thirdly, we ignore the non-orthogonality of the turntable axes, and the turntable has been factory calibrated. Further research on a solution without orthogonal axes could be performed.
• We can observe the convergence of each parameter in the EKF process. The experiment results show that the method is valid and is not restricted in the Kalman filter. Some optimal algorithms, such as the particle filter and Levenberg–Marquardt algorithm can also be used. The calibration parameters are obtained, and the complete visual/inertial integrated system is established. Future research may focus on the calibration in the navigation process, and the proposed method may be seen as a standard calibration step in factory production and user operation.

4. Conclusions

An extrinsic parameter calibration method for a visual/inertial integrated system is developed based on a swinging motion. A checkerboard corner detection algorithm is then utilized to detect checkerboard corners with a smearing effect. The extrinsic parameter calibration method is developed based on the imaging model and dynamic equation. This method is validated by performing a simulation and a real-world experiment, which results highlight the effectiveness of the proposed method. This method can also be seen as a standard calibration step and used for visual/inertial systems, especially for visual and inertial navigation integrated systems.