Calibration and Noise Identification of a Rolling Shutter Camera and a Low-Cost Inertial Measurement Unit

Abstract

A low-cost inertial measurement unit (IMU) and a rolling shutter camera form a conventional device configuration for localization of a mobile platform due to their complementary properties and low costs. This paper proposes a new calibration method that jointly estimates calibration and noise parameters of the low-cost IMU and the rolling shutter camera for effective sensor fusion in which accurate sensor calibration is very critical. Based on the graybox system identification, the proposed method estimates unknown noise density so that we can minimize calibration error and its covariance by using the unscented Kalman filter. Then, we refine the estimated calibration parameters with the estimated noise density in batch manner. Experimental results on synthetic and real data demonstrate the accuracy and stability of the proposed method and show that the proposed method provides consistent results even with unknown noise density of the IMU. Furthermore, a real experiment using a commercial smartphone validates the performance of the proposed calibration method in off-the-shelf devices.

1. Introduction

An inertial measurement unit (IMU) and a camera form a widely used sensor configuration for mobile platform localization. For example, visual-inertial SLAM [1,2,3,4,5,6] is an alternative navigation method in GPS-denied environments such as tunnels and indoor areas where GPS signals are not available. In particular, a MEMS-based IMU and a rolling shutter camera, which capture images line-by-line, are commonly used owing to their low-cost sensing capability.

To use inertial measurements and images simultaneously, calibration of these two heterogeneous sensors is very critical. The calibration of a low-cost IMU and a rolling shutter camera has been studied for last decades in robotics and computer vision community [7,8,9,10,11,12,13,14,15,16,17,18,19]. Previous studies [7,8,9,10,11,12,13,14,15] have separately estimated the IMU intrinsics (i.e., bias, scale factor, and misalignment) and rolling shutter camera intrinsics (i.e., focal length, principle point, and rolling shutter readout time), and then their extrinsics (i.e., relative rotation and translation and time delay) are obtained using the given intrinsic parameters. However, separate calibration is a cumbersome and time-consuming task because each calibration has different procedures. Moreover, intrinsic calibration errors are negatively propagated to extrinsic parameters, which are estimated using pre-computed intrinsics. For these reasons, joint intrinsic and extrinsic calibration of the IMU and the camera has been proposed [16,17,18,19]. These approaches simultaneously minimize the calibration errors of intrinsics and extrinsics.

Most IMU–camera calibration methods estimate the calibration parameters with fixed noise densities of the IMU and camera measurements, which are typically handled as tuning parameters. To achieve accurate calibration, we need to know such noise densities of the IMU and camera. Fortunately, for cameras, the variation in noise densities is not large because calibration patterns (e.g., a checkerboard) are used for corner extraction. Therefore, it is dependent on feature extraction algorithms, whose accuracy in terms of pixels can be easily evaluated and is well-known [20]. In contrast, setting the IMU noise density parameters is difficult and heuristic, because IMUs have various noise densities according to the types and costs of IMUs, and the noise density is not intuitive. Furthermore, noise information of low-cost IMUs is not provided in general.

To address these problems, we jointly estimate the noise density of the low-cost IMU and the intrinsic/extrinsic calibration parameters through the graybox system identification, which estimates the unknown parameters describing the prediction of the dynamic system well [21,22,23]. In general, the graybox method is implemented on a nonlinear optimization problem that minimizes the residual between the predicted and observed measurements of filtering (i.e., the Kalman filter). The proposed framework is composed of two types of graybox methods, which include filtering and optimization, for calibration and noise identification, respectively.

For calibration, rather than finding the calibration parameters in a single optimization step, we divide the calibration process into the filtering step for initialization and the optimization step for refinement. As a result, the convergence time of our framework becomes much faster than the single optimization method. This is because it finds sub-optimal calibration parameters in the filtering step, whose computation time is more efficient than the optimization, and then uses the parameters as an initial value of optimization. Hence, the good initial estimates from filtering reduces the convergence time of optimization for the calibration problem. In fact, it is difficult to design both the calibration and noise parameter estimation problems together in the single optimization step. In the filtering step, we utilize the unscented Kalman filter (UKF) to address the strong nonlinearity of our calibration problem, and its state vector consists of the intrinsic/extrinsic calibration parameters as well as the IMU motion parameters including the position, velocity, and orientation. In the optimization step, we refine the calibration parameters with the noise density estimated from the previous filtering step, while the UKF only estimates the IMU motion including the position, velocity, and orientation.

For noise identification, the optimizer estimates the noise density that attempts to make the UKF converge properly while enabling the estimation of calibration parameters by UKF. Unlike the previous graybox method [23], we define both the prediction errors and state error covariances, which is a constraint, as the cost function of optimization. The IMU noise density is parameterized in terms of system noise covariances in UKF, and both the prediction errors and state error covariances in the cost function involve the system noise covariance in each frame. Without the constraint, the optimization solver forces the system noise covariance to become very large, because it estimates the noise density, which is over-fitted to measurements minimizing the errors (i.e., residuals) in UKF. To avoid such divergence, the proposed method not only minimizes the residuals but also constrains the state error covariances. The initialization of the noise parameters for this optimization is performed using a grid search strategy with the cost function for noise estimation.

We demonstrate the overall procedure of the proposed method with the input and output in Figure 1, and summarize the outputs as follows.

Figure 1. Overall framework of the proposed algorithm. ${\mathbf{x}}_0$ is the initial state vector and ${\mathit{\theta }}_{i,c,e}$ indicates IMU intrinsics (i), camera intrinsics (c), and extrinsics (e). $\mathbf{u}$ and $\mathbf{z}$ denote inertial and visual measurements, respectively. $\mathbf{R}$ is a known measurement noise covariance. The notation $\widehat{·}$ denotes the final estimates and subscript ${}_0$ denotes an initial step.

• IMU intrinsics: biases, scale factors, and misalignments
• Camera intrinsics: focal length, radial distortion coefficients, and rolling shutter readout time
• IMU–camera extrinsics: relative rotation, translation, and time delay between an IMU and a camera
• IMU noise parameters: noise density of a gyroscope and an accelerometer

The remainder of this paper is organized as follows. Section 2 briefly reviews the previous studies related to IMU–camera calibration. Section 3 illustrates the problem formulation for our calibration method. A detailed description about the calibration system model and its measurement model is provided in Section 4. Section 5 describes the framework and the implementation of our calibration method. Section 6 analyzes the experimental results to validate the advantages of the proposed calibration method. Finally, we conclude the paper in Section 7.

2. Related Work

As mentioned in Section 1, the goal of calibration is to obtain the intrinsic parameters of an IMU and a camera and extrinsic parameters of these two sensors. In this section, we review the separate calibration methods of each sensor and the joint calibration methods of an IMU and a camera. In addition, we introduce several studies related to the noise identification of an IMU.

2.1. Separate Calibration

The intrinsic calibration of a camera is a well-known problem in computer vision [24]. Therefore, rather than reviewing conventional camera intrinsic calibration methods, we focus on rolling shutter calibration, which additionally finds the readout time delay between lines. Geyer et al. [10] proposed the exploitation of LED flashing at a high frequency to estimate the delay. They assumed constant velocity models and removed the lens for good illumination. In contrast, Ringaby et al. [11] introduced a more efficient approach that does not need to remove lens. O’Sullivan and Corke [25] designed a new rolling shutter camera model, which can handle arbitrary camera motions and projection models such as fisheye or panoramic model. Oth et al. [12] used video sequences with a known calibration pattern. They used a continuous-time trajectory model with a rolling shutter camera model, and applied a batch optimization approach to estimate the sequential camera poses and the line delay of the rolling shutter camera.

The intrinsic calibration of an IMU has been performed using mechanical platforms such as robotic manipulators, which precisely moves IMU along known motion trajectories [7,8,9]. The intrinsic parameters of an accelerometer and a gyroscope are estimated by comparing the output of IMU and the motion dynamics, i.e., acceleration and angular velocity, which represent the generated known motion. In [26,27], authors proposed to exploit a marker-based motion tracking system or GPS measurements rather than using the expensive mechanical platforms. Recently, calibration methods that do not require any additional device have also been proposed in [13,28].

The extrinsic calibration of an IMU and a camera has been studied for the last decade in robotics community. Mirzaei and Roumeliotis [14] first presented a filter-based IMU–camera calibration method. They exploited the extended Kalman filter (EKF) to approximate nonlinear system and measurement models. Hol and Gustafsson [23] adopted the graybox system identification with which they integrated a filter-based framework and an optimization method. Kelly and Sukhatme [2] employed UKF to handle the nonlinear calibration model. They also proposed a general self-calibration algorithm that does not require additional equipment. Unlike other approaches, they included gravity as one of the state parameters to be estimated considering geo-location dependency. Dong–Si and Mourikis formulated rotation calibration as a convex problem [29]. Fleps et al. [30] proposed a new cont function that includes alignment between each trajectory of an IMU and a camera. Furgale et al. [15] proposed spatial and temporal calibration by modeling the IMU and camera motion based on a spline function. In photogrammetry community, the calibration of multiple heterogeneous sensors is called bore-sight alignment. Blazquez and Colomina [31] utilized INS/GNSS and cameras on unmanned aerial vehicle (UAV), and estimated relative boresight (rotation) and lever-arm (translation) between these sensors. Cucci et al. [32] proposed self-calibration of the sensors and 3D pose estimation using a low-cost IMU, GNSS, and cameras in a small UAV.

2.2. Joint Calibration

Joint estimation of the intrinsic and extrinsic parameters of an IMU and a camera is efficient because it does not need to conduct calibration independently for each sensor. Furthermore, it generates more accurate results because it minimizes the intrinsic and extrinsic calibration errors simultaneously. Moreover, the joint estimation is more practical since most mobile devices are equipped with a low-cost IMU and a rolling shutter camera. The joint estimation has been introduced in [16], where a low-cost IMU and a pinhole camera model were used, and in [33], where a low-cost gyroscope and a rolling shutter camera were used. Li et al. additionally considered the temporal synchronization between the IMU and the camera and the rolling shutter camera model for calibration of a low-cost IMU and a rolling shutter camera [34,35]. Lee et al. proposed to suppress uncertain noises of low-cost IMU for IMU–camera calibration [18]. Rehder et al. proposed the calibration of multiple IMUs and a camera, and considered the transformation between individual axes of the IMUs [19]. Li et al. introduced self-calibration methods for a low-cost IMU–camera system while estimating sequential camera poses in vision-aided inertial navigation [17]. It estimates the intrinsic parameters, extrinsic parameters, rolling shutter readout time, and temporal offset between an IMU and camera measurements. We highlight the differences between the proposed method and the aforementioned methods in Table 1.

Table 1. Comparison of related studies: the parameters denote IMU–camera translation (${}_C^I\mathbf{t}$), IMU–camera rotation (${}_C^I\mathbf{q}$), IMU–camera time delay (${\lambda }_t$), translation between IMU axes (${}_I^B\mathbf{t}$), camera focal length (f), camera radial distortion ($\mathbf{d}$), rolling shutter readout time (${\lambda }_{rs}$), IMU bias (${\mathbf{b}}_{a,g}$), IMU scale factor(${\mathbf{s}}_{a,g}$), IMU misalignment(${\mathbf{m}}_{a,g}$), and IMU noise density (${\sigma }_{a,g}$). Refer to Figure 2 and Table 2 for the notations I, C, and W.

	Estimator	Extrinsics			Camera Intrinsics		IMU Intrinsics			IMU Noise Densities
		${}_{\mathit{C}}^{\mathit{I}}\mathbf{t}$, ${}_{\mathit{C}}^{\mathit{I}}\mathbf{q}$	${\mathit{\lambda }}_{\mathit{t}}$	${}_{\mathit{I}}^{\mathit{B}}\mathbf{t}$	f, d	${\mathit{\lambda }}_{\mathit{rs}}$	${\mathbf{b}}_{\mathit{a},\mathit{g}}$	${\mathbf{s}}_{\mathit{a},\mathit{g}}$	${\mathbf{m}}_{\mathit{a},\mathit{g}}$	${\mathit{\sigma }}_{\mathit{a},\mathit{g}}$
Zachariah [16]	UKF	√					√	√	√
Lee [18]	UKF	√					√	√	√
Rehder [19]	LM	√	√	√			√	√	√
Li [17]	MSCKF	√	√		√	√	√	√	√
Proposed	Graybox	√	√		√	√	√	√	√	√

2.3. Noise Identification

The Allan variance (AV) and the non-parametric power spectral density are the classical tools used to describe noise characteristics of an IMU. Refer to [36] for a detailed description of these procedures. Vaccaro et al., introduced the automatic noise identification by computing and matching the covariances of the AV [37]. In addition, the IMU noise parameter estimation methods were proposed in [38,39] by using a maximum likelihood with the KF in both online and offline manners.

3. Problem Formulation

Our system comprises a low-cost IMU and a rolling shutter camera, and these two sensors are rigidly connected as in Figure 2. The IMU measures 3-DOF acceleration and 3-DOF angular velocity, and a rolling shutter camera captures an image row-by-row. In this paper, we represent the coordinates of the IMU, the camera, and the world by $\left\{I\right\}$, $\left\{C\right\}$, and $\left\{W\right\}$, respectively. These notations are used as a left superscript to denote a reference coordinate and a left subscript to represent a target coordinate, as shown in Figure 2. For example, the 3D position $\mathbf{p}$ of the IMU coordinate $\left\{I\right\}$ from the world coordinate $\left\{W\right\}$ is represented by ${}_I^W\mathbf{p}$, and the 3D translation $\mathbf{t}$ of the camera $\left\{C\right\}$ coordinate from the IMU coordinate $\left\{I\right\}$ is denoted by ${}_C^I\mathbf{t}$.

Figure 2. Schematic overview and the calibration coordinate description for the rigidly connected low-cost IMU and rolling shutter camera. The notations ($\{I\}$, $\{C\}$, and $\{W\}$) denote the coordinates of the IMU, the rolling shutter camera, and the world, respectively.

To generate calibration data, we sequentially obtain inertial measurements and images by capturing a checkerboard along three-axis linear and angular motion. The inertial measurements $\mathbf{u}$ from the IMU are composed of acceleration $\mathbf{a}\in {\mathbb{R}}^3$ and angular velocity $\mathbf{w}\in {\mathbb{R}}^3$. Visual measurements $\mathbf{z}\in {\mathbb{R}}^{2\times M}$ comprise M corners of the checkerboard in the image coordinate. Given the visual and inertial measurements, we formulate the calibration and noise identification as a minimization problem as follows.

$$\widehat{\mathit{\theta }}=\underset{\mathit{\theta }}{min}E(\mathit{\theta };\mathbf{u},\mathbf{z}),$$

(1)

where the cost function $E(·)$ illustrates the alignment error between motions of the IMU and the camera, which is computed from the visual and inertial measurements $\mathbf{u},\mathbf{z}$, with the calibration and noise parameters denoted by $\mathit{\theta }=\{{\mathit{\theta }}_i,{\mathit{\theta }}_c,{\mathit{\theta }}_e,{\mathit{\theta }}_n\}$. The intrinsic parameter ${\mathit{\theta }}_i$ of the IMU describes the inherent difference between the raw measurements of sensors and the real-world value. The intrinsic parameter ${\mathit{\theta }}_c$ of the rolling shutter camera explains the mapping from the camera coordinate to the image coordinate. The extrinsic parameter ${\mathit{\theta }}_e$ between the IMU and the camera describes the relative geometric difference between the IMU coordinate and the camera coordinate. The noise parameter ${\mathit{\theta }}_n$ of the IMU represents noise characteristics of inertial measurements.

$$\left\{\begin{array}{lll}{\mathbf{u}}^{\prime }& =a(\mathbf{u},{\mathit{\theta }}_i)& :\mathrm{IMU}\mathrm{intrinsics}\\ \mathbf{z}& =b{(}^C\mathbf{X},{\mathit{\theta }}_c)& :\mathrm{Camera}\mathrm{intrinsics}\\ {}^I\mathbf{X}& =c{(}^C\mathbf{X},{\mathit{\theta }}_e)& :\text{IMU-camera}\mathrm{extrinsics}\\ {\mathit{\theta }}_n& =d\left(\mathbf{u}\right)& :\mathrm{IMU}\mathrm{noises}\end{array}\right.$$

(2)

where ${\mathbf{u}}^{\prime }$ indicates the intrinsically calibrated inertial measurements, and ${\mathbf{X}}^C\in {\mathbb{R}}^3$ indicates the position of a three-dimensional point in a camera coordinate. $a(·),b(·),c(·)$, and $d(·)$ are functions for describing each parameter. The calibration and noise parameters are composed of several variables. Table 2 shows a detailed description of the unknown variables in each parameter.

Table 2. The parameter description.

	Notation	Description	Dimension	Unit
Extrinsic parameters	${}_C^I\mathbf{t}$	Translation between IMU and camera	3	mm
	${}_C^I\mathbf{q}$	Rotation from IMU to camera (unit-quaternion)	4	-
	${\lambda }_d$	Time delay between IMU and camera measurements	1	ms
IMU intrinsic parameters	${\mathbf{b}}_a$	Bias of acceleration measurements	3	m/s${}^2$
	${\mathbf{b}}_g$	Bias of gyroscope measurements	3	rad/s
	${\mathbf{m}}_a$	Misalignment of accelerometer axes	3	-
	${\mathbf{m}}_g$	Misalignment of gyroscope axes	3	-
	${\mathbf{s}}_a$	Scale factors of accelerometer axes	3	-
	${\mathbf{s}}_g$	Scale factors of gyroscope axes	3	-
Camera intrinsic parameters	f	Focal length of camera	1	pixel
	$\mathbf{d}$	Radial distortion coefficients	2	-
	${\lambda }_{rs}$	Rolling shutter readout time	1	$\mathsf{\mu }$s
Noise parameters	${\sigma }_a$	Noise density of accelerometer	1	m/s${}^2/\sqrt{\mathrm{Hz}}$
	${\sigma }_g$	Noise density of gyroscope	1	rad/s/$\sqrt{\mathrm{Hz}}$

4. Model

In this section, we describe the system and measurement models for calibration as follows.

$$\begin{array}{c}{\mathbf{x}}_k=F\left({\mathbf{x}}_{k-1},{\mathbf{u}}_k-1\right)+{\mathbf{w}}_{k-1}\\ {\mathbf{z}}_k=H\left({\mathbf{x}}_k\right)+{\mathbf{v}}_k,\end{array}$$

(3)

where the state vector $\mathbf{x}$ represents the sensor motion state and calibration parameters, and are propagated with the inertial measurements $\mathbf{u}$ based on the system model $F\left(·\right)$. The nonlinear measurement model $H\left(·\right)$ describes the relation between the state vector and the measurement $\mathbf{z}$. $\mathbf{w}$ is a process noise and $\mathbf{v}$ is a measurement noise. Both are assumed to be white Gaussian noise, $\mathbf{w}\sim \mathcal{N}(0,\mathbf{Q})$ and $\mathbf{v}\sim \mathcal{N}(0,\mathbf{G})$ with the noise covariance $\mathbf{Q}$ and $\mathbf{G}$, respectively. Subscripts $k-1$ and k denote time steps. We explain the aforementioned notations in detail as follows.

We include the sensor motion parameters in the state vector. Therefore, the state vector $\mathbf{x}\in {\mathbb{R}}^{40}$ is composed of a motion state $\mathit{\rho }$, extrinsic parameters ${\mathit{\theta }}_e$, IMU intrinsic parameters ${\mathit{\theta }}_i$, and camera intrinsic parameters ${\mathit{\theta }}_c$.

$$\mathbf{x}={\left[\begin{array}{cccc}{\mathit{\rho }}^{\top }& {\mathit{\theta }}_e^{\top }& {\mathit{\theta }}_i^{\top }& {\mathit{\theta }}_c^{\top }\end{array}\right]}^{\top }.$$

(4)

Here, the motion state $\mathit{\rho }\in {\mathbb{R}}^{10}$ is represented by

$$\mathit{\rho }={\left[\begin{array}{ccc}{}_I^W{\mathbf{p}}^{\top }& {}_I^W{\mathbf{v}}^{\top }& {}_I^W{\mathbf{q}}^{\top }\end{array}\right]}^{\top },$$

(5)

where ${}_I^W\mathbf{p}\in {\mathbb{R}}^3$ and ${}_I^W\mathbf{v}\in {\mathbb{R}}^3$ indicate the position and velocity of the IMU coordinate from the world coordinate, respectively; and ${}_I^W\mathbf{q}\in {\mathbb{R}}^4$ is an orientation expressed as a unit quaternion of the IMU coordinate from the world coordinate.

The extrinsic parameter ${\mathit{\theta }}_e\in {\mathbb{R}}^8$ is denoted by

$${\mathit{\theta }}_e={\left[\begin{array}{ccc}{}_I^C{\mathbf{t}}^{\top }& {}_I^C{\mathbf{q}}^{\top }& {\lambda }_d\end{array}\right]}^{\top },$$

(6)

where ${}_I^C\mathbf{t}\in {\mathbb{R}}^3$ is a translation from the IMU coordinate to the camera coordinate, and ${}_I^C\mathbf{q}\in {\mathbb{R}}^4$ is a rotation represented as a unit quaternion from the IMU coordinate to the camera coordinate. ${\lambda }_d$ is a temporal delay between the IMU and the camera.

The IMU intrinsic parameter ${\mathit{\theta }}_i\in {\mathbb{R}}^{18}$ is represented by

$${\mathit{\theta }}_i={\left[\begin{array}{cccccc}{\mathbf{b}}_a^{\top }& {\mathbf{b}}_g^{\top }& {\mathbf{m}}_a^{\top }& {\mathbf{m}}_g^{\top }& {\mathbf{s}}_a^{\top }& {\mathbf{s}}_g^{\top }\end{array}\right]}^{\top },$$

(7)

where ${\mathbf{b}}_a\in {\mathbb{R}}^3$, ${\mathbf{m}}_a\in {\mathbb{R}}^3$, and ${\mathbf{s}}_a\in {\mathbb{R}}^3$ indicate a bias, a misalignment, and a scale factor of the accelerometer, respectively. ${\mathbf{b}}_g\in {\mathbb{R}}^3$, ${\mathbf{m}}_g\in {\mathbb{R}}^3$, and ${\mathbf{s}}_g\in {\mathbb{R}}^3$ are the parameters of the gyroscope.

The camera’s intrinsic parameter ${\mathit{\theta }}_c\in {\mathbb{R}}^4$ is represented by

$${\mathit{\theta }}_c={\left[\begin{array}{ccc}f& \mathbf{d}& {\lambda }_{rs}\end{array}\right]}^{\top },$$

(8)

where $f\in \mathbb{R}$ is the focal length, $\mathbf{d}\in {\mathbb{R}}^2$ is radial distortion, and ${\lambda }_{rs}\in \mathbb{R}$ is the rolling shutter readout time of the rolling shutter camera.

4.1. System Model: Low-Cost IMU

We describe the system model $F(·)$ in Equation (3) performing the transition of the state vector $\mathbf{x}$. The visual and inertial measurements are obtained at different frame rates and their time intervals have some variation. An example of this time interval is demonstrated in Figure 3.

Figure 3. Time diagram for visual and inertial measurements. The green and red vertical lines indicate the timestamps of the IMU and camera measurements, respectively.

The state transition (or prediction) is sequentially performed with the inertial measurements ${\mathbf{u}}_s$ until a new visual measurement is obtained, as demonstrated in Figure 3. The number of predictions becomes $\#prediction=\#{\mathbf{u}}_s+1$. We define the time interval $\Delta t$ for each prediction as follows.

$$\Delta t=\left\{\begin{array}{cc}{t}_{imu,1}-(t_{img,k-1}+{\lambda }_d)& :\mathrm{First}\hfill \\ t_{imu,s}-t_{imu,s-1}& :\mathrm{Middle}\hfill \\ (t_{img,k}+{\lambda }_d)-t_{imu,end}& :\mathrm{Last}\hfill \end{array}\right.$$

(9)

where k is the time-step of the visual measurements, while s is that of the inertial measurements.

The nonlinear system model in Equation (3) is defined as follows.

$$\begin{array}{cc}\hfill F\left({\mathbf{x}}_{k-1},{\mathbf{u}}_s\right)=& \left[\begin{array}{c}\begin{array}{c}{F}_{mot}({\mathit{\rho }}_{k-1},{\mathbf{u}}_s)\\ {\mathit{\theta }}_{e,k-1}\\ {\mathit{\theta }}_{i,k-1}\\ {\mathit{\theta }}_{c,k-1}\end{array}\end{array}\right].\hfill \end{array}$$

(10)

Here, the motion state is sequentially predicted with the IMU measurement ${\mathbf{u}}_s$ and the time interval Δt, and the other calibration parameters are considered as constant because they are static and two sensors are rigidly connected. Note that we do not regard the bias states as smoothly time varying parameters, unlike [19] because the input inertial measurements for calibration are obtained for a short time (about 1–2 min).

$$\begin{array}{c}\hfill F_{mot}\left({\mathit{\rho }}_{k-1},{\mathbf{u}}_s\right)=\left[\begin{array}{c}\begin{array}{c}{}_I^W{\mathbf{p}}_{k-1}+{}_I^W{\mathbf{v}}_{k-1}\Delta t\\ {}_I^W{\mathbf{v}}_{k-1}+{}_I^W{\mathbf{a}}_s\Delta t\\ {}_I^W{\mathbf{q}}_{k-1}+\mathsf{\Omega }\left(-{\mathbf{w}}_s\frac{\Delta t}{2}\right){}_I^W{\mathbf{q}}_{k-1}\end{array}\end{array}\right],\end{array}$$

(11)

where the quaternion kinematic function $\mathsf{\Omega }:{\mathbb{R}}^3\to {\mathbb{R}}^{4\times 4}$ and the angular velocity ${\mathbf{w}}_s$ explain the time variation of IMU orientation. The linear acceleration ${}_I^W{\mathbf{a}}_s$ and angular velocity ${\mathbf{w}}_s$ are converted from the raw measurements ${\mathbf{a}}_m$ and ${\mathbf{w}}_m$ via the intrinsic parameters of the IMU as follows.

$$\left[\begin{array}{c}{}_I^W{\mathbf{a}}_s\\ {\mathbf{w}}_s\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}{}_I^W\mathbf{R}\left({\mathbf{S}}_a{\mathbf{M}}_a{\mathbf{a}}_m+{\mathbf{b}}_a\right)-\mathbf{g}\\ {\mathbf{S}}_g{\mathbf{M}}_g{\mathbf{w}}_m+{\mathbf{b}}_w\end{array}\end{array}\right],$$

(12)

where ${}_I^W\mathbf{R}\in {\mathbb{R}}^{3\times 3}$ is a rotation matrix expressing the orientation of the IMU, which is converted from ${}_I^W\mathbf{q}$, and $\mathbf{g}$ is the gravity vector in the world coordinate. The scale factor matrices ${\mathbf{S}}_a$ and ${\mathbf{S}}_g$ are generated by the scale factor states ${\mathbf{s}}_a$ and ${\mathbf{s}}_g$ as

$$\begin{array}{cc}\hfill {\mathbf{S}}_a=& \left[\begin{array}{c}\begin{array}{ccc}{\mathbf{s}}_{a,x}& 0& 0\\ 0& {\mathbf{s}}_{a,y}& 0\\ 0& 0& {\mathbf{s}}_{a,z}\end{array}\end{array}\right],{\mathbf{S}}_w=\left[\begin{array}{c}\begin{array}{ccc}{\mathbf{s}}_{g,x}& 0& 0\\ 0& {\mathbf{s}}_{g,y}& 0\\ 0& 0& {\mathbf{s}}_{g,z}\end{array}\end{array}\right]\hfill \end{array}$$

and the misalignment matrices (i.e., ${\mathbf{M}}_a$ and ${\mathbf{M}}_g$) between the orthogonal IMU coordinate and the accelerometer/gyroscope coordinates are represented as below

$$\begin{array}{cc}\hfill {\mathbf{M}}_a=& \left[\begin{array}{c}\begin{array}{ccc}1& {\mathbf{m}}_{a,x}& {\mathbf{m}}_{a,y}\\ 0& 1& {\mathbf{m}}_{a,z}\\ 0& 0& 1\end{array}\end{array}\right],{\mathbf{M}}_g=\left[\begin{array}{c}\begin{array}{ccc}1& {\mathbf{m}}_{g,x}& {\mathbf{m}}_{g,y}\\ 0& 1& {\mathbf{m}}_{g,z}\\ 0& 0& 1\end{array}\end{array}\right].\hfill \end{array}$$

The lower triangular elements become zero because we assume that the x-axes of the accelerometer and the gyroscope coincide to one of the orthogonal IMU coordinates as in [13,19].

4.2. Measurement Model: Rolling Shutter Camera

The measurement model in Equation (3) describes the relation between the state vector $\mathbf{x}$ and the measurements $\mathbf{z}$. The visual measurement $\mathbf{z}$ is a set of calibration pattern corners obtained from an image. Unlike a global shutter camera, a low-cost rolling shutter camera captures an image row-by-row. The projective geometry of a rolling shutter camera is formulated as follows.

$$\left[\begin{array}{c}{u}^{{}^{\prime }}\\ v^{{}^{\prime }}\end{array}\right]\sim \mathbf{K}{\left[\begin{array}{cc}{}_{C_v}^{C_{\frac{h}{2}}}\mathbf{R}& {}_{C_v}^{C_{\frac{h}{2}}}\mathbf{t}\end{array}\right]}_{rs}\left[\begin{array}{cc}{}_W^C\mathbf{R}& {}_W^C\mathbf{t}\\ \mathbf{0}& 1\end{array}\right]{}^W\mathbf{X},$$

(13)

where ${}^W\mathbf{X}\in {\mathbb{P}}^3$ denotes a 3D point in the world coordinate. The intrinsic matrix is represented by $\mathbf{K}\in {\mathbb{R}}^{3\times 3}$. The notation ∼ denotes normalization of a projected point. The rotation matrix and translation vector of the world coordinate from a camera coordinate are represented by ${}_W^C\mathbf{R}\in \mathrm{SO}\left(3\right)$ and ${}_W^C\mathbf{t}\in {\mathbb{R}}^3$, respectively. They are computed from our state vector, which is the position ${}_I^W\mathbf{p}$ and orientation ${}_I^W\mathbf{q}$ of the IMU from the world coordinate and transformation ${}_C^I\mathbf{t}$, ${}_C^I\mathbf{q}$ between the IMU and the camera. ${}_{C_v}^{C_{\frac{h}{2}}}\mathbf{R}$ and ${}_{C_v}^{C_{\frac{h}{2}}}\mathbf{t}$ denote rolling shutter transformation from the center row $(\frac{h}{2})$ to a row $\left(v\right)$ of the projected point because we use the center row as the reference row of an image when defining the rolling shutter distortion. Here, h is the height of the image. To compute the rolling shutter transformation, we use the Cayley transform model described in [40].

$${}_{C_v}^{C_{\frac{v}{2}}}\mathbf{R}\simeq \mathbf{R}\left(\left(\frac{h}{2}-v\right){\lambda }_{rs}{}^C\mathbf{w}\right),{}_{C_v}^{C_{\frac{v}{2}}}\mathbf{t}=\left(\frac{h}{2}-v\right){\lambda }_{rs}{}^C\mathbf{v},$$

(14)

where ${\lambda }_{rs}$ is the rolling shutter readout time included in the state vector and ${}^C\mathbf{w}$ and ${}^C\mathbf{v}$ are angular and linear velocities of the camera. The angular and linear velocities are computed from gyroscope measurements and states.

$${}^C\mathbf{w}={}_I^C\mathbf{R}{}^I{\mathbf{w}}_s,{}^C\mathbf{v}\simeq {}_W^C\mathbf{R}{}_I^W{\mathbf{v}}_{k|k-1},$$

(15)

where ${}_I^C\mathbf{R}$ is the rotation of the camera from the IMU, ${}^I{\mathbf{w}}_s$ is an intrinsically calibrated angular velocity of the IMU in Equation (12), and ${}_I^W{\mathbf{v}}_{k|k-1}$ is a predicted velocity of the IMU. We use them at the closest point to the time step $t_{img}+{\lambda }_d$, as follows.

Finally, each corner ${\mathbf{z}}_i\in \mathbf{z}$ in the visual measurement is obtained by considering radial distortion.

$${\mathbf{z}}_i=\left[\begin{array}{c}u\\ v\end{array}\right]=H(\mathbf{x};{}^W\mathbf{X})=(1+d_1{r}^2+d_2{r}^4)\left[\begin{array}{c}{u}^{{}^{\prime }}\\ v^{{}^{\prime }}\end{array}\right],$$

(16)

where the subscript i is an index of a projected point and $\{d_1,d_2\}\subset \mathbf{d}$ is the radial distortion state. To utilize multiple points, we concatenate the corner positions in the image coordinate as follows.

$$\begin{array}{c}\mathbf{z}={\left[{{\mathbf{z}}_1}^{\top }\cdots {{\mathbf{z}}_M}^{\top }\right]}^{\top },\hfill \end{array}$$

(17)

where M is the number of corner points.

5. Proposed Method

Our calibration and noise identification is based on the graybox system identification [22,41], which is a nonlinear optimization method minimizing the residuals of the Kalman filter (KF). Figure 1 demonstrates the overall framework of the proposed method. We first initialize the noise parameters by grid search because we do not have any prior knowledge on the noise. Then, we estimate the calibration and noise parameters with our novel graybox method, whose cost function considers not only residuals but also state error covariances of KF. Here, the noise parameters are estimated by nonlinear optimization, while the UKF estimates the motion state and the calibration parameters together. Finally, in the refinement step, the optimization module estimates the calibration parameters using the conventional graybox-based calibration [23], whose cost function considers only residuals, while the UKF module only estimates the motion state.

We define the cost function in Equation (1) as a square form of the state error covariances of the UKF and the residuals between the predicted and observed measurements.

$$E_1({\mathit{\theta }}_n)=\sum _{k=1}^M{\left({\mathbf{z}}_k-{\mathbf{z}}_{k|k-1}\right)}^2+{\left(diag({\mathbf{S}}_{k|k})\right)}^2,{\mathbf{S}}_{k|k}={\mathbf{H}}^{\top }{\mathbf{P}}_{k|k}\mathbf{H}$$

(18)

where $diag(·)$ is the diagonal of a square matrix and M is the number of visual measurements. The covariance ${\mathbf{S}}_{k|k}$ is computed in filtering with the updated state error covariance ${\mathbf{P}}_{k|k}$ and a Jacobian matrix of the measurement function $\mathbf{H}$. The computation of the predicted measurements and the state error covariances in UKF is described in Equation (22).

For the initialization of the noise parameters, we use the same cost function and a grid search strategy. The search range is $0<{\mathit{\theta }}_n<1$, and the grid size is 5 with log-scale intervals. Figure 4 demonstrates the cost map computed by the grid search. Then, we estimate the noise and calibration parameters using the graybox method. For the optimization, we exploit the Levenberg–Marquardt algorithm, which is a nonlinear least square solver.

$$arg\underset{{\mathit{\theta }}_n}{min}{E}_1({\mathit{\theta }}_n)$$

(19)

Figure 4. Cost map for noise parameter estimation. $\overline{E_1({\mathit{\theta }}_n)}$ is the normalized cost function defined in Equation (19).

At this time, the calibration parameters are simultaneously estimated by the UKF. Therefore, we estimate the noise parameters that result in the optimal convergence of the UKF for calibration.

5.1. Noise and Calibration Parameters in UKF

We adopt the UKF [42] owing to its efficient performance on the nonlinear system and measurement model. The prediction of the state ${\mathbf{x}}_{k-1|k-1}$ and its error covariance ${\mathbf{P}}_{k-1|k-1}$ are formulated as below.

$$\begin{array}{cc}{\chi }_{k-1|k-1}^i\hfill & =sp({\mathbf{x}}_{k-1|k-1},{\mathbf{P}}_{k-1|k-1},\mathbf{Q}({\mathit{\theta }}_n))\hfill \\ {\chi }_{k|k-1}^i\hfill & =F({\chi }_{k-1|k-1}^i,{\mathbf{u}}_{k-1},{\mathit{\theta }}_i)\hfill \\ {\mathbf{x}}_{k|k-1}\hfill & ={\sum }_i{W}_s^i{\chi }_{k|k-1}^i\hfill \\ {\mathbf{P}}_{k|k-1}\hfill & ={\sum }_i{W}_c^i[{\chi }_{k|k-1}^i-{\mathbf{x}}_{k|k-1}]{[{\chi }_{k|k-1}^i-{\mathbf{x}}_{k|k-1}]}^{\top }\hfill \end{array}$$

(20)

where $sp$ is the sigma point generation function, $\chi$ is the generated sigma points of the state, the superscript i is the index of the sigma point ($i=1,\cdots ,N$), and $W_s,W_c$ are weights of sigma points for the state and covariance, respectively. The sigma points ${\chi }_{k-1|k-1}^i$ are generated with the state ${\mathbf{x}}_{k-1|k-1}$, the state error covariance ${\mathbf{P}}_{k-1|k-1}$, and the system noise covariance $\mathbf{Q}$. Each sigma point is predicted using system model $F(·)$ with the inertial measurement ${\mathbf{u}}_{k-1}$ and the IMU intrinsic parameter ${\mathit{\theta }}_i$. The system noise covariance ${\mathbf{Q}}_{\rho }$ for motion states is defined from the IMU noise parameter $\{{\sigma }_a,{\sigma }_g\}\subset {\mathit{\theta }}_n$. The system noise covariances for other states are set to zero because they are static. As mentioned before, we regard the bias states as constant parameters, unlike [19].

$${\mathbf{Q}}_{\rho }=\left[\begin{array}{ccc}{\left({\sigma }_a{\Delta t}^2\right)}^2{\mathbf{I}}_3& {\mathbf{0}}_3& {\mathbf{0}}_{3\times 4}\\ {\mathbf{0}}_3& {\left({\sigma }_a\Delta t\right)}^2{\mathbf{I}}_3& {\mathbf{0}}_{3\times 4}\\ {\mathbf{0}}_{4\times 3}& {\mathbf{0}}_{4\times 3}& {\left({\sigma }_g\Delta t\right)}^2{\mathbf{I}}_4\end{array}\right]$$

(21)

The update of the predicted state ${\mathbf{x}}_{k|k-1}$ and the predicted state error covariance ${\mathbf{P}}_{k|k-1}$ are formulated as follows.

$$\begin{array}{cc}{\chi }_{k|k-1}^i\hfill & =sp({\mathbf{x}}_{k|k-1},{\mathbf{P}}_{k|k-1},\mathbf{R})\hfill \\ {\gamma }_{k|k-1}^i\hfill & =H({\chi }_{k|k-1}^i,{\mathit{\theta }}_c,{\mathit{\theta }}_e)\hfill \\ {\mathbf{z}}_{k|k-1}\hfill & ={\sum }_i{W}_s^i{\gamma }_{k|k-1}^i\hfill \\ {\mathbf{P}}_{z_k{z}_k}\hfill & ={\sum }_i{W}_c^i[{\gamma }_k^i-{\mathbf{z}}_{k|k-1}]{[{\gamma }_k^i-{\mathbf{z}}_{k|k-1}]}^{\top }\hfill \\ {\mathbf{P}}_{x_k{z}_k}\hfill & ={\sum }_i{W}_c^i[{\chi }_{k|k-1}^i-{\mathbf{x}}_{k|k-1}]{[{\gamma }_k^i-{\mathbf{z}}_{k|k-1}]}^{\top }\hfill \\ {\mathbf{K}}_k\hfill & ={\mathbf{P}}_{x_k{z}_k}{\mathbf{P}}_{z_k{z}_k}^{-1}\hfill \\ {\mathbf{x}}_{k|k}\hfill & ={\mathbf{x}}_{k|k-1}+{\mathbf{K}}_k({\mathbf{z}}_k-{\mathbf{z}}_{k|k-1})\hfill \\ {\mathbf{P}}_{k|k}\hfill & ={\mathbf{P}}_{k|k-1}-{\mathbf{K}}_k{\mathbf{P}}_{k|k-1}{\mathbf{K}}_k^{\top }\hfill \end{array}.$$

(22)

The predicted sigma point ${\chi }_{k|k-1}^i$ is transformed through the measurement model $H(·)$ with the camera intrinsic parameter ${\mathit{\theta }}_c$ and the IMU–camera extrinsic parameter ${\mathit{\theta }}_i$. Then, the Kalman gain ${\mathbf{K}}_k$ is computed with the predicted measurement covariance ${\widehat{\mathbf{P}}}_{z_k{z}_k}$ and state-measurement cross-covariance matrix ${\widehat{\mathbf{P}}}_{x_k{z}_k}$. The predicted state and state error covariance are updated with the Kalman gain. The measurement noise covariance $\mathbf{G}$ is defined as a block-diagonal matrix.

$$\mathbf{G}=\left[\begin{array}{ccc}{\sigma }_z^2{\mathbf{I}}_2& {\mathbf{0}}_2& {\mathbf{0}}_2\\ {\mathbf{0}}_2& \ddots & {\mathbf{0}}_2\\ {\mathbf{0}}_2& {\mathbf{0}}_2& {\sigma }_z^2{\mathbf{I}}_2\end{array}\right]$$

(23)

5.2. Refinement

We refine the estimated calibration parameters with the conventional graybox method. Here, we only use the residuals between the predicted measurement and the observed measurement in the cost function, and the filter module only estimates the IMU motion state. As a result, the fixed system noise covariance ${\mathbf{Q}}_{\rho }\in {\mathbb{R}}^{10}$, instead of $\mathbf{Q}\in {\mathbb{R}}^{40}$, for motion states is used in UKF. However, unlike [23], which estimated only extrinsic parameters, we estimate the intrinsic parameters of the IMU and the camera as well as the extrinsic parameters. Therefore, the cost function in Equation (1) is defined as follows:

$$arg\underset{{\mathit{\theta }}_{i,c,e}}{min}{E}_2({\mathit{\theta }}_{i,c,e}),E_2({\mathit{\theta }}_{i,c,e})=\sum _{k=1}^N{\left({\mathbf{z}}_k-{\mathbf{z}}_{k|k-1}\right)}^2.$$

(24)

Since this optimization is performed in batch manner, unlike calibration parameter estimation by filtering in the previous step, outliers or disturbances such as abrupt motion or illumination changes can be corrected in this step.

5.3. UKF Initialization

In this section, we describe the initialization of the states, the state error covariance, and the measurement noise covariance for the UKF. The initial position and orientation of the IMU are computed by transforming the position and the orientation of the camera at the first frame through the initial relative rotation and translation between the IMU and the camera. The velocity of the IMU is initialized to zero. The initial focal length f is set to the width of the image. The relative rotation between the IMU and the camera ${}_I^C\mathbf{q}$ is initialized by the angular velocities of the IMU and estimated rotation from the camera [15]. The IMU provides the raw angular velocity measurements. To compute angular velocities of the camera, we estimate the camera orientations with respect to the checkerboard pattern using homography [24]. Then, the angular velocity is obtained from the derivative of the orientation of the camera. The other parameters—the relative translation ${}_I^C\mathbf{t}$ and time delay ${\lambda }_d$ between the IMU and the camera, biases ${\mathbf{b}}_a,{\mathbf{b}}_g$, misalignments ${\mathbf{m}}_a,{\mathbf{m}}_g$, distortion coefficient $\mathbf{d}$, and rolling shutter readout time ${\lambda }_{rs}$—are initially set to zero, and scale factors ${\mathbf{s}}_a,{\mathbf{s}}_g$ are set to one. The initial state error covariance matrix is empirically set as shown in Table 3. Based on the pixel localization error of the corner extraction algorithm, the standard deviation ${\sigma }_z$ for the measurement noise covariance is set to 1. The gravity $\mathbf{g}$ in the world coordinate is set to ${[0,0,9.81]}^{\top }$, and the z-axis of the world coordinate is the vertical direction. The initial calibration parameters for the refinement step are obtained from the estimated calibration parameter from the UKF in the calibration and noise identification step.

Table 3. Initial state error covariance. The elements in a variable are set to equal values.

${\mathbf{P}}_{{}_{\mathit{I}}^{\mathit{W}}\mathbf{p}}$	${\mathbf{P}}_{{}_{\mathit{I}}^{\mathit{W}}\mathbf{v}}$	${\mathbf{P}}_{{}_{\mathit{I}}^{\mathit{W}}\mathbf{q}}$	${\mathbf{P}}_{{}_{\mathit{C}}^{\mathit{I}}\mathbf{t}}$	${\mathbf{P}}_{{}_{\mathit{C}}^{\mathit{I}}\mathbf{q}}$	${\mathbf{P}}_{{\mathit{\lambda }}_{\mathit{d}}}$	${\mathbf{P}}_{{\mathbf{b}}_{\mathit{a}}}$	${\mathbf{P}}_{{\mathbf{b}}_{\mathit{g}}}$	${\mathbf{P}}_{{\mathbf{m}}_{\mathit{a}}}$	${\mathbf{P}}_{{\mathbf{m}}_{\mathit{g}}}$	${\mathbf{P}}_{{\mathbf{s}}_{\mathit{a}}}$	${\mathbf{P}}_{{\mathbf{s}}_{\mathit{g}}}$	${\mathbf{P}}_{\mathit{f}}$	${\mathbf{P}}_{\mathbf{d}}$	${\mathbf{P}}_{{\mathit{\lambda }}_{\mathit{rs}}}$
${0.05}^2$	${0.05}^2$	${0.05}^2$	${100}^2$	${0.05}^2$	${100}^2$	${0.2}^2$	${0.003}^2$	${0.05}^2$	${0.05}^2$	${0.1}^2$	${0.1}^2$	${10}^2$	${0.1}^2$	${0.05}^2$

6. Experimental Results

We evaluate the proposed method on the synthetic and real data. The synthetic data contain synthetically generated motion information of an IMU–camera system and corresponding calibration pattern points. Note that the two sensors are rigidly connected. Using the synthetic data, we validate the accuracy and stability of the proposed method by running 100 Monte-Carlo simulations with fixed noise parameters. Furthermore, we compare the calibration results with and without noise identification using random noise parameters to demonstrate the effect of noise estimation. In real-data experiments, we utilize two experimental setups: a self-designed system, which is equipped with two rolling shutter cameras and one low-cost IMU, and a commercial smartphone, which has a rolling shutter camera and a low-cost IMU. The first setup is specially designed to evaluate the extrinsic calibration accuracy. Since the ground truth cannot be obtained for real-data experiments, we indirectly measure the performance based on loop-closing errors between the two cameras and the IMU. With these two setups, we first analyze the performance of the proposed method based on the standard deviation of the estimate. We also compare the proposed method with hand-measure estimates in terms of extrinsics and existing calibration methods in terms of intrinsics and extrinsics for reference. These comparisons show that the proposed method produces comparable results on the camera intrinsic calibration. In addition, the smartphone experiments show that the proposed method is more robust to measurement noises than the existing calibration method.

6.1. Synthetic Data

We generate the synthetic measurements for a frame length of 60 s. For this, we synthetically make 20 corners of a checkerboard pattern in the world coordinate and a smooth trajectory of the mobile platform by applying a sinusoidal function, as shown in Figure 5a. The points in the world coordinate are projected to the image coordinate as in [40], and are used as synthetic visual measurements for IMU–camera calibration. We set the image resolution to 1280 × 960. The focal length is set to 700 and the radial distortions are set to [0.1 −0.1]${}^{\top }$. The rolling shutter readout time ${\lambda }_{rs}$ is set to 41.8 $\mathsf{\mu }$s. The standard deviation of Gaussian noises for the visual measurements is set to 1 pixel. The IMU acceleration and angular velocity measurements along the world coordinate are obtained by differentiating the trajectory of the IMU. Then, to generate uncalibrated measurements, intrinsic parameters including a scale factor, a misalignment, a bias, and gravity in the world coordinate are used to perform the inertial measurements based on Equation (12). The scale factors (${\mathbf{s}}_a$, ${\mathbf{s}}_g$), misalignments (${\mathbf{m}}_a$, ${\mathbf{m}}_g$), and biases (${\mathbf{b}}_a$, ${\mathbf{b}}_g$) are set to 1.1, 1.1, 0.03, 0.03, 0.1, and 0.1, respectively. These parameters are represented by a 3D vector and the elements of each vector are set to the same value. For instance, ${\mathbf{m}}_a={[0.03,0.03,0.03]}^{\top }$. Finally, we apply additive white Gaussian noise to the inertial measurements. Figure 5b,c shows an example of the synthetic inertial measurements. The frame rates of the IMU and the camera measurements are set to 100 and 25 Hz, respectively. For the extrinsic parameters, the relative rotation, relative translation, and time delay between the IMU and the camera are set to [50 −50 50]${}^{\circ }$, [50 50 −50]${}^{\top }$ mm, and 100 ms, respectively. The calibration parameters are initialized as described in Section 5.3.

Figure 5. Synthetic data for the IMU–camera calibration: (a) trajectory of the rigidly connected IMU and camera; (b) synthesized acceleration measurements; and (c) synthesized angular velocity measurements. The blue, orange, and yellow color lines in (b,c) indicate the x-, y-, and z-axis measurements.

In the first experiment, we evaluate the proposed method with fixed IMU noise parameters, which represent the noise density of the IMU set to ${\sigma }_a$ = 0.1 m/s${}^2$/$\sqrt{\mathrm{Hz}}$, and ${\sigma }_g$ = 0.1${}^{\circ }$/s/$\sqrt{\mathrm{Hz}}$. The purpose of this experiment is to show how accurately and stably the proposed method estimates the calibration and noise parameters. Figure 6 shows the estimates of the calibration parameters and the standard deviation of their errors, which are a square root of diagonals of the state error covariance, over the 60-s frame length in the calibration and noise identification step. In our framework, the calibration parameters and their covariances are estimated by the UKF while estimating the noise parameters using nonlinear optimization. The graphs in Figure 6 show the calibration estimates obtained by the UKF at optimal noise parameter estimates by nonlinear optimization. In Figure 6a, the errors of the states, including the rotation ${}_C^I\mathbf{R}$, translation ${}_C^I\mathbf{t}$, time delay ${\lambda }_d$, biases ${\mathbf{b}}_a$ and ${\mathbf{b}}_g$, misalignments ${\mathbf{m}}_a$ and ${\mathbf{m}}_w$, scale factors ${\mathbf{s}}_a$ and ${\mathbf{s}}_w$, and focal length f, the radial distortion $\mathbf{d}$ and the rolling shutter readout time ${\lambda }_{rs}$, are rapidly converged to zero. The errors are computed using the ground truth calibration parameters. The rotation errors are displayed in the Euler angle form for intuitive representation. In Figure 6b, the error standard deviations obtained by the UKF are rapidly converged.

Figure 6. State errors and their covariances: The state contains calibration parameters, which are estimated by the filter of the proposed method on the synthetic data. The noise parameter is set to the mean of the estimates by the proposed method (${\sigma }_a=0.1$ m/s${}^2$/$\sqrt{\mathrm{Hz}}$, ${\sigma }_g=0.1$ deg/s/$\sqrt{\mathrm{Hz}}$). The blue, orange, and yellow color lines of the 3-dimensional states and covariances ($\varphi$, $\mathbf{t}$, ${\mathbf{b}}_a$, ${\mathbf{b}}_g$, ${\mathbf{s}}_a$, ${\mathbf{s}}_g$, ${\mathbf{m}}_a$, and ${\mathbf{m}}_g$) mean the x-, y-, and z-axis estimates. The blue and orange color lines of the 2-dimensional state and covariance $\mathbf{d}$ mean ${\mathbf{d}}_1$, ${\mathbf{d}}_2$. The 1-dimensional states and covariances (${\lambda }_d$, ${\lambda }_{rs}$, and f) are represented by the blue color lines: (a) state error; and (b) state error covariance.

Table 4 summarizes the final calibration and noise parameter estimates of the proposed method by running 100 Monte-Carlo simulations. We describe the mean and standard deviation of the estimates to show statistically meaningful results. Table 4d shows the noise parameter estimates obtained by the proposed method. We find that the ground truth of the noise parameter is within the error bound of the noise parameter estimates. This result indicates that the proposed noise identification reasonably estimates the noise parameters. Therefore, the estimated noise parameters can be used to further refine the calibration parameters in the refinement step. Table 4a–c shows the extrinsic and intrinsic parameter estimates of the IMU and the camera. The differences between the estimates and the ground truth are smaller than their standard deviations. This validates that the proposed method accurately estimates all calibration parameters and noise parameters.

In the second experiment, we generate 100 sequences based on different IMU noise parameters, and, using these sequences, we analyze the proposed calibration method with and without the noise density estimation. The purpose of this experiment is to show that how the proposed noise identification robustly estimates the calibration parameters under unknown IMU noise density. The noise parameters ${\sigma }_a,{\sigma }_w$ are randomly selected from 0.001 to 0.2, and the generated noises are added to the inertial measurements. These values cover most of the noise characteristics from low- to high-cost IMUs. We run the proposed method with noise estimation and fixed noise parameters (${\sigma }_a/{\sigma }_w$ = 0.2/0.2 and 0.001/0.001). Table 5 shows the calibration parameter estimates of the proposed method with and without noise estimation. With noise estimation, the proposed calibration method estimates accurate calibration parameters. The mean of errors between the estimated results and the ground truth is smaller than the standard deviation of the estimates. On the contrary, fixed noise densities cause inaccurate and unstable estimates. The mean of the translation estimates without noise estimation is out of the error bound, and their standard deviation is about two times larger than that of the proposed method with noise estimation. For bias estimates, the mean of the estimates is not within the error bound and their standard deviation is larger. Besides, the estimates of the focal length and rolling shutter readout time are largely affected by the IMU noise parameters, whereas the rotation, misalignments, scale factors, and radial distortions are not dependent on the IMU noise parameters. These results validate that the proposed noise identification approach improves the accuracy and stability of the calibration parameter estimation, especially under unknown IMU noise density.

6.2. Real Data

For the real-data experiments, we use two setups: a self-designed system and a commercial smartphone. Since it is difficult to obtain ground truth for real data, we indirectly evaluate the performance of the proposed method using the results of existing methods and the hand-measured values. In addition, we use the standard deviation of estimates as an evaluation metric to analyze the algorithm stability. Moreover, the loop closing error metric with two cameras and the IMU is utilized to evaluate extrinsic calibration performance.

At first, we evaluate the proposed method using the self-designed system, which consists of two rolling shutter cameras and a low-cost IMU. Figure 7a shows our experimental setup. The rolling shutter cameras are Logitech C80 (Logitech, Lausanne, Switzerland) and the low-cost IMU is Withrobot myAHRS+. The image resolution is 640 × 480, and the frame rates of the camera and the IMU are, respectively, 30 and 100 Hz. We record a set of measurements for approximately 90 s and perform experiments on the 10 datasets.

Figure 7. Two types of the low-cost IMU and the rolling shutter camera setup for real-data experiments: (a) our self-designed system having two rolling shutter cameras and a low-cost IMU; and (b) Samsung Galaxy Alpha (Samsung, Seoul, South Korea) having a rolling shutter camera and a low-cost IMU.

Table 6 demonstrates calibration parameter estimates of camera 1 and the IMU. For extrinsic and IMU intrinsic parameters, we compare our method with the existing calibration method (i.e., the Kalibr-imucam method [19]) and the hand-measured values. The noise density of the IMU for the Kalibr-imucam method is set to the noise density estimated by the proposed method. Table 6a describes the extrinsic parameter estimates of the proposed method, the Kalibr-imucam method, and the hand-measured values. The rotation parameter ${}_C^I\mathbf{q}$ is parameterized by Euler angle $\mathit{\varphi }$ instead of unit quaternion for intuitive understanding. The average rotation, translation, and time delay estimates of the proposed and Kalibr-imucam methods are close to each other, and the average rotation and translation estimates of both methods are within the error bound of the hand-measured value. In addition, the standard deviation of the extrinsic parameter estimates from both methods is very small. This comparison shows that the proposed method successfully estimates the extrinsic parameters; moreover, the estimated noise parameters from the proposed method lead to the successful calibration of the existing calibration method.

Furthermore, with the two rolling shutter cameras, we evaluate the extrinsic calibration performance using the loop-closing error metric, which is defined as the transformation error $\delta \mathbf{T}={}_{C_1}^I\mathbf{T}{}_{C_2}^{C_1}\mathbf{T}{{}_{C_2}^I\mathbf{T}}^{-1}$ between the low-cost IMU and the two cameras (i.e., $C_1$ and $C_2$). Here, ${}_{C_1}^I\mathbf{T}$ represents the relative transformation from the IMU to camera 1, ${}_{C_2}^{C_1}\mathbf{T}$ represents the relative transformation between the two cameras, and ${}_I^{C_2}\mathbf{T}$ denotes the relative transformation from camera 2 to the IMU. ${}_{C_2}^{C_1}\mathbf{T}$ is estimated through the stereo camera calibration algorithm [43], and ${}_{C_1}^I\mathbf{T}$ and ${}_{C_2}^I{\mathbf{T}}^{-1}$ are estimated through the IMU–camera calibration algorithm. We run 10 sequences with the left camera–IMU and right camera–IMU. Then, we compute 10 × 10 loop closing errors by using bipartite matching (i.e., total 100 sets). Table 6b shows the mean and standard deviation of the loop closing errors on rotation and translation motions. The proposed and Kalibr-imucam methods produce very small errors on rotation and translation. Since the error bound of ${}_{C_2}^{C_1}\mathbf{T}$ is about ±2 mm, the errors in our method and the Kalibr-imucam method are negligible.

Table 6c describes the IMU intrinsic parameter estimates of the proposed and Kalibr-imucam methods. We do not compare classic IMU intrinsic calibration methods [7,8,9] because they require special equipment such as robot arms to estimate intrinsic parameters. Besides, since the Kalibr-imucam method regards biases as time-varying parameters, unlike the proposed method, we do not compare it in this table. In both methods, the standard deviations of the misalignments and the scale factors are small enough, which indicates that the estimates are reliable. The results show that the z-axis bias of the accelerometer is larger than the x,y-axis biases due to the effect of the gravity vector located on the z-axis of the IMU coordinate (i.e., z-axis values of inertial measurements are much larger than those of x,y-axis). Interestingly, the misalignments are close to zero and the scale factors are close to one. This means that the IMU is somewhat intrinsically calibrated, although it is a low-cost sensor. The average estimates of the misalignments and scale factors from both methods have small differences because the Kalibr-imucam method also considers the effects of linear accelerations on gyroscopes. However, there are negligible numerical errors in the misalignments and scale factors.

For camera intrinsic parameters, we compare our method with the Kalibr-rs method [12] and the MATLAB calibration toolbox, as shown in Table 6d. The Kalibr-rs method uses image sequences, which are the same measurements as those used for the proposed method, and the MATLAB toolbox uses 20 still images of the same checker board. The focal length and radial distortion estimates of the three methods are close to each other. In addition, the average rolling shutter readout time estimates of the proposed method are close to the average estimates of the Kalibr-rs method. Although there are small differences between the estimates obtained from the three methods, they are negligible. Besides, the smaller standard deviation of the estimates from the proposed method indicates that the proposed method is more reliable than the Kalibr-rs method. This comparison of the intrinsic parameter estimates from the proposed and the two existing methods, which use various measurements, validates the camera intrinsic calibration of the proposed method.

Table 6e describes the estimated noise parameters. The standard deviation of IMU noise density estimates is small enough and their mean can be used for other sensor fusion algorithms such as IMU–camera calibration and visual-inertial SLAM methods. Although we do not compare the noise density provided from the other algorithms or manufacturers, the noise estimation performance of the proposed method is indirectly verified through comparison with extrinsic parameters of the hand-measured value and camera intrinsic parameters from the MATLAB Toolbox. In addition, the successful calibration of the Kalibr-imucam method with the noise density estimates obtained from the proposed method validates the noise estimation of the proposed method.

Second, we evaluate the proposed method using a smartphone “Samsung Galaxy Alpha”, which is equipped with a rolling shutter camera and a low-cost IMU; its configuration is shown in Figure 7b. The image resolution is 720 × 480, and the frame rates of the camera and the IMU are, respectively, 30 and 50 Hz. We record 10 sets of measurements whose frame length is about 90 s.

Table 7 shows the calibration parameter estimates of the smartphone. We compare the proposed method with the existing calibration method (i.e., Kalibr-imucam [19]) and hand-measured values, similarly to the above experiment. The average rotation estimates of the proposed method and the Kalibr-imucam method are close to those of the hand-measured value. The standard deviation of the rotation estimates from the proposed method is smaller than that obtained from the Kalibr-imucam method, and this result indicates that the proposed method is more reliable than the Kalibr-imucam method. Besides, the translation estimates of the Kalibr-imucam method are converged to unrealistic values that are zero, but our estimates are close to the hand-measured value. The average time delay estimates of the proposed method are close to those of [19]; however, the standard deviation is large because an irregular time delay occurs due to the camera module activation of the smartphone. In summary, in this experiment, the extrinsic calibration of the proposed method outperforms the Kalibri-imucam method unlike the first experiment, which uses the self-designed setup. We argue that the proposed method is more robust than the noisy measurements because our framework is based on the graybox method, which internally uses the Kalman filter. In practice, the rolling shutter camera and the IMU of the commercial smartphone contain larger noises than those of the self-designed setup, as described in Table 7d.

Table 7d demonstrates the IMU intrinsic estimates of the proposed method. Similar to the results of the first experiment, the z-axis bias of the accelerometer is larger than those of the other axes. The misalignment and scale factor estimates of the accelerometer from the proposed and Kalibri-imucam methods are, respectively, close to zero and one, and the biases and misalignments of the gyroscope are close to zero. This result indicates that they were already intrinsically calibrated. However, the scale factors of the gyroscope are calibrated through the proposed and Kalibr-imucam methods, as shown in the table. Although the scale factor estimates from two methods are slightly different, the low standard deviation of our estimates indicates that the proposed method is more reliable.

For camera intrinsic parameters, we compare our estimates to the Kalibr-rs method [12] and the MATLAB calibration toolbox, as shown in Table 7c. The focal length, radial distortion, and rolling shutter readout time estimates of the proposed method are close to the estimates of two existing methods. Although the estimates of the second coefficient of radial distortion are different for each method, the first coefficient is more important than the second. Besides, the proposed method provides the lower standard deviation of the rolling shutter readout time estimates, as in the first experiment.

Table 7d describes the estimated noise densities. They are two times larger than the noise density estimated on the smartphone setup. This means that the IMU noise of the smartphone setup is more severe than that of the self-designed setup. This experiment shows that a full calibration of off-the-shelf devices is possible with the proposed method.

Furthermore, we compare the operating time of the existing method (Kalibr [12,19]) and our method. The experimental environment for the comparison is Intel i7-4790K CPU running at a single core 4.00 GHz. Table 8 demonstrates the operating time of the Kalibri [12,19] and the proposed method. The timing statistics are measured on the 10 datasets used for the above “Samsung Galaxy Alpha” experiment. In case of the existing method, it is required to use both the Kalibr-rs [12] and the Kalibr-imucam [19] together because they separately estimate the intrinsic and extrinsic parameters, whereas, the proposed method does not. The total operating time of the Kalibr method is about 2 h in average for the full calibration. In particular, the operating time of Kalibri-rs [12] is longer than 1 h because of the iterative batch optimization for adaptive knot placement. However, the proposed method estimates noise parameters as well as calibration parameters, and it is 1.733 times faster than the Kalibr method in average.

Table 8. Timing comparison of the existing and proposed methods. Mean and standard deviation of the operating times are described together.

		Time (min)
Kalibri	Kalibri-imucam [19]	24.80 ± 2.89
	Kalibri-rs [12]	102.03 ± 42.94
	Total	126.83 ± 42.03
Proposed	Noise Parameter Initialization	24.62 ± 0.21
	Calibration & Noise Identification	27.18 ± 10.11
	Calibration Parameter Refinement	21.36 ± 12.10
	Total	73.16 ± 12.38

We also compare the prediction errors of the UKF for motion estimation with calibration parameters estimated by the Kalibr method and the proposed method. In this experiment, the states of the UKF are a position, velocity, and orientation of the IMU, and they are predicted by inertial measurements and corrected by visual measurements, which are corners of a checkerboard pattern. We record a set of inertial and visual measurements using the smartphone “Samsung Galaxy Alpha” for 60 s. The prediction errors using the Kalibr are accumulated due to the IMU noise and they result in the motion drift. However, Figure 8 shows that the calibration parameters of the proposed method reduce the mean and variance of prediction errors. The average RMSE in terms of pixel for all frames decreases from 3.03 to 2.45 (about 19.2%).

Figure 8. Prediction errors in the UKF for motion estimation with the calibration parameters of the Kalibr and the proposed method. The prediction errors of visual measurements (corners) for all frames are recorded, and a cross represents a prediction error. The color of crosses means the index of a frame (i.e., blue indicates the start of the sequence and red does the end of the sequences): (a) Kalibr-imucam [19] and Kalibr-rs [12] methods, with average RMSE of 3.03 pixel; and (b) proposed method, with average RMSE of 2.45 pixel.

Finally, we compare the localization accuracy with calibration parameters estimated by the Kalibr method and the proposed method. In this experiment, we capture image sequences and inertial measurements using the smartphone “Samsung Galaxy Alpha” in an outdoor vehicle driving environment. The trajectories are estimated by VINS-mono [44], which is an open source visual-inertial SLAM algorithm. In this experiment, we do not use loop closure correction and online calibration on the extrinsic parameter and temporal delay. Figure 9 shows the estimated trajectory with the proposed method is close to the ground truth trajectory, whereas the trajectory estimates with the Kalibr method suffers from scale drift as the trajectory becomes longer. The experimental results validate that the proposed method improves the performance of the real-world system as well.

Figure 9. Outdoor localization results with the calibration parameters of the Kalibr and the proposed method. The estimated trajectories are overlaid on Google Map. The visual and inertial measurements are collected by the smartphone “Samsung Galaxy Alpha” under vehicle driving condition. The length of the whole trajectory is about 0.8 km. The ground truth trajectory is obtained from GPS measurements.

7. Conclusions

This paper proposes a robust and accurate calibration method for a low-cost IMU and a rolling shutter camera. The proposed joint calibration estimates not only the intrinsic and extrinsic parameters but also the IMU noise parameters. To improve calibration efficiency including runtime, we divide the framework into two steps. In the first step, we roughly estimate the intrinsic and extrinsic parameters through filtering while estimating the noise parameters in the optimization module. In the second step, we refine the intrinsic and extrinsic parameters via the optimization module while estimating the sensor motion in filtering. The experimental results of the synthetic data demonstrate the superiority of our framework, and, in particular, the experiments on two real-data setups validate the performance of the proposed method in off-the-shelf devices.

As a result, the proposed method enhances the runtime about 73.3% and reduces IMU drift by 19.2% in comparison with those of the Kalibr method [12,19]. In particular, the results of the visual-inertial SLAM on the real-world system demonstrates that the proposed method outperforms the Kalibr method.