Structural Parameters Calibration for Binocular Stereo Vision Sensors Using a Double-Sphere Target

Abstract

Structural parameter calibration for the binocular stereo vision sensor (BSVS) is an important guarantee for high-precision measurements. We propose a method to calibrate the structural parameters of BSVS based on a double-sphere target. The target, consisting of two identical spheres with a known fixed distance, is freely placed in different positions and orientations. Any three non-collinear sphere centres determine a spatial plane whose normal vector under the two camera-coordinate-frames is obtained by means of an intermediate parallel plane calculated by the image points of sphere centres and the depth-scale factors. Hence, the rotation matrix R is solved. The translation vector T is determined using a linear method derived from the epipolar geometry. Furthermore, R and T are refined by nonlinear optimization. We also provide theoretical analysis on the error propagation related to the positional deviation of the sphere image and an approach to mitigate its effect. Computer simulations are conducted to test the performance of the proposed method with respect to the image noise level, target placement times and the depth-scale factor. Experimental results on real data show that the accuracy of measurement is higher than 0.9‰, with a distance of 800 mm and a view field of 250 × 200 mm².

1. Introduction

As one of the main structures of machine vision sensors, BSVS acquires 3D scene geometric information through one pair of images and has many applications in industrial product inspection, robot navigation, virtual reality, etc. [1,2,3]. Structural parameter calibration is always an important and concerning issue in BSVS. Current calibration methods can be roughly classified into three categories: methods based on 3D targets, 2D targets and 1D targets. 3D target-based methods [4,5] obtain the structural parameters by placing the target only once in the sensor field of view. However, its disadvantages lie in the fact that large size 3D targets are exceedingly difficult to machine, and it is usually impossible to maintain the calibration image with all feature points at the same level of clarity. 2D target-based methods [2,6] require the plane target to be placed freely at least twice with different positions and orientations, and different target calibration features are unified to a common sensor coordinate frame through the camera coordinate frame. Therefore, calibration operation becomes more convenient than in 3D target-based methods. However, there are also weaknesses in two primary aspects. One is that repeated calibration feature unification will increase transformation errors. The other is that when the two cameras form a large viewing angle or the multi-camera system requires calibration, it is difficult to simultaneously maintain the calibration image with all features at the same level of clarity for all cameras. Regarding 1D target-based methods [7], which are much more convenient than 2D target-based methods, the target is freely placed no less than four times with different positions and orientations. The image points of the calibration feature points are used to determine the rotation matrix R and the translation vector T, and the scale factor of T is obtained by the known distance constraint. Unfortunately, 1D target-based methods have the same weaknesses as 2D target-based methods. Moreover, in practice, 1D targets should be placed many times to obtain enough feature points.

The sphere is widely used in machine vision calibration owing to its spatial uniformity and symmetry [8,9,10,11,12,13,14,15,16,17]. Agrawal et al. [11] and Zhang et al. [16] both used spheres to calibrate intrinsic camera parameters using the relationship between the projected ellipse of the sphere and the dual image of the absolute conic (DIAC). Moreover, they also mentioned that the structural parameters between two or more cameras could be obtained by using the 3D points cloud registration method. However, this method requires many feature points to guarantee high accuracy. Wong et al. [17] proposed two methods to recover the fundamental matrix, and then structural parameters could be deduced when the intrinsic parameters of the two cameras are known. One method is to use sphere centres, intersection points and visual points of tangency to compute the fundamental matrix. The other is to determine the fundamental matrix using the homography matrix and the epipoles, which are computed via plane-induced homography. However, the second method requires an extra plane target to transfer the projected ellipse from the first view to the second view.

In this paper, we propose a method using a double-sphere target to calibrate the structural parameters of BSVS. The target consists of two identical spheres fixed by a rigid bar of known length and unknown radii. During calibration, the double-sphere target is placed freely at least twice in different positions and orientations. From the projected ellipses of spheres, the image points of sphere centres and a so-called depth-scale factor for each sphere can be calculated. Because any three non-collinear sphere centres determine a spatial plane π_s, if we have at least three non-parallel planes with their normal vectors obtained in both camera coordinate frames, the rotation matrix R can be solved. However, π_s could not be directly obtained. We obtained its normal vector by an intermediate plane paralleling to the plane π_s, which is recovered by the depth-scale factors and the image points of sphere centres. From the epipolar geometry, a linear relation between the translation vector T including a scale factor and the image points is derived, and then SVD is used to solve it. Furthermore, the scale factor is determined based on the known distance constraint. Finally, R and T are combined to be refined by Levenberg-Marquardt algorithm. Due to the complete symmetry of the sphere, wherever the sphere is placed in the sensor vision field, all cameras can capture the same high-quality images of the sphere, which are essential to maintain the calibration consistency, even if the angle between the principal rays of the two cameras is large. Moreover, regarding multi-camera system calibration, it is often difficult to make the target features simultaneously visible in all views because of the variety of positions and orientations of the cameras. In general, the cameras are divided into several smaller groups, and each group is calibrated separately, and finally all cameras are registered into a common reference coordinate frame [17]. However, using the double-sphere target can obtain the relationship of the cameras with common view district by once calibration. A kind of terrible configuration of two cameras mentioned above will be often taken place in multi-camera calibration. Therefore, using the double-sphere target can reduce the times of calibration and make calibration operation easy and efficient.

The remaining sections of this paper are organized as follows: Section 2 briefly describes a few basic properties of the projected ellipse of the sphere. Section 3 elaborates the principles of the proposed calibration method based on the double-sphere target. Section 4 provides detailed analysis of the impact on the image point of the sphere centre when the projected ellipse is not accurately extracted with positional deviation. Section 5 presents computer simulations and real data experiments to verify the proposed method. The conclusions are given in Section 6.

2. Basic Principles

This section mainly describes some related properties of the projected ellipse of sphere.

2.1. Derivation of the Projected Ellipse

Agrawal [11] and Zhang [14] each give the formula of the projected ellipse of the sphere. We further synthesize the two different derivation approaches to gain an easily understood explanation, which is briefly described as follows:

Consider a camera $P=K\left[I_{3\times 3}|0\right]=\left[K|0\right]$ viewing a sphere $Q$ with radius R₀ centered at X = [X₀ Y₀ Z₀]^T in the camera coordinate frame O~XYZ, where $K$ is the camera intrinsic matrix. Q is expressed as (X − X₀)² + (Y − Y₀)² + (Z − Z₀)² = $R_0^2$.

Denoting $\sqrt{X_0^2+Y_0^2+Z_0^2}$ by h₀; then we have:

$$X={\left[\begin{array}{ccc}{X}_0& Y_0& Z_0\end{array}\right]}^{\mathrm{T}}=h_{0}d$$

(1)

where d is the unit vector of X.

Sphere Q is further expressed by the following coefficient matrix:

$$Q=\left[\begin{array}{cc}{I}_{3\times 3}& -X\\ -X^{\mathrm{T}}& X^{\mathrm{T}}X-R_0^2\end{array}\right]=\left[\begin{array}{cccc}1& 0& 0& -X_0\\ 0& 1& 0& -Y_0\\ 0& 0& 1& -Z_0\\ -X_0& -Y_0& -Z_0& h_0^2-R_0^2\end{array}\right]$$

(2)

Thus, the dual Q* of Q is defined as:

$$Q^{*}=Q^{-1}$$

(3)

Next, we obtain the dual C* of the projected ellipse C of sphere Q under camera P [4]:

$$C^{*}=P{Q}^{*}{P}^{\mathrm{T}}=K{K}^{\mathrm{T}}-\frac{h_0^2}{R_0^2}Kd{d}^{\mathrm{T}}{K}^{\mathrm{T}}$$

(4)

Denoting h₀/R₀ by μ. From Equation (4), we have:

$$C^{*}=K{K}^{\mathrm{T}}-{\mu }^{2}Kd{d}^{\mathrm{T}}{K}^{\mathrm{T}}=K{K}^{\mathrm{T}}-o{o}^{\mathrm{T}}$$

(5)

with o = μKd, which is the image point of sphere centre X.

2.2. Derivation of the Image Point of the Sphere Centre

From Equation (5), C* can also be written as:

$$\rho C^{\ast }={\omega }^{\ast }-o{o}^{\mathrm{T}}$$

(6)

where $\rho$ is an unknown scale factor, and ${\omega }^{\ast }=K{K}^{\mathrm{T}}$.

Let ${C_1}^{\ast }$, ${C_2}^{\ast }$ be the dual of projected ellipses of spheres $Q_1$, $Q_2$ under camera $P$, respectively; then we have:

$$\{\begin{array}{c}{\rho }_1{C_1}^{\ast }={\omega }^{\ast }-o_1{o_1}^{\mathrm{T}}\\ {\rho }_2{C_2}^{\ast }={\omega }^{\ast }-o_2{o_2}^{\mathrm{T}}\end{array}$$

(7)

where ρ₁, ρ₂ are two unknown scale factors, o₁ = μ₁Kd₁, o₂ = μ₂Kd₂, and μ₁, μ₂, d₁, d₂ have the same meanings as μ in Equation (5) and d in Equation (1).

Let $X_{Q_{1}O}, X_{Q_{2}O}$ denote the centres of spheres Q₁, Q₂, respectively. These two points and the camera centre O determine a plane. Denote the vanishing line of this plane by l₁₂; then we know from [14] that:

$$C_2{C_1}^{\ast }{l}_{12}=\frac{{\rho }_2}{{\rho }_1}{l}_{12}$$

(8)

From Equation (8), it is observed that $l_{12}$ is the eigenvector corresponding to the eigenvalue ${\rho }_2/{\rho }_1$ regarding matrix $C_2{C_1}^{\ast }$, which has two real intersections with each projected ellipse $C_1$ and $C_2$.

Because $l_{12}$ passes through the image points $o_1$ and $o_2$, $l_{12}=o_1\times o_2$. Hence, if we know three projected ellipses $C_1$, $C_2$ and $C_3$ of spheres $Q_1$, $Q_2$ and $Q_3$, the image points $o_1$, $o_2$ and $o_3$ of these three spheres centres will be given by:

$$o_1=l_{12}\times l_{13},\hspace{0.17em}\hspace{0.17em}{o}_2=l_{12}\times l_{23},\hspace{0.17em}\hspace{0.17em}{o}_3=l_{13}\times l_{23}$$

(9)

2.3. Computation of the Depth-Scale Factor μ

Motivated by [18], we give a simple method to solve the depth-scale factor. As known, there are two mutually orthogonal unit vectors ${\overline{d}}_1$ and ${\overline{d}}_2$ perpendicular with $d$ in Equation (5). Denote $\left[\begin{array}{ccc}{\overline{d}}_1& {\overline{d}}_2& d\end{array}\right]$ by $\overline{R}$; then, the dual $C^{*}$ of the ellipse is also expressed as follows:

$$\begin{array}{cc}\hfill C^{*}& =K\left(I-{\mu }^{2}d{d}^T\right)K^{\mathrm{T}}=K\left(\overline{R}{\overline{R}}^{\mathrm{T}}+\overline{R}\hspace{0.17em}diag\{0,0,-{\mu }^2\}\hspace{0.17em}{\overline{R}}^{\mathrm{T}}\right)K^{\mathrm{T}}\hfill \\ \hfill & =K\overline{R}\hspace{0.17em}diag\{1,1,-{\mu }^2+1\}\hspace{0.17em}{\overline{R}}^{\mathrm{T}}{K}^{\mathrm{T}}\hfill \end{array}$$

(10)

Ellipse $C$ is then given by:

$${\rho }_{c}C=K^{-\mathrm{T}}\overline{R}diag\{1,1,\frac{1}{-{\mu }^2+1}\}{\overline{R}}^{\mathrm{T}}{K}^{-1}$$

(11)

where ${\rho }_c$ is an unknown scale factor. If $K$ is known, Equation (11) will be rewritten as:

$${\rho }_c{K}^{\mathrm{T}}CK = \overline{R}diag\{1,1,\frac{1}{-{\mu }^2+1}\}{\overline{R}}^{\mathrm{T}}$$

(12)

Denoting $K^{\mathrm{T}}CK$ by $A$; then we have:

$$A=\frac{1}{{\rho }_c}\overline{R}diag\{1,1,\frac{1}{-{\mu }^2+1}\}{\overline{R}}^{\mathrm{T}}$$

(13)

As known, $\overline{R}$ is an orthogonal matrix, so the singular values of matrix $A$ are $1/\left|{\rho }_c\right|$, $1/\left|{\rho }_c\right|$ and $1/\left[\left|{\rho }_c\right|\left(-{\mu }^2+1\right)\right]$, and $\mu$ can be obtained by SVD. For $\mu =h_0/R_0$ and $h_0>R_0$, $\mu$ is greater than 1. For different spheres with the same radius, μ is proportional to the corresponding h₀.

3. Calibration Principles

3.1. Acquisition of the Rotation Matrix

If $K$ is known, the normalized back-projected vector $d$ of the sphere centre in the camera coordinate frame will be:

$$d=\frac{K^{-1}o}{\Vert K^{-1}o\Vert }$$

(14)

Denote $\mu d$ by $D$; then:

$$D=\mu d=\frac{h_0}{R_0}d=\frac{1}{R_0}{X}_{QO}$$

(15)

where $X_{QO}$ is the sphere centre.

From Section 2.3, we can obtain the depth-scale factor $\mu$, and when there are three spheres Q₁, Q₂ and Q₃ with the same radius centered at $X_{Q_{1}O}$, $X_{Q_{2}O}$ and $X_{Q_{3}O}$, we can obtain three vectors D₁, D₂ and D₃ to determine a plane $\overline{D_1{D}_2{D}_3}$. The plane $\overline{D_1{D}_2{D}_3}$ is parallel to the plane ${\pi }_{\mathrm{s}}$ formed by $X_{Q_{1}O}$, $X_{Q_{2}O}$ and $X_{Q_{3}O}$. Therefore, the normal vector $n$ of the plane ${\pi }_{\mathrm{s}}$ is calculated by:

$$n=(D_2-D_1)\times (D_3-D_1)$$

(16)

Refer to Figure 1; for the BSVS, we can obtain the normal vectors $n_{\mathrm{l}}$ and $n_{\mathrm{r}}$ of the same plane ${\pi }_{\mathrm{s}}$ in the left camera coordinate frame (LCCF) and the right camera coordinate frame (RCCF) respectively. Thus, the following equation stands:

$$n_{\mathrm{r}}=R{n}_{\mathrm{l}}$$

(17)

where R is the rotation matrix between LCCF and RCCF.

Figure 1. Using a double-sphere target to calibrate a binocular stereo vision system.

If there are $m$ spheres with non-coplanar centres and $m\ge 4$, we can obtain the ${\mathrm{C}}_m^3$ equations of Equation (17) to solve R.

3.2. Acquisition of the Translation Vector

In the BSVS, suppose that the left camera is $K_{\mathrm{l}}\left[I_{3\times 3}|0\right]$ and the right camera is $K_{\mathrm{r}}\left[R|T\right]$, and $x_{\mathrm{l}}$, $x_{\mathrm{r}}$ are the image points of the 3D point $X$; then we have:

$$\{\begin{array}{l}{s}_{\mathrm{l}}{x}_{\mathrm{l}}=K_{\mathrm{l}}\left[I|0\right]X\hfill \\ s_{\mathrm{r}}{x}_{\mathrm{r}}=K_{\mathrm{r}}\left[R|T\right]X\hfill \end{array}$$

(18)

where $s_{\mathrm{l}}$, $s_{\mathrm{r}}$ are two unknown scale factors, and R, T are the rotation matrix and translation vector of the BSVS, respectively.

Define a skew-symmetric matrix ${\left[T\right]}_{\times }$ by $T$ as ${\left[T\right]}_{\times }=\hspace{0.17em}\left[\begin{array}{ccc}0& -T_{\mathrm{z}}& T_{\mathrm{y}}\\ T_{\mathrm{z}}& 0& -T_{\mathrm{x}}\\ -T_{\mathrm{y}}& T_{\mathrm{x}}& 0\end{array}\right]$. Denote $p_{\mathrm{l}}=\hspace{0.17em}{K}_{\mathrm{l}}^{-1}{x}_{\mathrm{l}}$, $p_{\mathrm{r}}=\hspace{0.17em}{K}_{\mathrm{r}}^{-1}{x}_{\mathrm{r}}$, and $s=s_{\mathrm{r}}/s_{\mathrm{l}}$; then from Equation (18), we have:

$${\left[T\right]}_{\times }R{p}_{\mathrm{l}}=s{\left[T\right]}_{\times }{p}_{\mathrm{r}}$$

(19)

Denote $R{p}_{\mathrm{l}}$ by ${\widehat{p}}_{\mathrm{l}}$, and $R$ is known, so we can obtain the final expression as:

$$p_{\mathrm{r}}^{\mathrm{T}}{\left[T\right]}_{\times }{\widehat{p}}_{\mathrm{l}}=0$$

(20)

Obviously, Equation (20) is a homogenous equation of $T$. Given at least three pairs of image points of the sphere centres, we can solve $T$ with a scale factor $\kappa$ as $T_0$ (see Appendix A for more details); i.e., $T=\kappa T_0$. Furthermore, based on the known distance ${\mathrm{L}}_0$ between the two sphere centres of the target and the common sense of a positive for the Z coordinate of the sphere centre, $\kappa$ is determined.

3.3. Nonlinear Optimization

Consider that $R$ and $T$ are separately obtained; in this section, we take them as the initial values to obtain more accurate results by combining them.

Establish the object function as:

$$\min F\left(x\right)={\displaystyle \sum _{i=1}^n\left({\displaystyle \sum _{j=1}^{2}d\left(p_{jl}^i,{\widehat{p}}_{jl}^i\right)}+{\displaystyle \sum _{j=1}^{2}d\left(p_{jr}^i,{\widehat{p}}_{jr}^i\right)}+\lambda \left(L^i-{\mathrm{L}}_0\right)\right)}$$

(21)

where $x=\left\{R,T\right\}$, $d()$ represents the Euclidean distance, $p_{jl}^i$, $p_{jr}^i$ are the real non-homogeneous image coordinates of the sphere centres, ${\widehat{p}}_{jl}^i$, ${\widehat{p}}_{rl}^i$ are the non-homogeneous reprojection image coordinates of the sphere centres, $L^i$ is the calculated distance of the two sphere centres, ${\mathrm{L}}_0$ is the known distance of the two sphere centres, $n$ is the number of placement times, and $\lambda$ is the weight factor.

To maintain the orthogonal constraint of the rotation matrix, parameter $R$ is transformed into the Rodriguez vector $r=\hspace{0.17em}{\left(r_x,r_y,r_z\right)}^{\mathrm{T}}$, so $x=\left[r;T\right]$. Considering the principle of error distribution, $\lambda$ is taken to be 10. The Levenberg-Marquardt optimization algorithm is used to obtain the final results of $R$ and $T$.

3.4. Summary

The implementation procedure of our proposed calibration is as follows:

• Calibrate the intrinsic parameters of two cameras.
• Take enough images of the double-sphere target with different positions and orientations by moving the target.
• Extract the subpixel contour points of the projected ellipses using Steger’s method [19], and then perform ellipse fitting [20].
• Compute the image points of each sphere centre, and then conduct image points matching.
• Compute the scale factor $\mu$ of each sphere.
• Solve the structural parameters $R$ and $T$ using the algorithm described in Section 3.1 and Section 3.2.
• Refine the parameters by solving Equation (21).

4. Error Analysis

The general equation of the ellipse is $A{x}^2+Bxy+C{y}^2+Dx+Ey+1=0$, and the coordinates of the ellipse centre are given as follows:

$$\{\begin{array}{c}{x}_c=\frac{BE-2CD}{4AC-B^2}\\ y_c=\frac{BD-2AE}{4AC-B^2}\end{array}$$

(22)

The matrix form of the ellipse is written as $C=\left(\begin{array}{ccc}A& B/2& D/2\\ B/2& C& E/2\\ D/2& E/2& 1\end{array}\right)$. The dual $C^{*}$ of $C$ is given by:

$$C^{*}={\rho }_c^{*}{C}^{-1}=\left(\begin{array}{ccc}\frac{4C-E^2}{4AC-B^2}& \frac{DE-2B}{4AC-B^2}& \frac{BE-2CD}{4AC-B^2}\\ \frac{DE-2B}{4AC-B^2}& \frac{4A-D^2}{4AC-B^2}& \frac{BD-2AE}{4AC-B^2}\\ \frac{BE-2CD}{4AC-B^2}& \frac{BD-2AE}{4AC-B^2}& 1\end{array}\right)$$

(23)

where ${\rho }_c^{*}$ is an unknown scale factor. Combining Equations (22) and (23), we can then obtain the following relationship between the ellipse centre $(x_c,y_c)$ and the elements of matrix $C^{*}$:

$$\{\begin{array}{l}{C}^{*}(1,3)=x_c\\ C^{*}(2,3)=y_c\end{array}$$

(24)

As known, there are many factors affecting the extraction of the ellipse contour points. Regarding the extracted points, the positional deviation may occur due to noise. We will then discuss how the computation of the image point of the sphere centre is influenced under this condition.

Suppose that the shape of the ellipse remains constant and the ellipse does not rotate; then we use the ellipse centre to represent the position of the ellipse. Let $Q$ denote the sphere, $C$ denote the projected ellipse of sphere $Q$, and $(x,y)$ denote the image point of the sphere centre.

To simplify the discussion, consider the condition in which the sphere centre is in the first quadrant of the camera coordinate frame. Because the sphere centre can be located in the first quadrant of the camera coordinate frame by rotating the camera, this discussion can be generalized.

First of all, let us discuss the element $a$ of $C^{\ast }$ (Note: $C^{\ast }(3,3)=1$). Expanding Equation (5) by the replacements $d\mapsto {(\begin{array}{ccc}{d}_x& d_y& d_z\end{array})}^{\mathrm{T}}$, $K\mapsto \left(\begin{array}{ccc}{\alpha }_x& 0& u_0\\ a_y& v_0\\ 1\end{array}\right)$, $o\mapsto \eta \left(\begin{array}{c}x\\ y\\ 1\end{array}\right)$ gives:

$$a=\frac{{\mu }^2{(u_0{d}_z+{\alpha }_x{d}_x)}^2-({\alpha }_x^2+u_0^2)}{{\mu }^2{d}_z^2-1}$$

(25)

The sphere is always in front of the camera, and the sphere centre is located in the first quadrant of the camera coordinate frame, so we have $Z>R_0>0$, $d_x>0$, $d_y>0$, $d_z>0$ and $u_0<x<2u_0$. Based on these equations, we can obtain:

$$a>0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}when\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}X\ge R_0$$

(26)

The details are described in Appendix B.

Next, we discuss the factors that have influences on calculating the image point of the sphere centre. Denoting $C^{\ast }$ by $\left(\begin{array}{ccc}a& b/2& d/2\\ b/2& c& e/2\\ d/2& e/2& 1\end{array}\right)$, we can obtain the following equation from Equation (5):

$$\left(2u_0-d\right)x^2-2\left({\alpha }_x^2+u_0^2-a\right)x+\left({\alpha }_x^2+u_0^2\right)d-2u_{0}a=0$$

(27)

Substituting $d=x_c/2$, $e=y_c/2$ into Equation (27), we get:

$$\left(u_0-x_c\right)x^2-\left({\alpha }_x^2+u_0^2-a\right)x+\left({\alpha }_x^2+u_0^2\right)x_c-u_{0}a=0$$

(28)

By Equation (28), we can obtain:

$$x=\frac{-\left({\alpha }_x^2+u_0^2-a\right)+\sqrt{{\left({\alpha }_x^2+u_0^2-a\right)}^2+4\left(x_c-u_0\right)\left[\left({\alpha }_x^2+u_0^2\right)x_c-u_0\mathrm{a}\right]}}{2\left(x_c-u_0\right)}$$

(29)

From Equation (29), computing the partial derivative of $x$ with respect to $x_c$ gives:

$$\frac{\partial x}{\partial x_c}=\frac{{\alpha }_x^2+u_0^2-a}{2{\left(x_c-u_0\right)}^2}\cdot \frac{\sqrt{M}-\left[2u_0\left(x_c-u_0\right)+\left({\alpha }_x^2+u_0^2-a\right)\right]}{\sqrt{M}}$$

(30)

where $M={\left({\alpha }_x^2+u_0^2-a\right)}^2+4\left(x_c-u_0\right)\left[\left({\alpha }_x^2+u_0^2\right)x_c-u_{0}a\right]$.

Let:

$${\rho }_{x{x}_c}=\frac{{\alpha }_x^2+u_0^2-a}{2{\left(x_c-u_0\right)}^2}\cdot \frac{\sqrt{M}-\left[2u_0\left(x_c-u_0\right)+\left({\alpha }_x^2+u_0^2-a\right)\right]}{\sqrt{M}}$$

(31)

and we can deduce that ${\rho }_{x{x}_c}$ satisfies $0<{\rho }_{x{x}_c}<1$ when ${\alpha }_x/u_0>\sqrt{3}$ is valid (see Appendix C for more details).

Suppose that $\Delta x_c$ is the positional deviation of the fitted ellipse, and $\Delta x$ is the computation error of image point $x$. Equation (30) is then written as:

$$\Delta x={\rho }_{x{x}_c}\Delta x_c$$

(32)

When ${\alpha }_x/u_0>\sqrt{3}$ and $X\ge R_0$ are valid, ${\rho }_{x{x}_c}$ satisfies $0<{\rho }_{x{x}_c}<1$, which shows that computation error $\Delta x$ caused by $\Delta x_c$ is reduced.

Because the extracted ellipse contour points have a positional deviation, the fitted ellipse also has a similar deviation. The following section discusses the solution for how to reduce the computation error of the image point of the sphere centre in this condition.

Firstly, consider the relationship between $a$ and ${\mu }^2$. From Equation (25), we can obtain:

$$\frac{\partial a}{\partial {\mu }^2}=\frac{{\alpha }_x^2{d}_z^2\left[-{\left(\frac{d_x}{d_z}\right)}^2-2\frac{u_0}{{\alpha }_x}\frac{d_x}{d_z}+1\right]}{{({\mu }^2{d}_z^2-1)}^2}$$

(33)

When ${\alpha }_x/u_0>\sqrt{3}$ is valid, we can deduce $\frac{\partial a}{\partial {\mu }^2}>0$. Hence, $a$ is a monotonically increasing parameter with respect to $\mu$ ($\mu >0$).

Second, from Equation (28), we have:

$$\frac{\partial x}{\partial x_c}=\frac{{\alpha }_x^2+u_0^2-x^2}{2\left(x_c-u_0\right)x+{\alpha }_x^2+u_0^2-a}$$

(34)

When ${\alpha }_x/u_0>\sqrt{3}$, we can obtain:

$$\frac{\partial x}{\partial x_c}>0$$

(35)

(see Appendix D for more details).

Suppose $\Delta x_c$ is the positional deviation of the fitted ellipse, and $\Delta x$ is the computation error caused by $\Delta x_c$. Equation (34) can then be written as:

$$\Delta x=\frac{{\alpha }_x^2+u_0^2-x^2}{2\left(x_c-u_0\right)x+{\alpha }_x^2+u_0^2-a}\Delta x_c$$

(36)

Based on Equations (26), (35) and (36), we can deduce that $\Delta x$ has a positive relationship with $\mu$.

Similarly, we can obtain:

$$\Delta y=\frac{{\alpha }_y^2+v_0^2-y^2}{2\left(x_c-v_0\right)y+{\alpha }_y^2+v_0^2-c}\Delta y_c$$

(37)

If $Y\ge R_0$, ${\alpha }_y/v_0<\sqrt{3}$ are valid, we can deduce that $c$ is a monotonically increasing parameter with respect to $\mu$, and also $\Delta y$ has a positive relationship with $\mu$.

Finally, based on the condition described above, we can obtain the following conclusion that the computation errors $\Delta x$, $\Delta y$ both have positive relationships with $\mu$. The smaller the value of μ, the smaller the computation error $\Delta x$, $\Delta y$. By reducing $h_0$ (the depth of the sphere centre) or increasing $R_0$ (the radius of the sphere), $\mu$ is smaller, so $\Delta x$,$\Delta y$ will be smaller. In this way, we can improve the computational accuracy of the image point ($x$,$y$) of the sphere centre.

5. Experiments

5.1. Computer Simulations

Using computer simulations, we analyse the following factors affecting the calibration accuracy: (1) image noise level $\sigma$; (2) the number of placement times N of the target; and (3) the depth-scale factor μ of the sphere.

Table 1 shows the intrinsic parameters of two simulation cameras. The camera distortions are not considered. Suppose that the LCCF is the world coordinate frame (WCF), and set the simulation BSVS structural parameters as r = [−0.03,0.47,0.07]^T, T = [−490,−49,100]^T. The working distance of BSVS is approximately 1000 mm, and the field of view is approximately 240 × 320 mm. The relative deviation of the calibration results and the truth values are used for the evaluation of accuracy. The rotation matrix R is expressed as Rodrigues vector r; then, both the rotation vector r and translation vector T have dimensions 3 × 1. The Euclidean distances of the simulation vectors and true vectors of r and T are used to represent absolute errors; the ratio of absolute error and the corresponding mould of the truth value are then the relative error.

Table 1. Intrinsic parameters of the simulation cameras.

Parameter List	Principal Distance (pixel/mm)	Principal Point (pixel)	Skew	Resolution (pixel)
Left Camera	(5100.0, 5100.0)	(800.0, 600.0)	0	1600 × 1200
Right Camera	(5100.0, 5100.0)	(800.0, 600.0)	0	1600 × 1200

5.1.1. Performance w.r.t. the Noise Level and the Number of Placement Times of the Target

In this experiment, Gaussian noise with 0 mean and $\sigma$ (0.05–0.50 pixel or 0.05–1.00 pixel) standard deviation is added to the contour points of the projected image. For each noise level and the number of placement times N (2, 3, 4) of the target, we perform 200 independent trials, and Figure 2 and Figure 3 show the relative error of R and T under different conditions. As we can see, errors increase with increasing noise level. The relative errors of R and T are even less than 5% with the minimum number of placement times (namely N = 2), and the relative errors are drastically reduced when increasing the number of placement times. For $\sigma$ = 1, N = 4, the calibration errors of R and T are less than 1‰. However, it is clear that the noise level is less than 1 pixel in practical calibration.

Figure 2. Relative errors vs. the noise level of the image points when N = 2.

Figure 3. Relative errors vs. the noise level of the image points when N = 3 and 4.

5.1.2. Performance w.r.t. the Depth-Scale Factor μ

This experiment studies the performance with respect to the depth-scale factor μ, which is the ratio of the depth $h_0$ of sphere centre and sphere radius $R_0$. To ensure the same orientation, we change the value of μ by varying only the sphere radius. Gaussian noise with 0 mean and standard deviation $\sigma$ = 0.50 pixel is added to the contour points of the projected ellipse, and the target is placed N = 3 times. We vary the radius from 4 mm to 36 mm and perform 200 independent trials for each radius. Figure 4 shows the results that the relative errors decrease with increasing radius (that is, the depth-scale factor μ decreases). Note that in practice, if the radius increases too much, the image of sphere may be too large to display in the image plane.

Figure 4. Relative errors vs. the sphere radius of the double-sphere target.

5.2. Real Data

In the experimental results on real data, the BSVS is composed of two AVT-Stingray F504B cameras, a 17 mm Schneider lens and support structures. The image resolution of the cameras is 1600 × 1200 pixels. Figure 5 shows the structure of the BSVS.

Figure 5. Structure of the BSVS.

5.2.1. Intrinsic Parameters Calibration

The Matlab toolbox and a checkerboard target (see Figure 6) are used to calibrate the intrinsic parameters. There are 10 × 10 corner points on the checkerboard target, and the distance between any two adjacent corner points is 10 mm with 5 µm accuracy. Twenty images of different orientations are taken for intrinsic parameters calibration of each camera. Table 2 shows the calibration results.

Figure 6. Checkerboard target and double-sphere target.

Table 2. Intrinsic parameters of the cameras.

Intrinsic Parameters	${\alpha }_x$	${\alpha }_y$	$\gamma$	$u_0$	$v_0$	$k_1$	$k_2$
Left Camera	5086.806	5086.827	0	787.205	595.726	−0.243	1.662
Right Camera	5087.828	5087.638	0	831.764	562.411	−0.240	0.330

5.2.2. Structural Parameters Calibration

The double-sphere target (see Figure 6) is composed of two spheres with the same radius and support structure. The distance between these two spheres is 149.946 mm with 0.003 mm accuracy. Set the LCCF as the WCF. To explore the best of number of placement times, a double-sphere target is placed freely 28 times in the measurement space, and 28 pairs of images are obtained. For evaluating the accuracy of the calibration, another 15 pairs of images of the double-sphere target are captured.

We then randomly select 8, 10, 12, 14, 16, 18, 22 and 28 pairs of images for calibration using our method and obtain several sets of structural parameters. Figure 7 illustrates the extraction and fitting details of a pair of target images.

Figure 7. Extraction and ellipse fitting of images (a,b) are the origin images and (c,d) are the processed images.

The calibrated BSVS is used to measure the distance between two sphere centres of the double-sphere target by means of another 15 pairs of measured images. Root-mean-square (RMS) errors of these measured values are taken as evaluation criteria of calibration accuracy. Table 3 shows the results, and Figure 8 displays the relative errors and absolute errors of RMS.

Table 3. Comparison of measured values (mm).

Place Times	8	10	12	14	16	18	22	28
Image 1	149.916	149.923	149.906	149.953	149.898	149.950	149.879	149.928
Image 2	149.769	149.811	149.800	149.817	149.776	149.799	149.727	149.785
Image 3	149.930	149.877	149.852	149.885	149.842	149.915	149.904	149.900
Image 4	149.843	149.895	149.934	149.972	149.939	149.847	149.870	149.883
Image 5	149.887	149.926	149.959	150.000	149.970	149.897	149.908	149.920
Image 6	149.911	149.913	149.939	149.971	149.971	149.922	149.932	149.915
Image 7	149.925	149.999	150.010	149.957	149.931	149.856	149.914	149.926
Image 8	149.854	149.904	149.908	149.854	149.825	149.769	149.844	149.840
Image 9	150.068	150.087	150.082	150.028	149.995	149.961	150.058	150.036
Image 10	149.934	149.909	149.888	149.841	149.800	149.809	149.920	149.878
Image 11	149.896	149.969	149.980	150.027	149.973	149.953	149.872	149.948
Image 12	149.987	150.049	150.047	150.084	150.009	150.009	149.956	150.028
Image 13	149.998	150.045	150.036	150.074	149.992	150.009	149.963	150.031
Image 14	149.940	149.896	149.879	149.928	149.864	149.918	149.926	149.925
Image 15	149.914	149.866	149.850	149.892	149.846	149.901	149.899	149.893
RMS (mm)	0.073	0.075	0.079	0.080	0.084	0.084	0.079	0.071

Figure 8. Relative errors and absolute errors of RMS errors with different number of placement times.

From Figure 8, we can see that when the number of placement times is greater than 16, the errors begin to monotonically decrease. Consequently, we must place the double-sphere target approximately 16 times in the experiment.

In this experiment, we have the calibration parameters with 18 times as the final result. By using the calibration parameters, we reconstruct the target positions in space, and the results are shown in Figure 9. For comparison, the Matlab toolbox method is also carried out for structural parameters calibration. Table 4 shows the calibration results of these two methods.

Figure 9. 3D reconstruction of the spatial position of the target by our method.

Table 4. Comparison of the structural parameters.

Structure Parameters	$r_x$	$r_y$	$r_z$	$t_x$	$t_y$	$t_z$
Our Method	−0.0046	0.5993	0.0302	−473.95	−5.622	121.70
Toolbox Method	−0.0071	0.6014	0.0290	−474.97	−7.181	122.36

5.2.3. Accuracy Evaluation

To evaluate the accuracy of the calibration, another 10 pairs of images of the checkerboard target are captured. In addition, 15 pairs of previous captured images of the double-sphere target are also used for accuracy evaluation.

Using the calibrated BSVS by these two methods, we measure the distance between two sphere centres of the double-sphere target and each distance between two adjacent corner points of the checkerboard target. The RMS errors of these measured values are taken as the evaluation criteria of calibration accuracy.

(a)

Measure the double-sphere target

Figure 10 displays the measured results of 15 distances, and Table 5 shows a comparison of the errors.

Figure 10. Results of measurements by two methods.

Table 5. Comparison of measurement accuracy of the double-sphere target (mm).

Methods	RMS	Relative Error (%)	Standard Error
Our method	0.084	0.056	0.073
Toolbox method	0.111	0.074	0.091

The results in Table 5 show that the RMS error of our algorithm is 0.084 mm, and the relative error is approximately 0.06%; the RMS error of the toolbox method is 0.111 mm, and the relative error is approximately 0.07%. Consequently, it is obvious that our method is slightly better than the toolbox method in measuring the distance of two sphere centres. The standard errors of the measured values show that the measured results by our method are more stable.

(b)

Measure the checkerboard target

The details of the checkerboard target have been introduced in the preceding paragraph. Because the measured values are too numerous, we represent these values by the scatter plots in Figure 11. Table 6 shows the comparison of errors. For an intuitive display of the calibration results of our method, we reconstruct the 3D points of the checkerboard target, and Figure 12 shows the results.

Figure 11. Results of measuring the checkerboard target by two methods.

Table 6. Comparison of the measurement accuracy of the checkerboard target (mm).

Method	RMS	Relative Error (%)	Standard Error
Our method	0.008	0.084	0.0055
Toolbox method	0.005	0.053	0.0053

Figure 12. Reconstructed 3D points by our method.

As we can see in Table 6, the RMS errors are 0.008 and 0.005 mm, and the relative errors are 0.08% and 0.05%, respectively. When measuring the distance of the corner points of the checkerboard, the toolbox method is slightly better. According to the standard errors, we can find that these two methods are both reasonably stable.

As we have observed from the accuracy evaluation, our method exhibits similar calibration accuracy to the toolbox method. The measured errors of both methods are less than 0.9‰. The sphere has an excellent property of complete symmetry, which can effectively avoid the simultaneously visible problem of target features in multi-camera calibration.

The toolbox method based on the plane target is a typical method. However, the algorithm usually requires the plane target to be placed in different orientations so as to provide enough constraints, which increases the possibility of occurrence of the simultaneous visibility problem. When the angle between the principal rays of the two cameras is large, it is difficult to capture high quality image at the same time, so the calibration accuracy would be heavily influenced. Figure 13 shows a comparison of the plane target and the double-sphere target in calibration when the angle between the principal rays is large. As seen in Figure 13, the right image of the plane target is so tilt that the corners cannot be accurately extracted, while both images of the double-sphere target have the same high level of clarity and the contours can be accurately extracted. Therefore, it is obvious that the double-sphere target will perform better than the plane target.

Figure 13. Images of the plane target and the double-sphere target (a,c) are the left images; and (b,d) are the right images.

6. Conclusions

In the paper, we describe a method to calibrate structural parameters. This method requires a double-sphere target placed a few times in different positions and orientations. We utilize the normal vectors of spatial planes to compute the rotation matrix and use a linear algorithm to solve the translation vector. The simulations demonstrate how the noise level, the number of placement times and the depth-scale factor influence the calibration accuracy. Real data experiments have shown that when measuring the object with a length of approximately 150 mm, the accuracy is 0.084 mm, and when measuring 10 mm, the accuracy is 0.008 mm.

If the sphere centres are all coplanar, our method will fail. Therefore, the double-sphere target should be placed in different positions and orientations to avoid this degradation. Because the calibration characteristic of the sphere is its contour, we should prevent the double-sphere target from completely mutual occlusion. As mentioned above, the two spheres should have the same radius. However, if the two sphere centres are unequal, our method can still work. If the ratio of the two radii is known, the ratio value should be considered when recovering the intermediate parallel planes; other computation procedures remain unchanged. If the ratio is unknown, three arbitrary projected ellipses with the same sphere should be selected to recover the intermediate parallel plane. Furthermore, this target must be placed at least four times. Obviously, such a target provides fewer constraints than the target with a known ratio of sphere radii when solving the rotation matrix. To calibrate the BSVS with a small public view while guarantee high accuracy, we can couple these two spheres with large radii to form a double-sphere target. In multi-camera calibration, using the double-sphere target can avoid the simultaneous visibility problem and performs well.