Camera Calibration with Phase-Shifting Wedge Grating Array

Abstract

Planar targets with known features have been widely used for camera calibration in various vision systems. This paper utilizes phase-shifting wedge grating (PWG) arrays as an active calibration target. Features points are encoded into the carrier phase, which can be accurately calculated using the phase-shifting algorithm. The 2π-phase points are roughly extracted with edge detection and then optimized by windowed bicubic fitting with sub-pixel accuracy. Two 2π-phase lines for each PWG are obtained using linear fitting method. The PWG centers that are used as feature points are detected through computing the intersections of 2π-phase lines. Experiment results indicate that the proposed method is accurate and reliable.

1. Introduction

Camera calibration has been extensively studied for few past decades, which attempts to determine the intrinsic and extrinsic parameters of cameras [1,2,3]. Through capturing calibration targets with known features, one can acquire an exact set of one-to-one correspondences between the world and the image coordinates. Camera parameters are then estimated from these correspondences. Thus, the feature detection will directly affect the calibration accuracy. Numerous studies have focused on developing patterns with distinctive features that can be accurately localized in the images [4]. Among various existing patterns, squares [5,6,7,8], and circles [9,10,11,12] have become popular due to their ease of use. In order to avoid the influence of the fabrication to the calibration results and further simplify the calibration procedures, the digital displays have been applied to the camera calibration, which provide an alternative way for planar targets [13,14,15,16,17,18,19].

When compared with the conventional calibration object (a planar target marked by printed patterns), digital displays have many advantages, including adjustable brightness, guaranteed flatness, and known pixel sizes in spite that they may be more expensive and inconvenient to move with power and data cables. Arbitrary features can be realized through simple programming on the computer. Most importantly, digital displays make active targets be possible. Therefore, more and more researchers used phase-shifting patterns, which that could solve the problem of the calibration imprecision that is caused by the coordinate extraction errors of the feature points [14,17] as calibration targets with the advantages of digital displays. Schmalz et al. [14] performed camera calibration with two phase-shifting sequences, one horizontal, and one vertical. Huang et al. [15] also applied horizontal/vertical phase-shifting fringe patterns for camera calibration, and made improvement on feature detection and optimization. Ma et al. [16] used horizontal/vertical fringe patterns as calibration targets, but extracted the two-dimensional (2D) phase-difference pulses as feature points. Xue et al. [17] applied concentric circles and wedge gratings for solving the vanishing points, and then estimated the camera parameters that were based on the orthogonal vanishing point conjugate. Ha et al. [18] proposed an accurate camera calibration method for defocused images. The calibration target included two pairs of horizontal/vertical binary patterns and one black pattern. Bell et al. [19] encoded the feature points into the carrier phase of horizontal/vertical binary patterns, which can be accurately recovered even under severe defocus. These above methods could achieve sufficient accuracy with active targets; yet they require five-frame patterns at least. In other words, we need capture five or more images at each viewpoint, which is tedious and laborious.

In order to obtain accurate calibration results and reduce the workload of capturing images marked by phase-shifting patterns. This paper presents an active calibration target that consists of phase-shifting wedge grating (PWG) arrays. The approach takes advantage of the high precision of the phase-shifting fringes extraction [14,16,17] and uses PWG arrays as calibration pattern to obtain the wrapped phase of the captured images. Meanwhile, the intersections of 2π-phase points that were regarded as feature points can be extracted accurately. The adoption of three-frame patterns could significantly reduce the workload and improve the efficiency of camera calibration. A liquid crystal display (LCD) is served as the planar target to display these patterns, which is flat, effortless and flexible. The three-step phase-shifting algorithm is used for the wrapped phase calculation. The 2π-phase points are roughly extracted with edge detection and are then optimized by windowed bicubic fitting with sub-pixel accuracy. Two 2π-phase lines for each PWG are obtained with linear fitting method. The PWG centers are located through computing the intersections of 2π-phase lines. Camera parameters are estimated using these PWG centers as feature points. The PWG arrays were compared with checkerboard, circle patterns and horizontal/vertical phase-shifting fringe patterns with the feature detection method in [16]. The calibration results confirmed the accuracy and the efficiency of the proposed method.

The rest of the paper is organized as follows: Section 2 explains the principle of the proposed method, including camera model, calibration pattern, and feature detection. Section 3 presents the experimental results to verify the proposed method. Lastly, Section 4 summarizes the paper.

2. Principle

2.1. Camera Model

The camera is modeled as the classic pinhole. Let $\mathit{P}={\left(X_W,Y_W,Z_W,1\right)}^T$ and $\mathit{P}={(u,v,1)}^T$ be the homogeneous coordinates of the 3D world point and the 2D image point, respectively. The relationship between $\mathit{P}$ and $\mathit{P}$ can be mathematically described as [1]:

$$\lambda \mathit{P}=\mathit{K}[\mathit{R}\mathit{t}]\mathit{P},$$

(1)

$$\mathit{K}=\left[\begin{array}{ccc}{f}_u& \gamma & u_0\\ 0& f_v& v_0\\ 0& 0& 1\end{array}\right],$$

(2)

$$\mathit{R}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right],\mathit{t}=\left[\begin{array}{c}{t}_1\\ t_2\\ t_3\end{array}\right],$$

(3)

where $\lambda$ is an arbitrary scale factor, $\mathit{K}$ called the intrinsic parameters contains focal length $\left(f_u,f_v\right)$, principal point $\left(u_0,v_0\right)$, and skew angle parameter $\gamma$. The rotation matrix $\mathit{R}$ and the translation vector $\mathit{t}$ denote the extrinsic parameters. Generally, camera lens consisting of several optical elements does not obey the ideal pinhole model, which will introduce nonlinear distortions to correct the captured images [3]. Radial and tangential distortions are the two most common that can be approximated as [19]:

$$\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\end{array}\right]=\left[\begin{array}{c}u\left(1+k_1{r}^2+k_2{r}^4\right)+2p_{1}uv+p_2\left(r^2+2u^2\right)\\ v\left(1+k_1{r}^2+k_2{r}^4\right)+p_1(r^2+2v^2)+2p_{2}uv\end{array}\right],$$

(4)

where $\left(u,v\right)$ and $\left(u^{\prime },v^{\prime }\right)$ are the distorted and undistorted image point in normalized image coordinates, $k_1$ and $k_2$ are the radial distortion coefficients, $p_1$ and $p_2$ are tangential distortion coefficients, and $r=\sqrt{{\left(u-u_0\right)}^2+{\left(v-v_0\right)}^2}$.

2.2. Calibration Pattern

Phase-shifting methods are widely used in optical metrology, which can provide very high precision and dense coding [14,20]. This paper encodes the feature points into the carrier phase of three PWG arrays with a phase shift of $2\pi /3$. The wrapped phase of the patterns can be recovered with phase-shifting algorithm. The intensities of the PWG are, respectively, described as [17]:

$$\{\begin{array}{l}{I}_1\left(x,y\right)=A+B\cos (2\pi f\theta -2\pi /3)\hfill \\ I_2\left(x,y\right)=A+B\cos (2\pi f\theta )\hfill \\ I_3\left(x,y\right)=A+B\cos (2\pi f\theta +2\pi /3)\hfill \end{array},$$

(5)

$$\theta \left(x,y\right)=\arctan \left(\frac{y-y_0}{x-x_0}\right),$$

(6)

where $\left(x,y\right)$ denotes one point on the PWG; A is the average intensity, and B is the intensity modulation; A and B are constant, usually we set $A=B=0.5$; $2\pi f\theta$ is the phase to be determined, and f is the frequency, $\theta \left(x,y\right)$ is the angle with x-axis that is expressed as Equation (6); the radius $r=\sqrt{{\left(x-x_0\right)}^2+{\left(y-y_0\right)}^2}$ is the Euclidean distance between the points $\left(x,y\right)$ and the PWG center $\left(x_0,y_0\right)$, and $r_{\min }<r<r_{\max }$ is used to restrict the size of the PWG. Figure 1a shows the PWG arrays contains 3 × 3 uniform PWGs with $f=2/\pi$ and a phase shift of $2\pi /3$. Figure 1b shows a single PWG, also the basic unit of the second PWG array, the corresponding wrapped phase of the PWG is shown in Figure 1c.

Obviously, the 2π-phase points are distributed on two straight lines that intersect at the PWG center, as shown in Figure 1d. If we can detect the two 2π-phase lines in the image, the PWG center can be located through computing the intersection. Since one PWG only has one center, some identical PWGs are arranged to generate PWG array, and their centers are used as feature points. Figure 2a shows the PWG array consisting of 3 × 3 PWGs. Specifically, for the $M\times N$ PWG array, the PWG at $m-th$ row and $n-th$ column is centered at:

$$\{\begin{array}{l}{x}_0^{mn}=x_0+(n-1)d\\ y_0^{mn}=y_0+(m-1)d\end{array},$$

(7)

where $m=0,1,\cdots ,M-1$; $n=0,1,\cdots ,N-1$; $d$ indicates the distance between adjacent PWG centers between the horizontal or vertical direction, which must be greater than $2r_{\max }$ to avoid interference between adjacent PWGs. Figure 2b shows its wrapped phase image.

2.3. Feature Detection

The emphasis of feature detection is how to detect the PWG centers. The detailed procedures of our method can be summarized, as follows:

• Let $J_1\left(u,v\right)$, $J_2\left(u,v\right)$, and $J_3\left(u,v\right)$ respectively represent the three PWG array images that are captured at the same viewpoint. Adding up the three images, the phase-modulated area $\mathsf{\Omega }$ (Figure 3a) for PWG arrays can be obtained with a suitable gray threshold $T$:

  $$\mathsf{\Omega }=\{\begin{array}{cc}1,\hfill & if{J}_1+J_2+J_3>T\\ 0,\hfill & otherwise\end{array}.$$

  (8)

  Label the connected components in the phase-modulated area $\mathsf{\Omega }$, the sub-masks ${\mathsf{\Omega }}_k$ for each PWG can be achieved. Calculate the centroid $\left(u_c^k,v_c^k\right)$ of each ${\mathsf{\Omega }}_k$, which can be regarded as the rough location of the PWG center.
• Based on the three-step phase-shifting algorithm, the wrapped phase ranging from [0, 2π] can be calculated as:

  $$\varphi (u,v)=\arctan \left(\sqrt{3}\frac{J_1-J_3}{2J_2-J_1-J_3}\right).$$

  (9)

  Clearly, $\varphi (u,v)$ abruptly changes at the 2π-phase jump areas. Thus, through edge detection (e.g., Sobel, Canny), we can easily extract the 2π-phase points $\left({\widehat{u}}_i,{\widehat{v}}_i\right)$ with pixel-level accuracy. These edge points $\left({\widehat{u}}_i,{\widehat{v}}_i\right)$ are distributed on several lines, as shown in Figure 3b.
• For each $\left({\widehat{u}}_i,{\widehat{v}}_i\right)$, we select its $7\times 7$ neighborhood points $\left({\widehat{u}}_{ij},{\widehat{v}}_{ij}\right)$ with wrapped phase $\varphi ({\widehat{u}}_{ij},{\widehat{v}}_{ij})$. The $\varphi ({\widehat{u}}_{ij},{\widehat{v}}_{ij})$ with 2π-phase jumps can be unwrapped as:

  $$\varphi \prime (x_{ij},y_{ij})=\{\begin{array}{cc}\varphi (x_{ij},y_{ij}),\hfill & \hfill \varphi (x_{ij},y_{ij})\ge \pi \\ \varphi (x_{ij},y_{ij})+2\pi ,\hfill & \hfill otherwise\end{array}.$$

  (10)

  Let $R_{ij}$ be the Euclidean distance between $\left({\widehat{u}}_{ij},{\widehat{v}}_{ij}\right)$ and its corresponding center $(u_c,v_c)$. Then, we can establish the function relations $x=f_x(\mathsf{\Phi },R)$ and $y=f_y(\mathsf{\Phi },R)$ by the windowed bicubic fitting algorithm. The optimized 2π-phase points $(\tilde{u},\tilde{v})$ can be obtained through substituting $\mathsf{\Phi }=2\pi$ and $R=\sqrt{{({\widehat{u}}_{ij}-u_c)}^2+{({\widehat{v}}_{ij}-v_c)}^2}$. Those 2π-phase points with sub-pixel accuracy are utilized to locate the real center $\left(u_0,v_0\right)$.
• When all of the 2π-phase points are optimized, two 2π-phase straight lines for each PWG can be obtained with linear fitting algorithm. The intersection point of the two straight lines can be treated as the PWG center, as well as the feature point. Using the one-to-one correspondences between the world and image coordinates of the PWG centers, the camera parameters can be estimated.

3. Experiments

The performance of the proposed method has been evaluated with some experiments. The experimental setup mainly includes a camera to be calibrated and an LCD used to display the calibration patterns. The camera model is IOI Flare 2M360-CL (IO Industries Inc., London, ON, Canada) with a resolution of 2048 × 1088 pixels. The camera lens has a zoom lens with a focal length of 12–35 mm. The LCD model is Philips 226V4L (Koninklijke Philips N.V., Amsterdam, Holland, The Netherlands), with a resolution of 1920 × 1080 pixels and a pixel pitch of 0.248mm. To begin with, the camera fixed on a device was placed at a suitable distance from the LCD and the optical axis of the camera was perpendicular to the screen. Then, a suitable focal length was adjusted to capture the sharp image of the pattern. The XY plane of the world reference frame was located on the LCD, and the Z axis was perpendicular to the planar monitor. Therefore, the feature points had the same $z=0$, which could simplify the calculation procedures. All of the feature detection methods were implemented in MATLAB R2014a (MATLAB 8.3, MathWorks, Inc., Natick, MA, USA, 2014) and Visual C++ 2012 (Visual Studio 2012 Update 5, Microsoft Corporation, Redmond, Washington, United States, 2015). The camera intrinsic parameters were estimated using the standard calibration method implemented in the MATLAB environment, with the assistance of Camera calibration toolbox [21] and Image Processing Toolbox [22]. The calibration accuracy can be assessed on the basis of the root-mean-square re-projection error (RMSE) of feature points [5,14], which can be computed by the equation:

$$\mathrm{RMSE}=\sqrt{\frac{1}{L}{\displaystyle \sum _{l=1}^L\left[{\left(u_l-{\widehat{u}}_l\right)}^2+{\left(v_l-{\widehat{v}}_l\right)}^2\right]}},$$

(11)

where $L$ is the total number of the feature points, $\left(u_l,v_l\right)$ are the extracted feature point locations, $\left({\widehat{u}}_l,{\widehat{v}}_l\right)$ are the re-projection point locations, which were computed from the known world coordinates of the feature point.

The PWG array used to calibrate the camera consists of 6 × 6 uniform PWGs, which can produce 36 feature points per viewpoint. The minimum and maximum radii of the PWG are $r_{\min }=20$ pixels and $r_{\max }=65$ pixels. The distance between adjacent PWG centers is $d=150$ pixels. The frequency is $f=2/\pi$.

3.1. PWG Arrays VS Checkerboard and Circle Patterns

In this experiment, checkerboard and circle patterns were selected as contrasting objects, which also designed to have 6 × 6 feature points with same locations. Both of the performances for three-step and four-step PWG arrays were explored. The four patterns were displayed on the LCD separately. Through adjusting the location of the camera, we collected four groups of pattern images from 10 different viewpoints. We obtained nine images per viewpoint: one for checkerboard, one for circle patterns, three for three-step PWG arrays, and four for four-step PWG arrays. Figure 4 shows the captured images of the four patterns.

After capturing the calibration target images, we extracted the feature points from the images: the corners for checkerboard with Camera Calibration Toolbox for Matlab [21], the centers for circles with OpenCV function findCirclesGrid [23], and the centers for three-step and four-step PWG arrays with the feature detection method in Section 2.3. Figure 5a–c show the three-step PWG array images and Figure 5d shows their wrapped phase with detected feature points.

Figure 6 shows the re-projection errors of feature points for the four patterns. The results clearly show that PWG arrays have an improvement in RMSEs than checkerboard and circle patterns.

Table 1. Camera intrinsic parameters estimated from different patterns.

	Parameter	${\mathit{f}}_{\mathit{u}}$ (Pixel)	${\mathit{f}}_{\mathit{v}}$ (Pixel)	${\mathit{u}}_0$ (Pixel)	${\mathit{v}}_0$ (Pixel)	${\mathit{k}}_1$	${\mathit{k}}_2$	RMSE (Pixel)
Pattern
Checkerboard		2964.7	2964.3	1011.3	545.5	−0.145	0.755	0.066
Circles		2963.5	2962.8	1012.8	544.8	−0.146	0.754	0.063
Three-step PWG arrays		2968.2	2968.0	1010.3	546.5	−0.155	0.827	0.054
Four-step PWG arrays		2965.1	2964.8	1012.1	546.3	−0.156	0.832	0.045

shows the intrinsic parameters estimated from the four patterns. The calibration results are very close to each other. The tangential coefficients as well as the skew angle parameter are extremely small. It is sufficient to keep radial coefficients for nonlinear distortion. The RMSEs in the presented method are smaller than checkerboard and circles patterns and the four-step PWG arrays are more accurate than three-step PWG arrays. To our knowledge, four-step pattern always recover the wrapped phase with higher precision than three-step pattern. Thus, the calibration results of PWG arrays are more accurate than checkerboard and circle patterns and the calibration accuracy will be higher and higher with the step increases.

3.2. PWG Arrays VS Horizontal/Vertical Phase-Shifting Fringe Patterns

In this experiment, fringe patterns generated with the same phase shift of $2\pi /3$ and feature point locations were chosen to compare with three-step PWG arrays. Both of them were phase-shifting patterns, and the fringe patterns consisted of horizontal and vertical phase-shifting fringe patterns. We performed the experimental procedures in the same way that is described in Section 3.1. Nine images per viewpoint consisted of three PWG images, three horizontal fringe pattern images and three vertical phase-shifting fringe pattern images were captured and ninety images were obtained in total. Figure 7 shows the captured images of the two phase-shifting patterns.

Then, we extracted the feature points from the images: the centers for three-step PWG arrays and the centers for three-step phase-shifting fringe patterns with the feature detection method in paper [16], which extracted the two-dimensional (2D) phase-difference pulse signals that were optimized by interpolation as feature points. The workload of capturing images for PWG arrays was reduced by half when compared with the fringe patterns. Figure 8 shows the 2D phase-difference pulses.

Table 2. Camera intrinsic parameters estimated from different phase-shifting patterns.

	Parameter	${\mathit{f}}_{\mathit{u}}$ (Pixel)	${\mathit{f}}_{\mathit{v}}$ (Pixel)	${\mathit{u}}_0$ (Pixel)	${\mathit{v}}_0$ (Pixel)	${\mathit{k}}_1$	${\mathit{k}}_2$	RMSE (Pixel)
Pattern
Three-step phase-shifting fringe patterns		2588.2	2588.4	1008.7	544.4	−0.015	0.081	0.085
Three-step PWG arrays		2561.6	2561.5	1010.3	546.8	−0.005	0.077	0.046

shows the intrinsic parameters estimated from the two phase-shifting fringe patterns. The RMSE for the presented method is smaller than that for the method in paper [16]. The difference in RMSEs was caused by the precision of feature detection. The method in paper [16] extracted the feature points based on the sum of two 2π-phase point images. While the addition operation brought more noise, which directly influenced the accurate extraction of feature points than the presented method that used one 2π-phase point image. Therefore, the proposed method is accurate and reliable.

4. Conclusions

In this study, PWG arrays are used as calibration patterns for accurate camera calibration, which is displayed on an LCD. The 2π-phase points are detected and optimized to obtain sub-pixel accuracy. Then, the PWG centers used as feature points are precisely extracted from the phase by using linear fitting method rather than directly from the intensity. When compared to horizontal/vertical phase-shifting fringe patterns, it is convenient and time-saving with the workload of capturing images reduced by half. Experimental results indicate that the proposed method is accurate and reliable.