This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
A novel and practical self-calibration method for misaligned non-central catadioptric cameras is proposed in this paper. Without the aid of any special calibration patterns in the scene, the developed method is able to automatically estimate the pose parameters of the mirror with respect to the perspective camera. First it uses the ellipse corresponding to the mirror boundary in the image to compute the possible solutions for mirror postures. With two pose candidates, thereafter we propose a novel selection method to find the correct solution by using the image of the lens boundary. The whole calibration process is automatic and convenient to carry out since nothing more than a single image acquired from the catadioptric camera is needed. Experimental results both on synthesized and real images prove our success.
A catadioptric camera system usually consists of a revolutionary symmetric reflective mirror and a conventional perspective camera observing a scene reflected by the mirror. Such vision systems featuring the advantage of large field of view are being increasingly used in many applications [1–3], such as mobile robot navigation, video surveillance, virtual reality, outer space exploration and 3D reconstruction. Depending on whether they pose a single viewpoint [4,5], catadioptric cameras can be classified as central or non-central imaging systems.
Since camera calibration is a preliminary step in most applications, a variety of calibration methods for catadioptric systems have been reported. Currently most of these calibrations focus on central systems due to their popularity and relatively mature computing theories. However there are only a few combinations of mirror type and camera can have the opportunity to meet the single viewpoint requirement, which are insufficient for lots of applications. Furthermore when misalignment happens, all of the central catadioptric cameras become non-central, aside from when the mirrors are not the type in the central list [6]. Therefore non-central catadioptric systems are more general and the researches of calibration on them are of greater importance.
Calibration works on non-central systems can be divided into two categories. The first one focuses on non-central mirrors with unknown parameters and tries to model and calibrate them [5,7–9].The model is called caustic surface which represents the actual locus of the viewpoints. They use known light patterns [5] or known camera motion and point correspondence [7] to calculate the caustic. In [9], it was reported that three polarization images taken with different orientations could estimate the caustics of any combination of specular mirror and lens. During the calibration they assume the perspective camera is co-axial with the symmetric axis of the mirror. The second category [3,6,10,11] models the non-central system as a generalized camera where each pixel in the image corresponds to an incident ray reflected by a special point on the mirror. Micusik et al. [6] calibrated non-central cameras in two steps. First an approximate central camera is calibrated and then the non-central model was used to finish the 3D reconstruction. In [3] and [11] the forward projection model to calculate the reflective point on the mirror by non-iterative algorithms was proposed and used in motion estimation and 3D reconstruction. They all assume the coaxial mounting of the mirror and the perspective camera. Another major part of the researches in this category can deal with the misalignment between the camera and the mirror. Fabrizio et al. [10] presented a method which uses the internal and external mirror boundaries to calibrate the perspective camera and its posture relative to the mirror. In their configuration, a black needle was specially designed and mounted at the bottom of the mirror to provide the internal calibration circle. Mashita et al. [12] used the mirror boundary and a number of lines at infinity to estimate the mirror posture. Nonlinear optimization algorithms are commonly used in the calibration [11,13–15]. Strelow et al. [14] and Goncalves and Araujo [15] proposed methods for calibrating part and full parameters of the entire non-central camera systems by using bundle adjustment. They use preset calibration patterns in the scene and calculate the unknown parameters by minimizing the re-projection error. The accuracy of these methods mainly relies on the goodness of initial values because of nonlinear optimization. The majority of those misalignment-calibration methods can only deal with slight misalignment due to the assumptions they make in their algorithms. Caglioti [16] proposed a calibration algorithm for large off-axis non-central cameras by using line patterns. However they need the profile of the mirror to be seen in the image, which is not very practical in lots of cases. Recently Agrawal [3] extended their original work [17] to an off-axis forward projection model.
In this paper, we propose a novel self-calibration method for non-central catadioptric systems by using the camera itself as calibration pattern. Our method belongs to the second category listed above and is not subject to the constraint of slight misalignment. Like [10] and [12], we use the mirror boundary as the main reference pattern. However, unlike [10] and [12], where specially-designed needles or lines are used as extra calibration patterns, we do not need any extra calibration patterns apart from camera itself. We use the lens boundary of the perspective camera as extra condition. To our knowledge it is the first time that the calibration of anon-central system without using any extra calibration patterns apart from the camera itself is reported.
We focus on calibrating the relative pose between the perspective camera and the mirror. First, we use the ellipse-shaped mirror boundary image to estimate the four possible mirror postures based on Chen's extrinsic calibration method [18]. In order to remove the ambiguity of mirror posture, we then present a novel selection method making use of the imaged lens boundary to find the correct solution. Experiments conducted both on simulated data and real images confirm the performance of the proposed method.
In the following section the general model of non-central camera is briefly explained. After giving the algorithm idea in Section 3, Sections 4, 5, and 6 describe the three main steps of our algorithm in detail. Experimental results both on simulated data and real image are represented in Section 7. Finally, conclusions are given in the last section.
Camera ModelGeneral Configuration of the Camera System
Figure 1 shows the general configuration of a catadioptric camera system, where the camera and the mirror coordinate systems are denoted with the subscripts “C” and “M” respectively. Due to the misalignment the rigid body transformation between the two coordinate systems, i.e., the rotation R_{M} and translation T_{M}, drifts from the ideal configuration and makes the system non-central.
The full model of the non-central system should include the parameters of the mirror and the conventional perspective camera as well as the relative posture between the mirror and the camera. Generally the manufacture of the mirror can be fairly accurate and the deviation from the theoretic design could be very small. Meanwhile the intrinsic parameters of the perspective camera can also be computed in advance by some mature algorithms like the calibration toolbox from Jean-Yves Bouguet [19], and they do not change when misalignment of catadioptric system happens. Therefore we believe it is reasonable and valuable to find a good self-calibration method by computing the relative posture between the mirror and camera given their intrinsic parameters.
Perspective Camera Model
Let X_{C} = (X_{C},Y_{C},Z_{C})^{T} be the coordinates of a 3D point in the camera coordinate system and ũ=(u,v,1)^{T} be the homogenous coordinates of the image point respectively, according to the pinhole model we have:
su∼=KXcwhere s is a scale factor and K is the camera intrinsic matrix. For off-the-shelf camera the radial distortion in the image has to be removed before calibration with Equation (2):
{xC=xCd/(1+k1(xC2+yC2)+k2(xC2+yC2)2)yC=yCd/(1+k1(xC2+yC2)+k2(xC2+yC2)2)where k_{1}, k_{2} are radial distortion coefficients (x_{c}, y_{c}) and (x_{Cd}, y_{Cd}) are the undistorted and distorted normalized image coordinates, respectively. The intrinsic parameters of the perspective camera can be calibrated independently and are assumed known in advance throughout the paper.
Algorithm Idea
The idea of the calibration algorithm will now be described. Before calibration a calibrating image should be acquired from the catadioptric camera. Different from most of the existing calibration methods, there is no any special calibration pattern in the environment. The only requirement for the image is the mirror boundary and the lens boundary or at least part of them should be clearly visible in the image, which is true for most of the catadioptric cameras. Now it is required to determine the mirror posture relative to the camera coordinate system. The steps of the algorithm are as follows.
Image preprocessing and robust ellipse extracting. The images of the mirror and lens boundary can be extracted and fitted with ellipses, which actually encodes the mirror posture (Section 6).
Mirror posture computing based on the imaging ellipse of the mirror boundary. It is a non-iterative method and finally two posture candidates are obtained (Section 4).
Correct posture selecting based on the image of lens boundary. During the selection process, the unknown position of the real lens boundary relative to the optical center should also be estimated simultaneously (Section 5).
The steps of this algorithm will be described in detail in the subsequent sections of the paper. The main subject of this paper comprises the last two steps of this algorithm, which will be described first. The first step is of peripheral interest, and a description of the robust method used for automatic ellipse extracting and fitting will be postponed to a later section.
Computing Mirror Posture Candidates
As show in Figure 1, the mirror posture can actually be represented by the transformation between the camera coordinate system O_{C} − X_{C}Y_{C}Z_{C} and the mirror coordinate system O_{M} − X_{M}Y_{M}Z_{M}. With intrinsic parameters of the perspective camera in hand, the next step is to compute the possible solutions of mirror postures by using the image of the mirror boundary. We apply Chen's method [18] to accomplish this. Considering a camera coordinate system O_{C} − X_{C}Y_{C}Z_{C} that the origin O_{C} is the optical center and the Z_{C}-axis is the optical axis asshown in Figure 1, the imaged mirror boundary can be described as an ellipse with the following form:
Au2+Bv2+2Cu+2Dv+2Euv+F=0
The quadratic form of Equation (3) is:
u∼TQIu∼=0where ũ=(u,v,1)^{T} represents the image coordinates of the points in the ellipse and:
QI=[AECEBDCDF]
Substituting ũ in Equation (4) by Equation (1), an oblique ellipse cone in the camera coordinate system is obtained as follows:
s2XCTQCXC=0where:
QC=KTQIK.
As shown in Figure 1, a mirror boundary coordinate system O_{mb} − X_{mb}Y_{mb}Z_{mb} (the subscript “mb” stands for “mirror boundary”) is defined where the origin O_{mb} overlapswith the optic center O_{C} and Z_{mb}-axis parallel with the unitnormal vector of the supporting plane of the circle to beviewed. The circle of mirror boundary centered at C_{mb}=(x_{0}, y_{0}, z_{0})^{T} on Z_{mb}=Z_{0} plane with known radius r_{0} can be described as:
XmbTMXmb=0where:
M=[10-x0/z001-y0/z0-x0/z0-y0/z0(x02+y02-r2)/z02]and X_{mb} is the mirror boundary points represented in the mirror boundary coordinate system.
Based on the definition of the two coordinate systems, only a rotation R_{M} exists between the camera and mirror boundary coordinate system and the relationship can be expressed as:
XC=RMXmbIntegrating Equations (6), (8) and (10) we have:
s2RMTQCRM=M
To solve the above equation, first we convert Q_{c} into a diagonal matrix by eigenvalue decomposition:
QC=VΛVTwhere Λ = diag{λ_{1},λ_{2},λ_{3}}, V = (v_{1},v_{2},v_{3}). From Equations (11) and (12), we get:
s2RTΛR=Mwhere R = V^{T}R_{M}. We assume λ_{1}λ_{2} > 0 and |λ_{1}| > |λ|_{2} Solving R^{T}R = I and Equation (13), we finally get R according to [18], where θ is a free variable, S_{1} and S_{2} are undetermined signs, and
g1=(λ2-λ3)/(λ1-λ3),g2=(λ1-λ2)/(λ1-λ3):R=[g1cosθS1g1sinθS2g2sinθ-S1cosθ0S1S2g2cosθS2g2sinθ-S1g1]
And from Equations (12), (13) and (14), the rotation from the mirror boundary coordinate to the camera coordinate is:
RM=VR
Finally we can describe the center of the circle and unit normal vector of the supporting plane in the camera coordinate system by the following expression:
{z0=S3λ2r0-λ1λ3nC=RM(0,0,1)T=V(S2g2,0,-S1g1)TCC=RMCmb=z0V(S2g2,0,-S1g1λ1λ2)Twhere S_{3} is an undetermined sign. Equation (16) gives a total of four sets of possible solutions although some of them are not physically possible, as shown in Figure 2.
In the mirror coordinate system O_{M} − X_{M}Y_{M}Z_{M}, being the central symmetric axis of the mirror, the Z_{M}-axis is parallel to the vector n_{C} while the origin O_{M} does not overlap with the optical center. The translation between the mirror and the camera coordinate system is obtained by Equation (17), where dh is the distance from the mirror boundary center to O_{M}.
TM=Cc-RM(0,0,dh)T
Constraining n_{C} points back towards the camera and C_{C} lies in front of the camera as follows:
{z0>0nC•(0,0,1)T>0CC•(0,0,1)T>0we can get two candidate solutions
{RMi,TMi}(i=1,2) of the mirror posture which are both physically possible.
Mirror Pose Selection
Obviously more constraints are necessary to finally select the correct solution from the obtained posture candidates. In order for the calibration method to be independent of any special designed calibration patterns, we propose a novel method to achieve this by using the image of the lens boundary.
In the calibrating image, the lens boundary of the camera can also be easily observed in the center part of the image. The observed lens boundary in the image, whose size and position depends on the posture and the shape of the mirror, can be represented by a closed curve. Normally the curve is difficult to describe strictly by some special type of conics. For the revolutionary symmetric mirrors used in the catadioptric system, we find fitting the curve into an ellipse is accurate enough for the pose selection purpose, as we will see in the experiments. Theoretically, given the 3D position and the size of the lens boundary in the camera coordinate system, the correct posture can be selected by checking the similarity between the “observed” and the “predicted” lens boundary. There are two ways to define the “observed” and the “predicted” boundaries, each of which corresponds to checking the similarity under 2D image plane or 3D camera coordinate. In the 2D image plane, the “observed” one is the real ellipse we obtained and the “predicted” one is an ellipse computed by projecting the lens reflection on the mirror to the image plane. That is what our previous work has done [20] and it needs to calculate the reflective point on the mirror for each point on the lens boundary. The calculation has to be finished by a nonlinear optimization process and is very time-consuming. Here we propose a better way to check the similarity, where the “observed” is the real lens boundary in 3D camera coordinate and the “predicted” is the intersecting curves of a “cutting plane” with the reflective “cone” composed of the incident rays from the lens boundary. The “cutting plane” is actually the plane where the lens boundary lies in the camera coordinate system. For each pixel in the imaged lens boundary, the corresponding mirror points can be computed straightly and consequently the reflective “cone” can be obtained. Therefore this algorithm is much more effective.
However, in practice not all of the parameters of the lens boundary are known. Among those unknown the most important are the position parameters. Only the radius of the lens boundary can be found in the data sheet of the lens. By reasonably assuming the plane where the lens boundary lies be parallel with the X-Y plane of the camera coordinate system, the distance of the plane to the optical center is still left as an unknown. Therefore we need to estimate the distance of the lens boundary and select the correct mirror posture solution simultaneously.
Apparently this selection method can be applied to any type of the mirror as long as the mirror parameters are known. For simplicity we take the common hyperboloidal mirror as an example to explain the computing process.
The hyperboloidal mirror can be expressed by Equation (19):
Z2am2-X2+Y2bm2=1Transforming Equation (19) into the canonical form yields:
XMTQMXM=0where:
QM=[1000010000-bm2/am20000bm2]and X_{M}=[X,Y,Z,1]^{T} the homogeneous coordinates of the mirror points. Then the quadric mirror can be expressed in the camera coordinate system as:
QCi=(Hi)-TQM(Hi)-1=[RMiTMi01]-T⋅[1000010000-bm2/am20000bm2]⋅[RMiTMi01]-1
Given the mirror parameter
QCi(i=1,2), the next step is to obtain a reflective cone emitting from the mirror corresponding to the image of lens boundary. Due to the nonlinearity of the mirror shape, it's difficult to obtain a linear matrix form for the reflective cone. However as shown in Figure 3, the cone can be represented by a set of discrete reflective rays vr emitting from the mirror pointS, where each ray vr_{ij} corresponds to a sampling point u_{j}(j = 0,1,2,…,n) in the imaged lens boundary.
In camera coordinates, from the perspective projection model, we have:
u∼ij=u∼j=1λK[I|0]SCijTherefore the incident ray
vij=[v1ij,v2ij,v3ij]T corresponds to u_{ij} can be parameterized as:
vij=λK-1u∼ij=λK-1u∼jWith the normalized incident ray
v¯ij=vij‖vij‖ in hand, the reflective mirror point
SCij(λ)=[λv¯ijT1]T can be determined by the factor λ which satisfies Equation (25):
(SCij(λ))TQCiSCij(λ)=0
Expanding Equation (25) yields a quadric equation about λ. By imposing a positive depth constraint the correct root λ_{0} can be selected from the solution and hence
SCij can be determined. The normal vector to the quadric mirror at the reflective point is given by the first three coordinates of the tangent plane
ΠNij=QCijSCij. The normalized normal vector is thus given by:
nij=[I3|0]QCiSCij‖[I3|0]QCiSCij‖where I_{3} is a 3×3 identity matrix. The reflective vector vr_{ij} can then be obtained according to the law of reflection:
vrij=v¯ij-2(v¯ij•nij)nij
Now we have two set of rays vr_{ij}(i=1,2; j=1,2,…,n) which represent two reflective cones corresponding to the two posture candidates respectively. As shown in Figure 3, the lens boundary with radius r_{1} lies ahead of the optical center O_{C} with distance h_{1}. Two sets of intersecting points
PCij can be obtained by intersecting the rays vr_{ij} with the “cutting plane” at a distance h_{1} to O_{C}. Writing
SCij=[S1ij,S2ij,S3ij,1]T and
vrij=[vr1ij,vr2ij,vr3ij]T,
PCij can be computed by Equation (28):
PCij=λ0v¯ijT+h1-S3ijvr3ijvrij
Taking h_{1} as an unknown variable, one dimensional searching on h_{1}'s supporting region
[0,am2+bm2] could be carried out. For each possible combination of posture labeled i and h_{1}, one set of
PCij can be obtained by Equation (28). The set of the points can be fitted into an ellipse whose center
O(i,h1)=[Ox(i,h1),Oy(i,h1),h1]T and the length of the major axis a(i, h_{1}) and minor axis b(i, h_{1}) are all the functions of variables i and h_{1} and can all be computed, respectively. Obviously the correct i and h_{1} should result in a circle with radius r_{1} and central coordinates (0,0,h_{1})^{T}. Therefore it is reasonable to construct an error function E measuring the difference between the intersecting ellipse and the real lens boundary as follows:
E(i,h1)=(Ox(i,h1))2+(Oy(i,h1))2+(a(i,h1)-r1)2+(b(i,h1)-r1)2(i=1,2;h1∈[0,am2+bm2])
In Equation (29) the first term and the second term of E represent the central position error and the radius error respectively. Finally, the correct i and h_{1} can be obtained byminimizing the function, that is:
(i∗,h1∗)=argminE(i,h1)i,h1
The function E is difficult to differentiate analytically, therefore a derivative-free optimizer is preferred. The downhill simplex method is a good candidate for this type of optimization [21].
Robust Ellipse Extracting and Fitting
In order to estimate the mirror posture, ellipses of the mirror and lens boundaries in the image need to be extracted accurately. Given a calibrating image, we propose a robust method to finish the ellipse extraction task automatically. First the Canny operator is applied to obtain an edge image. Then by using two Regions of Interest (ROIs) where the mirror boundary and lens boundary should appear respectively, most of the edge pixels outside the ROIs are removed. Meanwhile edge pieces with small length are also deleted from the image. After that, we do iterative least mean square ellipse fitting to all the remaining pixels within each ROI until convergence. As can be seen from Figure 10, the ideal mirror and lens boundary in the image appear to be the outmost and inmost elliptical contour. For mirror boundary, the initial fitted ellipse always sits inside the real boundary due to the existence of non-boundary pixels. Therefore during each iteration those edge pixels staying inside the fitted ellipse are removed so as to “expand” the fitted ellipse in the next iteration. A similar process is applied to the lens boundary except that a “shrinkage” strategy is used. The iteration continues until the fitting error is less than a threshold. To further improve the fitting accuracy, 5-point RANSAC ellipse fitting is applied to the rest edge pixels and the final optimal ellipse parameters can be obtained [22,23]. The advantage of this technique is that it can automatically extract the ellipse and obtain the ellipse parameters with high accuracy.
Experiments
To verify the proposed self-calibration method, some experiments based on simulation data and real images were carried out.
Experiment with Simulation Data
Based on the simulated camera configuration listed in Table 1, the synthesized imaging ellipses of the mirror and lens boundaries can be easily generated for calibration, respectively.
The calibration results are summarized in Table 2. They show that the calibration method is effective and the proposed selection method does find the correct pose solution and the height h_{1}.
Figure 4 shows the typical variance of the position and the size of the two predicted lens boundaries with respect to the actual one within h_{1}'s supporting region. Figure 5(a–d) shows the central error, radius error, and combined average error between the predicted and the actual lens boundary respectively. From the two figures we can confirm that the minimum average error is reached when h_{1} is near 0.02 m, and the correct pose solution has been selected out.
The results listed in Table 2, Figures 4 and 5 did not consider the noises which usually exist in the process of the boundary imaging, edge extraction and ellipse fitting. To see the robustness of the calibration method in the presence of noise, we added zero-mean Gaussian noise with standard deviation σ to the sampled points on the image of mirrorboundary and lensboundary. σ varies from 0 to 5 pixels in 0.5 pixel steps. For each noise level, the mirror posture and h_{1} areestimated by our algorithm. The difference between the estimated parameters and the ground truth were recorded as an error measurement. The resulting error of n_{C} (angles, unitin degree), the error of C_{C} (Euclidean distance, unit in meters) and the relative error of the detected h_{1} are shownin Figure 6. In all of the test cases, the calibration algorithm produced stable posture choices and the solution closer to the ground truth was correctly found. The results show that the proposed algorithm is robust in the condition of different noise levels.
Experiment with Real ImagesCatadioptric Camera Setup
The catadioptric camera system we used for real data experiment is made by NEOVISION and is shown in Figure 7. It consists of a H3S hyperbolic mirror and a Sony XCD-SX910CR camera and was originally made as a central single viewpoint camera. More specifications of the system are listed in Table 3.We deliberately changed the relative position between the mirror and the camera, making it bias from the factory configuration. Therefore it was not a central camera again and actually became a new non-central catadioptric system. The intrinsic parameters of the conventional camera XCD-SX910CR are listed as follows: f_{x} = 1455.07, f_{y} = 1459.51, k_{s} = 0, u_{0} = 639.2, v_{0} = 482.2, the image resolution is 1,280 × 960. The radius of the lens boundary is: r_{1}=0.0185m.
Ellipse Extracting and Fitting Results
Figure 8 shows the calibration image and its canny detection result. Figure 9(a,b) shows the average fitting error of two boundaries with respect to the iteration number, respectively. It demonstrates that our iterative ellipse fitting process can refine the boundaries and quickly leads to convergence.
After using the remaining edge pixels for 5-point RANSAC, we get the final extracted and fitted ellipses, as shown in Figure 10. The ultimate average fitting errors of mirror and lens boundaries are 0.0022 and 0.0071 pixels, respectively.
Real Image Calibration
As we have the ellipse parameters of the mirror and the lens boundaries in hand, we apply our calibration algorithm to Figure 10. The final solution obtained is listed in Table 4. The resulting combined error between the actual and predicted lens boundary was 2.263 mm, which is very satisfying considering the existence of imaging noises.
ApplicationsImage Transformation
In this section, we evaluate the performance of our calibration method by image transformation. Since our catadioptric camera system does not maintain the single viewpoint characteristic, we cannot transform the whole omnidirectional image into a perspective one, but it is still possible to transform a patch of acquired image by assuming an approximate single viewpoint.
As shown in Figure 11, we try to find a virtual single viewpoint vp by minimizing the sum of the angle error between each real ray vr and the virtual ray vr' originated from the viewpoint vp, as expressed in Equation (31):
vp∗=argminvp{∑i180πarccos(vri•vr′i‖vr‖‖vr′i‖)}.Figure 12(a,b) shows the original omni-directional image and the zoom-in view of the sub-image area to be converted, respectively. Figure 12(c) shows the perspective image transformed by the ideal mirror posture from factory configuration. Figure 12(d) shows the perspective image transformed from our calibration results. In the images one can easily observe the difference caused by the change of viewpoint, which corresponds to the bias of the “real” viewpoint from the ideal one. Other rectifying effects such as removal of distortion of the lines are comparable and not very distinct due to the low resolution of the image.
3D Reconstruction
To further verify the performance of our calibration algorithm quantitatively, we employed a trihedral object composed of three orthogonal checker patterns of known size (Figure 13(a)) and computed the angles between normal vectors of each checkerboard plane. First a sub-image containing the trihedral object was transformed into a perspective one using the method described above. Then a traditional camera calibration toolbox [19] was used to compute the normal directions of the three checkerboard planes respectively. Finally the three angles Θ1, Θ2 and Θ3, as shown in Figure 13(a) can be obtained.
Two different non-central configurations are designed for the experiment. The first configuration is denoted as “slightly non-central” which means the mirror focus is only biased a little from the ideal position. The second configuration is denoted as “medium non-central” which has several centimeters in the translation and several degrees in the rotation away from the ideal case. The posture parameters from the default factory configuration, Mei's calibration method [24] and our method are used for computing the angles, respectively. In each configuration 10 images with the trihedral object at different positions around the camera are acquired and the average values of the computed angles are recorded in Table 5. The three angles should all be 90 degrees in an ideal situation. From the table, we can see that angles calculated from our calibration algorithm are better than results from the default factory configuration in both configurations. For the slightly non-central case the results from our method and Mei's are comparable, while for the second case our method shows superior performance than Mei's. The reason is Mei's method is only designed for the central camera while our method can deal with the central and non-central situation equally.
Conclusions
A novel self-calibration method for non-central catadioptric cameras is proposed in this paper. We use the mirror boundary in the image to obtain the possible mirror posture candidates, and then select the correct solution by using the image of the lens boundary. In the implementation stage we also presented a robust ellipse extraction algorithm based on iterative outlier rejection followed by RANSAC. Both the computer simulation and real data have been used to test the proposed technique, and very satisfying results have been obtained. The calibration method is not subject to the constraint of slightly non-central misalignment and is able to calibrate the non-central camera in a single image using only the catadioptric camera itself. This also makes the method qualified for on-the-fly calibration processes and is particularly beneficial for the situation where no calibration patterns are available, such as off-road and planet robot navigation.
This research work is supported by National Science Foundation of China (NSFC) 61071219 and the Fundamental Research Funds for the Central Universities.
ReferencesNayarS.K.Catadioptric Omnidirectional CameraProceedings of IEEE Computer Society Conference on Computer Vision and Pattern RecognitionSan Juan, Puerto Rico17–19 June 1997482488FernandoC.G.MunasingheR.ChittooruJ.Catadioptric Vision Systems: SurveyProceedings of the Thirty-Seventh Southeastern Symposium on System TheoryTuskegee, AL, USA20–22 March 2005443446AgrawalA.TaguchiY.RamalingamS.Beyond Alhazen's Problem: Analytical Projection Model for Non-Central Catadioptric Cameras with Quadric MirrorsProceedings of IEEE Conference on Computer Vision and Pattern RecognitionColorado Springs, CO, USA21–23 June 201129933000SrinivasanM.V.A New Class of Mirrors for Wide-Angle ImagingProceedings of the 2003 Conference on Computer Vision and Pattern Recognition WorkshopMadison, WI, USA16–22 June 2003Volume 78590SwaminathanR.GrossbergM.NayarS.Non-Single Viewpoint Catadioptric Cameras: Geometry and AnalysisMicusikB.PajdlaT.Autocalibration & 3D Reconstruction with Non-Central Catadioptric CamerasProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern RecognitionWashington, DC, USA27 June–2 July 2004Volume 15865GrossbergM.D.NayarS.K.The Raxel Imaging Model and Ray-Based CalibrationMorelO.FofiD.Calibration of Catadioptric Sensors by Polarization ImagingProceeding of IEEE International Conference on Robotics and AutomationRoma, Italy10–14 April 200739393944Ainouz-ZemoucheS.MorelO.MosaddeghS.FofiD.Adapted Processing of Catadioptric Images Using Polarization ImagingProcessing of 16th IEEE International Conference on ImageCairo, Egypt7–11 November 2009217220FabrizioJ.TarelJ.BenosmanR.Calibration of Panoramic Catadioptric Sensors Made EasierProceeding of Third Workshop on Omnidirectional VisionWashington DC, USA2 June 20024552GoncalvesN.AraujoH.Projection Model, 3D Reconstruction and Rigid Motion Estimation from Non-Central Catadioptric CamerasProcessing of 2nd International Symposium on 3D Data, Visualization and Transmission ConferenceThessaloniki, Greece6–9 September 2004325332MashitaT.IwaiY.YachidaM.Calibration Method for Misaligned Catadioptric CameraKangS.B.Catadioptric Self-CalibrationProceedings of the Conference on Computer Vision and Pattern RecognitionHilton Head, SC, USA13–15 June 2000Volume 1201207StelowD.MishlerJ.KoesD.SinghS.Precise Omnidirectional Camera CalibrationProceedings of IEEE Conference on Computer Vision and Pattern RecognitionKauai, HI, USA8–14 December 2001Volume 1689694GoncalvesN.AraujoH.Estimating Parameters of Noncentral Catadioptric Systems Using Bundle AdjustmentCagliotiV.TaddeiP.BoracchiG.GaspariniS.GiustiA.Single-Image Calibration of Off-Axis Catadioptric Cameras Using LinesProceedings of IEEE 11th International Conference on Computer VisionRio de Janeiro, Brazil14–20 October 200716AgrawalA.TaguchiY.RamalingamS.Analytical Projection for Axial Non-Central Dioptric and Catadioptric CamerasProceedings of 11th European Conference on Computer VisionCrete, Greece5–11 September 2010Volume 6313129143ChenQ.WuH.WadaT.Camera Calibration with Two Arbitrary Coplanar CirclesProceedings of European Conference on Computer VisionWashington DC, USA27 June–2 July 2004Volume 3521532BouguetJ.Y.Camera Calibration Toolbox for MatlabAvailable online: http://www.vision.caltech.edu/bouguetj/calib_doc/ (accessed on 7 January 2012)FitzgibbonPiluM.FisherR.Direct Least Square Fitting of EllipsesFischlerM.A.BollesR.C.Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated CartographyXiangZ.SunB.A Novel Self-Calibration Method for Non-Central Catadioptric Camera SystemProcessing of the Third IEEE International Conference on Intelligent Computing and Intelligent SystemsGuangzhou, China24–26 October 2011Volume 1735739PressW.H.VetterlingW.T.TeukolskyS.A.FlanneryB.P.MeiC.RivesP.Single View Point Omnidirectional Camera Calibration from Planar GridsIEEE International Conference on Robotics and AutomationRoma, Italy10–14 April 200739453950Figures and Tables
General configuration of the non-central catadioptric camera system.
Illustration of four possible solutions.
Relative position among lens boundary and three other coordinate systems: camera (blue), mirror (red) and mirror boundary (green).
Illustration of the actual (black) and the two sets of the predicted (red and blue) lens boundaries.
(a) Central position error with respect toh_{1}. (b) Radius error with respect to h_{1}. (c) Combined error E with respect to h_{1}.
Calibration errors w.r.t. noise level: (a) direction error of the normal vector.(b) position error of the mirror center. (c) relative percentage error of the detected h_{1}.
(a) The NEOVISION catadioptric camera system used in our experiment. (b) The omnidirectional image acquired by the system.
(a) Average fitting error of mirror boundary with respect to the fitting iterations. (b) Average fitting error of lens boundary with respect to the fitting iterations.
Final result of ellipse fitting.
Illustration of finding an approximate single viewpoint.
(a) Original omnidirectional image. (b) Sub-image area to be converted. (c) Perspective image produced by ideal configuration. (d) Perspective image produced by our calibration results.
(a) Trihedron used in experiment. (b) Real image with trihedron in the scene.