^{1}

^{*}

^{1}

^{1}

^{2}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

Time-of-flight cameras, based on Photonic Mixer Device (PMD) technology, are capable of measuring distances to objects at high frame rates, however, the measured ranges and the intensity data contain systematic errors that need to be corrected. In this paper, a new integrated range camera self-calibration method via joint setup with a digital (RGB) camera is presented. This method can simultaneously estimate the systematic range error parameters as well as the interior and external orientation parameters of the camera. The calibration approach is based on photogrammetric bundle adjustment of observation equations originating from collinearity condition and a range errors model. Addition of a digital camera to the calibration process overcomes the limitations of small field of view and low pixel resolution of the range camera. The tests are performed on a dataset captured by a PMD[vision]-O3 camera from a multi-resolution test field of high contrast targets. An average improvement of 83% in RMS of range error and 72% in RMS of coordinate residual, over that achieved with basic calibration, was realized in an independent accuracy assessment. Our proposed calibration method also achieved 25% and 36% improvement on RMS of range error and coordinate residual, respectively, over that obtained by integrated calibration of the single PMD camera.

Providing three-dimensional information about the real environment is an essential factor for various applications in industry, computer vision, automation, multimedia, robotics, mobile mapping and many more fields. In recent years, Time-of-Flight (ToF) devices, based on Photonic Mixer Device (PMD) technology, are becoming increasingly popular solutions in 3D imaging applications [

The main advantages of range cameras in comparison with traditional 3D data acquisition systems such as laser scanners or stereo cameras are as follows:

No scanning mechanism is required,

Only one sensor is needed to capture 3D data without getting involved in different stereo analysis problems, and

Rapid imaging at a high frame rate provides the possibility of real time mapping and localization.

Due to several systematic error sources, however, proper calibration of such cameras is obligatory in order to perform reliable range sensing [

Karel was the first to consider self-calibration of range cameras to overcome the problems with instability in camera parameters [

This article presents an integrated range camera self-calibration method utilizing a low-cost custom-made system, composed of a PMD range camera (PMD[vision]-O3) and a digital camera (Canon Power Shot SX1 IS). Rigid combination of cameras and estimation of the PMD camera positions relative to the RGB camera along with integration of the range calibration into the photogrammetric bundle adjustment of the intrinsic and external parameters are the main characteristics of this work. It is suggested that this method can overcome the problems inherent to the small FoV and low resolution of the range camera. It is also assumed that the method can provide optimum estimates of the model parameters by incorporating the systematic errors and external parameters into the bundle adjustment based on weighted least squares criteria.

The following parameters are simultaneously estimated as the outcomes of the calibration process: interior orientation, radial and decentring lens distortion, sensor affinity and shear parameters, and also circular distance related error, signal propagation delay error and intensity related error parameters on range observations.

In the following section the operation principles of range imaging sensors based on PMD technology are briefly explained. Then, the geometric models are thoroughly described in Section 3. The details of calibration experiments including network and test field design are represented in Section 4. Section 5 reports the results of calibration method in terms of parameters accuracy and correlations, effectiveness of the systematic error models and noticeable impact of joint calibration with RBG camera. Finally, conclusions and future work are discussed in Section 6.

Range cameras based on PMD technology operate on the Time of Flight (ToF) concept to provide distance information. Typically, a PMD camera consists of a PMD chip and its peripheral electronics, emitter and receiver optics and other standard camera parts. The emitter, which is an illumination source, emits near infrared light. The reflected light is then received to measure the distance to the object. In contrast to typical ToF devices, e.g., laser scanners, all pixels in the PMD array simultaneously analyze the received optical signal to measure the depths of the corresponding points in space. The PMD chip is based on CMOS technology, which also provides an automatic suppression of background light, allowing the device to be utilized outdoor as well as indoor [

In order to provide range information, a reference electrical signal is applied to the modulation gates of each pixel on PMD array. Additionally, the incident light on photo-gates of pixels generates a second signal. The received optical signal differs from the reference one by a phase shift proportional to the depth of the reflecting target. To calculate the distance, the autocorrelation function of electrical and optical signal is analyzed by phase-shift algorithm (

In addition to the phase shift, two other values are extracted; signal strength of the received signal (amplitude) and the offset

The distance _{mod} is the modulation frequency of the emitted signal [

In the following subsections, the models used to calibrate range and lateral parameters of PMD camera in combination with digital camera are described. Initially, the procedures that have to be executed once before the main range camera calibration are explained. This is followed by a description of the imaging geometry of two cameras, systematic error models and integrated bundle adjustment process.

As described in Section 1, a joint setup is used to calibrate the range camera. It contains a digital camera and a PMD range camera. In order to get more accurate results, individual photogrammetric calibration of the RGB camera has to be performed before the main process. It is worth noting that the calibration process of the digital camera is done only once and its results are applied as inputs to the main calibration process of the range camera. The specifications of the RGB camera used in this work are listed in Section 4.

Calibration of the digital camera in this study is performed by the conventional photogrammetric calibration method, a computational method whereby camera parameters are estimated in a bundle adjustment solution [

In this study, orientation of the PMD camera is estimated relative to the digital camera. This means that there would be six parameters of exterior orientation at each imaging station for the digital camera and only six parameters of relative orientation at all stations for the range camera. _{1}_{2}

Assume that (_{1}_{1}_{1}_{1}_{2}_{2}_{2}_{2}

Therefore, the object space coordinates of point _{2}_{1}_{2}_{3}

The observations equations for range camera bundle adjustment logically stem from three kinds of observations:

Two image point collinearity equations for each pixel, which is the projection of an observed target on the digital camera image.

Two image point collinearity equations for the pixel on the intensity image captured by range camera.

One range equation for the pixel on the range image captured by range camera.

For object point _{ij,1}_{ij,1}_{i}_{i}_{i}_{p1}_{p1}_{1}_{ij,1}_{ij,1}

There will be two collinearity equations for object point

Therefore (_{ij,2}_{ij,2}_{ij,2}

In the above equations, (_{ij,2}_{ij,2}_{2}_{2}_{2}_{p2}_{p2}_{2}_{ij}_{ij,2}_{ij,2}_{ij}

The lateral camera systematic error model used for range camera calibration is same as the standard photogrammetric model for digital cameras [_{1,n}_{2,n}_{1,n}_{2,n}_{1,n}_{1,n}

The range error model for PMD camera used in this study is a combination of models presented by Fuchs and May [

In _{0} is the rangefinder offset [_{1}_{2}_{3}_{4}_{5}_{ij}_{ij}_{6}_{7}_{8}_{ij}

All the unknown parameters of integrated self-calibration are simultaneously estimated in combined method of least squares sense. Observations comprise the target image coordinates and range measurements. The image measurements are made at the corners of the squares on the designed test field. However, the corners are not good references to investigate intensity related range measurement errors. Therefore, only _{0}–c_{5}_{6}_{7}_{8}_{0}–c_{5}

Since the number of observations is more than that of the unknowns, a least squares solution is used. The mathematical model of observation equations expresses the relationship between observation and unknown vectors,

The vector function

Renaming the above matrices leads to:

The least squares estimation of unknown and residual vectors is obtained by the following equations:

These solutions,

To optimize the weighting of observations with different noise levels, robust estimation by adaptive weight determination is applied in this study [^{V}_{1-α,1,r}

The following subsections provide essential information on experimental aspects of the study. First, the camera specifications and their joint setup are represented. The target field used for the calibration procedure and the network designed for the purpose are described accordingly. Finally, the method of automatic target extraction from intensity images is explained in the last subsection.

As mentioned in previous sections, the calibration approach utilizes a PMD[vision]-O3 range camera and a Canon Power Shot SX1 IS digital camera. More specifications of each sensor are listed in

Cameras are rigidly mounted on a tripod as shown in

The test field consists of 24 multi resolution white squares on a black background rigidly attached to a flat wall, as indicated in

The network is configured to include two image sets [

In order to simplify the range camera self calibration process, the object coordinates of the targets (corners of the squares) are determined prior to main calibration process. The procedure utilizes a few (11) images of the digital camera, considering that calibration parameters of the digital camera have been determined in a pre-calibration process. These image stations are highlighted by magenta color in

It is important to clarify that after determining the object space coordinates of targets, they are used as pseudo observations in the main integrated calibration with their standard deviations as their weights.

The fundamental step in all the processes mentioned so far is correct detection of observed targets in intensity images. The targets are corners of white squares on the black background of the test field. In order to accomplish the task automatically and more accurate, the programs developed by Rufli

The PMD[vision]-O3 additional parameters and range error terms are determined using the procedures described in Sections 3 and 4. The algorithms are executed via the programs written particular to this approach. They can be applied to the data acquired with the standard software supplied with the PMD camera. The additional parameters of the digital camera are determined at the pre-calibration process described in Section 3.1. The results are reported in

Through the integrated calibration of range camera based on the joint setup, the internal parameters of the camera, including lateral parameters and range error terms, are reported in

In order to evaluate the test results, additional images are captured and the calibration parameters are applied to their observations. The check observations are made at the centers of the target squares. Differences of the corrected image coordinates and range observations from their true expected values are called residuals henceforth. For point

The first parentheses of both equations correct the observations by the estimated calibration parameters. The second parentheses are the true values of the observations that are computed directly from object coordinates and camera orientation parameters, as derived in Section 3. Since the object coordinates and the PMD camera orientations are determined based on RGB images, therefore the true values are independent of the PMD image or range observations. In order to simplify further denotations, the following symbols will be used hereafter:

[A]: Integrated calibration of range camera in joint setup with digital camera which is the proposed method of this study.

[B]: Integrated calibration of a single range camera without additional digital camera,

[C]: Basic calibration of a single range camera excluding range observations equations and their corresponding error parameters,

In the following analysis, we will evaluate the results of our proposed calibration method, scheme [A], against schemes [B] and [C]. The residuals on image observations (_{6}–c_{8}

As an improvement of our method, scheme [A], over schemes [B] and [C] can be investigated as the percentage of reduction in RMS of range error and image coordinate residuals after calibration by each method.

As discussed in [_{p}_{p}

The correlation coefficient between parameters _{i}_{j}

The correlation value of 0 means two parameters are fully de-correlated while 1 means they are thoroughly correlated.

Other concerning dependencies may occur between the rangefinder offset (_{0}_{0}_{0}

There also exists an average correlation of −0.0816 between the parameter _{1}

According to [_{0}_{1}

ToF cameras, based on PMD technology, provide range images at high frame rates which can be a valuable 3D data source for many applications. The errors associated with such measurements cannot be fully eliminated, but can be reduced to some extent by means of appropriate calibration procedures.

In this paper the integrated self-calibration of a range camera via a joint setup with a digital camera has been proposed and evaluated. The self-calibration bundle adjustment is performed based on observation equations of image point coordinates on intensity images of both cameras as well as range measurements of the PMD camera. The calibration results are improved by the presented approach regarding that:

Most pre-researched systematic error sources affecting the range accuracy are taken into account,

Image and range observations are adjusted in an integrated bundle adjustment,

Problems coherent to small FoV of range camera such as high correlation between interior and exterior orientation parameters are overcome, and

Specific equipment such as robot arms or laser scanners are not required.

Based on independent accuracy assessments, the proposed method achieved an average improvement of 83% in RMS of range error and 72% in RMS of coordinate residuals, over that achieved with basic calibration,

Autocorrelation function, phase shift, amplitude and intensity.

Coordinate systems and their relationships.

The joint setup of PMD and RGB cameras.

The self-calibration target field.

The self-calibration network and check images for independent accuracy assessment.

Target coordinates in object space.

Range error reduction ratio provided by calibration schemes [A] and [B].

Correlations between principal distance and perspective centre of PMD camera

Correlations between rangefinder offset and perspective centre of PMD camera

PMD[vision]-O3 camera specifications.

64 (v) × 48 (h) |
| |

40 (v) × 30 (h) | ||

850 | ||

0.2–4 | ||

25 | ||

0.1 |

Canon Power Shot SX1 IS camera specifications.

1,080 (v) × 1,920 (h) |
| |

0.005 | ||

CMOS | ||

Set to 5 | ||

Set to 1/125 | ||

Set to 400 |

Digital camera parameters from pre-calibration process.

_{1} |
0.003658 | 0.000006 |

_{2} |
−0.000057 | 0.000000 |

_{1} |
−0.000215 | 0.000005 |

_{2} |
0.000141 | 0.000005 |

_{p} |
−0.054075 | 0.000266 |

_{p} |
0.084998 | 0.000208 |

7.536374 | 0.001152 | |

_{1} |
−0.007454 | 0.000023 |

_{1} |
−0.000147 | 0.000022 |

Range camera calibration parameters from integrated joint calibration.

_{1} |
−0.027688 | 0.001897 | _{2} |
−0.022743 | 0.003890 |

_{2} |
0.007355 | 0.000137 | _{3} |
0.003958 | 0.000078 |

_{3} |
−0.000693 | 0.000169 | _{4} |
0.000074 | 0.000000 |

_{4} |
0.000022 | 0.000007 | _{5} |
−0.000072 | 0.000000 |

_{1} |
0.000635 | 0.000017 | _{6} |
0.003516 | 0.000985 |

_{2} |
0.000627 | 0.000016 | _{7} |
−0.000038 | 0.000000 |

_{p} |
0.017057 | 0.001764 | _{8} |
0.000000 | 0.000000 |

_{p} |
0.072515 | 0.001738 | _{2} |
0.011210 | 0.004765 |

8.041601 | 0.001563 | _{2} |
0.083192 | 0.00547 | |

_{1} |
0.000785 | 0.000355 | _{2} |
0.001816 | 0.000535 |

_{1} |
0.001542 | 0.000317 | 0.183448 | 0.000731 | |

_{0} |
−0.120160 | 0.008466 | −0.015591 | 0.000492 | |

_{1} |
0.032551 | 0.017607 | 0.017933 | 0.000265 |

Residuals on observations with calibration schemes [A], [B] and [C].

8.0 | 6.0 | 8.6 | 10.4 | 9.7 | 14.2 | 27.4 | 18.3 | 32.9 | |

9.1 | 7.6 | 10.1 | 12.2 | 8.7 | 15.0 | 27.6 | 18.1 | 33.0 | |

8.844 | 6.514 | 10.963 | 12.147 | 8.212 | 14.637 | 56.582 | 32.380 | 65.102 | |

13.067 | 10.482 | 16.715 | 12.764 | 10.257 | 16.339 |

Improvements achieved by the proposed calibration method.

71.6% | 36.0% | |

83.2% | 25.1% |