^{*}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

Projector-camera systems are currently used in a wide field of applications, such as 3D reconstruction and augmented reality, and can provide accurate measurements, depending on the configuration and calibration. Frequently, the calibration task is divided into two steps: camera calibration followed by projector calibration. The latter still poses certain problems that are not easy to solve, such as the difficulty in obtaining a set of 2D–3D points to compute the projection matrix between the projector and the world. Existing methods are either not sufficiently accurate or not flexible. We propose an easy and automatic method to calibrate such systems that consists in projecting a calibration pattern and superimposing it automatically on a known printed pattern. The projected pattern is provided by a virtual camera observing a virtual pattern in an OpenGL model. The projector displays what the virtual camera visualizes. Thus, the projected pattern can be controlled and superimposed on the printed one with the aid of visual servoing. Our experimental results compare favorably with those of other methods considering both usability and accuracy.

3D reconstruction has been in use for more than 30 years and is still an active research topic in the computer vision community. One of the reasons for the popularity of such systems is the large application field they cover, such as manufacturing, security systems, medical systems and games. Surface reconstruction can be divided into two different categories. The best known example of the first class, in which we find passive scanners, is stereovision, which consists of two or more cameras viewing a scene from different angles. A set of frames is acquired, and 3D information can be extracted by triangulation after resolution of the 2D–3D correspondence problem. The major drawback of this technique lies in establishing correspondences, in particular, with textureless scenes.

To overcome such difficulties, new techniques called active scanners were developed 20 years ago. In these techniques, an emitter source is used, such as light, X-ray or ultra-sound. One of the methods of this class is called time-of-flight (TOF). It uses a laser to emit a pulse of light, and the amount of time before the reflected light is seen by a detector is timed. The biggest drawback is the time processing. Another method for active scanners uses structured light. This technique consists in augmenting the 3D scene by the projection of a coded pattern. The pattern is detected in the images, and the 3D reconstruction can be achieved by triangulation. The coding of the pattern is crucial in retrieving 3D information with robustness. Salvi

As reported in many articles, the accuracy of such systems greatly depends on the calibration task. Calibration consists in computing intrinsic parameters of both the camera and the projector and the 3D transformation between them. The biggest challenge in camera–projector calibration is the expression of 2D–3D correspondences for the projector. At present, several different methods exist to accomplish this task, and they can be separated into two groups. There are methods that first calibrate the camera, then the projector and deduce the rigid motion matrix between the two devices. One classical method is to use a printed checkerboard pattern as a projection plane [

There are other methods that integrate both intrinsic and extrinsic calibrations. Zhang and Huang [

Some authors have proposed alternative solutions to overcome these constraints. Recently, Drareni

In this paper, we describe an automatic calibration technique that is easy to implement and does not require any additional devices and a static scene. The first study dedicated only to intrinsic parameters estimation of the projector has been published in [

Here, the projector is viewing like a moving camera, which allows us to make the pattern of translation and rotation. The projector displays the same intrinsic parameters what the virtual camera sees. Thus, our method consists in projecting a calibration pattern and superimposing it on a known printed pattern with the aid of visual servoing (by moving the virtual camera,

The paper is structured as follows. Section 2 describes the system model and the calibration method, Section 3 discusses visual servoing control loop, and in Section 4, we compare our method with existing techniques.

In our calibration method, system calibration is separated into two sequenced procedures: camera calibration and projector calibration. Camera calibration, using Zhang's method [

Calibration of a perspective projection system (such as a camera or projector) consists of estimating the intrinsic and extrinsic parameters and in defining the rigid motion between the two devices.

Let us consider a point, ^{w}y^{w}z^{w}^{T}^{c}y^{c}^{T}_{c}_{c}_{c}_{c}_{c} in the world coordinate system is given by translation _{c}

The rotation between the camera and the world frames is defined by matrix _{c}_{c}_{uc}_{u}_{vc}_{v}_{c} = _{u}

The intrinsic parameters are:

Focal lens _{c}

_{u}_{v}

(_{0}_{c}_{0}_{c}

The extrinsic parameters are defined by [_{c}_{c}

The projector is modeled like an inverted camera. Thus, its intrinsic and extrinsic parameters are similar to those of the camera. Consequently, point ^{p}y^{p}^{T}^{w}y^{w}z^{w}^{T}_{p}

The 3D transformation (expressed in the camera frame) between the two devices is given by:

Thus, the estimation of

_{c}_{c}

Of course, estimating 2D–3D correspondences for the projector is not straightforward. Our method consists in obtaining the 2D–3D correspondences for the projector with a virtual model of the scene coupled with a visual servoing control loop. A global overview of our method is given in

To recover the correspondences, we use a virtual model (modeled under OpenGL) of our scene. In this virtual world, an “OpenGL camera” takes pictures and sends them to the projector. In other words, the projector shows what the “OpenGL camera” is looking at. If we then use a chessboard pattern, a virtual chessboard pattern is set and the “OpenGL camera” films it. The key idea of our method is to control the “OpenGL Camera” pose by visual servoing, so as to achieve the same rigid motion between _{v}_{p}

Let us consider a 3D point, ^{u}y^{u}z^{u}^{T}^{v}y^{v}^{T}_{v}

In this equation, _{p}_{v}^{−1}, where _{p}_{v}_{3}. Thus,

From this equation, it is clear that if the vectors, _{p}_{v}

The following section describes the fundamentals of visual servoing and breaks down our approach into different steps.

In the classical visual servoing approach [

The aim of classical visual servoing is essentially to minimize error

A significant difficulty in minimizing the previous equation is to derive the relation
_{c}

To minimize error _{c}

In our method, the visual servoing task is to fit the printed pattern and the projected pattern. Consequently, the desired value

There is no robot end-effector to control only the OpenGL camera. Moreover, and unlike a classical visual servoing, the OpenGL camera is controlled, but cannot measure the image error _{×} is the skew symmetric matrix associated with the translation vector ^{T}^{+} is then used in the control law (

It has been shown that if we consider only a translation between the projector and the real camera,

These steps, which consists in changing the position of the checkerboard, must be repeated (at least 10 times) to obtain different point of views. Finally, we obtain a set of 2D–3D points required to perform the calibration. Matrix _{p}^{th}_{p}. Thus, _{p}^{T}_{p}_{p}_{p}_{p}_{p}_{p}_{p}_{p}_{p}

If we rewrite _{p}^{−1} gives ^{−1} = ^{−1}.^{−1}. ^{−1} can be identified to _{p}^{−1} can be interpreted as _{p}^{−1} is written as:
_{p}_{33}, we obtain:

We assessed our proposed calibration method with a MARLIN F080C camera, which uses the IEEE 1394 protocol (FireWire protocol). Its maximum resolution is 1,024 × 768 pixels. The camera is mounted on a metrology positioning system that allows it to be translated with micrometer accuracy. It consists of a linear and rotating stage from Newport©, controlled by a universal motion driver (

We will compare our method with the two other most commonly used techniques, that of Audet and Okutomi [

Set up the position of the projector and camera.

Calibrate using the three different methods.

Translate the camera (10 cm) with the calibration bench.

Recalibrate the system with each method.

Measure the translation error with the rigid motion matrix.

The first step is considered crucial. Many calibration methods for camera–projector systems need a special set-up. For instance, the camera must be carefully set and placed, since it can acquire frames containing the projected and printed patterns for the plane-based method. Thus, it must have a large field of view or be correctly placed to acquire all the frames needed. In another approach [

A typical example of the visual servoing process implemented in our method is given in

In terms of accuracy, our method falls in between the two already existing methods. The translation error was 0.47 mm, which is good when dealing with structured lighting applications, for instance. We used the same number of images as Audet and Okutomi. Of course, the addition of more images increases accuracy. The best result we obtained was 0.3 mm with 17 images. Our method was the easiest to use, since it does not impose any constraints on the position of the devices. In addition, we used no additional material, and the calibration pattern was a classical checkerboard. The method of Audet and Okutomi [

We present a new calibration method for projector-camera systems based on visual servoing. This technique simplifies the process, since the same 3D points are used for both camera and projector calibration. The method is simple and requires no special hardware or calibration artifacts. The experimental results show that the accuracy of the method is comparable to that of existing techniques. In the future, we would like to develop an adaptive gain for the visual servoing task to improve time processing at the end of the fitting process. Another feature we would like to investigate is the effect of the initial pose of the projected pattern. At present, this position must be set manually, and so, if the projected pattern is far from the printed pattern, the process will take longer.

The work reported in this paper was supported by ANR under the Fish-Parasit Project.

The authors declare no conflict of interest.

Setup of the system.

Overview of the setup: The virtual camera looks at a virtual printed pattern displayed by the projector.

Projector-camera system modeling. A classical pinhole model is used for both devices. The two systems are related by the rigid motion matrix, [

Flowchart summarizing the steps of the full procedure.

System model: The model consists of three frames representing each device. _{c}_{p}_{p}_{v}_{v}_{p}

Classical visual servoing scheme.

Calibration based on visual servoing.

System configuration.

Fitting process example.

Reprojection error: Difference in pixel coordinates between the position of grid corners observed in the image and that predicted by the computed calibration parameters. Each color corresponds with the position of a checkerboard. In this figure, only six positions have been considered for the purpose of clarity.

Calibration results.

1,956 | 1,984 | 1,948 | |

0.25 | 1.18 | 0.38 | |

0.25 mm | 8.9 mm | 0.47 mm | |

10 | 20 | 10 | |

++ | + | +++ |