Automatic Registration of Multi-Projector Based on Coded Structured Light

Abstract

Multi-projector display systems are widely used in virtual reality, flight simulators, and other entertainment systems. Geometric distortion and color inconsistency are two key problems to be solved. In this paper a geometric correction principle is theoretically demonstrated and a consistency principle of geometric correction is first proposed. A new method of automatic registration of a multi-projector on a curved screen is put forward. Two pairs of binocular-cameras are used to reconstruct the curved screen. To capture feature points of the curved screen precisely, a group of red-blue coded structured light images is designed to be projected onto the screen. Geometric homography between each projector and the curved screen is calculated to gain a pre-warp template. Work which can gain a seamless display is illustrated by a six-projector system on the curved screen.

1. Introduction

Tiling multiple projectors on curved displays is a common way to build high resolution, large-scale, and seamless vision systems in virtual reality. In this vision system, viewers can receive an immersive visual experience. Multi-projector systems form a seamless rendering view by overlapping projections onto each other. The challenges are to calibrate the geometric distortions due to projections onto the curved surface and to align and blend the overlapped parts of the projections both in geometry and color.

In the study of the geometric correction of multi-projector systems, the content of research has developed from multi-projector geometric correction on plane screens [1,2] to multi-projector geometric correction on curved screens [3,4], and the method of geometric correction has also developed from mechanical structure correction [5,6] to camera-based geometric correction [7,8]. However, the researchers mainly focus on the method of geometric correction, and there is a lack of theoretical research on the principle of off-line geometric correction. In this paper, the geometric correction of a multi-projector system is analyzed theoretically and the theoretical basis of off-line geometric correction (the consistency of geometric correction) is demonstrated for the first time.

Camera-based geometric correction is a kind of method with high accuracy. In this method the camera is used to capture the feature image of the system to get the geometric description of the screen. There are many different feature images which have developed from the physical mark on the screen [9,10] to the projection mark on the screen [11,12]. Among these, structured light image is a focus of research. Researchers use a variety of structured light methods to calibrate the features of the projector system [13,14]. These works, however, which involve projection on a curved surface, tackle the problem in a one-dimensional fashion, using multiple projections horizontally on the curved screen [15]. This paper presents a geometric correction method based on red-blue structured light in a two-dimensional (horizontal and vertical) manner.

In this paper, the geometric correction principle of a multi-projection system is theoretically demonstrated and the consistency principle of geometric correction is proposed. The work adopts binocular vision to achieve the registration of a multi-projector on a parameterized surface in a two-dimensional (horizontal and vertical) manner. It proposes a novel geometric calibration approach based on red-blue gray code structured light. Section 2 analyzes the principle of geometric registration and proposes the consistency principle of geometric correction. In Section 3, the method of geometric registration is presented. Section 4 gives results and performance and Section 5 provides our conclusion.

2. Geometric Registration Principle

Multi-projector systems are an effective way to achieve a large-scale display with sufficient resolution. The pictures $S_{i,j}$ projected by several projectors $P_{i,j}\left(i=1,2,\cdots ,n,j=1,2,\cdots ,m\right)$ on the display surface (screen) $S$ are connected with each other to form a unified large-scale picture. This large-scale picture realizes a wide field of view. Each picture projected on the curved screen is warped and the overlapping part of the projections is diverged. The issue is how to generate images projected on a curved screen that appear correctly to the viewer.

2.1. Coordinate System

In order to describe the geometric distortion and correction, the following definitions of the relevant coordinate system are given.

1.  Virtual world coordinate system $C_W$

Depending on $C_W$, the location of the model in a three-dimensional scene is represented.

2.  Eye point coordinate system $C_E$

$C_E$ describes the spatial position of the monocular observation point. For discussion convenience, we hold that the $Z$ axis of $C_E$ is orthogonal to the image plane ${\pi }_E$ and passes through the geometric center of ${\pi }_E$.

3.  Camera coordinate system $C_C$

$C_C$ describes the spatial position of the cameras used in automatic correction.

4.  Projector coordinate system $C_P$

$C_P$ describes the spatial position of the projector. For discussion convenience, we hold that the $Z$ axis of $C_P$ is orthogonal to the image plane ${\pi }_P$ and passes through the geometric center of ${\pi }_P$.

The affine transformation $A\left(R\_ij,t\_ij\right)$ is used between any two coordinate systems $C_i$, $C_j\left(i,j\in \left\{E,P,C,W\right\}\right)$, and $\left(R\_ij\in ℝ^\left(3\times 3\right),t\_ij\in ℝ^3\right)$.

2.2. Consistency of Geometric Correction

In a multi-projector system, the basic calculation and display process is divided into two steps. Firstly, through perspective projection centered on an eye point $E$, a two-dimensional projection picture $I$ of a three-dimensional space scene is generated on image plane ${\pi }_E$. Secondly, picture $I$ is projected through projection plane ${\pi }_P$ onto the display surface by the projector, as shown in Figure 1.

The perspective projection centered on eye point $E$ represents the viewer’s perspective with orientation on the curved screen. The perspective projection centered on projection point $P$ represents the projectors’ perspective. Geometric distortion of the multi-projector system is caused by the inconsistency of the projectors’ perspective and the viewer’s perspective. If the system has the same projectors’ perspective and viewer’s perspective, the correct image can be seen at the eye point, regardless of the screen shape. The process of geometric distortion correcting is defined as geometric registration.

Any point $x\in {\Re }^3$ in the frustum can be expressed as $x^E={\left(x^E,y^E,z^E\right)}^T$ in $C_E$. Define equivalent classes ${\left[x\right]}^E$ in projection space $P\left(C_E\right)$ based on $C_E$ as

$${\left[x\right]}^E=\left(\begin{array}{c}{x}^E/\omega \\ y^E/\omega \\ z^E/\omega \end{array}\right),\omega \in ℝ$$

where $z^E/\omega =d^E$. The corresponding point of $x$ on the image plane ${\pi }^E$ is

$$x_{{\pi }_E}^E={\pi }^E\cap {\left[x\right]}^E=d^E\left(\begin{array}{c}{x}^E/z^E\\ y^E/z^E\\ 1\end{array}\right)=d^E{\mu }^E\left(x\right)$$

Then, the corresponding point of $x$ on the screen $S$ in $C_E$ is

$$x_S^E={\left[x\right]}^E\cap S=d_S^E{\mu }^E\left(x\right)$$

The process of transferring an image from image plane ${\pi }_E$ to projection plane ${\pi }_P$ is defined as ${\pi }^E\stackrel{copy}{\to }{\pi }^P$, and we define the transformation

$$S^P=\left[\begin{array}{ccc}{S}_x& 0& 0\\ 0& S_y& 0\\ 0& 0& d^P/d^E\end{array}\right]$$

where $S_x=\frac{unitlengthof{X}_P}{unitlengthof{X}_E}$, $S_y=\frac{unitlengthof{Y}_P}{unitlengthof{Y}_E}$, and the corresponding point $x$ on the projection plane ${\pi }_P$ can be expressed as $x_{{\pi }_P}^P=S^P{x}_{{\pi }_E}^E=d^P{\mu }^P\left(x\right)$. Though the projector, the point on the screen $S$ in $C_P$ is $x_S^P=d_S^P{\mu }^P\left(x\right)$, where ${\mu }^P\left(x\right)={\left(\frac{d^E}{d^P}\frac{x^E}{z^E},\frac{d^E}{d^P}\frac{y^E}{z^E},1\right)}^T$.

Geometric distortion on screen $S$ can be quantified by geometric error ${\delta }_S^e\left(x\right)$:

$${\delta }_S^e\left(x\right)=\Vert x_S^E-\left(R_{EP}{x}_S^P+p_{EP}\right)\Vert$$

${\delta }_S^e\left(x\right)$ is the error observed on screen $S$ in the eye point coordinate system $C_E$. Geometric correction of this error requires the processing of geometric error on image plane ${\pi }_E$.

Point ${\left[x\right]}^E\cap S$ in $C_P$ is expressed as

$${\left({\left[x\right]}^E\cap S\right)}^P=R_{PE}{x}_S^E+p_{PE}$$

and its projective point on projection plane ${\pi }_P$ is

$$\left[{\left({\left[x\right]}^E\cap S\right)}^P\right]\cap {\pi }_P=\frac{d^P}{d_S^P}\left(R_{PE}{x}_S^E+p_{PE}\right)$$

Then, through the process ${\pi }^P\stackrel{copy}{\to }{\pi }^E$, the position of this point on image plane ${\pi }_E$ is

$${\tilde{x}}_{{\pi }_E}^E=S^{P^{-1}}\left(\frac{d^P}{d_S^P}\left(R_{PE}{x}_S^E+p_{PE}\right)\right)$$

We can gain the geometric error ${\delta }_{{\pi }_E}^e\left(x\right)$ on image plane ${\pi }_E$

$${\delta }_{{\pi }_E}^e\left(x\right)=\Vert x_{{\pi }_E}^E-{\tilde{x}}_{{\pi }_E}^E\Vert$$

where

$$\begin{array}{ll}{\tilde{x}}_{{\pi }_E}^E& =S^{P^{-1}}\left(\frac{d_S^E}{{\epsilon }_3^T\left(d_S^E{R}_{PE}{\mu }^E\left(x\right)+p_{PE}\right)}{d}^P\left(R_{PE}{x}_S^E+\frac{1}{d_S^E}{p}_{PE}\right)\right)\\ & =\phi \left(d_S^E,{\mu }^E\left(x\right)\right)\end{array}$$

and $e_3^T=\left(0,0,1\right)$.

When the ray direction ${\mu }^E\left(x\right)$ is determined, the geometric error ${\delta }_{{\pi }_E}^e\left(x\right)$ is only related to $d_S^E$, which is the coordinate in the $Z$ axis of the intersection of the ray and the screen $S$ in $C_E$.

From the above formula, the following conclusion can be drawn.

• When ${\mu }^E\left(x\right)$ remains unchanged, for each point on the screen $S$ and on the image plane ${\pi }_E$, the geometric error is determined and only related to the structural parameters of the multi-projector system, not to the virtual scene (consistency of geometric correction).

The consistency principle of geometric correction ensures that geometric correction can be carried out by determining the structural parameters of the multi-projector system and that the process can be done offline.

3. Geometric Correction

One of the most important tasks in determining structural parameters is to determine the geometric description of the screen $S$ in the eye point coordinate system $C_E$. We use binocular cameras to calculate the geometric description of screen $S$. Figure 2 shows that the binocular cameras capture the projections on the curved screen.

3.1. Geometric Distortion Correction

To correct geometric distortion, three components should be known: (a) equivalent classes ${\left[x\right]}^E$, which express the viewer’s perspective with orientation on the curved screen, and can be calculated by the geometric dimensions of the system; (b) the geometric description of screen $S$, which can be obtained by three-dimensional reconstruction of feature points; and (c) equivalent classes ${\left[x\right]}^{P_i}$ which express the projectors’ perspective. The mosaic method for a seamless multi-projector system with these three components in two-dimensionality is given as follows:

Firstly, render a 3D mesh $S$ of the curved screen using the viewer’s perspective specified.

Secondly, for each projector $P_j$, map the correspondences’ portion of the 3D mesh to the projection plane ${\pi }_{Pj}$ by the transformation $H_{Pj}^{-1}$, after which the pre-warp template $x_T$ in 2D is obtained.

Thirdly, through projecting the pre-warp template $x_T$ on the screen, a geometric seamless image can be seen at the location of the viewer.

Note that these calculations are registered to a common coordinate frame.

3.2. Geometric Description of Screen

3.2.1. Feature Points

With a multi-projector system, in the case of two-dimensionality projection, it is hard to capture and reconstruct the feature points, since there are overlapping parts of adjacent projections distributing in the vertical and horizontal directions.

To keep the curved screen clean without physical markers, we employ the red-blue gray code (RBGC) structured light images as the feature images. The gray code for decimal 0 rolls over to decimal 15 with only one switch change, so the coding has excellent anti-interference performance. The designed sequence of 4 bits RBGC is shown in Figure 3, in which the red and blue grids represent 0 and 1, respectively, in RBGC, and the number expresses the sequence of this structured light image.

The RBGC images projected to the curved screen are the feature images to be captured by the calibrated cameras. The RBGC images decode to the gray grading images, for which the number of the gray grades is $2^R$, where $R$ is the bit of the gray code. The gray value of the nth gray grade is $n\cdot 2^{8-R}$, and if the value is decoded as $G$, $n$ can be obtained as [16], i.e.,

$$n=\frac{G}{2^{8-R}}\hspace{1em}\left(n\in \left\{0,1,2,\cdots ,2^R-1\right\}\right)$$

The frame buffer coordinate of the nth gray grade’s edge which neighbors the lower gray grade is

$$b_l=n\cdot 2^{-R},\hspace{1em}\hspace{1em}0\le b_l<1$$

Substituting Equation (11) into Equation (12), the value of $b_l$ can be determined as

$$b_l=G/256$$

The decoded RBGC data is gained.

When projecting the structured light images to the screen in horizontal and vertical orientations, respectively, for each projector, the intersections of horizontal and vertical stripes are the feature points. An edge detection method for RBGC is introduced in the next section.

3.2.2. Edge Detection for RBGC

As the RBGC images are projected on the screen, deformation of the red-blue stripes appears. There are two reasons for which the accuracy of deformation capturing is affected: one is that the curved screen causes dislocation of the projected pixels on the red-blue edge and the other is that the camera pixels capturing the red-blue edge are dislocated and blurred. In order to improve the accuracy of edge detection, we design here an edge detection method for RBGC.

A red-blue stripe simulation pattern is constructed and the red-blue edge of the image is processed by divergence as shown in Figure 4. The divergence edge is established as

$$\{\begin{array}{cc}\begin{array}{c}{R}_{\left(x_0/2+i,y\right)}=R_0\cos \left(i\pi /2n\right)\\ B_{\left(x_0/2-i,y\right)}=B_0\cos \left(i\pi /2n\right)\end{array}& \left(i\in [0,n]\right)\end{array}$$

where $R_0$ and $B_0$ are the initial value of the red and blue components, respectively, $n=m\left(1+\sin y\right)$, and $m$ is the initial value of the scattered amplitude of the pixels.

To detect the edge of the red-blue stripe accurately, the red and blue components of the image are obtained by hierarchical processing. In order to reduce noise interference, the red component and the blue component are filtered separately by Gauss filtering to gain images $G_R$ and $G_B$. Then, scanning the image point by point, when the red component of the pixel is larger than the blue component $G_R>G_B$, the pixel is recorded as $R=1$. At the time $G_R<G_B$, the pixel is recorded as $R=0$, and the binary image of the red component is obtained. In the same row, the value of two adjacent pixels whose value has a step is kept and the value of the other adjacent pixels without a step is made to be 0.

Figure 5 shows the partial pixel step change in binary image. The edge of the red and blue stripe image should be at 400 column pixels. According to the data, the maximum error of the edge pixels is 3 pixels, the average error is 0.69 pixels, and the variance is $2.9396\times {10}^{-6}$.

After obtaining the discrete pixels, the continuous edge is obtained by curve fitting. The least squares method is used to fit the edge curve of the red-blue stripe.

The RBGC images are projected to screen $S$, respectively, in horizontal and vertical orientation for each projector, and the edge intersections of horizontal and vertical stripes are the feature points $s_{l_i{p}_j}$. The coordinate of $s_{l_i{p}_j}$ in the frame buffer can be expressed by $b_l$ and $b_p$ as $s_{{}_{l_i{p}_j}}^b$, and the coordinate of $s_{l_i{p}_j}$ in the camera can be expressed as $s_{{}_{l_i{p}_j}}^c$. By decoding the gray code, the corresponding relationship between $s_{{}_{l_i{p}_j}}^b$ and $s_{{}_{l_i{p}_j}}^c$ of the same point can be obtained, and the points on the frame buffer and those captured by the camera can be accurately matched.

3.2.3. Three-Dimensional Reconstruction

In order to obtain a geometric description of screen $S$, the space coordinates of the feature points are obtained by three-dimensional reconstruction. Expressing an opaque quadric surface in 3D by two arbitrary cameras needs a quadric transfer function $\psi$. The quadric transfer can be computed from nine or more correspondences.

A space point $X$ which can be expressed as a $4\times 1$ vector lies on a quadric surface $Q$ which can be expressed as a symmetric $4\times 4$ matrix, whereupon $X^{T}QX=0$ can be gained. The transfer function between the corresponding pixels $x$ in one camera and $x^{\prime }$ in the other camera can be related by

$$x^{\prime }\cong Bx-\left(q^{T}x\pm \sqrt{{\left(q^{T}x\right)}^2-x^T{Q}_{33}x}\right)e$$

There are 21 unknowns in this equation: the quadric $Q=\left[\begin{array}{cc}{Q}_{33}& q;\begin{array}{cc}{q}^T& 1\end{array}\end{array}\right]$, a $3\times 3$ homography matrix $B$, and the epipole in homogeneous coordinates, $e$. These 21 unknowns can be computed by the given pixel correspondences $\left(x,x^{\prime }\right)$. The epipole is the image of the center of projection of the first camera in the second camera. Part of this ambiguity is removed by defining

$$\begin{array}{cc}\begin{array}{cc}A=B-e{q}^T& \end{array}& E=q{q}^T-Q_{33}\end{array}$$

and the transfer is gained which is [7]

$$x^{\prime }=Ax\pm e\sqrt{x^{T}Ex}$$

Here,$x^{T}Ex=0$ defines the outline conic of the quadric in the first camera and $A$ is the homography via the polar plane between the binocular cameras. Note that this equation contains (apart from the overall scale) only one ambiguous degree of freedom resulting from the relative scaling of $E$ and $e$. This can be removed by introducing an additional normalization constraint such as $E\left(3,3\right)=1$. Furthermore, the sign in front of the square root is fixed within an outline conic in the image. As the internal parameters of the binocular cameras are known, a triangulation of corresponding pixels in binocular cameras can be used to compute parameters of quadric transfer $\left\{A,E,e\right\}$. This involves estimating the quadric $Q$ directly from point correspondences.

3.3. Geometric Description of Projector

In this multi-projector system each projector has its own matrix $H_{Pj}$ which reflects the transformation between reconstruction point $X_i$ in space and correspondence points $x_i$ on the projection plane ${\pi }_{P_j}$(equivalent classes ${\left[x\right]}^{P_j}$) as

$$x_i=H_{pj}{X}_i$$

According to projection geometry, for the projector $P_j$, a transformation consisting of a rotation $R_{Pj}$, a translation $T_{Pj}$ and a projection matrix $K_{Pj}$ can be obtained by QR decomposition of matrix $H_{Pj}$

$$H_{pj}=K_{pj}\left[R_{pj}|T_{pj}\right]$$

The projection matrix $H_{Pj}$ is an upper triangular matrix and the rotation matrix $R_{Pj}$ is an orthogonal matrix.

4. Results

4.1. Feature Points Identification Result

An RBGC structured light image used as a feature is one of the focuses of this paper. In the geometric correction of a multi-projector system, structured light images have been encoded by black-white gray code (BWGC) [17]. When using the camera to capture the feature image, the experimental environment cannot achieve the ideal conditions without ambient light which affects the accuracy of capturing. BWGC structured light is sensitive to the influence of ambient light, so RBGC structured light images are utilized in this paper to reduce the interference of ambient light and to improve the accuracy of capturing.

In the experiment, RBGC structured light and BWGC structured light were respectively projected onto the same screen and the same binocular cameras were used in the same position to capture the feature images. Because the ambient light was different at different times, two kinds of structured light images were captured 100 times at random times. Figure 6 is a schematic diagram of this process, in which a group of vertical RBGC structured light images are shown which are projected onto a curved screen to be captured by binocular cameras.

After reconstructing the feature points of the screen, the parameters of the screen can be fitted. The coordinate of the spherical center of the screen which has its own physical location and measurement location is selected as the evaluation criterion to compare the method based on RBGC with the method based on BWGC, with the experimental results is shown in Figure 7.

In Figure 7, the red pentagon is the center of the sphere in physics and the unit of measurement of the coordinate is the meter. The blue marks in Figure 7a are the center of the sphere calculated using BWGC structured light as the feature image and the green marks in Figure 7b are the center of the sphere calculated using RBGC structured light as the feature image.

From this figure we can ascertain that the green marks are closer to the red pentagon. This means that the results calculated by RBGC structured light are closer to the physical coordinates of the spherical center. In order to measure the results more clearly, the distance between the measured spherical center and the physical spherical center is calculated. The normal probability plot diagram shown in Figure 8 was drawn for the distance obtained from RBGC and the distance obtained from BWGC. The abscissa in the figure is the distance between the measurement center and the physical center. It can be seen from Figure 8a that the results of the calculation of the BWGC structured light are mostly distributed in $\left(0–0.01\right)$. From Figure 8b we can see that the results of the calculation of the RBGC structured light are mostly distributed in $\left(0–0.005\right)$. It can be judged that the calculation accuracy of the RBGC is higher.

4.2. Geometric Registration Result

A six-channel multi-projector system was used to verify the algorithm. On a $1/4$ spherical screen with a radius of 3 m, $3\times 2$ projectors were used to project the picture on a sphere, as shown in Figure 9a. Each projection image has geometric distortion and the overlapped parts of the images cannot be aligned. The method in this paper is used to register the geometry of the multi-projector system, and a seamless picture is obtained, as shown in Figure 9b.

5. Discussion

Through theoretical analysis of the geometric correction of a multi-projector system, the following conclusion is obtained: geometric error is determined and only related to the structural parameters of the multi-projector system, not to the virtual scene. To gain geometric registration, determining structural parameters is an important task. Red-blue gray code structured light images were used to calculate the geometric description of screen $S$ by binocular cameras. The geometric correction method and results are given in this paper. It is proven that this method is effective and can obtain a seamless, large-scale picture.