Reconstruction of Isolated Moving Objects by Motion-Induced Phase Shift Based on PSP

Abstract

The reconstruction of moving objects based on phase shifting profilometry has attracted intensive interests. Most of the methods introduce the phase shift by projecting multiple fringe patterns, which is undesirable in moving object reconstruction as the errors caused by the motion will be intensified when the number of the fringe pattern is increased. This paper proposes the reconstruction of the isolated moving object by projecting two fringe patterns with different frequencies. The phase shift required by the phase shifting profilometry is generated by the object motion, and the model describing the motion-induced phase shift is presented. Then, the phase information in different frequencies is retrieved by analyzing the influence introduced by movement. Finally, the mismatch on the phase information between the two frequencies is compensated and the isolated moving object is reconstructed. Experiments are presented to verify the effectiveness of the proposed method.

1. Introduction

Phase shifting profilometry (PSP) is one of the most popular technologies for 3D reconstruction [1,2,3]. PSP projects multiple sinusoidal fringe patterns with phase shift by a projector and then captures the reflected fringe patterns from the object surface by the camera. The phase information existing in the sinusoidal fringe pattern is retrieved by eliminating the influence of the background light and object surface reflectivity, leading to the advantages of high accuracy and robustness [4,5]. However, the measured object must keep static when implementing the reconstruction. The motion of the object not only introduces position mismatch, but unknown phase shift is also generated, resulting in errors in the reconstruction [6].

Recently, much work has focused on the issues caused by the motion in PSP [7,8,9]. Feng et al. [10] proposed compensating the motion errors for rigid moving object reconstruction. The motion speed is limited to be constant, and its influence is removed by using the statistical nature of the fringe patterns. However, it cannot reconstruct the object with rotation movement, as the motion-induced phase shifts are different for different points on the object surface. Li et al. [11] proposed reconstructing the dynamic object based on a modified three-wavelength phase unwrapping algorithm and phase error compensation method. The sequence of the projected fringe pattern is adjusted and less fringe patterns are required. Then, the Hilbert transform is employed to remove the motion-induced errors. Wu et al. [12] reconstructed a moving object based on feature correspondence. One fringe pattern is projected, and five fringe patterns of the object are captured. Based on the oriented fast and rotated brief feature algorithm, the correspondence of the object fringe pattern can be defined. Then, the phase value is retrieved by the traditional PSP. The method requires the equivalent phase variation, which means the object should have one direction of constant speed movement. Guo et al. [13] address the motion-induced error by separating the fringe patterns of four-step PSP into two groups. The first group is the first three fringe images, and the other is the last three images. As the frequency of the error caused by motion is twice of the fringe pattern frequency, the error is compensated by the average phase of the two groups. Guo et al. [14] estimate the motion from the object images in the beginning and end state. Based on the phase map and the obtained initial displacement data, the displacement for each point is retrieved. Then, the phase jump caused by the object motion is compensated. However, the method limits the object’s uniform motion. Duan et al. [15] proposed solving the motion-induced error issue by using an adaptive reference. Two planes are put in the up and down sides, and the object is put in the middle. Their phase maps are used to compensate the phase error. Our group [16] reconstructed multiple objects by motion tracking. Through object identification, the area of interest for each object is defined. Then, the object motion is tracked with the kernel correlation filter algorithm. The mathematical descriptions of movement for all objects are calculated. At last, the 3D information of the multiple objects is obtained. Recently, our group reconstructed the object by projecting a single fringe pattern [17]. Multiple fringe patterns with motion-induced phase variation were captured and the phase information was retrieved. However, the method can only reconstruct the object with a smooth surface, as the spatial phase unwrapping method is employed.

Based on [17], this paper focuses on the reconstruction of multiple objects with two-dimensional movement by using motion-induced phase shift based on PSP. The motion information is not only used to generate the motion-induced phase variation, but it is also employed to compensate the phase variations between the two frequencies. As multiple objects are reconstructed, the double-frequency phase unwrapping method is employed. Instead of projecting multiple phase-shifted fringe patterns, the proposed method only projects two fringe patterns with different frequencies. The phase shift required by the PSP is introduced by the object motion. The model describing the motion-induced phase shift is presented. Then, the movement influence on the fringe pattern is analyzed. Through movement tracking, the phase information of the moving object in different frequencies is retrieved. Finally, the mismatch on the phase information between the two frequencies is compensated, and the isolated moving object is reconstructed.

In Section 2 of the paper, the traditional PSP method is presented. Section 3 shows the principle of the motion-induced phase shift, and the influence caused by object movement is presented. In Section 4, experiments are presented to prove the effectiveness. Section 5 shows the conclusions of the paper.

2. The Principle of the Traditional PSP

Assuming that N-step PSP is employed, the captured fringe patterns of reference plane and object can be described as [18]:

$$S_n(x,y)=a+b\cos (\varphi (x,y)+2\pi (n-1)/N)$$

(1)

$$I_n(x,y)=a+b\cos (\varphi (x,y)+\phi (x,y)+2\pi (n-1)/N)$$

(2)

where $n=1, 2, 3, \cdots , N$; $S_n(x,y)$ and $I_n(x,y)$ are the intensity values for the $n\mathrm{th}$ fringe pattern on the reference plane and object, respectively; $a$ is the ambient light and is the fringe pattern intensity amplitude; $\varphi (x,y)$ is the phase distribution of the reference plane; $\phi (x,y)$ is the phase difference introduced by the object shape.

The parameters $a$, $b$, $\varphi (x,y)$ and $\phi (x,y)$ are unknown; $S_n(x,y)$ and $I_n(x,y)$ are known. Therefore, when the number of the fringe pattern is more than three, the phase value of the reference plane and object can be obtained by:

$$\varphi (x,y)=\arctan \frac{-{\displaystyle \sum _{n=1}^N{S}_n(x,y)\sin 2\pi (n-1)/N}}{{\displaystyle \sum _{n=1}^N{S}_n(x,y)\cos 2\pi (n-1)/N}}$$

(3)

$$\varphi (x,y)+\phi (x,y)=\arctan \frac{-{\displaystyle \sum _{n=1}^N{I}_n(x,y)\sin 2\pi (n-1)/N}}{{\displaystyle \sum _{n=1}^N{I}_n(x,y)\cos 2\pi (n-1)/N}}$$

(4)

In Equations (3) and (4), the phase value is wrapped into $-\pi$ to $\pi$ because of the operation $\arctan (\cdot )$. The wrapped phase value has ambiguity and phase jump in different fringe periods. Phase unwrapping [19,20] is applied to obtain the unwrapped phase map with a monotonous value. Finally, the object is reconstructed based on the unwrapped phase information and the calibration parameters of the system.

From Equations (1) and (2) we can see that the traditional PSP introduces the phase shift by projecting multiple phase-shifting fringe patterns. A constant phase shift is set among the fringe patterns. In Equations (3) and (4), the intensity values for the same point are used directly for the phase value calculation. The phase shift is also exactly set in advance. Therefore, when the object with motion is reconstructed, position mismatch and unknown phase shift will be introduced.

3. The Proposed Method

3.1. Phase Retrieval Based on Motion-Induced Phase Shift

This paper proposes to project two fringe patterns based on the dual-frequency phase unwrapping method to reconstruct the isolated object. The fringe pattern with high frequency is projected first, and a set of fringe pattern images is captured; then, the fringe pattern with low frequency is projected and another set of fringe pattern images is captured. When the object with motion is reconstructed, multiple fringe patterns are captured for each frequency, and the captured object fringe patterns are:

$$I_n^h(x,y)=a+b\cos ({\varphi }^h(x,y)+{\phi }^h(x,y))$$

(5)

$$I_n^l(x,y)=a+b\cos ({\varphi }^l(x,y)+{\phi }^l(x,y))$$

(6)

where $I_n^h(x,y)$ and $I_n^l(x,y)$ are the captured fringe patterns of high frequency and low frequency, respectively; ${\varphi }^h(x,y)$ and ${\varphi }^l(x,y)$ are the phase map on the reference plane of the two frequencies; ${\phi }^h(x,y)$ and ${\phi }^l(x,y)$ are the phase variation caused by the height of the object in different frequencies. Different from the traditional PSP in Equations (1) and (2), there is no phase shift among the projected fringe patterns. The same fringe patterns will be captured. However, for the object with motion, the fringe patterns can be described as [17]:

$${\tilde{I}}_n^h(f_n^h(x,y),g_n^h(x,y))=a+b\cos ({\varphi }^h(f_n^h(x,y),g_n^h(x,y))+{\phi }^h(x,y))$$

(7)

$${\tilde{I}}_n^l(f_n^l(x,y),g_n^l(x,y))=a+b\cos ({\varphi }^l(f_n^l(x,y),g_n^l(x,y))+{\phi }^l(x,y))$$

(8)

where ${\tilde{I}}_n^h$ and ${\tilde{I}}_n^l$ are the fringe patterns with movement of the high and low frequency, respectively. $(f_n^h(x,y),g_n^h(x,y))$ and $(f_n^l(x,y),g_n^l(x,y))$ are the coordinates after movement, which can be calculated with the rotation matrix and translation vector by movement tracking [18]. In Equations (7) and (8), ${\tilde{I}}_n^h$, ${\tilde{I}}_n^l$, ${\varphi }^h(x,y)$ and ${\varphi }^l(x,y)$ can be obtained in advance, and $a$, $b$, ${\phi }^h(x,y)$ and ${\phi }^l(x,y)$ are the unknown parameters. The phase variation of high frequency ${\phi }^h(x,y)$ and low frequency ${\phi }^l(x,y)$ can be calculated by the least-square method. Take the high frequency fringe patterns as the example to illustrate the derivation of the phase information.

Rewrite Equation (7) as:

$${\tilde{I}}_n^h(f_n^h(x,y),g_n^h(x,y))=a+B(x,y)\cos \delta +C(x,y)\sin \delta$$

(9)

where $B(x,y)=b\cos {\phi }^h(x,y)$, $C(x,y)=-b\sin {\phi }^h(x,y)$ and $\delta ={\varphi }^h(f_n^h(x,y),g_n^h(x,y))$. Define ${\tilde{I}}_n^{hT}(f_n^h(x,y),g_n^h(x,y))$ as the real captured fringe pattern. the sum of the squared error for each pixel between the fringe pattern distribution in theory ${\tilde{I}}_n^h$ and the real captured fringe pattern ${\tilde{I}}_n^{hT}$ can be described as:

$$S(x,y)={\displaystyle \sum _{n=1}^N({\tilde{I}}_n^h(f_n^h(x,y),g_n^h(x,y))-{\tilde{I}}_n^{hT}(f_n^h(x,y),g_n^h(x,y)){)}^2}$$

(10)

Employing the least-square criteria to minimizing Equation (10), the following relationship can be obtained:

$$X(x,y)=A^{-1}(x,y)B(x,y)$$

(11)

where

$$A(x,y)=\left[\begin{array}{ccc}N& {\displaystyle \sum _{n=1}^N\cos \delta }& {\displaystyle \sum _{n=1}^N\sin \delta }\\ {\displaystyle \sum _{n=1}^N\cos \delta }& {\displaystyle \sum _{n=1}^N{\cos }^2\delta }& \frac{1}{2}{\displaystyle \sum _{n=1}^N\sin 2\delta }\\ {\displaystyle \sum _{n=1}^N\sin \delta }& \frac{1}{2}{\displaystyle \sum _{n=1}^N\sin 2\delta }& {\displaystyle \sum _{n=1}^N{\sin }^2\delta }\end{array}\right]$$

(12)

$$X(x,y)={[\begin{array}{ccc}a& B(x,y)& C(x,y)\end{array}]}^T$$

(13)

$$B(x,y)=[\begin{array}{ccc}{\displaystyle \sum _{n=1}^N{\tilde{I}}_n^h(f_n^h(x,y),g_n^h(x,y))}& {\displaystyle \sum _{n=1}^N\cos \delta \times {\tilde{I}}_n^h(f_n^h(x,y),g_n^h(x,y))}& {\displaystyle \sum _{n=1}^N\sin \delta \times {\tilde{I}}_n^h(f_n^h(x,y),g_n^h(x,y))}\end{array}]$$

(14)

Based on Equations (11)–(13), the unknown parameters $a$, $B(x,y)$ and $C(x,y)$ can be obtained and the phase variation ${\phi }^h(x,y)$ can be obtained by:

$${\phi }^h(x,y)={\tan }^{-1}[-C(x,y)/B(x,y)]$$

(15)

The same operation can be applied to the low frequency fringe patterns and ${\phi }^l(x,y)$ can be obtained.

3.2. Movement Mismatches Compensation for the Dual-Frequency Unwrapping Method

With Equation (15), the phase maps of the two frequencies can be obtained. However, because of the object movement, position mismatches also exist between the phase maps. Assuming that a point $(x,y)$ is moved to $(u,v)$, the wrapped phase values of the two frequencies are:

$${\Phi }^h(x,y)={\varphi }^h(x,y)+{\phi }^h(x,y)$$

(16)

$${\Phi }^l(u,v)={\varphi }^l(u,v)+{\phi }^l(u,v)$$

(17)

When the dual-frequency phase unwrapping method is implemented directly, the position between the two phase maps is mismatched, and the preset relationship between ${\Phi }^h(x,y)$ and ${\Phi }^l(u,v)$ is violated, leading to the failure of unwrapping. Therefore, the position of the object should be moved back firstly. The object position in the low frequency is moved to the same position of high frequency as follows:

$${\Phi }^{l\prime }(x,y)={\Phi }^l(f^{hl}(x,y),g^{hl}(x,y))$$

(18)

where ${\Phi }^{l\prime }(x,y)$ is the phase distribution of low frequency after position alignment, and $(u=f^{hl}(x,y),v=g^{hl}(x,y))$ is the relationship of the positions before movement and after movement, which is described by the rotation matrix and translation vector. The position of the object is same between the high frequency and low frequency with Equation (18).

As the movement also introduces the phase shift, therefore the phase value in ${\Phi }^{l\prime }(x,y)$ cannot be used directly. The phase shift caused by the movement between the two frequencies also needs to be compensated by Equation (19) [6]:

$${\Phi }^{l″}(x,y)={\Phi }^{l\prime }(x,y)+{\varphi }^l(x,y)-{\varphi }^l(u,v)$$

(19)

where ${\Phi }^{l″}(x,y)$ is the phase value with movement influence compensation. With the phase map after position alignment and phase compensation, the dual-frequency phase unwrapping method can be applied as follows:

$${\Phi }^{h\_unwrap}(x,y)={\Phi }^h(x,y)+2\pi n(x,y)$$

(20)

$$n(x,y)=INT\left[\frac{k{\Phi }^{l^{″}unwrap}(x,y)-{\Phi }^h(x,y)}{2\pi }\right]$$

(21)

where ${\Phi }^{l^{″}unwrap}(x,y)$ and ${\Phi }^{h\_unwrap}(x,y)$ are the unwrapped phase map of the low frequency and high frequency, and $n(x,y)$ is the fringe order of the high frequency.

4. Experiments

The experimental setup is shown in Figure 1a. It includes two color cameras (Manta 504C with 2452 × 2056 resolution, Allied Vision Technologies Inc., Freistaat Thuringen State, Germany) and a projector (Wintech DLP PRO 4500 with 912 × 1140 resolution, Wintech Digital System Technology Corp., Beijing, China). Please note, only one camera is used in the proposed method. The isolated models shown in Figure 1b are reconstructed by the proposed method based on three-step PSP. As the isolated objects are reconstructed, the dual-frequency phase unwrapping method is employed. Two-dimensional movement is applied on the object during the reconstruction. The same with the method in [16], a red fringe pattern is projected for convenient movement tracking. With the captured images, the red channel provides the sinusoidal fringe information, and the blue channel contains the pure object image without the fringe pattern. Then, the phase information and object movement information can be retrieved from the images in different channels. Three images are captured, respectively, during the projection of the high frequency (30 pixel/fringe) and low frequency (150 pixel/fringe), and Figure 1c,d show their first image.

At the beginning, the high frequency fringe pattern is projected and three images with object motion are captured, as shown in Figure 2a–c. Then, the low frequency fringe pattern is projected and three images with motion are captured, as shown in Figure 2d–f.

As the fringe pattern is projected in red color, therefore, in the object image shown in Figure 3a, the sinusoidal fringe pattern is contained in the red component (as shown in Figure 3b). In the blue component, the pure object image can be found as presented in Figure 3c.

The SIFT (scale-invariant feature transform) [21] algorithm is implemented on the blue component images to track the object movement, as shown in Figure 4. Then, the rotation matrix and translation vector are calculated by the SVD (singular value decomposition) algorithm [18].

Based on the movement information, the wrapped phase maps for the two frequencies are obtained. The results are shown in Figure 5. A significant position mismatch is introduced by the object motion.

In order to unwrap the phase value of the high frequency, the phase value of the low frequency was unwrapped, as given in Figure 6a. The position mismatch between the wrapped phase maps of the two frequencies should be removed. The object position in the low frequency is moved to the object position in the high frequency. Then, the phase variation caused by the motion is compensated by Equation (18) (shown in Figure 6b). Then, the dual-frequency unwrapping method is applied, and the result is shown in Figure 6c.

With the correct phase map, the object is reconstructed successfully, as presented in Figure 7a,b. It is apparent that the object is reconstructed well. The object is reconstructed by the traditional PSP for comparison (as given in Figure 7c,d). It can be seen that a high-accuracy result is achieved by the proposed method, and the motion-induced error is removed.

The RMS (root mean square) error is employed to evaluate the accuracy of the proposed method. The true value of the object is obtained by the eight-step traditional PSP algorithm with the static object. With the proposed method shown in Figure 7a,b, the RMS error is 0.073 mms and 6.775 mm for the traditional PSP in Figure 7c,d.

The error source of the proposed method may come from the following aspects. (1) Movement tracking: As the proposed method utilizes the movement information to compensate the motion-induced error, the accuracy of the movement tracking is important for the reconstruction accuracy. This paper employs the SIFT algorithm to track the movement. The SIFT algorithm can achieve a subpixel level tracking accuracy. Other movement tracking methods can be used to pursue higher reconstruction accuracy. (2) Movement mismatch: When the object movement is tracked, the corresponding points on the object before movement and after movement can be obtained. However, affected by the tracking accuracy and difference in the view of the camera, the movement mismatch will still exist. Improving the tracking accuracy and minimizing the difference of the view before movement and after movement will be helpful for reducing the movement mismatch.

Please note that, as the proposed method introduces the phase shift by motion, the object can have movement in any direction except parallel to the fringes. When the movement direction is parallel to the fringes, there is no phase shift generated. The details can be found in [17]. From the derivation of the proposed method, there is no limitation on the selection of the frequency. However, as two frequencies are employed to unwrap the phase value, the frequency of the fringe pattern should be designed according to the rules of the dual-frequency phase unwrapping method. The proposed method requires capturing the moving object without blur. Therefore, the camera capture speed should be increased when the object’s movement speed is fast. There are also no specific limitations on the object size. However, the object is required to stay within the scene of the camera, which is also a requirement of traditional PSP.

5. Conclusions

The isolated objects are reconstructed by projecting only two fringe patterns with different frequencies. The phase shift required in traditional PSP is introduced by object motion. The phase information of the object in the two frequencies is retrieved first. Then, the position mismatches and phase variation between the wrapped phase maps at different frequencies are compensated. The isolated objects are reconstructed successfully.