A Photometric Stereo Using Re-Projected Images for Active Stereo Vision System

Abstract

In optical 3D shape measurement, stereo vision with structured light can measure 3D scan data with high accuracy and is used in many applications, but fine surface detail is difficult to obtain. On the other hand, photometric stereo can capture surface details but has disadvantages, in that its 3D data accuracy drops and it requires multiple light sources. When the two measurement methods are combined, more accurate 3D scan data and detailed surface features can be obtained at the same time. In this paper, we present a 3D optical measurement technique that uses re-projection of images to implement photometric stereo without an external light source. 3D scan data is enhanced by combining normal vector from this photometric stereo method, and the result is evaluated with the ground truth.

1. Introduction

Optical 3D measurement technology is used in many applications, such as reverse engineering, quality inspection, restoration of cultural properties, face recognition, and dental computer aided design/computer aided manufacturing (CAD/CAM), because of its high measurement speed and precision [1,2,3,4,5]. Many passive stereo vision technologies have been proposed [6,7], but active stereo vision using structured light is one of the most popular optical 3D measurement techniques. It has high 3D scan data accuracy and fast data processing speed, but it has difficulties in expressing fine surface features. On the other hand, photometric stereo—another optical 3D measurement technique—can achieve fine surface feature but provides less accurate 3D scan data and requires multiple light sources. The output of active stereo vision is point data, and normal vector is calculated based on this point data, while the output of photometric stereo is normal vector, and point data is calculated by integrating this normal vector. Therefore, point data obtained by active stereo vision is more precise than that of photometric stereo, and normal vector obtained by photometric stereo is more detailed than that of active stereo vision. In this paper, we aimed to acquire more precise and detailed 3D scan data by combining active stereo vision and photometric stereo without external light sources in an active stereo vision hardware setup. The final result was verified by comparing distance deviation of point data and angular deviation of normal vector acquired with ground truth.

2. Related Works

2.1. Active Stereo Vision Using Structured Light

Active stereo vision using structured light is a common method to ensure that the accuracy of measured 3D scan data is high enough for its use in commercial devices [8,9,10]. The equipment for active stereo vision consists of two cameras and one projector, as shown in Figure 1a. The 3D scan data (Figure 1d) can be obtained by capturing left and right camera images of the object upon which the structured light is projected (Figure 1b,c), and by using epipolar geometry between the calibrated two cameras, 3D scan data is generally smoothed because of the high-frequency noise that it contains. In this process, the surface detail is also smoothed, making it difficult to obtain fine surface features.

2.2. Photometric Stereo

Photometric stereo is widely used for applications which require the surface detail of a target object to be obtained [11,12,13]. Photometric stereo requires one camera and three or more light sources, which must not be located in one straight line, as shown in Figure 2a. This method is a technique for acquiring normal vector through the object’s reflection characteristics by using images taken from a single camera with different light positions (Figure 2b–d). The normal vector (Figure 2e) acquired are relatively accurate, but a depth map obtained with the same technique is not accurate. This is because the depth map is obtained by continuously integrating the normal vector, causing accumulated error.

2.3. Data Combining Method

Active stereo vision and photometric stereo are techniques suitable for acquiring accurate 3D scan data and detailed surface features. By combining the data from each method, more accurate 3D scan data and detailed surface features can be acquired. Nehab’s algorithm combines 3D scan data from the active stereo vision with normal vector from the photometric stereo to obtain improved 3D scan data [14]. Figure 3 shows the experimental setup for Nehab’s algorithm, which consists of two cameras, a single projector, and external light sources.

2.4. Previous Works

Through the Nehab algorithm, precise and detailed 3D scan data can be obtained by combining active stereo vision and photometric stereo. However, combining an active stereo vision system and a photometric stereo system increases the size and complexity of the equipment configuration, incurring additional equipment costs [15,16,17]. To solve this problem, we propose a photometric stereo system that uses re-projected images instead of images taken under different light conditions to obtain detailed surface data.

3. Our Approach

3.1. Active Stereo Vision Measurement System Analysis

In general, the stereo vision measurement method is used to measure an object at various viewpoints by solving the field of view problem [18]. To obtain a complete 3D model of an object, 3D scan data from various viewpoints are registered to a single coordinate system and merged as shown in Figure 4. In this registration process, transformation matrices between these 3D scan data from various viewpoints are obtained.

3.2. Re-Projection Method

To use the photometric stereo measurement method, three or more images from a single camera with different light sources are needed. For the re-projection method, such images can be obtained by re-projecting the images obtained at various viewpoints in the stereo vision measurement process. If a reference view image (Figure 5b) and an image from a different view (Figure 5c) are obtained at two different viewpoints (as shown in Figure 5), a new image (Figure 5d) can be generated by re-projecting an image from view 2 to reference view 1. The projector in view 2 works as an external light source. As a result, the re-projected image (Figure 5d) can be assumed to be an image taken at view 1, but with a light source from view 2.

In the image generation process, the 3D scan data ($X$) obtained at another viewpoint in the local coordinate system is converted into 3D scan data ($X^{\prime }$) in the world coordinate system using the transformation matrix ($\mathrm{P}$).

$$X^{\prime }\left(x^{\prime }, y^{\prime }, z^{\prime }\right)=\mathrm{P}X$$

(1)

The converted 3D scan data are applied to the function ($g$), including the radial and tangent distortion of the lens in accordance with the camera model, and are re-projected onto the image plane at the reference point.

$$u^{\prime }=f\frac{x^{\prime }}{z^{\prime }}$$

(2)

$$v^{\prime }=f\frac{y^{\prime }}{z^{\prime }}$$

(3)

$$\left(k_1,k_2\right)=g\left(u^{\prime },v^{\prime }\right)$$

(4)

Since the re-projected coordinates ($k_1,k_2$) do not fit the camera pixel coordinates, they are converted to integer coordinates through bilinear interpolation. Because the corresponding points of the 3D scan data ($X$) obtained at the other viewpoints and the other viewpoint image pixels (u$,v$) can be known in the 3D scan data measurement process of the stereo vision, the corresponding points can be found between the other viewpoint image (u$,v$) and the reference viewpoint image pixel ($u^{\prime },v^{\prime }$). Thereafter, the intensity values ($I_1,I_2, I_3, I_4$) of the image corresponding to the corresponding point are bilinearly interpolated to obtain a re-projected image as shown in Figure 6. Through this process, images obtained from various positions can be re-projected to the reference coordinate system to obtain images for use in photometric stereo.

$$I_p=\left(1-t\right) · I_1+t · I_3$$

(5)

$$I_q=\left(1-t\right) · I_2+t · I_4$$

(6)

$$I_c=\left(1-t\right) · I_p+t · I_q$$

(7)

4. Experimental Results and Evaluation

To obtain the 3D data, an active stereo vision system was built, which consists of the same hardware configuration of the state of the art and two cameras (acA1300-200um-Basler ace, Basler, Ahrensburg, Germany), two lenses (HF16HA-1B, Fujifilm, Tokyo, Japan), and a single projector (LC3000-PRO 100, Texas Instruments, Dallas, TX, USA), as shown in Figure 7. In addition, a photometric stereo system was built with five external LED light sources and a single camera (acA1300-200um-Basler ace). The target object was a knight statue (50 mm in width, 150 mm in height, and 30 mm in depth). The proposed algorithm was implemented in C++ language and implemented on a personal computer with an i7-4770K processor (Intel, Santa Clara, CA, USA) with 8 GB memory and a Windows 7 operating system. The computational cost of Nehab’s method and ours has no difference in terms of acquiring the 3D point cloud, since we use the same algorithm to enhance the 3D scan data. The main difference is hardware setup and time cost of taking images for the algorithm. Nehab’s method takes five images per point of view to implement photometric stereo, while our method takes only a single image per point of view. This difference increases as the number of views to obtain scan data increase. Creating a re-projected image of 800 × 600 resolution only takes 20 ms per view on our hardware setup, which is minor compared to the extra time spent taking images for photometric stereo in Nehab’s configuration with a 30 fps camera.

The normal data (N^ps) from the traditional photometric stereo method which uses images under different light conditions were obtained through the procedure described in Section 2.2. 3D point data (P^sv) and normal vectors (N^sv) were obtained from five different views via the stereo vision system, and registration was done through the procedure described in Section 3.1. The registration errors of five scan patches was 0.022 mm, which proves high precision of an active stereo vision system. With this registration result, images from four different views were re-projected to a reference view, creating four re-projected images as described in Section 3.2. These four re-projected images and the reference view image work as images under different light conditions. Normal vector (N^rp) can be obtained via the photometric stereo method from these five images.

When there is a unit normal vector with a value of −1 to 1, this value is converted from 0 to 255, and is mapped to the RGB field of the image to create the normal map, as shown in Figure 8. The normal vector of the photometric stereo (N^ps) in Figure 8b represents more detailed surface features than the normal vector (N^sv) of the stereo vision in Figure 8a. The normal vector (N^rp) in Figure 8c obtained from re-projected images also represents more detailed surface features than that of the of stereo vision system. The difference is shown more clearly in the shield portion of the knight model.

We used these detailed normal vectors (N^ps) and (N^rp) to improve the stereo vision 3D point data (P^sv) by means of Nehab’s algorithm, obtaining more precise 3D data, as described in Section 2.3. In order to see errors of our re-projection algorithm’s result only, we applied Nehab’s algorithm on a single scan patch, which does not contain registration error and point difference from the merging process. Figure 9 shows 3D rendering comparisons between each method described above. Figure 9a is 3D scan data obtained from the stereo vision ((P^sv), (N^sv)) without Nehab’s algorithm, which contains high-frequency noise. Figure 9b shows 3D scan data obtained from Nehab’s algorithm with the normal vector from traditional photometric stereo method ((P^sv), (N^ps))^Nehab; this shows more surface details, and the high-frequency noise is removed. Figure 9c shows 3D scan data obtained from/via Nehab’s algorithm with the normal vector from the re-projection method proposed in this paper ((P^sv), (N^rp))^Nehab. The result also shows more surface details, and the high-frequency noise is removed, while it is not significantly different from the 3D scan data obtained from the method which uses an external light source.

For quantitative comparison analysis, high-quality 3D scan data obtained by measuring a target object with a high-precision 3D scanner (Identica Hybrid, Medit Company, Seoul, Republic of Korea) which has 7 μm accuracy was defined as ground truth. This ground truth was compared with the 3D scan data measured previously using the Geomagic Control X (3D Systems, Rock Hill, SC, USA) [19]. Figure 10 shows the point distance deviation comparison to the ground truth.

Table 1. Distance deviation of point data.

Error (mm)	Active Stereo Vision	Nehab’s Configuration	Ours
RMS (root mean square) Estimate	0.0490	0.0450	0.0454
Standard deviation	0.0489	0.0450	0.0454

shows the root mean square (RMS) error and standard deviation. Applying Nehab’s algorithm to the 3D scan data from active stereo vision resulted in a decrease in RMS error.

A comparison between normal vectors is also needed to see how much the surface detail is improved. Figure 11 shows the angular deviation between the normal vector of the ground truth and the other normal vector.

Figure 12 shows the angular deviation with the normal vector in terms of the number of points per deviation. Both Nehab’s configuration with external light sources and our configuration without external light sources show an increase in the number of points, with the angular deviation under 12°, thereby showing that the surface detail has improved, compared to the original stereo vision scan data. To use Nehab’s algorithm, an external light source is required, increasing the size and complexity of the equipment. However, our configuration does not require any external light source in order to use Nehab’s algorithm, which improves the quality of the 3D scan data.

5. Conclusions

In this study, we implemented a photometric stereo method using re-projected images under a stereo vision system, which does not use any external light sources. Comparing the normal vectors obtained from the traditional photometric stereo method (which uses external light sources), we confirmed that our re-projection method also works well in obtaining the normal vectors, even without external light sources. We applied each of these normal vectors to Nehab’s method to improve the quality of 3D scan data, and found that the precision of the 3D scan data and surface details are better than those of the stereo vision system. Our proposed method requires no external light sources; thus, it can be implemented in the existing stereo vision system without any additional equipment. This brings an advantage in size and complexity compared to the traditional photometric stereo system or Nehab’s configuration. Another advantage of our method is that time for taking images required for the algorithm is shorter, while having no significant difference in terms of computational cost. We expect that this study can help improve the quality of 3D scan data with minimal additional cost.