# High-Capacity Spatial Structured Light for Robust and Accurate Reconstruction

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Key Technology and Algorithm

#### 2.1. Pseudo-2D Coding Method

#### 2.2. Detection of the Coded Feature Points

- (1)
- Collection of training Samples.

- (2)
- End-to-end corner detection with U-Net.

#### 2.3. Decoding of the Pseudo-2D Coding Pattern

## 3. Experiments and Results

^{4}+ x

^{2}+ Ax + A

^{2}, with four coding elements and a coding window of the size of $2\times 2$. In the first pattern, the coding elements were designed as square blocks with embedded geometrical shapes, and the square blocks were specially colored with a black or white background, which thereby comprises a typical checkerboard pattern. The embedded geometric shapes adopted a simple ‘L’ shape with four different rotation angles (${0}^{\xb0},{90}^{\xb0},{180}^{\xb0}$ and ${270}^{\xb0}$), as shown in Figure 10b. In the second pattern, speckle dots in different distributions were tucked into the blocks that were formed by a series of horizontal and vertical lines, as shown in Figure 10c. In both patterns, the grid corners formed by the horizontal and vertical lines were taken as the main feature points of SL. One of the irreplaceable advantages of these feature corners is that they can be extracted in sub-pixel precision. Based on the coding strategy in Section 2.1, given the number of coding elements ($q=4$) and the size of the coding window ($m=r\times s=2\times 2$), the dimension of the pseudo-2D sequence

**S’**was $r\times W=r\times \u230aL(\mathit{S})/r\u230b=r\times \u230a({q}^{m}-1)/r\u230b=2\times 127$. Afterward, repeating

**S’**by $n$ times, resulted in the pattern with a theoretical maximum coding capacity of $n\times r\times W=n\times 2\times 127$ being generated. In this work, the value of W was empirically set to be 65. Therefore, the whole coding capacities of both patterns are 16,510, which have been greatly improved compared with all previously reported work, as far as we know.

#### 3.1. Performance Evaluation of the Developed End-to-End Corner Detection Algorithm

#### 3.1.1. Performance w.r.t. (with Respect to) the Noise Level

#### 3.1.2. Performance w.r.t. the Density of the Feature Points

#### 3.2. Accuracy Evaluation of the Developed System

#### 3.3. Reconstruction of Surfaces with Rich Textures

#### 3.4. Reconstruction of Surfaces with Large Mutations

## 4. Discussions

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Tian, X.; Liu, R.; Wang, Z.; Ma, J. High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision. Inf. Fusion
**2022**, 77, 19–28. [Google Scholar] [CrossRef] - Samavati, T.; Soryani, M. Deep learning-based 3D reconstruction: A survey. Artif. Intell. Rev.
**2023**, 1–45. [Google Scholar] [CrossRef] - Kartashov, O.O.; Chernov, A.V.; Alexandrov, A.A.; Polyanichenko, D.S.; Ierusalimov, V.S.; Petrov, S.A.; Butakova, M.A. Machine Learning and 3D Reconstruction of Materials Surface for Nondestructive Inspection. Sensors
**2022**, 22, 6201. [Google Scholar] [CrossRef] [PubMed] - Nguyen, H.; Tran, T.; Wang, Y.; Wang, Z. Three-dimensional shape reconstruction from single-shot speckle image using deep convolutional neural networks. Opt. Lasers Eng.
**2021**, 143, 106639. [Google Scholar] [CrossRef] - Wang, J.; Yang, Y. Phase extraction accuracy comparison based on multi-frequency phase-shifting method in fringe projection profilometry. Measurement
**2022**, 199, 111525. [Google Scholar] [CrossRef] - Salvi, J.; Fernandez, S.; Pribanic, T.; Llado, X. A state of the art in structured light patterns for surface profilometry. Pattern Recognit.
**2010**, 43, 2666–2680. [Google Scholar] [CrossRef] - Yin, W.; Hu, Y.; Feng, S.; Huang, L.; Kemao, Q.; Chen, Q.; Zuo, C. Single-shot 3D shape measurement using an end-to-end stereo matching network for speckle projection profilometry. Opt. Express
**2021**, 29, 13388–13407. [Google Scholar] [CrossRef] [PubMed] - Gu, F.; Song, Z.; Zhao, Z. Single-shot structured light sensor for 3d dense and dynamic reconstruction. Sensors
**2020**, 20, 1094. [Google Scholar] [CrossRef] [PubMed] - Khan, D.; Kim, M.Y. High-density single shot 3D sensing using adaptable speckle projection system with varying preprocessing. Opt. Lasers Eng.
**2021**, 136, 106312. [Google Scholar] [CrossRef] - Salvi, J.; Pages, J.; Batlle, J. Pattern codification strategies in structured light systems. Pattern Recognit.
**2004**, 37, 827–849. [Google Scholar] [CrossRef] - Kawasaki, H.; Furukawa, R.; Sagawa, R.; Yagi, Y. Dynamic scene shape reconstruction using a single structured light pattern. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2008), Anchorage, AK, USA, 23–28 June 2008; pp. 1–8. [Google Scholar]
- Albitar, C.; Graebling, P.; Doignon, C. Robust Structured Light Coding for 3D Reconstruction. In Proceedings of the 11th IEEE International Conference on Computer Vision (ICCV2007), Rio de Janeiro, Brazil, 14–21 October 2007; pp. 1–6. [Google Scholar]
- Tang, S.; Zhang, X.; Song, Z.; Jiang, H.; Nie, L. Three-dimensioanl surface reconstruction via a robust binay shape-coded structured ligh method. Opt. Eng.
**2017**, 56, 014102. [Google Scholar] [CrossRef] - Song, Z.; Tang, S.M.; Gu, F.F.; Shi, C.; Feng, J.Y. DOE-based structured-light method for accurate 3D sensing. Opt. Lasers Eng.
**2019**, 120, 21–30. [Google Scholar] [CrossRef] - Gu, F.; Cao, H.; Song, Z.; Xie, P.; Zhao, J.; Liu, J. Dot-coded structured light for accurate and robust 3D reconstruction. Appl. Opt.
**2020**, 59, 10574–10583. [Google Scholar] [CrossRef] [PubMed] - Sagawa, R.; Furukawa, R.; Kawasaki, H. Dense 3D reconstruction from high frame-rate video using a static grid pattern. IEEE Trans. Pattern Anal. Mach. Intell.
**2014**, 36, 1733–1747. [Google Scholar] [CrossRef] [PubMed] - Morano, R.A.; Ozturk, C.; Conn, R.; Dubin, S.; Zietz, S.; Nissano, J. Structured light using pseudorandom codes. IEEE Trans. Pattern Anal. Mach. Intell.
**1998**, 20, 322–327. [Google Scholar] [CrossRef] - Lei, Y.; Bengtson, K.R.; Li, L.; Allebach, J.P. Design and decoding of an M-array pattern for low-cost structured light 3D reconstruction systems. In Proceedings of the IEEE International Conference on Image Processing (ICIP2013), Melbourne, Australia, 15–18 September 2013; pp. 2168–2172. [Google Scholar]
- Hall-Holt, O.; Rusinkiewicz, S. Stripe boundary codes for real-time structured-light range scanning of moving objects. In Proceedings of the Eighth IEEE International Conference on Computer Vision (ICCV 2001), Vancouver, BC, Canada, 7–14 July 2001; Volume 2, pp. 359–366. [Google Scholar]
- Trif, N.; Petriu, E.M.; McMath, W.S.; Yeung, S.K. 2-D pseudo-random encoding method for vision-based vehicle guidance. In Proceedings of the Intelligent Vehicles’ 93 Symposium, Tokyo, Japan, 14–16 July 1993; pp. 297–302. [Google Scholar]
- Gu, F.; Feng, J.; Xie, P.; Cao, H.; Song, Z. Robust feature detection method in high-density structured light system. In Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence, Normal, IL, USA, 6–8 December 2019; pp. 191–195. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Proceedings of the 18th International Conference, Munich, Germany, 5–9 October 2015; Proceedings, Part III 18; Springer International Publishing: New York, NY, USA, 2015; pp. 234–241. [Google Scholar]
- Neubeck, A.; Van Gool, L. Efficient non-maximum suppression. In Proceedings of the 18th International Conference on Pattern Recognition, Hong Kong, China, 20–24 August 2006. [Google Scholar]
- Feng, Z.; Man, D.; Song, Z. A pattern and calibration method for single-pattern structured light system. IEEE Trans. Instrum. Meas.
**2019**, 69, 3037–3048. [Google Scholar] [CrossRef]

**Figure 1.**Some representative coding patterns in spatial SL. (

**a**)

**Speckle-coded SL.**From left to right are the speckle pattern used in Microsoft Kinect v1, Intel RealSense D435, Orbbec Astra Pro, and our previous work [8] respectively. (

**b**)

**Shape-coded SL**. From left to right are the 2D coding pattern designed in [12,13,14,15] respectively.

**Figure 2.**The overall coding capacity in 2D coding strategies is limited by the number of coding elements and the size of the coding window. (Reprinted/adapted with permission from Ref. [10]. April 2004, Elsevier”).

**Figure 3.**Schematic diagram of the proposed pseudo-2D coding method (taking $r=2,s=2$ as an example).

**Figure 5.**The projected SL pattern and some examples of the grid corners’ labeling results: (

**a**) the projected SL pattern; (

**b**) the corners’ labeling results of a vase; (

**c**) the corners’ labeling results of a ball; (

**d**) the corners’ labeling results of a face model. (Red dots represent automatic labeling results, and green dots represent manual labeling results).

**Figure 6.**The U-Net architecture. (

**a**) Original version; and (

**b**) a simplified version adopted in this work.

**Figure 7.**One example of grid corners’ detection results by U-Net. (

**a**) The input captured image; (

**b**) the output segmentation map; and (

**c**) initial corner detection results.

**Figure 9.**Diagram of the decoding process. (

**a**) The search for all candidate matching blocks based on the epipolar constraint, (

**b**) coarse matching based on the judgment of blocks’ codewords, (

**c**) fine matching based on the constraint of the coding window.

**Figure 10.**Experimental setup and two different pseudo-2D SL patterns. (

**a**) Experimental setup; (

**b**) part of the first pseudo-2D pattern with four coding primitives of ‘L’; and (

**c**) part of the second pseudo-2D pattern with four coding primitives of mahjong dots.(Inside the red rectangular frame in Figure 10b,c is the designed coding sequence primitive).

**Figure 12.**Corner detection data under different noise levels of the (

**a**) vase, (

**b**) face model, and (

**c**) the ball, respectively.

**Figure 13.**Projected SL patterns displayed in Figure 5a with different coding densities onto the surfaces of a vase, a face model, and a ball, respectively.

**Figure 14.**Reconstruction accuracy of the proposed system. (

**a**) Schematic diagram of the accuracy evaluation experiment; and (

**b**) reconstruction errors in different working distances.

**Figure 15.**Target objects with complex textures. (

**a**,

**b**) Raw image and corner detection results of a face model with varying geometrical textures, respectively; (

**c**,

**d**) Raw image and corner detection result of a chessboard with varying color textures, respectively.

**Figure 16.**Reconstruction results. (

**a**,

**b**) Point cloud and 3D model of the face model; (

**c**,

**d**) Point cloud and 3D model of the chessboard.

**Figure 17.**Reconstruction results of the deformed surfaces based on the SL pattern in Figure 5a. (

**a**) the captured frame; (

**b**) the enlarged details of (

**a**); (

**c**) the reconstructed results; and (

**d**) the enlarged details of (

**c**).

m (m = r × s) | q = 3 | q = 4 | q = 5 | q = 6 |
---|---|---|---|---|

2 | x^{2} + x + 2 | x^{2} + x + A | x^{2} + Ax + A | x^{2} + x + A |

3 | x^{3} + 2x + 1 | x^{3} + x^{2} + x + A | x^{3} + x + A | x^{3} + x + A |

4 | x^{4} + x+ 2 | x^{4} + x^{2} + Ax + A^{2} | x^{4} + x+A^{3} | x^{4} + x + A^{5} |

Focal Length/Pixels | Principal Points/Pixels | Lens Distortion Coefficients | |
---|---|---|---|

Camera | (8980.67, 8975.48) | (1991.58, 1511.24) | (−0.077,0, 0, 0, 0) |

Projector | (6805.51, 6797.03) | (1942.82, 2946.19) | (−0.009, 0, 0, 0, 0) |

Translation vector T/mm: (156.79, −108.33, 41.52) | |||

Rotation vector om: (0.18, −0.17, 0.08) |

**Table 3.**Accuracy comparison between the traditional method and the proposed method with different coding densities.

Different Sizes of Pattern Blocks/Pixels | 51 × 51 | 41 × 41 | 31 × 31 | 21 × 21 | ||
---|---|---|---|---|---|---|

Number of detected feature corners | Vase | Ground truth | 135 | 251 | 449 | 845 |

Traditional method | 110 | 197 | 350 | 661 | ||

The proposed method | 134 | 250 | 447 | 835 | ||

Ratio | ↑21.8% | ↑26.9% | ↑27.7% | ↑26.3% | ||

Face model | Ground truth | 285 | 525 | 903 | 2218 | |

Traditional method | 224 | 410 | 701 | 1710 | ||

The proposed method | 285 | 523 | 902 | 2196 | ||

Ratio | ↑27.2% | ↑27.6% | ↑28.7% | ↑27.8% | ||

Ball | Ground truth | 643 | 1103 | 1932 | 4898 | |

Traditional method | 551 | 935 | 1620 | 4128 | ||

The proposed method | 642 | 1101 | 1926 | 4834 | ||

Ratio | ↑16.5% | ↑17.8% | ↑18.9% | ↑17.1% |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gu, F.; Du, H.; Wang, S.; Su, B.; Song, Z.
High-Capacity Spatial Structured Light for Robust and Accurate Reconstruction. *Sensors* **2023**, *23*, 4685.
https://doi.org/10.3390/s23104685

**AMA Style**

Gu F, Du H, Wang S, Su B, Song Z.
High-Capacity Spatial Structured Light for Robust and Accurate Reconstruction. *Sensors*. 2023; 23(10):4685.
https://doi.org/10.3390/s23104685

**Chicago/Turabian Style**

Gu, Feifei, Hubing Du, Sicheng Wang, Bohuai Su, and Zhan Song.
2023. "High-Capacity Spatial Structured Light for Robust and Accurate Reconstruction" *Sensors* 23, no. 10: 4685.
https://doi.org/10.3390/s23104685