# PDE-Based 3D Surface Reconstruction from Multi-View 2D Images

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

## 3. PDE Model

## 4. PDE-Based Surface Reconstruction from Multi-View Images

#### 4.1. Multi-View Images Generation

#### 4.2. Point Cloud Reconstruction from Multi-View Images

#### 4.3. PDE-Based Surface Reconstruction from Point Clouds

#### 4.3.1. Segmentation and Parameterization of Point Cloud

#### 4.3.2. Fitting PDE Model to Point Cloud

- 1
- PDE model with 16 variables

- 2
- PDE model with 64 variables

## 5. Empirical Results

## 6. Conclusions and Future Work

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Nocerino, E.; Stathopoulou, E.K.; Rigon, S.; Remondino, F. Surface Reconstruction Assessment in Photogrammetric Applications. Sensors
**2020**, 20, 5863. [Google Scholar] [CrossRef] - Nicolae, C.; Nocerino, E.; Menna, F.; Remondino, F. Photogrammetry Applied to Problematic Artefacts. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2014**, 40, 451. [Google Scholar] [CrossRef] [Green Version] - Toschi, I.; Ramos, M.M.; Nocerino, E.; Menna, F.; Remondino, F.; Moe, K.; Poli, D.; Legat, K.; Fassi, F. Oblique photogrammetry supporting 3D urban reconstruction of complex scenarios. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, 8, 519–526. [Google Scholar] [CrossRef] [Green Version] - Zeng, G.; Paris, S.; Quan, L.; Sillion, F.X. Progressive Surface Reconstruction from Images Using a Local Prior. In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), Beijing, China, 17–21 October 2005; Volume 1, p. 9. [Google Scholar]
- Lhuillier, M.; Quan, L. A Quasi-Dense Approach to Surface Reconstruction from Uncalibrated Iages. IEEE Trans. Pattern Anal. Mach. Intell.
**2005**, 27, 418–433. [Google Scholar] [CrossRef] [Green Version] - Maiti, A.; Chakravarty, D. Performance Analysis of Different Surface Reconstruction Algorithms for 3D Reconstruction of Outdoor Objects from Their Digital Images. SpringerPlus
**2016**, 5, 932. [Google Scholar] [CrossRef] [Green Version] - Deepika, S.; Sanjeev, K.; Gulati, T.R. NURBS-Based Geometric Inverse Reconstruction of Free-Form Shaped Objects. JKSU-Comput. Inf. Sci.
**2014**, 29, 116–133. [Google Scholar] - Zhao, C.; Mohr, R. Relative 3D Regularized B-Spline Surface Reconstruction through Image Sequences. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 1994; pp. 417–426. [Google Scholar]
- Zhao, C.; Mohr, R. Global Three-Dimensional Surface Reconstruction from Occluding Contours. Comput. Vis. Image Underst.
**1996**, 64, 62–96. [Google Scholar] [CrossRef] [Green Version] - Han, X.-F.; Laga, H.; Bennamoun, M. Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era. IEEE Trans. Pattern Anal. Mach. Intell.
**2021**, 43, 1578–1604. [Google Scholar] [CrossRef] [Green Version] - Ozyesil, O.; Voroninski, V.; Basri, R.; Singer, A. A survey of structure from motion. Acta Numer.
**2007**, 26, 305–364. [Google Scholar] [CrossRef] - Heinly, J.; Schonberger, J.L.; Dunn, E.; Frahm, J.-M. Reconstructing the World* in Six Days* (as Captured by the Yahoo 100 Million Image Dataset). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3287–3295. [Google Scholar]
- Schonberger, J.L.; Frahm, J.-M. Structure-from-Motion Revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 4104–4113. [Google Scholar]
- Snavely, N.; Seitz, S.M.; Szeliski, R. Photo Tourism: Exploring Photo Collections in 3D. In ACM Siggraph 2006 Papers; ACM: New York, NY, USA, 2006; pp. 835–846. Available online: http://phototour.cs.washington.edu/Photo_Tourism.pdf (accessed on 1 July 2006).
- Furukawa, Y.; Ponce, J. Accurate, Dense, and Robust Multi-View Stereopsis (PMVS). In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; Volume 2. [Google Scholar]
- Bailer, C.; Finckh, M.; Lensch, H.P. Scale Robust Multi View Stereo. In Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012; pp. 398–411. [Google Scholar]
- Shan, Q.; Curless, B.; Furukawa, Y.; Hernandez, C.; Seitz, S.M. Occluding Contours for Multi-View Stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 4002–4009. [Google Scholar]
- Schönberger, J.L.; Zheng, E.; Frahm, J.-M.; Pollefeys, M. Pixelwise View Selection for Unstructured Multi-View Stereo. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 501–518. [Google Scholar]
- Wang, S.; Xia, Y.; You, L.; Zhang, J. Reconstruction of Curve Networks from Unorganized Spatial Points. J. Univers. Comput. Sci.
**2020**, 26, 1265–1280. [Google Scholar] [CrossRef] - Berger, M.; Tagliasacchi, A.; Seversky, L.M.; Alliez, P.; Guennebaud, G.; Levine, J.A.; Sharf, A.; Silva, C.T. A Survey of Surface Reconstruction from Point Clouds. Comput. Graph. Forum
**2017**, 36, 301–329. [Google Scholar] [CrossRef] [Green Version] - Lei, J.; Sridhar, S.; Guerrero, P.; Sung, M.; Mitra, N.; Guibas, L.J. Pix2surf: Learning Parametric 3d Surface Models of Objects from Images. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 121–138. [Google Scholar]
- Kato, H.; Ushiku, Y.; Harada, T. Neural 3d Mesh Renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3907–3916. [Google Scholar]
- Sun, X.; Wu, J.; Zhang, X.; Zhang, Z.; Zhang, C.; Xue, T.; Tenenbaum, J.B.; Freeman, W.T. Pix3d: Dataset and Methods for Single-Image 3d Shape Modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 2974–2983. [Google Scholar]
- Wu, J.; Wang, Y.; Xue, T.; Sun, X.; Freeman, B.; Tenenbaum, J. Marrnet: 3d Shape Reconstruction via 2.5 d Sketches. arXiv
**2017**, arXiv:1711.03129. [Google Scholar] - Fu, K.; Peng, J.; He, Q.; Zhang, H. Single image 3D object reconstruction based on deep learning: A review. Multimed. Tools Appl.
**2021**, 80, 463–498. [Google Scholar] [CrossRef] - Mandikal, P.; Navaneet, K.L.; Agarwal, M.; Babu, R.V. 3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image. arXiv
**2018**, arXiv:1807.07796. [Google Scholar] - Navaneet, K.L.; Mathew, A.; Kashyap, S.; Hung, W.-C.; Jampani, V.; Babu, R.V. From Image Collections to Point Clouds with Self-Supervised Shape and Pose Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1132–1140. [Google Scholar]
- Mandikal, P.; KL, N.; Venkatesh Babu, R. 3D-PSRNet: Part Segmented 3D Point Cloud Reconstruction from a Single Image. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Lu, Q.; Xiao, M.; Lu, Y.; Yuan, X.; Yu, Y. Attention-Based Dense Point Cloud Reconstruction from a Single Image. IEEE Access
**2019**, 7, 137420–137431. [Google Scholar] [CrossRef] - Yi, H.; Wei, Z.; Ding, M.; Zhang, R.; Chen, Y.; Wang, G.; Tai, Y.-W. Pyramid Multi-View Stereo Net with Self-Adaptive View Aggregation. In Proceedings of the European Conference on Computer Vision, Salt Lake City, UT, USA, 18–23 June 2018; Springer: Berlin/Heidelberg, Germany, 2020; pp. 766–782. [Google Scholar]
- Zaheer, A.; Rashid, M.; Riaz, M.A.; Khan, S. Single-View Reconstruction using orthogonal line-pairs. Comput. Vis. Image Underst.
**2018**, 172, 107–123. [Google Scholar] [CrossRef] - Zou, Q. A PDE Model for Smooth Surface Reconstruction from 2D Parallel Slices. IEEE Signal Processing Lett.
**2020**, 27, 1015–1019. [Google Scholar] [CrossRef] - Duan, Y.; Yang, L.; Qin, H.; Samaras, D. Shape Reconstruction from 3D and 2D Data Using PDE-Based Deformable Surfaces. In Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, 11–14 May 2004; Springer: Berlin/Heidelberg, Germany, 2004; pp. 238–251. [Google Scholar]
- Othman, M.N.M.; Yusoff, Y.; Haron, H.; You, L. An Overview of Surface Reconstruction Using Partial Differential Equation (PDE). In Proceedings of the IOP Conference Series: Materials Science and Engineering; IOP Publishing: Bristol, UK, 2019; Volume 551, p. 012054. [Google Scholar] [CrossRef] [Green Version]
- Zhu, Z.; Chaudhry, E.; Wang, S.; Xia, Y.; Iglesias, A.; You, L.; Zhang, J.J. Shape Reconstruction from Point Clouds Using Closed Form Solution of a Fourth-Order Partial Differential Equation. In Proceedings of the International Conference on Computational Science, Krakow, Poland, 16–18 June 2021; Springer: Berlin/Heidelberg, Germany, 2021; pp. 207–220. [Google Scholar]
- Bianco, S.; Ciocca, G.; Marelli, D. Evaluating the Performance of Structure from Motion Pipelines. J. Imaging
**2018**, 4, 98. [Google Scholar] [CrossRef] [Green Version] - Wu, C. Towards Linear-Time Incremental Structure from Motion. In Proceedings of the 2013 International Conference on 3D Vision-3DV 2013, Seattle, WA, USA, 29 June–1 July 2013; pp. 127–134. [Google Scholar]
- Grilli, E.; Menna, F.; Remondino, F. A Review of Point Clouds Segmentation and Classification Algorithms. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, 42, 330–344. [Google Scholar] [CrossRef] [Green Version] - He, Y.; Yu, H.; Liu, X.; Yang, Z.; Sun, W.; Wang, Y.; Fu, Q.; Zou, Y.; Mian, A. Deep Learning Based 3D Segmentation: A Survey. arXiv
**2021**, arXiv:2103.05423. [Google Scholar] - Choi, G.P.T.; Liu, Y.; Lui, L.M. Free-Boundary Conformal Parameterization of Point Clouds. J. Sci. Comput.
**2022**, 90, 14. [Google Scholar] [CrossRef] - Azariadis, P.N. Parameterization of clouds of unorganized points using dynamic base surfaces. Comput. Aided Des.
**2004**, 36, 607–623. [Google Scholar] [CrossRef] - Ma, W.; Kruth, J.-P. Parameterization of randomly measured points for least squares fitting of B-spline curves and surfaces. Comput. Aided Des.
**1995**, 27, 663–675. [Google Scholar] [CrossRef] - Park, I.K.; Yun, D., II; Lee, S.U. Constructing NURBS surface model from scattered and unorganized range data. In Proceedings of the Second International Conference on 3-D Digital Imaging and Modeling (Cat. No.PR00062), Ottawa, ON, Canada, 8 October 1999; pp. 312–320. [Google Scholar]

**Figure 1.**The pipeline of the proposed PDE-based 3D surface reconstruction method from multi-view images.

**Figure 3.**(

**a**) Reconstructed cylinder point cloud with the Meshroom algorithm; (

**b**) magnified view of the reconstructed point cloud in (

**a**); (

**c**) magnified view of the reconstructed point cloud in Figure 4c with Colmap algorithm.

**Figure 4.**(

**a**) Input to Colmap: multi-view 2D images; (

**b**) 3D point cloud reconstruction from multi-view 2D images; (

**c**) reconstructed 3D point cloud.

**Figure 5.**(

**a**) Cylinder in the cylindrical coordinate system; (

**b**) parameterizing point cloud of cylinder shape.

**Figure 6.**(

**a**) Fitting plane to the point clouds; (

**b**) projecting points to the projecting plane (u, v plane).

**Figure 7.**(

**a**) Reconstructed 3D point cloud of a cylinder shape from multi-view 2D images; (

**b**) reconstructed PDE surface using a single PDE model with 16 variables; (

**c**) reconstructed PDE surface using a single PDE model with 64 variables; (

**d**) reconstructed PDE surface using two PDE models with 16 variables; (

**e**) segmented point cloud.

**Figure 8.**(

**a**) Point set of a bowl; (

**b**) surface reconstructed using Poisson; (

**c**) PDE-based surface using single 16-variables PDE model; (

**d**) PDE-based surface using single 64-variables PDE model.

**Figure 9.**(

**a**) The ground truth of a bench surface; (

**b**) point set of a bench surface; (

**c**) surface reconstructed using Poisson; (

**d**) PDE-based surface using a single 16-variables PDE model; and (

**e**) PDE-based surface using a single 64-variables PDE model.

**Figure 10.**(

**a**) The ground truth of a slide surface; (

**b**) point set of a slide surface; (

**c**) surface reconstructed using Poisson after segmentation; (

**d**) PDE-based surface using a single 16-variable PDE model; (

**e**) PDE-based surface using a single 64-variable PDE model.

**Figure 11.**(

**a**) The point cloud of a hat; (

**b**) segmented 2 subsets; (

**c**) reconstructed PDE-based surface using 2 PDE patches defined by the 64-variables PDE model; (

**d**) segmented 3 subsets; (

**e**) reconstructed PDE-based surface using 3 PDE patches defined by the 64-variable PDE mode.

**Figure 12.**(

**a**) The point cloud of a truck; (

**b**) segmented subsets; (

**c**) reconstructed PDE-based surface.

**Table 1.**The mean distance and its standard deviation between the ground truth surface and the PDE-based surface with polygon-based surface, respectively.

Methods Errors | Ground Truth to Polygon Surface | Ground Truth to PDE-Based Surface with 16 Variables | Ground Truth to PDE-Based Surface with 64 Variables |
---|---|---|---|

Mean distance | 0.039 | 0.031 | 0.029 |

Standard deviation | 0.016 | 0.021 | 0.015 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zhu, Z.; Iglesias, A.; Zhou, L.; You, L.; Zhang, J.
PDE-Based 3D Surface Reconstruction from Multi-View 2D Images. *Mathematics* **2022**, *10*, 542.
https://doi.org/10.3390/math10040542

**AMA Style**

Zhu Z, Iglesias A, Zhou L, You L, Zhang J.
PDE-Based 3D Surface Reconstruction from Multi-View 2D Images. *Mathematics*. 2022; 10(4):542.
https://doi.org/10.3390/math10040542

**Chicago/Turabian Style**

Zhu, Zaiping, Andres Iglesias, Liqi Zhou, Lihua You, and Jianjun Zhang.
2022. "PDE-Based 3D Surface Reconstruction from Multi-View 2D Images" *Mathematics* 10, no. 4: 542.
https://doi.org/10.3390/math10040542