Next Article in Journal
Data Sampling Methods to Deal With the Big Data Multi-Class Imbalance Problem
Previous Article in Journal
Research on the Internal Thermal Boundary Conditions of Concrete Closed Girder Cross-Sections under Historically Extreme Temperature Conditions
Previous Article in Special Issue
Semantic 3D Reconstruction for Robotic Manipulators with an Eye-In-Hand Vision System
Open AccessArticle

Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images

by Zizhuang Wei 1,2,†, Yao Wang 1,2,†, Hongwei Yi 1,2, Yisong Chen 1,2,3 and Guoping Wang 1,2,3,*
1
Graphics & Interaction Lab, School of Electronics Engineering and Computer Sciences, Peking University, Beijing 100871, China
2
Key Lab of Machine Perception and Intelligent, MOE, Department of Computer Sciences, Peking University, Beijing 100871, China
3
Beijing Engineering Technology Research Center of Virtual Simulation and Visualization, Peking University, Beijing 100871, China
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Appl. Sci. 2020, 10(4), 1275; https://doi.org/10.3390/app10041275 (registering DOI)
Received: 21 December 2019 / Revised: 8 February 2020 / Accepted: 10 February 2020 / Published: 14 February 2020
(This article belongs to the Special Issue Augmented Reality, Virtual Reality & Semantic 3D Reconstruction)
Semantic modeling is a challenging task that has received widespread attention in recent years. With the help of mini Unmanned Aerial Vehicles (UAVs), multi-view high-resolution aerial images of large-scale scenes can be conveniently collected. In this paper, we propose a semantic Multi-View Stereo (MVS) method to reconstruct 3D semantic models from 2D images. Firstly, 2D semantic probability distribution is obtained by Convolutional Neural Network (CNN). Secondly, the calibrated cameras poses are determined by Structure from Motion (SfM), while the depth maps are estimated by learning MVS. Combining 2D segmentation and 3D geometry information, dense point clouds with semantic labels are generated by a probability-based semantic fusion method. In the final stage, the coarse 3D semantic point cloud is optimized by both local and global refinements. By making full use of the multi-view consistency, the proposed method efficiently produces a fine-level 3D semantic point cloud. The experimental result evaluated by re-projection maps achieves 88.4% Pixel Accuracy on the Urban Drone Dataset (UDD). In conclusion, our graph-based semantic fusion procedure and refinement based on local and global information can suppress and reduce the re-projection error. View Full-Text
Keywords: semantic 3D reconstruction; deep learning; multi-view stereo; probabilistic fusion; graph-based refinement semantic 3D reconstruction; deep learning; multi-view stereo; probabilistic fusion; graph-based refinement
Show Figures

Figure 1

MDPI and ACS Style

Wei, Z.; Wang, Y.; Yi, H.; Chen, Y.; Wang, G. Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images. Appl. Sci. 2020, 10, 1275.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop