Next Article in Journal
Joint SAR Image Time Series and PSInSAR Data Analytics: An LDA Based Approach
Next Article in Special Issue
Mining Hard Negative Samples for SAR-Optical Image Matching Using Generative Adversarial Networks
Previous Article in Journal
Sea Surface Temperature (SST) Variability of the Eastern Coastal Zone of the Gulf of California
Previous Article in Special Issue
WeedMap: A Large-Scale Semantic Weed Mapping Framework Using Aerial Multispectral Imaging and Deep Neural Network for Precision Farming
Article Menu
Issue 9 (September) cover image

Export Article

Open AccessArticle
Remote Sens. 2018, 10(9), 1435;

3D Façade Labeling over Complex Scenarios: A Case Study Using Convolutional Neural Network and Structure-From-Motion

Remote Sensing Division, National Institute for Space Research, Av. dos Astronautas, 1758, São José dos Campos, SP 12227-010, Brazil
Institute for Photogrammetry (IfP), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174 Stuttgart, Germany
Institute of Geodesy and Geoinformatics, Wroclaw University of Environmental and Life Sciences, Norwida 25, 50375 Wroclaw, Poland
College of Life and Environmental Sciences, University of Exeter, Exeter EX4 4RJ, UK
Author to whom correspondence should be addressed.
Received: 1 June 2018 / Revised: 22 June 2018 / Accepted: 29 June 2018 / Published: 8 September 2018
(This article belongs to the Special Issue Deep Learning for Remote Sensing)
Full-Text   |   PDF [28181 KB, uploaded 8 September 2018]   |  


Urban environments are regions in which spectral variability and spatial variability are extremely high, with a huge range of shapes and sizes, and they also demand high resolution images for applications involving their study. Due to the fact that these environments can grow even more over time, applications related to their monitoring tend to turn to autonomous intelligent systems, which together with remote sensing data could help or even predict daily life situations. The task of mapping cities by autonomous operators was usually carried out by aerial optical images due to its scale and resolution; however new scientific questions have arisen, and this has led research into a new era of highly-detailed data extraction. For many years, using artificial neural models to solve complex problems such as automatic image classification was commonplace, owing much of their popularity to their ability to adapt to complex situations without needing human intervention. In spite of that, their popularity declined in the mid-2000s, mostly due to the complex and time-consuming nature of their methods and workflows. However, newer neural network architectures have brought back the interest in their application for autonomous classifiers, especially for image classification purposes. Convolutional Neural Networks (CNN) have been a trend for pixel-wise image segmentation, showing flexibility when detecting and classifying any kind of object, even in situations where humans failed to perceive differences, such as in city scenarios. In this paper, we aim to explore and experiment with state-of-the-art technologies to semantically label 3D urban models over complex scenarios. To achieve these goals, we split the problem into two main processing lines: first, how to correctly label the façade features in the 2D domain, where a supervised CNN is used to segment ground-based façade images into six feature classes, roof, window, wall, door, balcony and shop; second, a Structure-from-Motion (SfM) and Multi-View-Stereo (MVS) workflow is used to extract the geometry of the façade, wherein the segmented images in the previous stage are then used to label the generated mesh by a “reverse” ray-tracing technique. This paper demonstrates that the proposed methodology is robust in complex scenarios. The façade feature inferences have reached up to 93% accuracy over most of the datasets used. Although it still presents some deficiencies in unknown architectural styles and needs some improvements to be made regarding 3D-labeling, we present a consistent and simple methodology to handle the problem. View Full-Text
Keywords: façade feature detection; 3D reconstruction; deep-learning; structure-from-motion façade feature detection; 3D reconstruction; deep-learning; structure-from-motion

Graphical abstract

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).

Share & Cite This Article

MDPI and ACS Style

Lotte, R.G.; Haala, N.; Karpina, M.; Aragão, L.E.O.C.; Shimabukuro, Y.E. 3D Façade Labeling over Complex Scenarios: A Case Study Using Convolutional Neural Network and Structure-From-Motion. Remote Sens. 2018, 10, 1435.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Remote Sens. EISSN 2072-4292 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top