Next Article in Journal
A Low-Power WSN Protocol with ADR and TP Hybrid Control
Previous Article in Journal
Anomaly Detection in Discrete Manufacturing Systems by Pattern Relation Table Approaches
Article

Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image

1
Graduate School of Science and Engineering, Aoyama Gakuin University, 5-10-1 Fuchinobe, Chuo-ku, Sagamihara, Kanagawa 252-5258, Japan
2
Department of Integrated Information Technology, Aoyama Gakuin University, 5-10-1 Fuchinobe, Chuo-ku, Sagamihara, Kanagawa 252-5258, Japan
*
Author to whom correspondence should be addressed.
Sensors 2020, 20(20), 5765; https://doi.org/10.3390/s20205765
Received: 11 August 2020 / Revised: 2 October 2020 / Accepted: 9 October 2020 / Published: 12 October 2020
(This article belongs to the Section Physical Sensors)
This paper proposes a novel 3D representation, namely, a latent 3D volume, for joint depth estimation and semantic segmentation. Most previous studies encoded an input scene (typically given as a 2D image) into a set of feature vectors arranged over a 2D plane. However, considering the real world is three-dimensional, this 2D arrangement reduces one dimension and may limit the capacity of feature representation. In contrast, we examine the idea of arranging the feature vectors in 3D space rather than in a 2D plane. We refer to this 3D volumetric arrangement as a latent 3D volume. We will show that the latent 3D volume is beneficial to the tasks of depth estimation and semantic segmentation because these tasks require an understanding of the 3D structure of the scene. Our network first constructs an initial 3D volume using image features and then generates latent 3D volume by passing the initial 3D volume through several 3D convolutional layers. We apply depth regression and semantic segmentation by projecting the latent 3D volume onto a 2D plane. The evaluation results show that our method outperforms previous approaches on the NYU Depth v2 dataset. View Full-Text
Keywords: multi-task learning; latent 3D volume; depth estimation; semantic segmentation multi-task learning; latent 3D volume; depth estimation; semantic segmentation
Show Figures

Figure 1

MDPI and ACS Style

Ito, S.; Kaneko, N.; Sumi, K. Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image. Sensors 2020, 20, 5765. https://doi.org/10.3390/s20205765

AMA Style

Ito S, Kaneko N, Sumi K. Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image. Sensors. 2020; 20(20):5765. https://doi.org/10.3390/s20205765

Chicago/Turabian Style

Ito, Seiya, Naoshi Kaneko, and Kazuhiko Sumi. 2020. "Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image" Sensors 20, no. 20: 5765. https://doi.org/10.3390/s20205765

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop