Skip to Content
SensorsSensors
  • Article
  • Open Access

29 January 2019

18 Pages

High Level 3D Structure Extraction from a Single Image Using a CNN-Based Approach

and
1
Department of Computer Science, Instituto Nacional de Astrofísica, Óptica y Electrónica, 72840 Puebla, Mexico
2
Department of Computer Science, University of Bristol, Bristol BS8 1TH, UK
*
Author to whom correspondence should be addressed.
This article belongs to the Special Issue Depth Sensors and 3D Vision

Abstract

High-Level Structure (HLS) extraction in a set of images consists of recognizing 3D elements with useful information to the user or application. There are several approaches to HLS extraction. However, most of these approaches are based on processing two or more images captured from different camera views or on processing 3D data in the form of point clouds extracted from the camera images. In contrast and motivated by the extensive work developed for the problem of depth estimation in a single image, where parallax constraints are not required, in this work, we propose a novel methodology towards HLS extraction from a single image with promising results. For that, our method has four steps. First, we use a CNN to predict the depth for a single image. Second, we propose a region-wise analysis to refine depth estimates. Third, we introduce a graph analysis to segment the depth in semantic orientations aiming at identifying potential HLS. Finally, the depth sections are provided to a new CNN architecture that predicts HLS in the shape of cubes and rectangular parallelepipeds.

1. Introduction

In computer vision, High-Level Structure (HLS) extraction consists of recognizing 3D elements from a set of images. There are several HLS that can be extracted (lines, planes and polyhedrons), and several approaches for HLS extraction have been proposed. In general, the use of HLS provides rich scene information since in man-made scenes (urbanized environments), there exist abundant HLS. In addition, HLS reduces computational processing by covering large areas with a few parameters. Due to these characteristics (rich scene information and computational processing reduction), several tasks use HLS in order to perform improvements, for example: robotics [1], augmented reality [2], navigation [3], 3D reconstruction [4] and Simultaneous Localization and Mapping (SLAM) [5].
There exist several approaches for HLS extraction: the first analyses two or more images captured from different camera views [6,7]. This approach has high performance under image sequences (collections of images related by time, such as frames in a movie or magnetic resonance imaging). Unfortunately, sufficient parallax is necessary, i.e., some difference between camera views to reach accurate results. Other approach associates HLS with a 3D point cloud [8,9,10]. These methods rely on fitting algorithms, typically RANSAC and some optimization techniques to fit 3D structures within 3D point clouds. Nevertheless, several thresholds and a specific setup are required in order to guarantee high performance for a specific scene. This is an important limitation because in several cases, it is difficult to set appropriate thresholds for setting up values.
The use of depth sensors is another approach to HLS extraction [11,12,13]. This approach in most case uses algorithms to build the 3D model on depth information, i.e., this uses algorithms that can complete the unobserved geometry using a prediction computed from the observed depth. Unfortunately, RGBD sensors often deliver low stability under outdoor scenarios. In addition, they are not present in personal devices (cell phones, personal assistants, personal computers, etc.). Finally, the power computation, cost and size are higher than RGB sensors.
Another approach, and which we are interested in this research, is the extraction of HLS from a single image [14]. Unlike the other trends (using two views or using 3D point clouds), this approach extracts HLS without parallax constraint and without threshold values. This is useful because in real-world applications, several data are limited to a single view from an unknown scene, for example historical images, Internet images, personal pictures, holiday photos and so on. Therefore, in the current work, HLS from a single image represents a promising solution with high performance. However, there are several challenges because there is insufficient information recorded in an image. In recent work [15,16], important progress in 3D structure interpretation has been made. This was achieved via learning algorithms that learn the relationship between visual appearance and scene structure. Motivated by the results of such techniques and the potential benefits that single-image perception provide (HLS extraction without parallax constraints and without threshold values), this work focuses on HLS extraction from a single image. We believe this is a very interesting task since, despite the considerable challenges involved, some kinds of single-image structure interpretation do indeed seem to be possible.
Following, in Section 2, we present the related works that determine the location of the research; Section 3 contains our proposed method; Section 4 describes the experiments designed to evaluate the feasibility of the proposed method and the results achieved; finally, the conclusions and future work are indicated in the last section.

3. The Proposed Method

Although learning algorithms can predict a depth map, this depth map presents several challenges to HLS extraction such as low sharpness of depth information, erroneous depth in different image sections, etc. For that, we propose a new method for HLS extraction from a single image. Our method has four steps: the use of a CNN to predict the depth, a depth analysis to remove uncertain depth sections, a graph analysis to segment the depth and a new CNN architecture that predicts HLS. Figure 1 shows the block diagram of our proposed method.
Figure 1. Block diagram of the proposed method. HLS, High-Level Structure.

3.1. Depth Analysis

In this subsection, we present the proposed depth analysis to remove uncertain depth. The depth analysis elaborates, removes and replaces depth sections. Furthermore, the depth sections are labelled with a semantic orientation. For that, the depth analysis uses a decision tree and a probability technique.

3.1.1. Depth Sections

We analyse the depth in sections, i.e., we do not use the depth points as independent elements; we analyse the behaviour of the depths using depth sets. For that, we use the depth images predicted by CNN [26]. Element D ε denotes a depth image. We divide the image D ε into a grid Δ . For that, the grid Δ consists of sections Δ w . Section Δ w is a finite set of pixels Δ w = { x 1 , … , x n } , Δ w ∈ Δ , where n is the number of pixels in a section and Δ w ⇔n is an odd number. Each section Δ w has a patch Λ α , β . Patch Λ α , β is a finite set of pixels Λ α , β = { x 1 , … , x u } , Λ α , β ∈ Δ , where, u is the number of pixels in a patch and Λ α , β ⇔u is an odd number; where w denotes the w th section in grid Δ , α is the abscissa from grid Δ and β is the ordinate from grid Δ . Figure 2a shows a grid example Δ of 3 × 3 .
Figure 2. Depth analysis. Each image shows a step of the depth analysis.

3.1.2. Semantic Orientation

We consider different analysis in the depth section to obtain semantic orientation. Figure 2c shows an image with the nine orientations. We use the ID3decision tree algorithm [34] to decide which analysis to use. The analyses selected were the key points analysis χ α , β i and section analysis γ α , β i . Key points analysis χ α , β i uses eight depth points. For that, the depth image D ε is divided into patches Λ α , β as in Figure 2a. In the patch Λ α , β , we obtain the depth points τ 1 , τ 2 , τ 3 , …, τ 8 . Figure 2b shows the depth points τ 1 , τ 2 , τ 3 , …, τ 8 for a patch Λ α , β of 17 × 17 pixels. To analyse the behaviour of the depth points τ i , we propose Equations (1) and (2).
χ α , β 1 = 1 if   τ 1 > τ 7 , τ 2 > τ 6 , τ 3 > τ 5 , 2 if   τ 1 < τ 7 , τ 2 < τ 6 , τ 3 < τ 5 , 3 if   τ 1 = τ 7 , τ 2 = τ 6 , τ 3 = τ 5 , 0 o t h e r w i s e ,
χ α , β 2 = 1 if   τ 5 > τ 7 , τ 4 > τ 8 , τ 3 > τ 1 , 2 if   τ 5 < τ 7 , τ 4 < τ 8 , τ 3 < τ 1 , 3 if   τ 5 = τ 7 , τ 4 = τ 8 , τ 3 = τ 1 , 0 o t h e r w i s e ,
The section analysis γ α , β i divides the depth section into two sessions. For that, the depth image D ε is divided into patches Λ α , β , as in Figure 2a. Equations (3) and (4) are used to compute the section analysis γ α , β i . Equation (3) divides the depth section into two horizontal sections, and they are subtracted. Equation (4) divides the depth section into two vertical sections, and they are subtracted; where the pixel of a depth image D ε is denoted by k i , j and u is the pixel number (row∖column) for the patch Λ α , β .
γ α , β 1 = ∑ i = 0 u ∑ j = 0 u / 2 ( k i , j - k i , j + u 2 ) u 2 / 2
γ α , β 2 = ∑ j = 0 u ∑ i = 0 u / 2 ( k i , j - k i + u 2 , j ) u 2 / 2
To analyse the behaviour of the depth sections, we obtain their semantic orientation ϑ α , β k . For that, we obtain the semantic orientation using a decision tree with the ID3 algorithm. Figure 3 shows our decision tree. Furthermore, Figure 2c shows an image with the nine orientations; where the patch Λ α , β is painted in orange colour if it has an orientation with a right and down view ϑ α , β 1 . The patch Λ α , β is painted in yellow colour if it has an orientation with a down view ϑ α , β 2 . The patch Λ α , β is painted in dark green colour if it has an orientation with a left and down view ϑ α , β 3 . The patch Λ α , β is painted in red colour if it has an orientation with a right view ϑ α , β 4 . The patch Λ α , β is painted in blue colour if it has an orientation with a front view ϑ α , β 5 . The patch Λ α , β is painted in green colour if it has an orientation with a left view ϑ α , β 6 . The patch Λ α , β is painted in purple colour if it has an orientation with a right and upward view ϑ α , β 7 . The patch Λ α , β is painted in brown colour if it has an orientation with an upward view ϑ α , β 8 . The patch Λ α , β is painted in sky blue colour if it has an orientation with a left and upward view ϑ α , β 9 . Finally, if the patch Λ α , β does not have an orientation ϑ α , β k , the patch Λ α , β has an uncertain orientation or null orientation ⊘. The patch Λ α , β is painted in black colour if it has a null orientation ⊘.
Figure 3. The obtained decision tree using the ID3algorithm.
We use the Markov chain [35] to label the patch Λ α , β with null orientation ⊘. The Markov chain analysis substitutes the patches Λ α , β with a null orientation considering the orientation neighbour. For that, we use a central patch Λ α , β to analyse its connection with the orientation neighbour. Figure 4 shows a central patch Λ 3 , 3 with a grid Δ of 5 × 5 .
Figure 4. Example of patch Λ 3 , 3 with a grid Δ of 5 × 5 .
A stochastic matrix P describes a Markov chain X t over a finite state space with cardinality S. We use a right stochastic matrix that is a real square matrix, with each row summing to 1. We use P i to name a row of the stochastic matrix. Each of its entries p i , j is a nonnegative real number representing a probability. In our stochastic matrix P, we consider in each row the probability that the patches with null orientation have an orientation of a neighbour patch.
Considering Figure 4, our stochastic matrix P is a 3 × 3 matrix. For that, in P 1 , we consider that the patches with null orientation have an orientation with a left view ϑ α , β 6 , P 1 = { 19 / 25 , 5 / 25 , 1 / 25 } . In P 2 , we consider that the patches with null orientation have an orientation with a front view ϑ α , β 5 , P 2 = { 17 / 25 , 7 / 25 , 1 / 25 } . In P 3 , we consider that the patches with a null orientation have an orientation with a right view ϑ α , β 4 , P 3 = { 17 / 25 , 5 / 25 , 3 / 25 } . In addition, we have three probability vectors, one for each orientation contemplated V 1 = { 1 , 0 , 0 } , V 2 = { 0 , 1 , 0 } and V 3 = { 0 , 0 , 1 } . Finally, every probability vector multiplies by the stochastic matrix P. If some element of the multiplication is greater than a threshold, the orientation of the element greater than the threshold enters central patch Λ α , β . Otherwise, the resulting vector is multiplied by the stochastic matrix P. If the image has the same number of patches Λ α , β with null orientation, the threshold is reduced. The iteration ends when the image does not have null orientation. We use a threshold = 1 as the initial value.

3.2. Orientation Segmentation

The orientation segmentation segments the image D ε into sections with similar orientation patches. The orientations ϑ α , β k of patches Λ α , β are connected to each other; where an orientation session ω m is a set of patches Λ α , β with similar orientation ϑ α , β k and connected to each other ω m = { ϑ a , u k , … , ϑ e , o k } , ω m ∈ Δ , m denotes the m th orientation session ω m , ( a , e ) are abscissas of the grid Δ and ( u , o ) are ordinates of the grid Δ .
The orientation session ω m has the following properties:
(1) Patches’ connection: ϑ a , u k and ϑ e , o k are connected if there is a patch sequence with similar orientation.
(2) Disjoint region: ω i and ω j are disjoint regions if ω i ∩ ω j = ⊘ for all i = { 1 , … , w } ; where i ≠ j , w is the number of regions in image D ε , ⊘ is the null set and { ω i , ω j } ∈ D ε .
(3) Segmented region: P ( ω m ) = T R U E if all pixels in ω m are of a similar orientation detected by our decision tree Figure 3, where P ( ω m ) is a logical predicate defined over the points in set ω m and ⊘.

3.3. Graph Analysis

The proposed model uses an undirected network G = ( N , A ) , consisting of a finite set of nodes N = { 1 , 2 , 3 , … , n } and a set of undirected edges A = { ( i , j ) : i , j ∈ N , i ≠ j } joining pairs of nodes in N. For all edges ( i , j ) ∈ A , let there be one nonnegative weight denoted by c i , j [36]. We consider the orientation session ω m on depth image D ε the nodes of our graph analysis. Furthermore, the connection between two sessions ω m is represented with an edge. We refer to c i , j as the number of connected pixels between two sessions ω m . Considering the edges (the different connections between two sessions ω m ), the graph analysis locates sections with possible polyhedrons (cube, half cube and rectangular parallelepipeds) (see Figure 7).
Furthermore, each section with a polyhedron is saved in an image, removing the sky and floor. The graph analysis analyses the area of the polyhedron section and classifies the polyhedron on a cube, a half cube, a horizontal rectangular parallelepiped or a vertical rectangular parallelepiped. Finally, the depth of the polyhedron section is provided to one of the four CNN, where every CNN provides the coordinate (x,y,z) vertices of one HLS.

3.4. CNN for HLS Extraction

Our network aims to provide the coordinate (x,y,z) vertices of HLS from a given depth map. The depth map is obtained from the mentioned graph analysis. The proposed network accepts as input a depth map with a size of 240 × 320 pixels. In addition, our network contains two stages. The first stage consists of 7 convolutional, 7 batch normalization and 4 max pooling layers, which extract feature maps with different resolutions from the input image. This stage is the backbone of the network since the extracted features are shared in the second stage. The second stage consists of combining all the found local features of the previous convolutional layers. For that, we use a flatten layer and batch normalization. Finally, we use dense layers to obtain the coordinate (x,y,z) vertices of HLS. Figure 5 shows the architecture of our CNN for HLS extraction. In addition, we have four architectures of our CNN, and every CNN extracts one structure (cube, half cube, horizontal rectangular parallelepiped or vertical rectangular parallelepiped). To know what CNN has to be used, we use graph analysis (see Section 3.3). The 3D models of the HLS extraction (see Figure 9e) are obtained plotting the coordinates (x,y,z) in Matlab.
Figure 5. The architecture of our CNN for HLS extraction.

4. Discussion and Results

In this section, we present the discussion and results of the proposed HLS extraction. These discussion and results are the integration of the proposed depth analysis to remove uncertain depth sections, a new graph analysis to locate possible 3D shapes and a new CNN architecture that predicts HLS. We evaluated our approach on our simulated dataset and two datasets that provide different urbanized scenes: Make3D [15,37] and KITTI [38]. Quantitative evaluation was performed using comparisons of our depth and ground-truth depth. Furthermore, we evaluated our segmentation orientation using pixel comparisons with the ground-truth. Finally, we used the simulated dataset to evaluate our HLS extraction.

4.1. Depth Evaluation

We compared our depth post-processing analysis with baseline methods. We evaluated it on two popular datasets, which are available online: Make3D dataset [15,37] and the KITTI dataset [38]. To provide quantitative results, we used two measures: the root mean squared error (rms), Equation (5), and average log 10 error ( log 10 ), Equation (6); where d p g t and d p are the ground-truth and predicted depths respectively at the pixel indexed by p and T is the total number of pixels in all of the evaluated images.
r m s = 1 T ∑ p ( d p g t - d p ) 2
log 10 = 1 T ∑ p log 10 d p g t - log 10 d p
Our post-processing analysis used a predicted depth. We used the depth images predicted by DCNF-FCSP [26]. Using these depth images, we used a post-processing analysis to obtain our refined depth. Figure 6 shows the input images, ground-truth, images predicted by DCNF-FCSP [26] and our refined depth. We can see the qualitative results of the depth post-processing analysis (see Figure 6d). As can be seen, our refined depth provided an HLS depth with higher sharpness, i.e., our refined depth easily located the HLS (cubes and rectangular parallelepipeds) of the buildings. Furthermore, in Table 1, we show quantitative comparisons concerning the state-of-the-art for the Make3D [15,37] and KITTI datasets [38]. We can see that our depth post-processing analysis improved the results predicted by DCNF-FCSP [26].
Figure 6. Examples of qualitative comparisons on the Make3D dataset. Colour indicates depths (red is far; blue is close). (a) Input image; (b) Ground-truth; (c) DCNF-FCSP [26]; (d) Our refined depth.
Table 1. Depth estimation using the state-of-the-art and our refined depth. We compare on the Make3D and KITTI datasets.

4.2. Segmented Orientations Evaluation

In this subsection, we use the Make3D dataset [15,37] and the KITTI dataset [38] to evaluate our segmented orientations. However, to analyse our segmented orientations, we performed ground truth labelling. This ground truth consists of orientation labelling. The quantitative evaluation was performed using pixel comparisons of the obtained segmented orientations with the ground-truth. To provide quantitative results, we used three measures ( r e c a l l , p r e c i s i o n and F-score) based on the numbers of true positives, true negatives, false positives and false negatives. The true positives T p count the number of pixels whose orientation label was predicted correctly w.r.t. to the ground truth. To count the number of true negatives T n , we proceed as follows: suppose that we are interested in the orientation label Down view, then all those pixels corresponding to other orientations rather than down view, according to the ground truth, should have received any other predicted label except Down view; if that is the case, each of these pixels are counted as true negatives. The false positives F p correspond to all those pixels whose orientation label is incorrect. Finally, false negatives F n correspond to those pixels that should have received a specific label, but the prediction did not assign it correspondingly, for instance those portions of the image corresponding to the floor, should have received an Upward view label for each pixel; however, if any floor pixels did not receive such a label, then those are counted as false negatives. In terms of the measures, we carried out an analysis by each orientation label. In this sense, we used the r e c a l l to measure the proportion of pixels whose respective orientation label was predicted correctly regarding the total amount of pixels in the ground truth labelled with such an orientation label, that is in simple terms, the amount of ground truth that was predicted correctly. The p r e c i s i o n measures the proportion of orientation labels that were predicted correctly. Finally, the F-score helps to summarise the performance of the predictions returned by the system. In sum, we could say that for a system with good performance, both the recall and precision should tend to one, meaning that most of the system’s predictions tend to be correct and that such predictions tend to cover most of the ground truth. If this is the case, then the F-score should tend to one.
r e c a l l = T p T p + F n
p r e c i s i o n = T p T p + F p
F - s c o r e = 2 1 r e c a l l + 1 p r e c i s i o n = 2 r e c a l l ∗ p r e c i s i o n r e c a l l + p r e c i s i o n
Table 2 shows the results of our different segmented orientations on the Make3D [15,37] and KITTI datasets [38]. To provide quantitative results, we use three measures ( r e c a l l , p r e c i s i o n and F-score) on the five principal orientations (down view, right view, front view, left view and upward view), because the KITTI dataset notably only contains these orientations. Our segmented orientations have on average: a r e c a l l of 0.879758, a p r e c i s i o n of 0.908150 and an F-score of 0.892359.
Table 2. Semantic orientation estimation using the Make3D and KITTI datasets. To provide quantitative results, we use three measures ( R e c a l l , P r e c i s i o n and F-score).
We deem that our results are adequate since using our orientation segmentation, we had a r e c a l l average of 0.879758, i.e., considering the ground-truth, we segmented 87.9% on average correctly. Furthermore, the results improved considering the p r e c i s i o n , and we had a p r e c i s i o n average of 0.908150, i.e., considering our segmented orientation, we segmented 90.8% on average correctly. In addition, the results of all our different segmented orientations had an F-score greater than 0.85. These results are important since the semantic orientation is an essential pre-processing step in our HLS method.
Finally, in our method, the orientation with a front view usually invades the orientations with a left and right view. This decreases the segmentation of the orientations with the left and right view (decreasing its r e c a l l ). On the other hand, by invading sections that do not belong to the front view, the orientation with a front view decreases in its p r e c i s i o n .

4.3. HLS Extraction Evaluation

We elaborated a new dataset using the Gazebo [39] simulator to evaluate the HLS extraction. The dataset consists of 152,000 images ( 480 × 640 pixels) with four 3D shapes (cube, half cube, horizontal rectangular parallelepiped and vertical rectangular parallelepiped) collected from a simulated environment. Figure 7 shows the four 3D shapes (cube, half cube, horizontal rectangle and vertical rectangle). We divided the dataset into two sets: the training set and the test set. The training set had 132,000 images, i.e., 33,000 images for every 3D shape. The test set had 20,000 images, i.e., 5000 images for every 3D shape. Therefore, there were four test sets with 5000 images and four training sets with 33,000 images. Every set (test sets or training sets) only had one 3D shape (cube, half cube, horizontal rectangular parallelepiped or vertical rectangular parallelepiped). For that, we created the 3D shape in the Gazebo simulator, and using Python, we rotated and moved the 3D shapes randomly. Finally, for every rotation and random move, we saved an RGB image, a txt with the ground-truth of depth and a txt with the ground-truth of the coordinates (x,y,z) of all vertices.
Figure 7. 3D shapes: we used four 3D shapes (cube, half cube and rectangular parallelepipeds) to represent the HLS extraction. (a) Cube; (b) Half cube; (c) Horizontal rectangular parallelepiped; (d) Vertical rectangular parallelepiped.
Furthermore, we used the training set to train the CNN and a test set to evaluate the CNN. Using depth information, the CNN provided the coordinate (x,y,z) vertices of a 3D shape. Finally, we extracted the HLS with the coordinates of the vertices. To evaluate the HLS extraction, we compared the coordinates of vertices predicted of the CNN with the ground-truth of the coordinates of vertices. For that, we used the root mean squared error (rms), Equation (5); where d p and d p g t are the coordinates predicted by our CNN and its ground-truth respectively at coordinates indexed by p and T is the total number of coordinates in all the test set.
We use the architecture of our network to learn the coordinates of one HLS specifically, i.e., we have four CNN, and every CNN extracts one structure (cube, half cube, horizontal rectangular parallelepiped or vertical rectangular parallelepiped). To know what CNN to use, we use graph analysis (see Section 3.3). In Table 3, we show the HLS extraction evaluation with different training (the images number used to train our CNN) for every structure (cube, half cube, horizontal rectangular parallelepiped or vertical rectangular parallelepiped). As can be seen, the network gets better generalization abilities with the increase of the images number, i.e., the CNN gets better coordinate prediction if we increase the image number used to train. Furthermore, the coordinate prediction by our four CNN architectures (trained with 33,000 images) had an rsm average between 0.35 and 0.36. This shows the stability in the prediction of HLS coordinates.
Table 3. We measured the coordinate (x,y,z) vertices of our HLS extraction with the ground-truth. Our CNN training used different simulated element numbers in the comparison. We measured the HLS extraction effectiveness with the root mean squared error (rms).
In addition, to analyse the robustness of our HLS extraction method with our four CNN trained with 33,000 images, we tested our HLS extraction method on three datasets that provided different outdoor scenes (Make3D [15,37], KITTI [38] and our simulated images). Figure 8, Figure 9 and Figure 10 show some qualitative results of our HLS extraction method on the three datasets. We can see that our HLS extraction presents a reliable 3D representation of the observed structure.
Figure 8. Our HLS extraction method on our simulated images. (a) Input image; (b) Orientation segmentation; (c) Graph analysis; (d) HLS extraction.
Figure 9. Our HLS extraction method on the Make3D dataset. (a) Input image; (b) Orientation segmentation; (c) Graph analysis; (d) Our depth; (e) HLS extraction.
Figure 10. Our HLS extraction method on the KITTI dataset. (a) Input image; (b) Orientation segmentation; (c) Graph analysis; (d) HLS extraction.

5. Conclusions

In this work, we have presented a novel method for HLS extraction from a single image. The images processed by our method correspond to outdoors urban scenes. Our method combines four elements: (i) we use a CNN to predict the depth for a single image; (ii) we proposed a depth analysis to removes uncertain depth sections; (iii) we proposed a new graph analysis to segment the depth in semantic orientations (namely: left, right, front, up, down), which enables the grouping of sections whose connectivity could be used to infer HLS; and (iv) the connected sections are used as input in a new CNN architecture, proposed by us, to predict HLS in the shape of cubes and rectangular parallelepipeds.
Regarding the experimental results, we carried out a set of evaluations to assess the classification of the semantic orientations in the single image. These evaluations returned a r e c a l l average of 0.879758, i.e., considering the ground-truth, we segmented the 87.9% on average correctly. Furthermore, we have an average p r e c i s i o n of 0.908150, i.e., considering our segmented orientation, we segmented 90.8% on average correctly. In addition, the results of all our different segmented orientations have an F-score greater than 0.85. We deem these results adequate since the semantic orientations are an essential pre-processing step in our method.
We continued our evaluations to assess the performance of the CNN used to recover the 3D HLS from joint segments detected in the single image. For this, we used a virtual environment based on the simulator Gazebo in order to generate the ground truth for the 3D shapes observed in the single test images. Thus, our four CNN architectures (trained with 33,000 images) obtained an r s m average between 0.35 and 0.36. This shows the stability in the prediction of HLS coordinates. Last, but not least, we carried out a qualitative evaluation of the whole method using three datasets with different outdoor scenes.
The results reported in this work indicate that our methodology is feasible and promising. We envisage that our proposed methodology can be extended to more complex polyhedral structures. In this sense, we should highlight that our goal is not that of fine/dense 3D reconstruction, but to generate a useful abstract representation that can be exploited in different applications ranging from robotics to computer vision. Some concrete application examples we can think of are: occlusion representation during navigation; semantic annotation (buildings, roads); city modelling (using HLS); and augmented and mixed reality applications.
We differentiate our approach from other CNN-based approaches for 3D object pose estimation in the sense that we focus on urban environments, where we aim at interpreting the scene in order to generate abstract representations of buildings. These representations involve the use of HLS in the shape of 3D volumetric primitives such as cubes and rectangular parallelepipeds, whose 3D position and orientation are also estimated w.r.t. to the optical centre of the camera view corresponding to the single image. As shown in some qualitative examples, our method enables the extraction of such parallelepipeds, which can be placed in the same reference coordinate system, thus enabling a high-level 3D representation of what is being observed in a single image.
Finally, we should note that our approach involves a combination of techniques based on CNN architectures with conventional methods for classification (from the machine learning point of view) and rule-based segmentation and data grouping, which in our opinion brings the best of the two worlds to address the difficult problem of a single-image interpretation, which in our case was focused on the task of extracting HLS.
Future work involves further investigation of the improvement of the CNN architectures used in this work. There is also an open question about what steps in our method can be replaced by CNN modules if all the modules or even if the whole problem can be solved using a single CNN architecture. In the mean time, we consider this work and its obtained results to be the first baseline to be overcome. We will also explore the use of more complex parallelepipeds to those used in this work, seeking to find the trade-off between representation and usefulness for the user and/or application.

Author Contributions

Conceptualization, J.A.d.J.O.-C. and J.M.-C. Investigation, validation and writing, original draft preparation: J.A.d.J.O.-C. and J.M.-C. Supervision and writing, review and editing: J.A.d.J.O.-C. and J.M.-C.

Funding

This research was funded by CONACYT grant number 594590.

Acknowledgments

The first author is thankful to the National Council on Science and Technology (CONACYT) for the scholarship (594590). The second author is thankful for the support received through his Royal Society-Newton Advanced Fellowship with Reference NA140454.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Dani, A.; Panahandeh, G.; Chung, S.J.; Hutchinson, S. Image moments for higher-level feature based navigation. In Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, 3–7 November 2013; pp. 602–609. [Google Scholar] [CrossRef] [Scilit]
  2. Osuna-Coutiño, J.A.D.J.; Cruz-Martínez, C.; Martinez-Carranza, J.; Arias-Estrada, M.; Mayol-Cuevas, W. I Want to Change My Floor: Dominant Plane Recognition from a Single Image to Augment the Scene. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), Merida, Mexico, 19–23 September 2016; pp. 135–140. [Google Scholar] [CrossRef] [Scilit]
  3. Li, W.; Song, D. Toward featureless visual navigation: Simultaneous localization and planar surface extraction using motion vectors in video streams. In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–7 June 2014; pp. 9–14. [Google Scholar] [CrossRef] [Scilit]
  4. Vosselman, G.; Dijkman, S. 3D building model reconstruction from point clouds and ground plans. In Proceedings of the International Society for Photogrammetry and Remote Sensing (ISPRS), Annapolis, MD, USA, 22–24 October 2001; pp. 37–43. [Google Scholar]
  5. Gee, A.; Chekhlov, A.; Calway, A.; Mayol-Cuevas, W. Discovering Higher Level Structure in Visual SLAM. IEEE Trans. Robot. 2008, 24, 980–990. [Google Scholar] [CrossRef] [Scilit]
  6. Bartoli, A.; Sturm, P. Constrained Structure and Motion From Multiple Uncalibrated Views of a Piecewise Planar Scene. Int. J. Comput. Vis. (IJCV) 2003, 52, 45–64. [Google Scholar] [CrossRef] [Scilit]
  7. Silveira, G.F.; Malis, E.; Rives, P. Real-time robust detection of planar regions in a pair of images. In Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China, 9–15 October 2006; pp. 49–54. [Google Scholar] [CrossRef] [Scilit]
  8. Gee, A.; Chekhlov, D.; Mayol, W.; Calway, A. Discovering planes and collapsing the state space in visual SLAM. Br. Mach. Vis. Conf. (BMVC) 2007, 6–12. [Google Scholar] [CrossRef] [Scilit]
  9. Aguilar-González, A.; Arias-Estrada, M.; Berry, F. Depth from a Motion Algorithm and a Hardware Architecture for Smart Cameras. Sensors 2019, 19, 53. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  10. Zhao, S.; Fang, Z. Direct Depth SLAM: Sparse Geometric Feature Enhanced Direct Depth SLAM System for Low-Texture Environments. Sensors 2018, 18, 3339. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  11. Firman, M.; Mac Aodha, O.; Julier, S.; Brostow, G.J. Structured prediction of unobserved voxels from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 5431–5440. [Google Scholar] [CrossRef] [Scilit]
  12. Maturana, D.; Scherer, S. 3D Convolutional Neural Networks for Landing Zone Detection from LiDAR. In Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA, 26–30 May 2015; pp. 3471–3478. [Google Scholar] [CrossRef] [Scilit]
  13. Wu, J.; Zhang, C.; Xue, T.; Freeman, W.T.; Tenenbaum, J.B. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In Proceedings of the Advances in Neural Information Processing Systems 29 (NIPS 2016), Barcelona, Spain, 5–10 December 2016. [Google Scholar]
  14. Hoiem, D.; Efros, A.A.; Hebert, M. Recovering surface layout from an image. Int. J. Comput. Vis. 2007, 75, 151–172. [Google Scholar] [CrossRef] [Scilit]
  15. Saxena, A.; Sun, M.; Ng, A.Y. Make3D: Learning 3D Scene Structure from a Single Still Image. IEEE Trans. Pattern Anal. Mach. Intell. 2009, 31, 824–840. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  16. Haines, O.; Calway, A. Recognising planes in a single image. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 1849–1861. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  17. Hoiem, D.; Efros, A.A.; Hebert, M. Geometric context from a single image. In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), Beijing, China, 17–21 October 2005; pp. 654–661. [Google Scholar] [CrossRef] [Scilit]
  18. Kosecká, J.; Zhang, W. Extraction, matching, and pose recovery based on dominant rectangular structures. Comput. Vis. Image Understand. 2005, 100, 274–293. [Google Scholar] [CrossRef] [Scilit]
  19. Micusik, B.; Wildenauer, H.; Kosecka, J. Detection and matching of rectilinear structures. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 23–28 June 2008; pp. 1063–6919. [Google Scholar] [CrossRef] [Scilit]
  20. McClean, E.; Cao, Y.; McDonald, J. Single Image Augmented Reality Using Planar Structures in Urban Environments. In Proceedings of the 2011 Irish Machine Vision and Image Processing Conference, Dublin, Ireland, 7–9 September 2011; pp. 1–6. [Google Scholar] [CrossRef] [Scilit]
  21. Haines, O.; Calway, A. Estimating Planar Structure in Single Images by Learning from Examples. In Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM), Algarve, Portugal, 6–8 February; pp. 289–294.
  22. Xiang, Y.; Schmidt, T.; Narayanan, V.; Fox, D. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Proc. Robot. Sci. Syst. 2018, 1–10. [Google Scholar] [CrossRef] [Scilit]
  23. Fan, H.; Su, H.; Guibas, L. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2016; pp. 1–12. [Google Scholar]
  24. Liu, M.; Salzmann, M.; He, X. Discrete-Continuous Depth Estimation from a Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 716–723. [Google Scholar] [CrossRef] [Scilit]
  25. Zhuo, W.; Salzmann, M.; He, X.; Liu, M. Indoor Scene Structure Analysis for Single Image Depth Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 614–622. [Google Scholar] [CrossRef] [Scilit]
  26. Liu, F.; Shen, C.; Lin, G.; Reid, I. Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields. IEEE Trans. Pattern Anal. Mach. Intell. 2016, 38, 2024–2039. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  27. Liu, D.; Liu, X.; Wu, Y. Depth Reconstruction from Single Images Using a Convolutional Neural Network and a Condition Random Field Model. Sensors 2018, 18, 1318. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  28. Cherian, A.; Morellas, V.; Papanikolopoulos, N. Accurate 3D ground plane estimation from a single image. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Kobe, Japan, 12–17 May 2009; pp. 2243–2249. [Google Scholar] [CrossRef] [Scilit]
  29. Rahimi, A.; Moradi, H.; Zoroofi, R.A. Single image ground plane estimation. IEEE Int. Conf. Image Process. (ICIP) 2013, 2149–2153. [Google Scholar] [CrossRef] [Scilit]
  30. Wu, Z.; Song, S.; Khosla, A.; Yu, F.; Zhang, L.; Tang, X.; Xiao, J. 3D ShapeNets: A Deep Representation for Volumetric Shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 1912–1920. [Google Scholar] [CrossRef] [Scilit]
  31. Sharma, A.; Grau, O.; Fritz, M. VConv-DAE: Deep Volumetric Shape Learning Without Object Labels. Eur. Conf. Comput. Vis. 2016, 9915, 236–250. [Google Scholar] [CrossRef] [Scilit]
  32. Ren, M.; Niu, L.; Fang, Y. 3D-A-Nets: 3D Deep Dense Descriptor for Volumetric Shapes with Adversarial Networks. arXiv, 2017; arXiv:1711.10108.
  33. Shi, X.; Chen, Z.; Wang, H.; Yeung, D.Y.; Wong, W.K.; Woo, W.C. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Comput. Vis. Pattern Recognit. 2015, 802–810. [Google Scholar]
  34. Chen, J.; Luo, D.-L.; Mu, F.-X. An improved ID3 decision tree algorithm. In Proceedings of the International Conference on Computer Science & Education, Nanning, China, 25–28 July 2009. [Google Scholar] [CrossRef] [Scilit]
  35. Norris, J.R. Markov Chains; Cambridge University Press: Cambridge, UK, 1998. [Google Scholar]
  36. Bazarra, M.S.; Jarvis, J.; Sherali, H. Linear Programming and Networks Flow; John Wiley & Sons: New York, NY, USA, 1990. [Google Scholar]
  37. Saxena, A.; Chung, S.H.; Ng, A.Y. Learning Depth from Single Monocular Images. In Proceedings of the Advances in Neural Information Processing Systems 19 (NIPS 2006), Vancouver, BC, Canada, 4–9 December 2006. [Google Scholar]
  38. Uhrig, J.; Schneider, N.; Schneider, L.; Franke, U.; Brox, T.; Geiger, A. Sparsity Invariant CNNs. In Proceedings of the International Conference on 3D Vision 2017, Qingdao, China, 10–12 October 2017. [Google Scholar]
  39. Howard, A.; Koenig, N. gazebosim.org; University of Southern California: Los Angeles, CA, USA, 2018. [Google Scholar]

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.