To represent a robot's surrounding terrain in a virtual environment, it is necessary to reconstruct a terrain model using an integrated dataset obtained from multiple sensors [8–12]. Rovira-Más [13] proposed a density grid for 3D reconstruction from information obtained from stereo cameras, a localization sensor, and an inertial measurement unit. Sukumar [3] provided a convenient visualization method by integrating sensed datasets into a textured terrain mesh. However, it is difficult for these systems to process the large datasets obtained in outdoor environments and achieve on-line rendering.

Other researchers have enhanced the performance of terrain reconstruction to provide on-line photo-realistic visualization. Kelly [9] describes real-world representation methods using video-ranging modules. In the near field, 3D textured voxel grids are used to describe the surrounding terrain, whereas a billboard texture in front of the robot is used to show scenes in the far field. However, a range sensor cannot sense all terrain information, often leaving empty spaces in the terrain model in practice.

Recovery of these “unsensed” regions plays a major role in obstacle avoidance. Some researchers apply interpolation algorithms to fill empty holes and smooth terrain [14–17]. For example, to estimate such unobserved data, Douillard [18] interpolates grids in empty regions in elevation maps in order to propagate label estimates. However, it is difficult to use these methods to recover missing information that is beyond the measurement range of the sensors.

Wellington [19] applies a hidden semi-Markov model to classify terrain vertical structure into ground, trees, and free space classes for each cell of a voxel-based terrain model. Then an MRF algorithm is used to estimate ground and tree height. However, this height estimation process simply averages across cells using neighbor data and cannot estimate actual height values.

In hardware design research, Früh [7] utilizes a vertical 2D laser scanner to measure large buildings and represent streetscapes in urban environments. When an object is located between the sensors and a building, some regions of the building cannot be sensed by the laser scanner as they are blocked by the object. These missing regions can be easily filled by planar or horizontal interpolation algorithm.

Ground segmentation is a widely studied topic necessary to determine the traversable regions in a terrain. Pandian [2] classifies terrain features into rocky, sandy, and smooth classes solely from 2D images. The segmented results take the form of a rectangular grid, instead of polygon shape. Therefore, this method lacks precision.

The MRF algorithm is effective in object segmentation from 2D images and 3D point clouds [20–26]. However, it is difficult to specify the probability density functions (PDF) in MRF. To solve this problem, the Hammersley-Clifford theorem proves an equivalence relationship between MRF and the Gibbs distribution [25]. However, computation of the Gibbs-MRF is too complicated for real-time ground segmentation.

Object segmentation is necessary to extract features, implement classification, and generate a semantic map. Weiss [27] utilizes a RANSAC algorithm to detect the ground and organize a point cloud into several clusters by segmenting plants and measuring plant positions. Segmented plants are estimated with high accuracy. However, this method can only be used for small plants, because it cannot be applied to objects outside the sensor's measurement range.

Golovinskiy [28] proposed a graph-based object segmentation method. The 3D points sensed by the range sensor are grouped into nodes of a graph using the k-nearest neighbor algorithm. The min-cut algorithm is then applied to segment the nodes into several objects. Lalonde [29] segments 3D points into scatter-ness, linear-ness, and surface-ness saliency features. In this method, an object model with a special saliency feature distribution is trained off-line by fitting a Gaussian mixture model (GMM) using the expectation-maximization (EM) algorithm. New data can be classified on-line into the model with a Bayesian classifier.

Huber [30] proposed a semantic representation method for building components. The floor and ceiling components are identified by finding the bottom-most and top-most local maxima in the height histogram. After low-density cells in the ground plane histogram are removed, the wall lines are detected using the Hough transform.

Nüchter [8] described a feature-based object detection method for 3D point cloud classification. First, the plans are extracted from the 3D point cloud using the RANSAC algorithm. Then, the wall, floor, ceiling, and other objects are labeled according to the defined scene interpretation. Finally, the objects are detected from a 2D image taken from the 3D rendering result.

In this paper, we discuss a multisensor integration method. For ground segmentation, we use the Gibbs-MRF and a flood-fill algorithm. Further, in contrast to interpolation methods, we propose a height estimation algorithm to recover unsensed regions, especially for objects at a height and outside the sensor's range of measurement.

We integrate the sensed dataset into a grid-based textured terrain mesh. First, we project the 3D points onto the 2D image in front of the robot and get a coordinate in 2D image, named UV vector, for each 3D point. Then, we transform the local 3D points into global coordinates, and register them on the terrain mesh. The terrain mesh is generated using several grids, each with 151 × 151 textured vertices. In this application, the cell size is 0.125 × 0.125 m². The height value of each cell is updated with the registered 3D points. If a new 3D point is to be inserted into the reconstructed terrain mesh but is outside the existing grids, we create a new grid to register this point, as shown in Figure 2.

After registration of all points, a ground segmentation algorithm is implemented to segment ground data and non-ground data in the 2D image. Then, a height estimation method is used to recover the missing regions outside the sensor's measurement range. Finally, the tree and building objects are classified using a classification operation. In our implementation, the user controls a virtual camera to study the reconstructed terrain from different viewpoints. A virtual robot model is loaded on the terrain to show the robot's navigation information in the real world.

We classify each pixel in the 2D image into ground and non-ground classes on the basis of the probability of it being in that configuration, which depends on its connected neighbors. Therefore, we can apply the MRF for ground segmentation. However, it is difficult to determine the probability because it must be computed from local and neighbor observations. According to the Hammersley-Clifford theorem, we can solve this problem using the Gibbs-MRF model.

Given observation d and configuration f, we find the best possible configuration f* for site s using the following optimum solution: (1) f ∗ ( s ) = arg max f p ( X s = f | X t = d , ∀ t ≠ s )

The probability of a site's configuration is calculated using the Gibbs distribution [22]: (2) p ( f ) = Z − 1 e − 1 T U ( f ) (3) U ( f ) = ∑ c ∈ C V c ( f ) (4) Z = ∑ f e − 1 T U ( f )

We define a clique as a neighboring set, and a clique set C as a collection of single-site and pair-site cliques. A potential function V_c(f) is defined to evaluate the effect of neighbor sites in clique c.

According to the Bayes' rule, the solution of Equation (1) is as follows: (5) f ∗ = arg max f p ( f | d ) = arg max f p ( d | f ) p ( f ) = arg min f U ( f | d ) = arg min f { U ( d | f ) + U ( f ) }

The energy function of U(d|f) + U(f) is defined to evaluate the effect of the neighbor sites in single-site and pair-site potential cliques, as follows: (6) U ( d | f ) + U ( f ) = ∑ s ∈ C 1 V 1 ( f s ) + ∑ { s , s ' } ∈ C 2 V 2 ( f s , f s ' ) + ∑ s ∈ C 1 V 1 ( d s | f s ) + ∑ { s , s ' } ∈ C 2 V 2 ( d s , d s ' | f s , f s ' )

The evaluations of the clique potential functions V₁(f_s) and V₁(d_s|f_s) depend on the local configuration and observations of clique C₁. The clique potential functions V₂(f_s, f_s') and V₂(d_s, d_s'|f_s, f_s') are evaluations of the pair-site consistency of clique C₂.

When we apply the Gibbs-MRF to ground segmentation in a 2D image, we first determine a set of pixels whose configurations are in the ground class with high confidence. We initially segment the 3D points as ground data using the robot vehicle's height h₁ as the standard. We assume that if the y coordinate of a 3D point is ranging from −h₁ − Δ to −h₁ + Δ, then this point is ground data, as shown in Figure 3. This step is a rough ground segmentation process, which produces a dataset G₁.

Then we find the projected pixels in the 2D image from the points in G₁, using the projection matrix as follows: (7) t = K R [ I | − Cam ] Twhere the homogeneous coordinates of image pixel t are projected from the homogeneous coordinates of the 3D point T. Cam is defined as the vector of the camera's position, the matrix R is defined as the mobile rotation matrix, and I is an identity matrix. The camera calibration matrix K is defined as follows: (8) K = [ l 0 p x 0 l p y 0 0 1 ]where l is the focal length of the camera, and the 2D coordinate (p_x, p_y) is the center position of the captured image. As shown in Figure 4, the 2D pixel dataset G 1 ' is mapped from the dataset G₁. We determine the configuration of site s ∈ G 1 ' as ground.

We apply the Gibbs-MRF algorithm to classify the configurations of other pixels into the ground or non-ground classes. We consider that:

If the configuration of site s is same as its observation, the probability of this configuration is high.

The clique potential functions are formulated as follows: (9) V 1 ( f s ) = { − α i f ( s ∈ G 1 and f s = ground ) o r ( s ∉ G 1 and f s = nonground ) + α i f ( s ∉ G 1 and f s = ground ) o r ( s ∈ G 1 and f s = nonground ) (10) V 1 ( d s | f s ) = { − α i f ( d s = f s ) + α i f ( d s ≠ f s ) (11) V 1 ( f s , f s ' ) = { − β i f ( f s = f s ) + β i f ( f s ≠ f s ) (12) V 1 ( d s , d s ' | f s , f s ' ) = { − γ e − ‖ d s − d s ' ‖ i f ( f s = f s ' ) + γ e − ‖ d s − d s ' ‖ i f ( f s ≠ f s ' )

Here, the constants α, β, and γ are positive numerical values. The configuration f_s depends on whether the pixel s belongs to the ground dataset G 1 '. The formula ‖d_s – d_s'‖ is defined as the color difference between observations d_s and d_s'.

We derive Equation (5) using the potential functions defined in Equations (9–12), and label the configuration of each pixel.

To reduce the computation load of Gibbs-MRF, we apply a flood-fill algorithm to compute the configurations of pixels inside the boundary between ground and non-ground. The pseudocode for ground segmentation using the flood-fill algorithm is as follows: for each site s in G 1 '  configuration f(s) = ground;  enqueue neighbour sites of s into a queue Q;  while (Q is not empty)   dequeue a site s′ from the Q;   if (f* (s′) =ground)    enqueue neighbor sites of s′ into Q;   endif;  endwhile; endfor;

Starting with the pixel set G 1 ', we estimate the configurations of the neighboring pixels. We apply the Gibbs-MRF algorithm to classify the configurations of other pixels into the ground or non-ground classes.

The pixels with a ground configuration are grouped into dataset G 2 ', which is shown as the blue region. The other regions contain objects and background textures. We classify the ground vertices in the 3D terrain mesh, which are mapped to the pixels in the dataset G 2 ', as shown in Figure 11(b).

When mobile robots detect surrounding terrain information, some parts of objects are outside the measurement of range sensors. We see that the top of the building is missing in the terrain reconstruction result, shown as Figure 5.

We propose a height estimation method to solve the problem of missing regions by estimating the y coordinate of an object's top boundary.

Using the ground data segmentation result, we assume that the non-ground vertices in the terrain mesh belong to objects, because background data, such as the sky, cannot be sensed by the range sensor. Next, we project these vertices onto pixels in a 2D image, whose configuration is determined as being part of an object. We apply the Gibbs-MRF method to classify the non-ground pixels into objects and background classes, in order to detect the boundary pixels between objects and background. The boundary detection results are shown as red pixels in Figure 6.

We find the boundary's y coordinates using an inverse process of projection from 2D pixels to 3D points. We place the camera centre at the origin. The projection ray from the origin to the object vertex gives an estimate of the height of that object vertex, as shown in Figure 7. Because the horizon coordinates of the 3D object vertex in the terrain mesh are fixed, we update the elevation value of each object vertex in the terrain mesh to obtain the results shown in Figure 8.

We consider tree objects, including both grass and trees, to have a porous surface that allows rays from the range finder to pierce through to the inside. This is in contrast to buildings, for which the 3D range finder only detects points on the outer surface. Therefore, the horizon shape of a building has a uniform distribution, whereas that for a tree has a normal distribution. As shown in Figure 9, we can see that the horizon structure of the buildings consists of the line-like components. We classify buildings by detecting these lines using the masks described in Figure 10.

The convolution function for the masking method is: (13) U ( i , j ) = ∑ n = − s s ∑ m = − s s h ( i − m , j − n ) f ( m , n )where h(i, j) is the elevation value of a vertex in the terrain mesh, f(m, n) is the value in a mask cell, and s is the size of the mask. If U(i, j) is larger than a threshold, we determine the vertex (i, j) belongs to a building. If not, we determine the vertex belongs to a tree. After classifying buildings in the terrain mesh, we map the building vertices onto the 2D images in order to identify the sensed buildings in the 2D images.