This article is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

An objective error criterion is proposed for evaluating the accuracy of maps of unknown environments acquired by making range measurements with different sensing modalities and processing them with different techniques. The criterion can also be used for the assessment of goodness of fit of curves or shapes fitted to map points. A demonstrative example from ultrasonic mapping is given based on experimentally acquired time-of-flight measurements and compared with a very accurate laser map, considered as absolute reference. The results of the proposed criterion are compared with the Hausdorff metric and the median error criterion results. The error criterion is sufficiently general and flexible that it can be applied to discrete point maps acquired with other mapping techniques and sensing modalities as well.

A truly autonomous robot should be able to build an accurate spatial model of its unknown physical environment based on the sensory data it acquires. This problem is addressed by the field of robotic mapping. The environment in which the robot operates could be a static one, or more realistically, a dynamically changing environment. Therefore, the selection of a suitable mapping scheme is an important task [

Since robot mapping is characterized by sensor measurement noise and uncertainty, most of the current robotic mapping algorithms in the literature are probabilistic in nature [

Feature-based approaches are based on extracting the geometry of the environment from sensor data as the first step in data interpretation process (e.g., edge detection, fitting straight-lines or curves to obstacle boundaries, differentiating features such as planar walls, corners, and edges, open and closed doors, and corridors). Important issues to consider are the representation of uncertainty, suitability of the selected feature to the environment and the type of data, the reliability of the feature extraction process, and the speed with which the model can be constructed. An alternative representation is the

The map of an unknown environment can be acquired by making range measurements with a variety of sensing modalities, such as ultrasonic sensors, laser range finders, or millimeter wave radar. However, the acquired map is usually erroneous, consisting of outliers, gaps, noisy or incorrect measurements. Some of the measurements are caused by multiple or higher-order reflections from specular (mirror-like) surfaces, or crosstalk between multiple sensors, making the data difficult to interpret. This is particularly a problem with airborne ultrasonics and underwater sonar sensing. Researchers have identified regions of constant depth (RCDs) [

In Section 2., we give a description of the proposed error criterion and provide two other criteria for comparison: the Hausdorff metric and the median error. The use of the criterion is demonstrated through an example from ultrasonic and laser sensing in Section 3. Section 4. provides details of the experimental procedure and compares the results of the proposed criterion with the Hausdorff metric and the median error. Section 5. discusses the limiting circumstances for the criterion that may arise when there are temporal or spatial differences in acquiring the maps. The last section concludes the paper by indicating some potential application areas and providing directions for future research.

Let ^{3} and ^{3} be two finite sets of arbitrary points with _{1} points in set _{2} points in set

The well-known Euclidean distance _{i}_{j}^{3} → ℝ^{≥}^{0} of the _{i}_{xi}_{yi}_{zi}^{T}_{j}_{xj}_{yj}_{zj}^{T}

There is a choice of metrics to measure the similarity between two sets of points, each with certain advantages and disadvantages:

A very simple metric is to take the minimum of the distances between any point of set _{i}_{j}_{i}

The error criterion we propose for measuring the closeness or similarity between sets _{1} = 3 and _{2} = 2, the error is

The error criterion we propose is sufficiently general that it can be used to compare any two arbitrary sets of points with each other. This makes it possible to compare the accuracy of discrete point maps acquired with different techniques or sensing modalities with an absolute reference, as well as among themselves, both in 2-D and 3-D. When curves or shapes (e.g., lines, polynomials, snakes, spherical or elliptical caps) are fitted to the map points, the criterion proposed here also enables us to make an assessment of goodness of fit of the curve or shape to the map points. In other words, a fitted curve or shape comprised of a finite number of points can be treated in exactly the same way.

A widely used measure for comparing point sets in the computer vision area is the Hausdorff metric [

More formally, the _{i}_{j}_{i}_{j}_{i}_{j}

A more general definition of Hausdorff distance is
_{3}, _{2}), _{2}, _{2}), and _{3}, _{2}), _{2}, _{2}) } = _{3},q_{2}) since _{3}, _{2}) > _{2}, _{2}).

Instead of taking the average or the maximum value among the minimum distance values, another possibility is to take the median of the minimum distances. In this case, a suitable error measure can be defined as:

For all of the criteria above, the two sets of points can be chosen as (i) two different sets of map points acquired with different mapping techniques or different sensing modalities (e.g., ultrasonic and laser, laser being the accurate reference), or (ii) two sets of fitted curve points (2-D) or shapes (3-D) (e.g., two polynomials, snake curves or spherical caps) to maps extracted by different mapping techniques or sensing modalities, or (iii) a set of extracted map points and a set of curve points fitted to them (i.e., assessment of goodness of fit).

To demonstrate the use of the criterion through an example, we consider three ultrasonic mapping techniques. Most commonly, the large beamwidth of the transducer is accepted as a device limitation which determines the angular resolving power of the system. In this naive approach (method 1), a range reading of

The Arc-Transversal Median Algorithm (method 2) was developed by Choset and his co-workers, and requires both extensive bookkeeping and considerable amount of processing [

Method 3 is based on directional processing of ultrasonic data and is detailed in [

The latter two methods are the most recently reported techniques for processing ultrasonic range measurements. All three methods use the same set of time-of-flight measurements acquired from the same environment. However, due to differences in processing the measurements, each one results in a different set of map points.

The experiments are performed using the Polaroid 6500 series ultrasonic transducers [_{o} = 49.4 kHz and aperture radius _{o}, which is the round-trip travel time of the transmitted pulse between the transducer and the point of reflection, from which the range _{o}/2, where

In the first experiment, we considered a 90° corner which is a typical feature of indoor environments. Due to its high curvature at the corner point, accurate mapping of this feature is usually difficult with ultrasonic sensors. The laser data obtained from the 90° corner and the locations of the five ultrasonic sensors are presented in

For the second experiment, we have constructed curved surfaces of varying curvature and dimensions in our laboratory, using thin cardboard (see

We also present experimental results from the indoor environment in

The errors are of the order of several pixels and are presented in

In _{mean} since this metric takes the maximum among the minimum values, whereas our proposed metric averages the minimum distance values. Consequently, the Hausdorff distance severely penalizes outlier points in the map. For this reason, method 3, which is successful at eliminating the outliers in the second and third experiments results in smaller errors in these experiments.

Since we are comparing ultrasonic map points with much more accurate laser data in this example,

The results of calculating the median error are tabulated in _{mean} presented in

In summary, it can be concluded that the Hausdorff metric severely penalizes outlier measurements since it takes the maximum value among the minimum distances for the set points. The median error filters out the outliers and under-penalizes them because of the nonlinear filtering involved in taking the median. Our proposed metric provides a reasonable balance between these two metrics by weighting each of the minimum distances equally and calculating their average value bi-directionally.

In this work, we have assumed that the robot or the sensing mechanism is well localized so that sufficiently accurate estimates of its pose are available. If this assumption is not true so that the localization errors are large, the acquired maps may be shifted and rotated with respect to the true map, resulting in increased error values. In this case, localization errors could be dominating the map error. If we are comparing simultaneously acquired maps not with the true map but among themselves, the relative error between the maps will not be much affected from the localization errors. Note that our criterion still remains applicable under these circumstances.

We have also assumed that the maps are acquired with the different sensing modalities under similar conditions. In the case of a dynamic environment, maps should be acquired simultaneously. For a static environment, the timing could be different. In the example that we have given, the two data sets were acquired simultaneously from the ultrasonic and structured-light laser systems on the mobile robot.

Another possibility that may result in large errors between the two point sets can occur when the two sensing modalities compared are so different in nature or at completely different positions in the environment that the corresponding views of the environment and the occluded or “shadowed” parts of the environment may be very different.

In case the sensing modalities take partial views or scans of the environment, we assume that they correspond to the same sectoral scan. Otherwise, the map points would have to be matched and their correspondence be found, which is beyond the scope of this work. It is worth emphasizing once again that in this work, we do not require the correspondence between the two sets of points to be known and do not make distinctions between the points of the same data set.

To summarize, the limiting circumstances for the proposed criterion could be caused by spatial or temporal differences in the acquirement of the two sets of map points.

We have presented an objective criterion to compare maps obtained with different techniques with an absolute reference as well as among themselves. This criterion can be used to compare two arbitrary sets of points with each other without requiring the correspondence between the two sets of points to be known. The two sets of points can correspond to: (i) two different sets of map points acquired with different mapping techniques or different sensing modalities, or (ii) two sets of fitted curve points to maps extracted by different mapping techniques or sensing modalities, or (iii) a set of extracted map points and a set of curve points fitted to them. We have demonstrated how the criterion can be used through an example from ultrasonic sensing based on time-of-flight measurements, considering accurate laser data as absolute reference and have compared the results with the Hausdorff metric and the median error results. We believe that the error criterion is sufficiently general so that it can be applied to both 2-D and 3-D range measurements of different scales, acquired with different mapping techniques and sensing modalities. Potential application areas for this work include millimeter wave radar for unmanned ground vehicles, mobile robotics, underwater sonar, optical sensing and metrology, remote sensing, ocean surface exploration, geophysical exploration, and acoustic microscopy. As part of the future work, it would be beneficial to demonstrate the applicability of the criterion to these sensing modalities at different scales.

This work is supported by The Scientific and Technological Research Council of Turkey (TÜBİTAK) under grant number EEEAG-105E065.

(a) Point set

Nomad 200 mobile robot. The ring of ultrasonic sensors can be seen close to the top rim of the turret, and the structured-light laser system is seen pointing rightwards on top.

Laser data acquired from the three environments (top row, red) and the sensor configuration (top row, green); resulting maps of method 1 (2nd row, blue), method 2 (3rd row, magenta), and method 3 (4th row, green).

(a) An example curved surface and a meter stick; (b) a view of the indoor environment in

Results of the three experiments based on the proposed criterion.

_{mean} (pixels) | |||
---|---|---|---|

experiment 1 | experiment 2 | experiment 3 | |

| |||

method 1 | 1.19 | 6.64 | 4.37 |

| |||

method 2 | 1.40 | 4.04 | 4.26 |

| |||

method 3 | 2.24 | 2.13 | 2.49 |

Results of the three experiments based on the Hausdorff metric. The Hausdorff distance is highlighted in boldface fonts.

Hausdorff distance (pixels) | ||||||
---|---|---|---|---|---|---|

experiment 1 | experiment 2 | experiment 3 | ||||

| ||||||

| ||||||

method 1 | 2.24 | 6.08 | 16.12 | |||

| ||||||

method 2 | 3.16 | 8.54 | 17.00 | |||

| ||||||

method 3 | 5.00 | 8.94 | 14.00 |

Results of the three experiments based on the median criterion.

_{median} (pixels) | |||
---|---|---|---|

experiment 1 | experiment 2 | experiment 3 | |

| |||

method 1 | 1.21 | 2.29 | 1.71 |

| |||

method 2 | 1.21 | 2.12 | 1.50 |

| |||

method 3 | 2.12 | 2.00 | 1.21 |