Environment perception is an indispensable component for autonomous vehicle as it provides real-time environment information. Driving safety and intelligence of autonomous vehicle is based on accurate and quick understanding of the driving environment. Among the elements that consist the driving environment, road curbs are especially important for being boundaries of the drivable area.
To detect road curbs quickly and accurately, a variety of sensors are used including cameras and lasers. In the past decades, great achievements have been made in computer vision and some of the proposed methods have been successfully applied to detect lane marks, traffic participants and traffic signs including vehicle, bicycle and pedestrian, and traffic lights [1
]. However cameras are sensitive to light changes and no distance information is provided. Compared with cameras, Lidars can provide accurate distance information and they are not affected by light conditions. 2D-Lidars were widely used to detect obstacles in advanced driver assistant system on vehicles [9
]. While the data is too sparse for environment perception of autonomous vehicles. In recent researches on autonomous vehicles, 3D Lidar plays a more and more import role as it provides sufficient information, in the form of point cloud, to model the 3D environment. 3D Lidar is a kind of multi-beam Lidar which percept the environment by transmitting and receiving multiple laser rays. In 2005 DARPA Grand Challenge, only the team from Velodyne, a famous 3D Lidar manufacturer, was equipped with a 3D Lidar. While 6 of the 11 teams entered the final competition of 2007 DARPA Urban Challenge were equipped with a 3D Lidar [10
]. Recently, Google, Ford and Baidu choose 3D Lidars as the main sensor to obtain environment information [12
Ground segmentation is another way of saying obstacle detection. Height difference based methods were proposed to detect obstacles [17
]. Besides, a combination of height difference and convexity criterion was proposed to segment the ground in [22
]. Cheng et al. [23
] proposed a Ring Gradient-based Local Optimal Segmentation (RGLOS) algorithm to segment the ground based on the gradient feature between the rings. The common problem for the aforementioned method is that a single or multiple features are adopted in ground segmentation, resulting in over segmentation with strict threshold while under-segmentation with loose threshold. To overcome this shortcoming, ground segmentation method based on the multi-feature with loose threshold is presented in [24
]. These two methods take advantage of local road features to detect curbs without considering the global road trend. However, our experiment found that the applied scope of different features should be considered in greater detail because the geometric properties vary with different features.
Road curbs can be deemed as special obstacles which are boundaries of the drivable area. Chen et al. [25
] selected the initial curb points from the individual scan lines by using the distance criterion, Hough Transform (HT), and the iterative Gaussian Process Regression (GPR). Zhao [26
] obtained the seed points that meet three criteria and modeled the curb as a parabola while using a random sample consensus (RANSAC) algorithm to remove the false positives that do not match the parabolic model. Least trimmed squares (LTS) regression method is applied to fit a function in the curb candidate points in [27
] while a sliding window is applied to obtain curb points in [28
]. In the study of Liu et al. [24
], distance transformation is performed on the obstacle map and threshold segmentation is used to increase the continuity of the map. Thereafter, the drivable area is obtained by region growing, and the road curb points are obtained by searching. All the aforementioned methods shared the drawback of being easily affected by the obstacle inside the road area, especially for road with considerable number of vehicles, because of the lack of the overall road shape information. They are based on local features of the road rather than the global road trend.
Based on the aforementioned considerations, a road curb detection method, which takes global trend of road curb into consideration, is proposed in this study and shown in Figure 1
. First, a multi-feature, loose-threshold, varied-scope ground segmentation method is presented in this work. Features used in the segmentation process are discussed in detail. Second, a road curb detection process is proposed. The road curb can be obtained through extraction or tracking, which is switched by evaluating the predicted road curb. If the predicted road curb based on the motion of the vehicle is evaluated to be the right curb, then a simple update of the road curb is sufficient. Otherwise, the road curb should be obtained through extraction.
The main contributions of this work are threefold. First, a multi-feature, loose-threshold, varied-scope ground segmentation method is presented to increase the robustness of ground segmentation. Second, a road curb extraction method based on the analysis of the road shape information, which consists of road trend information and road width distribution information, is proposed. Third, a framework for road curb detection is proposed based on the multi-index evaluation of the road curb in three aspects, namely, the shape of the road curb, the fit degree between the road curb and the scene, and the variance between the curve and history information.
The succeeding portions of this paper are organized as follows: The segmentation of the ground is described in Section 2
. The road curb detection method is discussed in Section 3
. Experiment and results are introduced in Section 4
. Section 5
provides our conclusions on this method and discusses future work that needs to be completed.
3. Road Curb Detection
The height of curbs and distance between curbs and the sensor make it difficult to detect the curbs. However, the application of road trend can be helpful in the detection process.
Two typical scenes are shown in Figure 10
to illustrate the difficulties in road curb detection. The green box in the left image shows that a car is parking on the roadside, which would be detected as a road curb if only the local information is applied. The green box in the right image shows that a gap exists between two segments of the road curb, and this gap would cause the leak detection of the road curb if only the local information is applied.
However, it is not difficult to learn the road’s overall shape information despite the existence of the obstacle inside the road area and the interval of the curb. The road’s overall shape information can be reflected by the road centerline, and the road width information. Besides, the real road width can be obtained by the road width distribution.
A road curb detection method based on the road’s overall shape information is presented in this section, and its flow diagram is shown in Figure 11
.The method can be divided into two parts: road shape evaluation and curb detection and update.
3.1. Road Shape Evaluation
In this part, a two-step, centerline extraction and road width estimation, method is proposed to estimate the road shape.
3.1.1. Centerline Extraction
After the ground segmentation, a grid map that only contains two values is created, obstacles are marked as 0 and other grids are marked as 255. Each grid cell of the map represents 20 cm. Then a distance transformation, which uses Algorithm 1, is conducted and the grid intensity increases with the distance between the grid and an obstacle grid. An example of the distance transformation performed on the obstacle map is shown in Figure 12
. As shown in Figure 12
c, the intensities of the points in the road centerline is the local maximum because the distance between the points in the road centerline and its nearest obstacle point is the maximum compared with its local neighboring points. Moreover, the road centerline reflects the trend of the road.
|Algorithm 1. Distance Map Generation Process|
|Distance Transform Algorithm|
|for i← 1 to MAP_HEIGHT − 2|
| for j← 1 to MAP_WITH − 2|
| DistanceMap [i + 1][j] = min(ObstacleMap [i][j] + 1, ObstacleMap [i + 1][j])|
| DistanceMap [i + 1][j + 1] = min(ObstacleMap [i][j] + 1, ObstacleMap [i + 1][j + 1])|
| DistanceMap [i][j + 1] = min(ObstacleMap [i][j] + 1, ObstacleMap [i][j + 1])|
| DistanceMap [i − 1][j + 1] = min(ObstacleMap [i][j] + 1, ObstacleMap [i − 1][j + 1])|
| end for|
|for i← MAP_HEIGHT − 2 to 1|
| for j← MAP_WITH − 2 to 1|
| DistanceMap [i − 1][j] = min(ObstacleMap [i][j] + 1, ObstacleMap [i − 1][j])|
| DistanceMap [i − 1][j − 1] = min(ObstacleMap [i][j] + 1, ObstacleMap [i − 1][j − 1])|
| DistanceMap [i][j − 1] = min(ObstacleMap [i][j] + 1, ObstacleMap [i][j − 1])|
| DistanceMap [i + 1][j − 1] = min(ObstacleMap [i][j] + 1, ObstacleMap [i + 1][j − 1])|
| end for|
In most cases, the Lidar is not found in the road’s center position. Therefore, a seed point should be obtained to generate the entire road centerline. As shown in Figure 12
c, the intensity in the distance map varies continuously, which means that the gradient information can be applied to find the nearest local maximum point. Search from the vehicle’s position to find the local maximum point in 5 × 5 model. The point is set as the seed point if it is the local maximum point in the 5 × 5 model centered on itself. Otherwise, the search is continued until a point is the local maximum point in the 5 × 5 model centered on itself. The result of obtaining the nearest local maximum point is shown in Figure 13
, where the upper right box is the position of the Lidar, whereas the lower left box is the position of the nearest local maximum point.
Starting from the nearest local maximum position, a search is performed on the distance map to find the local maximum points in the upward and downward directions. This step is followed by quadratic curve fitting and the result is in the form of (5).
a shows the extraction of the centerline in a winding road situation. The third column shows that the extracted centerline reflects the road trend. A situation where the distance map is affected by the obstacle inside the road area is shown in Figure 14
b. It can be seen that the extracted centerline is not in the center position of the road area. The centerline should satisfy two requirements. First, the line should reflect the road trend generally. Second, the line should be found inside the road area. Our experiment found that these two requirements are relatively easy to meet if a relatively large model is selected to lead to the detection of a long centerline, for example a 10 × 10 model. A longer centerline detected inside the road area means a more accurate quadratic curve fit for the road shape. As shown in the green box in Figure 14
b, the road centerline is continuous in the distance map despite the existence of the obstacle inside the road area. Thus, a long centerline inside the road area can be extracted.
3.1.2. Road Width Estimation
To increase the continuity of the obstacle map, instead of using a normally morphological close operation on the obstacle map, a threshold segmentation, which is less time consuming, is performed on the distance map. Pixels less than the threshold are set as 255 and the rest pixels are set as 0. The result is shown in Figure 15
. Figure 15
c shows that the road curb points are more continuous than the original obstacle map. Moreover, the road curb is smoother than the original obstacle map, and the high-frequency noise caused by the uneven sample is reduced.
Based on the aforementioned analysis, road trend and a more continuous obstacle map are obtained. Points in the road centerline, whose y value is in the interval of [STARTY, ENDY], are used as seed points and a search is performed from a seed point in left and right directions until an obstacle point is met. In this manner, the distances of the point to the left and the right nearest obstacles can be obtained. By summing these two distances up for each point in the road centerline, the road width distribution can be obtained. Based on statistics, the most likely width of the road can be obtained. The detailed algorithm is shown in Algorithm 2.
|Algorithm 2. Process for the Analysis of Width Distribution|
|For y← STARTY to ENDY|
| LeftWidth← 0|
| While ThreSegImg[y][x − LeftWidth] == 0|
| LeftWidth ++|
| End While|
| RightWidth← 0|
| While ThreSegImg[y][x + RightWidth] == 0 |
| RightWidth ++|
| End While |
| RoadWidth[y]← LeftWidth + RightWidth |
| Save the Point (x, y) to WidthPoints[RoadWidth[y]/10]|
| RoadWidthCount[RoadWidth[y]/10] ++|
| If(RoadWidthCount[RoadWidth[y]/10] > maxWidthCount)|
| maxWidthCount← RoadWidthCount[RoadWidth[y]/10]|
| MaxRoadWidthPortion = RoadWidth[y]/10;|
| End if|
In Algorithm 2, a search is performed between STRARTY and ENDY to find obstacle points of the road edge on both sides of the centerline. All the width values are assigned to partitions with a difference of 2 m. Thus, the points corresponding to the most possible width, i.e., points in the largest partition, can be selected as the seed points.
Two typical scenes are shown in Figure 16
, where the gray area in the third column shows the width distribution of the road. A winding road is shown in the first row. In spite of the curved shape of the road, the width distribution of the road is continuous. An obstacle inside the road area situation is shown in the second row. The third column shows that the width distribution is affected by two vehicles. However, based on the maximum possible width analysis, the true road width can be obtained. In this manner, the vehicle inside the road is not regarded as the road curb.
3.2. Curb Detection and Update
The seed points are applied to detect the entire road curb points by region growing in the original obstacle map. The candidate road curb points should satisfy the following criterion: the gradient in the x
-direction should be negative for the left curb candidate road curb points while positive for the right curb candidate road curb points. Thereafter, a quadratic curve fitting is performed on the candidate points. The result is shown in Figure 17
As shown in the white box in the third column of the first row, a gap exits in the road curb. However, based on the width distribution information, seed points in the two sides of the road curb are obtained, and they are grown to obtain the road curb candidate points in the two sides of the gap. The white boxes in the third column of the first row show two vehicles. The partially enlarged detail map shows that the curb points between the upper vehicles are detected, whereas the curb points between the lower vehicles are leak-detected. This difference is caused by the difference in the gaps between the vehicles, such that the curb points in the left side of the lower vehicle are not regarded as the road curb points. However, this difference does not affect the extraction of the road curb because most road curb points are detected and the shape of the road curb is reflected.
Then, a multi-index evaluation of road curb in three aspects, including the shape of the road curb, the fit degree between the road curb and the scene, and the variance between the curve and history information, is presented in this study. These aspects are described in detail in the subsequent subsections.
3.2.1. Shape of the Road Curb
The shape of the road curb can be partially reflected by its curvature, which can be denoted by the coefficient of the fitted quadratic curve. The road trend changes relatively slowly in a local frame grabbed by Velodyne LIDAR. Thus, the coefficient of the quadratic curve should not exceed a certain threshold.
3.2.2. Fit Degree between the Road Curb and the Scene
The fit degree between the road curb and the scene reflects the property how the detected curve reflects the true shape of the road. This aspect is denoted by three indices in this study.
The number of the detected curb points.
The maximum difference of the y value between the detected points.
The variance of the width distribution
Our experiment found that, if the number of the detected road curb points is excessively small, the points cannot reflect the shape of the road curb. Moreover, the more divergent these points are, the more accurate these points will reflect the road shape. The dispersion of the curb points can be reflected by their maximum difference of the y value. As the road is continuous, the road width distributions along the y-direction should be similar. Thus, the variance of the width distribution should be small.
3.2.3. Variance between the Curve and History Information
During driving, the road curb should be continuous in most cases. Thus, the curb shape between two consecutive frames should not change significantly. The consecutiveness of the road curb is denoted by the difference of the road width between two consecutive frames. The difference should not exceed a certain threshold, which is defined and tested with a lot of experiments conducted on our autonomous vehicle.
In our experiments, the position of the Lidar is (250, 500) in the grid map. Each cell represents 20 cm. As the obstacle information is more detailed near the Lidar, STARTY is set as 100 and ENDY is set as 400 in our experiment. The thresholds selected for the aforementioned indices are shown in Table 2
The prediction and update of the road curb are described in detail in [24
], which predict the initial position of the road curb of this frame based on the motion of the vehicle from the previous curb position and the update of the height difference map generated by this frame data using SNAKE Model.
4. Experimental Results and Discussion
Our experimental platform is the autonomous vehicle named “Intellgent Pioneer”, as shown in Figure 2
. Besides three-dimensional Lidar, it is equipped with a SPAN-CPT, an integrated navigation device combining inertial navigation and Global Positioning System, which provides accurate position and moving state of the intelligent vehicle.
To validate the robustness and accuracy of the ground segmentation algorithm presented in this work, experiments under various environments are conducted including straight road, winding road and crossroad in urban area and off-road area. Figure 18
a shows the heavy traffic situation. The existence of the obstacle influences the local geometric characteristics of the tangential neighboring points on the ground, which would be falsely detected based on some features alone. The four features are fused in different scopes and the ground segmentation is completely without over segmentation. Figure 18
b shows ground segmentation with a low irregular obstacle inside the road area. As shown in the real scene picture, three rows of cones are in front of the intelligent vehicle, which were low and difficult to detect. The detection of a negative obstacle is shown in Figure 18
c, where two pits exist in the scene. A round pit and a rectangular pit are marked out in the red box in the second row. As shown in the figure, with the ground cleanly segmented, the points projected on the round pit are mostly detected as obstacles, and several points projected onto the rectangular pit are detected as obstacles.
The comparison between the proposed ground segmentation method and that of [23
] is presented in Figure 19
, where a low stone curb situation is shown. The result of the ground segmentation algorithm presented in [23
] is shown in Figure 19
b, whereas the result of this study is shown in Figure 19
c. According to the gradient feature presented in [23
], although the curb points near the vehicle have been detected continuously, the curb points distant from the vehicle were leak-detected. As mentioned previously, when the curb points detected are more divergent, the curb shape is reflected to be more accurate. Figure 19
c shows that the curb detected by the algorithm presented in this study covered a broader scope despite the existence of gaps, which shows more information on the road trend.
The comparison between the ground segmentation of this study and that of [24
] is presented in Figure 20
, which contains the real scene image, yaw point cloud, result obtained by the algorithm presented in [24
], and result obtained by the algorithm presented in this study. The experiment found that the distance based features are insensitive to the influence exerted by the distant obstacles on the local geometric characteristic. Whereas, the angle based features are insensitive to the influence exerted by the nearby obstacles. Global thresholds are applied in [24
] to segment the ground. Therefore, leak detection is inevitable as shown in the red box in Figure 20
c. As an in-depth analysis is performed on the threshold and the fitted scope of each feature applied in this study, leak detection is reduced compared with the algorithm presented in [24
], as shown in Figure 20
The result of road curb detection in the three scenes are shown in Figure 21
, including the yaw point cloud, obstacle map, and detected road curb, respectively. A common difficulty for road curb detection in these scenes is the existence of the obstacle inside the road area, which may influence the detection of the road curb.
a shows an urban scene where the road width varied and the number of vehicles inside the road generated interference on road curb detection. Based on the analysis of the road’s overall trend information, the vehicles inside the road were identified as dynamic obstacles instead of the road curb. Moreover, the existence of the dynamic obstacles inside the road area caused the leak detection of the low curb, as shown in the red box in the second column, which did not affect the detection of the road curb. Figure 21
b shows the influence of snowy weather on road curb detection. As shown in the first column, the noise points on the road are generated by the snow. However, the road curb was correctly detected without being influenced by these noise points. A field scene is shown in Figure 21
c, where the coarse ground in the left of the vehicle was detected as an obstacle. Moreover, the curb is rough. The detection results shown in the third column shows that the curb was correctly detected.
The comparison between the results of road detection in [24
] and that in this study is shown in Figure 22
. The result of [24
] is shown in Figure 22
b and the obstacle inside the road was detected as a road curb as the lack of the road’s overall information. The result of this study is shown in Figure 22
c. As the road’s overall information was applied in this work, the obstacle inside the road was not designated as the road curb.
The results of an experiment measuring the time needed for road curb detection are shown in Figure 23
where the blue line shows the time consumption based on detection solely and the yellow line shows the time consumption based on the framework presented in this study. The average time of the detection solely method is 112 ms and the peak value is larger than 900 ms as a result of the limit of the multi-threaded framework used in the program. The proposed framework is much faster, the average time is 11 ms, and the value is stable.
To evaluate the accuracy of the proposed curb detection method, a comparison between the detection result and the road curb in a high precision map is conducted. The testing path is shown in Figure 24
with red lines. The blue lines are lane marks in the map. The path is about 9.2 km including 4.9 km straight road and 4.3 km winding road.
The curbs in high precision map can be treated as ground truth whose error is less than 10 cm. To reduce the impact of positioning errors RTK differential positioning system is used and the positioning error is less than 5 cm. Distance between the detected curbs and the ground truth can be worked out in the local coordinate of the intelligent vehicle. The intelligent vehicle is moving along the road and the horizontal distance between a detection curb point and the ground truth point with the same vertical coordinate can be obtained as the curb detection error. The accuracy of the proposed method and method of [24
] is shown in Table 3
. The average detection error of our method is smaller than method of [24
] on both straight road and winding road. Besides, the accuracy difference of the proposed method between on straight road and on winding road is smaller, which demonstrates a better robustness.
Two typical scenes on straight road and winding road are shown in Figure 25
. Red lines are road curbs in high precision map and blue lines are detected road curbs. The proposed method is robust to obstacles on road and discontinuity of road curbs which are shown in red boxes. The detected results are consistent with the ground truth.
This paper introduced a quick and accurate method for online road curb detection using multi-beam Lidar. In the process of ground segmentation, a multi-feature, loose threshold, varied-scope method is used to increase the robustness. Then, an extraction-update mechanism based road curb detection method is proposed and the overall road shape information is used in the extraction process.
Experimental results show that the proposed ground segmentation method performs much better than the method in [23
] and the method in [24
]. The robustness is proved by applying the method in different scenes including congested viaduct, urban road with obstacles at night and road with negative obstacles. Besides, the road curb detection method is 10 times faster than the detection solely method and the detection speed is stable. The accuracy is also evaluated and it is sufficient for autonomous driving.
Our future work will focus on improving the accuracy and rationality of thresholds used in ground segmentation and maybe an adaptive method will be adopted according to the road roughness. Another research direction is the fusion of data from Lidar and camera to increase the robustness and accuracy of curb detection.