Robust Object Segmentation Using a Multi-Layer Laser Scanner

The major problem in an advanced driver assistance system (ADAS) is the proper use of sensor measurements and recognition of the surrounding environment. To this end, there are several types of sensors to consider, one of which is the laser scanner. In this paper, we propose a method to segment the measurement of the surrounding environment as obtained by a multi-layer laser scanner. In the segmentation, a full set of measurements is decomposed into several segments, each representing a single object. Sometimes a ghost is detected due to the ground or fog, and the ghost has to be eliminated to ensure the stability of the system. The proposed method is implemented on a real vehicle, and its performance is tested in a real-world environment. The experiments show that the proposed method demonstrates good performance in many real-life situations.


Introduction
With the recent developments in vehicular technology, advanced driver assistance system (ADAS) concept has spread rapidly; however, many problems still remain to be addressed before the field of ADASs can be widely expanded. The biggest problem in ADAS is the use of sensor measurements and recognition of the surrounding environment. To this end, several types of sensors have been considered, including radar and a visual or infrared (IR) camera. Unfortunately, however, none of the sensors are sufficient for ADAS, and each has its own shortcomings.
For example, radar returns relatively accurate distances to obstacles, but its bearing measurements are not accurate. Radar cannot recognize object classes, and also suffers from frequent false detections [1][2][3][4]. A visual camera is another ADAS tool. This type of camera returns a relatively accurate bearing to the obstacle, but its distance measurement is not reliable. The camera is capable of object recognition, but it also exhibits a high false detection rate. Thus, most of the current systems combine several sensors to compensate for the drawbacks of each sensor and to obtain reliable information about the nearby environments [5][6][7][8][9].
Recently, the laser scanner has received attention within the ADAS community, and it is considered to be a strong candidate for the primary sensor in ADAS [1,10]. The strong points of the laser scanner are its ability to accurately determine both near and far distances as well as the bearing to an obstacle. In addition, the object detection of a laser scanner is reliable and robust, and it can recognize object classes to some extent through determination of the contour of the surrounding environment.
Thus, unlike a camera or radar, a laser scanner can be used as the sole sensor for ADAS without being combined with other sensors. Further, if the laser scanner is combined with other sensors, it can compensate for all the drawbacks, thereby improving the recognition accuracy.
Because the range and bearing measurements of the laser scanner are sufficiently accurate, in this paper, the laser scanner is considered as a single sensor for ADAS. In this paper, the laser scanners are divided into three types according to the number of layers; single-layer, multi-layer, and three-dimensional (3D) laser scanners. A single-layer laser scanner consists of only one layer, a multi-layer laser scanner is composed more than one but fewer than eight layers, and a 3D laser scanner is composed of eight or more layers. A single-layer laser scanner can obtain 2-dimensional (2D) information, a 3D laser scanner can get 3D information, and a multi-layer laser scanner can get limited 3D information. In general, the information from a laser scanner is proportional to the number of layers, but a 3D layer scanner is expensive and difficult to install on the vehicle. However, single-layer and multi-layer laser scanners can be implemented inside a vehicle's body [11]. Therefore, the 3D layer scanner is not yet suitable for ADAS, and the multi-layer laser scanner is currently more suitable for ADAS.
The remainder of this paper is organized as follows: in Section 2, an outline of the obstacle recognition system using a laser scanner is described. Related works are described in Section 3. The segmentation for a multi-layer laser scanner and the ghost elimination are explained in Section 4. The proposed system is installed on a vehicle and is applied to actual urban road navigation. The experimental results are presented in Section 5. The discussion about robustness is described in Section 6. Finally, some conclusions are drawn in Section 7.

System Outline
The system aims to detect obstacles through the processes of segmentation, classification, and tracking. Figure 1 shows the outline of the system developed in this paper. In the segmentation step, the proposed system receives a full scan (a set) of measurement points from a multi-layer laser scanner and decomposes the set into several segments, each of which corresponds to an object. In the segmentation step, the outliers are removed to avoid performance degradation. In the classification step, the segment features are computed and classified [12][13][14]. In the tracking step, the location and velocity of the segment are estimated over time. Segmentation is the essential step to execute classification or tracking [15][16][17][18]. In this paper, we focus on the methods for segmentation and outlier elimination.

Related Works
A laser scanner detects the closest obstacle for a bearing angle and returns the angle-wise distance to the obstacle. The output of the laser scanner can be modeled by the following set of pairs: where N denotes the number of scanner measurement points as shown in Figure 2; the superscript t denotes the measurement time; and t i r and t i θ denote the distance and bearing to the obstacles, respectively. Equation (3) refers to a property of the laser scanner, and it implies that the scanner scans the environment from left to right. As stated in Section 2, segmentation is the first step for the object detection by the laser scanner. A scan of the measurements, as given in Equations (1) and (2), is decomposed into several groups called segments, as shown in Figure 2.
In general, the segmentation methods can be classified into two kinds: the geometric shape method and the breakpoint detection (BD) method. The first method assumes the geometric shapes of the segments, such as a line or an edge, and decomposes the scanner measurements into the predetermined shapes [19,20]. The BD method decomposes the scanner measurements into segments based on the Euclidean distance between the consecutive points t i p and 1 t i p − [21] or using a Kalman filter [22]. Methods using BD based on the distances between consecutive points are the most popular and are widely used for laser segmentation [23][24][25][26][27]. In the distance-based BD, if the distance between two consecutive points is greater than a threshold thd D , two points are likely to belong to different objects and a breakpoint is selected between the two points as shown in Figure 2a.
In Figure 2, the laser scanner returns 12 data points ( 1 t p -12 t p ) and the segment n S originates from the n th object ( 1, 2,3 n = ). The performance of the BD segmentation depends on the choice of a threshold thd D , and several methods have been developed for the selection of thd D . In [23], the threshold thd D is determined by: Borges et al., recently proposed the adaptive breakpoint detector (ABD) in [25]. In the ABD, the threshold thd D is determined by: which adaptively depends on 1 t i r − and θ Δ as shown in Figure 3. In Equation (6), λ is chosen on the basis of user experience, and r σ is the sensor noise associated with r .

Figure 3. Adaptive Breakpoint Detector (ABD).
All of the previous segmentation methods, however, were based on a single-layer laser scanner, and to our knowledge, no research has been reported regarding the segmentation for a multi-layer laser scanner. This is one of the contributions of this paper.

ABD Segmentation for Multi-Layer Laser Scanner
A multi-layer laser scanner has multiple layers and returns the measurement points as shown in Figure 4.  Figure 4a shows the laser scanner measurements depicted on the x -y plane, and Figure 4b shows the scanner measurements superimposed on the camera image after calibration [28]. In the figure, the information of the different layers is represented by different colors. In the multi-layer laser, each data point t i p is not a pair but a triplet consisting of the distance t i r and bearing t i θ to the obstacle and the layer information t i l . The output of the multi-layer laser scanner is modeled by: where a four layer laser scanner, the IBEO LUX2010 [29], is used, and: Equations (8) and (9) refer to the property that the scanner scans the laser from left to right and from bottom to top, respectively. The left points are measured before the right points and at the same t i θ , and the lower layers are measured before the upper layers. The direct application of a standard ABD to multi-layer segmentation would lead to the loss of layer information and would lead to inefficient segmentation.
In order to develop a new segmentation method for the multi-layer scanner, two important properties of the multi-layer laser scanner should be considered: (1) Two points at the same bearing but on different layers can belong to different objects. An example is the situation given in Figure 4. In the figure, both a sedan and a bus lie in the same bearing, but the sedan is closer to the scanner than the bus. From the box bounded in red dotted lines in Figure 4b, the points in the lower three layers belong to the sedan, but the points in the top layer belong to the bus. Thus, it must be determined whether or not two data points, t i p and t j p , with consecutive bearings and consecutive layers belong to the same object. (2) The measurement sets are not complete, and there are many vacancies in the θl plane. When the scanner input is plotted on the θl plane, the ideal output will look like a grid as in Figure  5a, but the actual output appears as in Figure 5b. Thus, the grid-type segmentation using a nested for-loop cannot be used. A layer-wise independent segmentation process can be considered; however, in our experience, this method does not work well and requires multi-layer segmentation. In multi-layer laser segmentation, we will say that two points t i p and t j p are connected if they belong to the same object. Unlike single-layer scanner segmentation, we consider not only the connectivity between the points with consecutive bearings, but also the connectivity between the points with consecutive layers. The foremost requirement in multi-layer segmentation is that the algorithm should operate in a single scan with the running time ( ) O N , and the algorithm must not trace back to the old previous points, making it 2 ( ) O N or higher, where N is the number of measurement points. In this paper, an ( ) O N fast segmentation method is presented. When each data point t i p is given, a candidate set i  composed of previous data points t j p ( j i < ) is built, and the connectivity of the point t i p is tested only with the elements in i  , thereby implementing an ( ) O N implementation. In our segmentation method, the candidate set i  consists of the newest data points in each layer. Therefore, the maximum size of the candidate set i  is four in this case. Figure 6 illustrates the ABD segmentation process.
in Figure 6d, where ∪ and − denote set union and subtraction, respectively. As before, the connectivity of  Table 1 shows the proposed segmentation algorithm for the multi-layer laser scanner. In the table, Nseg denotes the number of segments, and N and L denote the number of data points and the number of layers, respectively. 1 S and 1  are initialized with empty sets, and the segmentation proceeds from

Robust Segmentation through Ghost Elimination
When the above ABD segmentation is applied to actual roads, ghost segments are sometimes detected. Here, a ghost segment refers to a segment that does not actually exist but that is detected by the laser scanner. Figures 7 and 8 show examples of ghost segments. These ghosts pose a serious risk to safe driving. Most ghost segments are caused by (1) laser reflections from the ground surface; or (2) lights from vehicles or fog. The false determination of a ghost segment seriously degrades the subsequent object classification performance.    Ghosts that are caused by reflections from the ground surface often occur when vehicles travel over bumpy roads or when they go up-or downhill. Ghost segments are detected on only one layer, usually the first layer, within a 40-m distance of the scanner, and experience rapid change, appearing and disappearing and changing shape. Ghosts that are caused by headlights or tail lights or by nearby fog are also detected on only a single layer within a 20-m distance of the scanner, do not have a uniform shape, and are detected intermittently.
The two kinds of ghosts both exist only on a single layer and are detected within a short distance. Thus, our robust segmentation method is developed by considering the first property and applying it within a limited 40-m distance around the vehicle. For distances greater than 40 m, the ABD segmentation method explained in Section 4.1 is used.
The robust segmentation method is similar to the ABD segmentation given in Section 4.1, with two main differences. The first difference is that when a point, t i p , is presented in robust segmentation, the point is not segmented with a point on the same layer in i  . The reason for this is that ghost points tend to gather only on a single layer. By not combining a new point with the points on the same layer, ghost points do not build a meaningful segment and thus are not considered. The second difference is that the candidate set i  consists of not one but two of the newest points in each other layer. To prevent an object from being divided due to a ghost, we determine the connectivity of up to two of the newest points in each other layer.
Thus, i  in robust segmentation can have up to eight ( 2 4 × ) elements. Figure 9 illustrates the robust segmentation process. When a new point, 9 t p , is presented, as in Figure 9a  The problem with this approach is that if an obstacle is very shallow and is detected only on a single layer, it may not be detected. However, this is rarely the case due to the sufficiently small angular resolution of the multi-layer laser scanner. In the case of the IBEO LUX2010, the vertical and horizontal angular resolutions are 0.8° and 0.125°, respectively, allowing an obstacle 20 m or more from the vehicle and larger than 0.56 m to be detected on more than two layers. If the obstacle is larger than 0.26 m, it will return more than six points. Table 2 shows the pseudo-code of the robust segmentation. In lines 4-14, when a new point, t i p , is within 40 m of the scanner, the connectivity determination with the point on the same layer is skipped, as Figure 9. In lines 15-24, when a new point is far from the scanner, its connectivity with the point on the same layer is determined as the ABD segmentation. The processes of making new segment and updating candidate set, 1 i+  , is same as the processes of ABD segmentation. End of this algorithm, small segments are eliminated. The small segments mean the number of point is smaller than min N , and min N denotes the minimum number of points required for an object.

Experiment
In this experiment, an IBEO LUX2010 multi-layer laser scanner and a camera are installed on a Kia K900 as shown in Figure 10. As previously stated, the LUX2010 has a total of four layers, and its horizontal and vertical resolutions are 0.125° and 0.8°, respectively. The camera is used to obtain the ground truth of the environment.  Figure 11 shows the segmentation results for six different scenarios. The first column shows the raw measurements from the IBEO scanner, and the second and third columns show the ABD and robust segmentation results, respectively. The fourth column contains the corresponding camera image with the scanner measurements superimposed. Figure 11a and b show the results when the road is flat and ghosts are not detected. Only vehicles appear in Figure 11a, while both vehicles and pedestrians appear in Figure 11b. In the two scenarios, ghosts are not observed, and it can be seen that the ABD and robust segmentations produce the same results. Figure 11c and d show the results when a ghost is detected that is created by the surface. In the figures, the dots in the red box are the ghost, detected by the bottom layer laser, which is indicated in blue. When the ABD segmentation method is applied (second column), the ghost forms an outlier segment and appears to be an obstacle. When the robust segmentation method is applied (third column), however, the ghost is successfully removed, leaving only the segments from the preceding vehicles. Figure 11e and f show the results in rainy, foggy test conditions. As in Figure 11c and d, the dots in the red box are detected by the second-layer laser, and appear to result from fog. When the ABD segmentation method is applied (second column), the ghost survives and could activate the brake system, which can lead to an accident. When the robust segmentation method is applied (third column), however, the ghost is successfully removed. For quantitative analysis, we gather samples from four scenarios as in Table 3 and apply the ABD and robust segmentation methods.  All the samples are clipped manually from the IBEO scan. In Tables 4-7, the results of the ghost elimination are described. The value of λ in Equation (6)    In the tables, the ABD and robust segmentation methods are compared in terms of (1) ghost elimination ratio; (2) inlier survival ratio and (3) computation time. Here, the ghost elimination ratio and the inlier survival ratio are defined as: The number of eliminated ghosts Ghost elimination ratio = 100 The number of ghosts × The number of survived inliers Inlier survival ratio = 100 The number of inliers × From the tables, the proposed robust method outperforms the ABD in all cases with the similar computation time.

Discussion
Obviously, the goal is to remove as many ghosts as possible while maintaining as many inliers as possible and, thus to keep both ratios high. It can be seen from Tables 4 to 7 that the results of robust segmentation are better than those of ABD segmentation in every condition. In particular, the proposed robust method demonstrates more than 95% of the ghost elimination ratio in a robust manner regardless of the weather or the road. The ABD, however, demonstrates 17% to 65% of the ghost elimination ratio depending on the weather or the road. When it rains or the car goes uphill and, thus, ghosts frequently occur, ABD fails in eliminating the ghosts but the robust method removes most of the ghosts well. Interestingly, the ABD also performs well in the foggy weather and the reason is that the ghosts are detected intermittently in the foggy weather and they tend not to form a segment. Further, the ghost elimination ratio is not much affected by the value of λ . The reason might be that the ghosts are very close to the sensors and they are far enough from the other obstacles.
The inlier survival ratio is also an important factor because if the inlier is accidently removed by the algorithm, it will lead to a serious accident. The result of the inlier survival ratio is also shown in Tables 3-6. It can be seen that both of the segmentation methods have sufficiently high inlier survival ratios and the both algorithms do not accidently remove the important measurement points.
The ABD and robust segmentation methods are also compared in terms of computation time. The computation time in Tables 4 through 7 is obtained by computing the average over 100 frames. It can be seen that the robust method takes slightly longer time than the ABD but the extra time is not much. The reason is that the ghost tends to form a number of small segment and the elimination of them takes some time.

Conclusions
In this paper, a new object segmentation method for a multi-layer laser scanner has been proposed. For robust segmentation, efficient connectivity algorithms were developed and implemented with ( ) O N complexity. The proposed method was installed on an actual vehicle, and its performance was tested using real urban scenarios. It was demonstrated that the proposed system works well, even under complex urban road conditions.