Multi-stage Hough Space Calculation for Lane Mark Detection via IMU and Vision Data Fusion

It’s challenging to achieve robust lane detection depending on single frame when considering complicated scenarios. In order to detect more credible lane markings by using sequential frames, a novel approach to fusing vision and Inertial Measurement Unit (IMU) is proposed in this paper. The hough space is employed as the space where lane markings are stored and it’s calculated by three steps. Firstly, a basic hough space is extracted by Hough Transform and primary line segments are extracted from it. In order to measure the possibility about line segments belong to lane markings, a CNNs based classifier is introduced to transform the basic hough space into a probabilistic space by using the networks outputs. However, this probabilistic hough space based on single frame is easily disturbed. In the third step, a filtering process is employed to smooth the probabilistic hough space by using sequential information. Pose information provided by IMU is applied to align hough spaces extracted at different times to each other. The final hough space is used to eliminate line segments with low possibility and output those with high confidence as the result. Experiments demonstrate that the proposed approach has achieved a good performance.

The final filtered probabilistic hough space is used to extract the final line segments. Line segments 62 with low possibility will be eliminated and those with high values will be kept and tracked in the 63 proposed probabilistic hough space. 64 Related works will be introduced in Section 2. In Section 3, we describe the construction of the 65 primary probabilistic hough space depending on single frame. In Section 4, the primary probabilistic 66 hough space is filtered across frames by the fusion of IMU and vision data. Finally, detailed experiments 67 are discussed in this paper. Fig.1 shows the workflow of the proposed method.   Workflow of the proposed approach: Hough Transform and Classification networks are used to extract the primary probabilistic hough space. Kalmen filtering is introduced to smooth the probabilistic hough space across frames, where sequential information is employed. Movement information provided by IMU is applied to make the previous line segemnts aligned in the current vehicle coordinate system. Lane structure is a higher-level feature than edge. Hough Transform is a classical and robust 86 approach to extract line segments from image. In order to purify these extracted line segments,  Convolutional neural networks free us from designing handicraft features and rules, which 95 have achieved state-of-art performance in many data sets. In [10], a multi-task network named 96 VPG-net is proposed where multi-task training is proved that can improve the network performance.  networks(GANs) are also studied in this field, for example, EL-GAN [13] uses a Generative adversarial 106 networks(GANs) and embedding loss to train an end-to-end network.

Single Frame: Primary Probabilistic Hough Space via Lane Markings Extraction
In this section, a primary probabilistic hough space is constructed by the line segments extraction 109 and classification. Firstly, a combination of Hough Transform and RANSAC algorithm is employed 110 to extract line segments efficiently. Then, the proposed CNNs networks is used to classify these line 111 segments and construct the primary probabilistic hough space by using the output confidence for each 112 line segment. An efficient Hough Transform [16]is used in this paper. Actually, traditional Hough Transform would bring much extra computation for its large voting range of direction which usually ranges from 0 to 360 degrees. Edge direction is employed to limit the voting range of direction. Defining the edge direction as φ and setting H(ρ, θ) as the Hough space, θ is limited by the right part of equation (1) in this paper.
This approach can make the extraction of line segments more efficient and reduce noise at the same  After line segments extraction, a post process is necessary to eliminate false line segments such as 123 those lie on rails or trucks. In this paper, we propose a novel probabilistic hough space to measure  Table 1 shows the structure of this network. Input image of this network is provided by each line 130 segment. The diagonal points of these input images will be calculated according to each line segments.
Line segment l has different position at different times in Hough Space because of the movement of 157 vehicle, and it's necessary for kalmen filtering to obtain its observed value y from sets of probabilistic 158 hough spaces which extracted at different times. So an alignment of l t−1 (ρ t−1 , θ t−1 ) and l t (ρ t , θ t ) 159 should be solved in the Hough Space( Figure .9).

160
The filtered probabilistic hough space describes the probability from sequence consistency 161 perspective about whether a line segments is belong to traffic lane markings or not and that is better 162 and more robust than the primary probabilistic hough space. The result of this smooth process by 163 using sequential information is showed by fig.7.
ρ=c * cos(θ) + r * sin(θ) However, precision alignment is hard to achieve due to some factors such as the noise of IMU and the error of perspective mapping. So we regard all the ( θ, ρ) as projection of l t−1 (ρ t−1 , θ t−1 ) be solved with more sequential information. It plays a role similar to curvature tracking in many 173 other works, but has more specific history information for decision. Equation (7) is employed to align 174 previous results in current coordinate system, the final result is displayed by fig.10. These kind of 175 lane-map will provide more global clues than single frame, which make the detection more stable.

176
In order to give the final outputs, a region-growth algorithm is used to divide these foreground points into different lane instances and a parabolic model is used to fit each lane in current vehicle coordinate. Fig.10 shows the whole process of this part. In order to limit the risk of over-fitting, L2 norm is added into our loss function displayed by equation (10). In equation (10), α 1 and α 2 are tradeoff coefficients. Figure 10. Local lane-map is constructed by connecting those recorded results from t-n to t in the same vehicle coordinate. It makes the final output more stable by providing useful information for the fitting stage in a larger spatial and time scale than single frame.

177
In order to perform detailed experiments, we create our own dataset include images and IMU To evaluate our algorithm, we choose four parts of road data to test the performance of our method, which contain rainy and sunlight conditions, and 667 pictures are annotated( fig.11). Those annotated pictures which size is [940,1824] are labeled in the form of line segments like what fig.11 displays. Standard to decide whether a line segment is valid or not is showed by equation (11), where we define Er is the total offset between the detected line segments {(x i , y i )} and the ground truth {(x i ,ŷ i )} and n represents the length of the labeled line segments.
If Er is smaller than T, then we regard this detected line segment as valid detection. In this paper, T is 182 set to 80.
183 Figure 11. Ground truth is labeled in the form of line segments   204 A comparison between the performance of the primary probabilistic hough space and the filtered 205 space by Kalmen Filtering is displayed by Fig. 14. It's easy to see the accuracy of line segments 206 classification has been enhanced by using sequential information. We test the proposed method on 207 four labeled datasets with the measurement metric of accuracy(ACC).   The performance of the proposed approach in this paper is compared with Neven's method [11] 216 by using the metrics described in equation.11. It can be seen from table 4 that the proposed method in 217 this paper perform better than Neven's method sometimes, especially, we have a lower false positive 218 rate than their method all the time due to the use of sequential information. However, both of Neven's 219 method and our method perform not very well on the metrics of TPR, it's hard to detect those in the 220 distance.

225
In this paper, a multi-stage hough transform is proposed for our lane detection task by fusing information to improve classification accuracy. More developments will be studied to improve the 237 performance of our algorithm in the future work.