Road Lane Detection Robust to Shadows Based on a Fuzzy System Using a Visible Light Camera Sensor

Recently, autonomous vehicles, particularly self-driving cars, have received significant attention owing to rapid advancements in sensor and computation technologies. In addition to traffic sign recognition, road lane detection is one of the most important factors used in lane departure warning systems and autonomous vehicles for maintaining the safety of semi-autonomous and fully autonomous systems. Unlike traffic signs, road lanes are easily damaged by both internal and external factors such as road quality, occlusion (traffic on the road), weather conditions, and illumination (shadows from objects such as cars, trees, and buildings). Obtaining clear road lane markings for recognition processing is a difficult challenge. Therefore, we propose a method to overcome various illumination problems, particularly severe shadows, by using fuzzy system and line segment detector algorithms to obtain better results for detecting road lanes by a visible light camera sensor. Experimental results from three open databases, Caltech dataset, Santiago Lanes dataset (SLD), and Road Marking dataset, showed that our method outperformed conventional lane detection methods.


Introduction
Detecting road lane markings is an important task in autonomous vehicles [1][2][3]. Most recent algorithms for lane detection are vision-based. Images captured from various types of cameras such as visible light camera sensors are processed to extract all meaningful feature data such as edges, lane orientation, and line boundaries, and they are combined with the distance information measured by radar sensors. A vision-based system requires camera calibration before operating, good environmental situations and road conditions, and high processing speed to detect lane boundaries in real time to match the speed of the vehicles. Therefore, most of the methods based on handcrafted features propose three main steps of processing [1,[4][5][6][7]: (1) pre-processing: enhancing illumination of the original image captured from the camera; (2) main-processing: extracting features of road lane markings such as edges, texture, and color; and (3) post-processing: removing outliers or clustering detected line segments.
Unlike traffic signs, severe shadows can exist on road lanes, and this factor leads to challenging problems for automatic recognition and classification of road lanes. For example, owing to the effect of overly bright or overly dark illuminations, a solid lane can be divided into smaller units; therefore, it can be falsely recognized as a dashed lane [6]. Therefore, we propose a method of road lane detection by using a fuzzy inference system (FIS) to overcome the effect of shadows on input images. Detailed explanations of previous approaches are provided in the Section 2.
-First, to evaluate the level of shadows in the ROI of the road image, we use two features as the inputs for FIS: hue, saturation, and value (HSV) color difference based on local background area (feature 1) and gray difference based on global background area (feature 2). Two features from different color and gray space are used for FIS to consider the characteristics of shadow in various color and gray spaces. -Second, using FIS based on these two features, we can estimate the level of shadows depending on the output of FIS after the defuzzification process. We modeled the input membership functions based on the training data of two features and maximum entropy criterion to enhance the accuracy of FIS. The procedure of intensive training which is required in training-based method such as neural network, support vector machine, and deep learning is not necessary for using FIS. -Third, by adaptively changing the parameters of the line segment detector (LSD) and CannyLines detector algorithms based on the output of FIS, more accurate line detection can be possible based on the fusion of the detection results by LSD and CannyLines detector algorithms, irrespective of severe shadows on the road image. -Previous researches did not discriminate the solid and dashed lanes in the detected road lanes although it is necessary for autonomous vehicle. However, even the solid and dashed lanes are discriminated (including the detection of starting and ending positions of dashed lanes) in the detected road lanes by our method.
In Table 1, we show the summarized comparisons of the proposed and existing methods. The remainder of this paper is organized as follows: in Section 3, our proposed system and methodology are introduced. In Section 4, the experimental setup is explained and the results are presented. Section 5 presents both our conclusions and discussions on ideas for future work.  Figure 1 depicts the overall procedure for our method. The input image is captured by the frontal-viewing camera, and has various sizes (640 × 480 pixels or 800 × 600 pixels). In order to reduce computational complexity as well as noise, ROI for lane detection is automatically defined based on the detected vanishing point from the input image only in case that the correct vanishing point is detected (see the condition of Figure 1 in Section 3.2). If it fails to detect the correct vanishing point, the predetermined ROI is empirically defined. Next, by using two input features such as HSV color difference based on local background area (feature 1) and gray difference based on global background area (feature 2), FIS outputs the level of shadow in the current selected ROI image. Based on the FIS output value, the parameters for line segment detector algorithms are changed adaptively to enhance the accuracy of line detection. Next, three steps focus on eliminating invalid line segments based on the properties of road lanes, such as angle and vanishing point, and the correct left and right boundaries of road lanes are finally detected. We detail each step in the next sections.

Detect Vanishing Point and Specify ROI
In the first step, the vanishing point is detected and the ROI where the road lane is detected is automatically defined in the input image only in case that the correct vanishing point is detected. If it is failed to detect the correct vanishing point, the ROI is empirically defined. By performing the road lane detection within the ROI instead of the whole image, various noises in the captured image by the frontal-viewing camera as shown in Figure 2, can be reduced in the procedure of lane detection. In addition, the effect of environmental conditions such as sunshine, rain, or extreme weather conditions can be lessened in the case using ROI compared to that using the whole image.
In general, the vanishing point is considered one of the most important keys to retaining a valid road lane, because road lanes are assumed to converge at one point within the captured image. As shown in Figure 2, lane markings always appear within the lower part of the image, but this depends on each camera configuration, and the input image can also include other objects (e.g., car hoods in Figure 2b-f).

Detect Vanishing Point and Specify ROI
In the first step, the vanishing point is detected and the ROI where the road lane is detected is automatically defined in the input image only in case that the correct vanishing point is detected. If it is failed to detect the correct vanishing point, the ROI is empirically defined. By performing the road lane detection within the ROI instead of the whole image, various noises in the captured image by the frontal-viewing camera as shown in Figure 2, can be reduced in the procedure of lane detection. In addition, the effect of environmental conditions such as sunshine, rain, or extreme weather conditions can be lessened in the case using ROI compared to that using the whole image.
In general, the vanishing point is considered one of the most important keys to retaining a valid road lane, because road lanes are assumed to converge at one point within the captured image. As shown in Figure 2, lane markings always appear within the lower part of the image, but this depends on each camera configuration, and the input image can also include other objects (e.g., car hoods in Figure 2b-f).  The vanishing point is detected as follows [24]: Left and right road lane markings usually appear like two sides of a trapezoid based on the perspective projection of the frontal-viewing camera. Therefore, we can assume that all left and right lane boundaries can converge at one point called the vanishing point. First, line segments are detected by algorithms called LSD [32,33] and CannyLines [34] using consistent texture orientation. Let S = {s 1 , s 2 , . . . , s k } be the set of line segments extracted from image. Each line segment s i , (i = 1, 2, . . . , k) is defined as: where (x 1i , y 1i ) and (x 2i , y 2i ) are the coordinates of the starting point and the ending point of line segment s i , respectively. θ i is the angle of line segment s i . Next, we define the length of line segment ith (len i ) as the length weight (W L ). The longer line segment represents more pixels in the same direction, as well as a higher voting weight which increases the voting score. Second, Gaussian weight is calculated in Equation (2) [24]. In the voting space image, we not only consider the intersected point between two line segments, but also its 5 × 5 neighboring points. Based on Gaussian distribution, those involved points have different values to make the lines vote more smoothly, and thus improve the accuracy of the detection of the vanishing point:

Detect Vanishing Point and Specify ROI
In the first step, the vanishing point is detected and the ROI where the road lane is detected is automatically defined in the input image only in case that the correct vanishing point is detected. If it is failed to detect the correct vanishing point, the ROI is empirically defined. By performing the road lane detection within the ROI instead of the whole image, various noises in the captured image by the frontal-viewing camera as shown in Figure 2, can be reduced in the procedure of lane detection. In addition, the effect of environmental conditions such as sunshine, rain, or extreme weather conditions can be lessened in the case using ROI compared to that using the whole image.
In general, the vanishing point is considered one of the most important keys to retaining a valid road lane, because road lanes are assumed to converge at one point within the captured image. As shown in Figure 2, lane markings always appear within the lower part of the image, but this depends on each camera configuration, and the input image can also include other objects (e.g., car hoods in Figure 2b The vanishing point is detected as follows [24]: Left and right road lane markings usually appear like two sides of a trapezoid based on the perspective projection of the frontal-viewing camera. Therefore, we can assume that all left and right lane boundaries can converge at one point called the vanishing point. First, line segments are detected by algorithms called LSD [32,33] and CannyLines [34] using consistent texture orientation. Let S = { 1 , 2 , … , } be the set of line segments extracted from image. Each line segment , ( = 1, 2, … , ) is defined as: where ( 1 , 1 ) and ( 2 , 2 ) are the coordinates of the starting point and the ending point of line segment , respectively. is the angle of line segment . Next, we define the length of line segment ith ( ) as the length weight ( ). The longer line segment represents more pixels in the same direction, as well as a higher voting weight which increases the voting score. Second, Gaussian weight is calculated in Equation (2) [24]. In the voting space image, we not only consider the intersected point between two line segments, but also its 5 × 5 neighboring points. Based on Gaussian distribution, those involved points have different values to make the lines vote more smoothly, and thus improve the accuracy of the detection of the vanishing point: where the candidate vanishing point ( , ) is computed in the neighborhood space 5 × 5 matrix, −2 ≤ , ≤ 2, = 1.5. In Equation (2), (x, y) is the candidate vanishing point. Because there can be errors in the detected (x, y) position just based on line segments, the neighborhood space of 5 × 5 pixels based on the (x, y) is also considered by using Gaussian distribution. By using the weight of where the candidate vanishing point (x, y) is computed in the neighborhood space 5 × 5 matrix, −2 ≤ x, y ≤ 2, σ = 1.5. In Equation (2), (x, y) is the candidate vanishing point. Because there can be errors in the detected (x, y) position just based on line segments, the neighborhood space of 5 × 5 pixels based on the (x, y) is also considered by using Gaussian distribution. By using the weight of Gaussian distribution, the less weight is assigned to the position (of candidate vanishing point) far from (x, y) when determining the final vanishing point as shown in Equation (3). In addition, the less weight is given to the position (of candidate vanishing point) which is determined based on shorter line segment (W L ) as shown in Equation (3). The score of the current selected pixel is then calculated as follows: Finally, we create a matrix space which is the same size as the input image and initialized to 0. Next, we update the score of each element in the matrix that corresponds to each pixel in the input image by adding I(x, y) score into the current value at the same position. Here, (x, y) is coordinate of current element in matrix and it is also a coordinate of current selected pixel in input image. The point that has the largest value is considered the vanishing point [24]. Figure 3b shows examples of detecting the vanishing point and defined ROI based on the vanishing point. Incorrect vanishing point caused by the car hood can be removed, and correct one is obtained, which produce the correct ROI as shown in Figure 3b. In addition, although incorrect line segments can be generated by shadows, the voting methods considering the Gaussian function-based weight and the length weight of line segment as shown in Equations (2)  into the current value at the same position. Here, ( , ) is coordinate of current element in matrix and it is also a coordinate of current selected pixel in input image. The point that has the largest value is considered the vanishing point [24]. Figure 3b shows examples of detecting the vanishing point and defined ROI based on the vanishing point. Incorrect vanishing point caused by the car hood can be removed, and correct one is obtained, which produce the correct ROI as shown in Figure 3b. In addition, although incorrect line segments can be generated by shadows, the voting methods considering the Gaussian function-based weight and the length weight of line segment as shown in Equations (2)     In order to prevent an incorrect ROI caused by inaccurate detection of the vanishing point, the y position of the vanishing point is compared to the upper y position of the predetermined ROI of Figure 3a (which is manually determined according to the database). If the difference between these two y positions is larger than the threshold (30 pixels), the predetermined ROI is used for lane detection, assuming that detection of the vanishing point fails. The diagram of these procedures are shown in Figure 4. In next Sections 3.3 and 3.4, we would explain the method of extracting features 1  In order to prevent an incorrect ROI caused by inaccurate detection of the vanishing point, the y position of the vanishing point is compared to the upper y position of the predetermined ROI of Figure 3a (which is manually determined according to the database). If the difference between these two y positions is larger than the threshold (30 pixels), the predetermined ROI is used for lane detection, assuming that detection of the vanishing point fails. The diagram of these procedures are shown in Figure 4. In next Sections 3.3 and 3.4, we would explain the method of extracting features 1 and 2 as the inputs to FIS to measure the level of shadows.
3.3. Calculating Feature 1 (HSV Color Difference Based on Local Background Area) Figure 5 shows the flowchart for determining shadow for feature 1. As the first step of Figure 5, the ROI of RGB color space is converted to that of HSV color space [35]. In the HSV color space, the V component is a direct measure of intensity. Pixels that belong to shadow should have a lower value of V than those in the nonshadow regions, and the hue (H) component of shadow pixels changes within a certain limited range. Moreover, shadow usually lowers the saturation (S) component. In conclusion, a pixel p is considered to be part of shadow if its value is satisfactory with the following three equations [36]: where I E p and B E p represent the specific channel of HSV color space (E = H, S, and V, respectively) for the pixel p in the current input image (I) and in the background ROI (B) (blue boxes of Figure 6a,c,e), respectively. The values thr Val pha , thr Vbeta , thr S , and thr H represent the threshold values, and these values are respectively 0.16, 0.64, 100, and 100. These optimal values were empirically determined by experiments with training data. It is unnecessary to recalculate the thresholds even if the camera is modified. In our experiment of Section 4, we used same thresholds with three different databases where the different cameras were used. Among these thresholds, those which affect shadow detection most are thr Val pha and thr Vbeta , because thr Val pha is used to define a maximum threshold for the darkening effect of shadows on the background pixel, whereas thr Vbeta prevents the system from incorrectly identifying the too dark (nonshadow) pixels as shadow pixels [37]. From the ROI of Figure 3, the ROI for lane detection is reduced by removing the left and right upper areas of the images as shown in Figure 6a,c,e to extract the features used as the input to FIS. Figure 6b,d,f shows the binarization image of extracted shadow within these ROIs based on Equations (4)-(6) and Figure 5. Thus, the average number of shadow pixels in this ROI is calculated as feature 1 in our research. From the ROI of Figure 3, the ROI for lane detection is reduced by removing the left and right upper areas of the images as shown in Figure 6a,c,e to extract the features used as the input to FIS. Figure 6b,d,f shows the binarization image of extracted shadow within these ROIs based on Equations (4)-(6) and Figure 5. Thus, the average number of shadow pixels in this ROI is calculated as feature 1 in our research.  From the ROI of Figure 3, the ROI for lane detection is reduced by removing the left and right upper areas of the images as shown in Figure 6a,c,e to extract the features used as the input to FIS. Figure 6b,d,f shows the binarization image of extracted shadow within these ROIs based on Equations (4)-(6) and Figure 5. Thus, the average number of shadow pixels in this ROI is calculated as feature 1 in our research. Figure 7 shows the flowchart for determining shadow for feature 2. While feature 1 is calculated in HSV color space, feature 2 is calculated in gray image to consider the characteristics of shadow in various color and gray spaces. Two thresholds for lower and upper bound threshold thr low and thr high are determined to calculate feature 2. According to the kinds of experimental databases, the threshold values are a little changed, and the ranges of these two thresholds are 16~17 and 48~50, respectively. These ranges of optimal thresholds were empirically determined by experiments with training data. Next, the mean value of all pixels whose value is in the range from thr low to thr high is calculated as µ mean . For example, there are four pixels inside the ROI of Figure 8, and their pixel values (gray levels) are 20, 15, 33, and 40, respectively. Because three pixels of 20, 33, and 40 (except for 15) belong to the range from thr low to thr high , µ mean is calculated as 31((20 + 33 + 40)/3). Finally, the pixel (x, y) which satisfied the condition of Equation (7) is determined as shadow:

Calculating Feature 2 (Gray Difference Based on Global Background Area)
where I(x, y) is the pixel value at coordinate x and y in the ROI for lane detection of Figure 8a,c; and the optimal threshold (thr medium ) was also empirically determined by experiments with training data. According to the kinds of experimental databases, the threshold value is a little changed, and the range of this threshold is 24~26, respectively. Next, the average number of shadow pixels in this ROI is calculated as feature 2 in our research.
respectively. These ranges of optimal thresholds were empirically determined by experiments with training data. Next, the mean value of all pixels whose value is in the range from ℎ to ℎ ℎ ℎ is calculated as . For example, there are four pixels inside the ROI of Figure 8, and their pixel values (gray levels) are 20, 15, 33, and 40, respectively. Because three pixels of 20, 33, and 40 (except for 15) belong to the range from ℎ to ℎ ℎ ℎ , is calculated as 31((20 + 33 + 40)/3). Finally, the pixel (x, y) which satisfied the condition of Equation (7) is determined as shadow: where ( , ) is the pixel value at coordinate x and y in the ROI for lane detection of Figure 8a,c; and the optimal threshold ( ℎ ) was also empirically determined by experiments with training data. According to the kinds of experimental databases, the threshold value is a little changed, and the range of this threshold is 24~26, respectively. Next, the average number of shadow pixels in this ROI is calculated as feature 2 in our research.   threshold values are a little changed, and the ranges of these two thresholds are 16~17 and 48~50, respectively. These ranges of optimal thresholds were empirically determined by experiments with training data. Next, the mean value of all pixels whose value is in the range from ℎ to ℎ ℎ ℎ is calculated as . For example, there are four pixels inside the ROI of Figure 8, and their pixel values (gray levels) are 20, 15, 33, and 40, respectively. Because three pixels of 20, 33, and 40 (except for 15) belong to the range from ℎ to ℎ ℎ ℎ , is calculated as 31((20 + 33 + 40)/3). Finally, the pixel (x, y) which satisfied the condition of Equation (7) is determined as shadow: where ( , ) is the pixel value at coordinate x and y in the ROI for lane detection of Figure 8a,c; and the optimal threshold ( ℎ ) was also empirically determined by experiments with training data. According to the kinds of experimental databases, the threshold value is a little changed, and the range of this threshold is 24~26, respectively. Next, the average number of shadow pixels in this ROI is calculated as feature 2 in our research.

Designing Fuzzy Membership Functions and Rule Table
For the next step, our method measures the level of shadow included in the ROI by using FIS using two features (features 1 and 2) as inputs as shown in Figure 1. The range of each feature is represented from 0 to 1 by min-max scaling to use two features as inputs to FIS. The input values are separated into two classes (low (L) and high (H)) in the membership function. In general, there is an overlapped area between these two value classes, and we define the shape of the input membership function as a linear function. Linear membership functions have been widely adopted in the FIS because the algorithm is less complex and the calculation speed is very fast compared to the nonlinear membership function [38][39][40]. With the training data, we obtained the distributions of features 1 and 2, and based on maximum entropy criterion, we designed the input member ship functions as follows: where a L_i is 1/(p L_i − q L_i ) and b L_i is q L_i /(q L_i − p L_i ). In addition, a H_i is 1/(p H_i − q H_i ) and b H_i is q H_i /(q H_i − p H_i ). In Equations (8) and (9), i = 1 and 2, and F L_ f eature i (x) is the L membership function of feature i, whereas F H_ f eature i (x) is its H membership function. Next, we can obtain the following equations: In Equations (10) and (11), i = 1 and 2. In addition, Dist L_ f eature i (x) is the L (data) distribution of feature i (nonshadow data of Figure 9), whereas Dist H_ f eature i (x) is the H (data) distribution of feature i (shadow data of Figure 9). Based on Equations (10) and (11), the entropy can be calculated as follows: where i = 1 and 2. Based on the maximum entropy criterion [41,42], the optimal parameters of (p L_i , q L_i , p H_i , q H_i ) of feature i are calculated by being selected when the entropy H( p L_i , q L_i , p H_i , q H_i ) is maximized. From this, the input membership functions of features 1 and 2 are defined as shown in Figure 9.
These membership functions are used to convert input values to a degree of membership. The output value of FIS is also described in the form of a linear function from the membership function to determine whether selected ROI contains more shadow or less. In our research, we designed the output membership function using three functions of low (L), medium (M), and high (H) as shown in Figure 10. We define the output fuzzy rule as "L" in the case when the level of shadow is close to 0 (minimum) and "H" when the level of shadow is close to 1 (maximum), as shown in Table 2. Thus, the optimal output value of FIS can be obtained using these output membership functions: the fuzzy rule table, and the combination of the defuzzification method with Min and Max rules. These membership functions are used to convert input values to a degree of membership. The output value of FIS is also described in the form of a linear function from the membership function to determine whether selected ROI contains more shadow or less. In our research, we designed the output membership function using three functions of low (L), medium (M), and high (H) as shown in Figure 10. We define the output fuzzy rule as "L" in the case when the level of shadow is close to 0 (minimum) and "H" when the level of shadow is close to 1 (maximum), as shown in Table 2. Thus, the optimal output value of FIS can be obtained using these output membership functions: the fuzzy rule table, and the combination of the defuzzification method with Min and Max rules.    These membership functions are used to convert input values to a degree of membership. The output value of FIS is also described in the form of a linear function from the membership function to determine whether selected ROI contains more shadow or less. In our research, we designed the output membership function using three functions of low (L), medium (M), and high (H) as shown in Figure 10. We define the output fuzzy rule as "L" in the case when the level of shadow is close to 0 (minimum) and "H" when the level of shadow is close to 1 (maximum), as shown in Table 2. Thus, the optimal output value of FIS can be obtained using these output membership functions: the fuzzy rule table, and the combination of the defuzzification method with Min and Max rules.

Determining Shadow Score Based on Defuzzification Methods
Using the two normalized input features, four corresponding values can be calculated using the input membership functions as shown in Figure 11. Four functions are defined as g L f 1 (·), g H f 1 (·), g L f 2 (·), and g H f 2 (·). The corresponding output values of the four functions with input values of f 1 (feature 1) and f 2 (feature 2) are shown by (g L f 1 , g H f 1 ) and (g L f 2 , g H f 2 ). For example, suppose that the two input values for f 1 and f 2 are 0.20 and 0.50, respectively, as shown in Figure 11. The values of (g L f 1 , g H f 1 ) and (g L f 2 , g H f 2 ) are (0.80(L), 0.20(H)) and (0.00(L), 1.00(H)), respectively, as shown in Figure 11. With these values,   Using four IVs, we can obtain the final output of FIS by one of the five defuzzification methods. In our research, we only consider five methods for defuzzification: first of maxima (FOM), last of maxima (LOM), middle of maxima (MOM), mean of maxima (MeOM), and center of gravity (COG) [38,43,44]. The FOM method selects the minimum value ( 1 ) among the values calculated using the maximum IV (( 1 ( ) and 2 ( ) of Figure 12a), LOM selects the maximum value ( 3 ) among the values calculated using the maximum IV (( 1 ( ) and 2 ( )). The MOM gets the middle value of the weight value from FOM and LOM (( 1 + 3 )/2), and MeOM gets the mean value (( 1 + 2 + 3 )/3). The output of FIS obtained by the COG is 5 as represented in Figure 12b, which is calculated from the COG of three regions ( 1 , 2 , and 3 ). We compared the five defuzzification methods and used one method (COG) which shows the best performance. That is, 5 is used as of the Equations (13) and (14) to adaptive change the parameters of LSD and CannyLines detector. With these four combinations, a value is selected by the Min or Max rule with the fuzzy rules in Table 2. In the Min method, the minimum value is selected from each combination, whereas the Max method selects the maximum value. For example, for (0.80(L), 1.00(H)), in the case of the Min rule, 0.80 is selected and M is determined (if "L" and "H," then "M" as shown in Table 2). Finally, the obtained value is 0.80(M). In the case of the Max rule, 1.00 is selected with M, and the obtained value is 1.00(M). These obtained values are called "inference values" (IVs). Table 3 shows the obtained  IVs by the Min or Max rule with the rule table of Table 2 from these four combinations of (0.80(L), 0.00(L)); (0.80(L), 1.00(H); (0.20(H), 0.00(L)); and (0.20(H), 1.00(H)).  Using four IVs, we can obtain the final output of FIS by one of the five defuzzification methods. In our research, we only consider five methods for defuzzification: first of maxima (FOM), last of maxima (LOM), middle of maxima (MOM), mean of maxima (MeOM), and center of gravity (COG) [38,43,44]. The FOM method selects the minimum value ( 1 ) among the values calculated using the maximum IV (( 1 ( ) and 2 ( ) of Figure 12a), LOM selects the maximum value ( 3 ) among the values calculated using the maximum IV (( 1 ( ) and 2 ( )). The MOM gets the middle value of the weight value from FOM and LOM (( 1 + 3 )/2), and MeOM gets the mean value (( 1 + 2 + 3 )/3). The output of FIS obtained by the COG is 5 as represented in Figure 12b, which is calculated from the COG of three regions ( 1 , 2 , and 3 ). We compared the five defuzzification methods and used one method (COG) which shows the best performance. That is, 5 is used as of the Equations (13) and (14) to adaptive change the parameters of LSD and CannyLines detector.

Adaptively Change Input Parameters for Line Segment Detector Algorithms
The obtained output of FIS in Section 3.6 represents the level of shadow in the input image, and then based on this output, the input parameters of line segment detector algorithms are changed adaptively, as shown in Equations (13) and (14). That is because more line segments are usually extracted from the boundaries of shadows in the case when the image including the larger level of Using four IVs, we can obtain the final output of FIS by one of the five defuzzification methods. In our research, we only consider five methods for defuzzification: first of maxima (FOM), last of maxima (LOM), middle of maxima (MOM), mean of maxima (MeOM), and center of gravity (COG) [38,43,44]. The FOM method selects the minimum value (w 1 ) among the values calculated using the maximum IV ((IV 1 (L) and V 2 (M) of Figure 12a), LOM selects the maximum value (w 3 ) among the values calculated using the maximum IV ((IV 1 (L) and IV 2 (M)). The MOM gets the middle value of the weight value from FOM and LOM ((w 1 + w 3 )/2), and MeOM gets the mean value ((w 1 + w 2 + w 3 )/3). The output of FIS obtained by the COG is w 5 as represented in Figure 12b, which is calculated from the COG of three regions (R 1 , R 2 , and R 3 ). We compared the five defuzzification methods and used one method (COG) which shows the best performance. That is, w 5 is used as f uzzy score of the Equations (13) and (14) to adaptive change the parameters of LSD and CannyLines detector.

Adaptively Change Input Parameters for Line Segment Detector Algorithms
The obtained output of FIS in Section 3.6 represents the level of shadow in the input image, and then based on this output, the input parameters of line segment detector algorithms are changed adaptively, as shown in Equations (13) and (14). That is because more line segments are usually extracted from the boundaries of shadows in the case when the image including the larger level of shadows is compared to the image including the lesser level of shadows.
In this paper, we combine two robust line segment detection algorithms to efficiently detect road lane markings boundaries from an input image. They are called LSD algorithm [32,33] in OpenCV library [45] and CannyLines detector [34], which are applied into the ROI of the input image, sequentially. The LSD method has several parameters to control meaningful line segments as follows; and the scale is adjusted in our research because it affects line segment detection more than sigma_scale: (1) Scale (α of Equation (13)): The scale of the image that is used to find the lines; its range is from 0 to 1. The 1 means that the original image is used for line segment detection. A smaller value shows that the image of a smaller size is used for line segment detection. For example, 0.5 means the image whose width and height are respectively half compared to those of the original image is used for line segment detection (2) Sigma_scale: Sigma value for Gaussian filter Based on the output of FIS, we update the LSD parameter (scale) dynamically based on Equation (13). In this Equation, α 0 is the default scale (0.8) of the LSD parameter, and f uzzy score is the output of FIS, whose range is from 0 to 1. The image of larger f uzzy score means that the larger levels of shadows are included. Therefore, in this case, we use the smaller α for LSD, which means the image size is reduced for line segment detection. With the image of smaller size, the high frequency edges of the image disappear compared to that of larger size. Therefore, the line segments from the boundary of the shadow tends to be reduced: Most of the parameters of the CannyLines detector related to the input image are determined by the image itself. However, there are still some parameters which can be adjusted, and µ v is adjusted in our research because it affects line segment detection more than other parameters: (1) µ v : Denotes the lower limit of a gradient magnitude (2) θ s : Represents the minimal length of an edge segment to be considered for splitting and equals twice that of the possibly shortest edge segment (3) θ m : Represents the maximal direction deviation tolerance of two close-direction line segments to be considered for merging Based on the output of FIS, the parameter of the CannyLines detector is also updated by Equation (14). The value µ 0 is the default value (70) of the lower limit of a gradient magnitude. As explained previously, the image of larger f uzzy score means that the larger levels of shadows are included. Based on Equation (14), consequently, µ v becomes larger. A larger µ v means that the higher limit of a gradient magnitude is used, which causes the reduction of the detected line segment by the CannyLines detector: As shown in Figure 13, through the adaptive adjusting of parameters of the LSD and CannyLines detector, we can find that the incorrect line segments are reduced in the result image. in our research because it affects line segment detection more than other parameters: (1) : Denotes the lower limit of a gradient magnitude (2) : Represents the minimal length of an edge segment to be considered for splitting and equals twice that of the possibly shortest edge segment (3) : Represents the maximal direction deviation tolerance of two close-direction line segments to be considered for merging Based on the output of FIS, the parameter of the CannyLines detector is also updated by Equation (14). The value 0 is the default value (70) of the lower limit of a gradient magnitude. As explained previously, the image of larger means that the larger levels of shadows are included. Based on Equation (14), consequently, becomes larger. A larger means that the higher limit of a gradient magnitude is used, which causes the reduction of the detected line segment by the CannyLines detector: As shown in Figure 13, through the adaptive adjusting of parameters of the LSD and CannyLines detector, we can find that the incorrect line segments are reduced in the result image.

Detecting Correct Lane Boundaries by Eliminating Invalid Line Segments Based on Angle and Vanishing Point
As shown in Figure 13b,d, there are still incorrect line segments after adaptively adjusting the parameters by the output of FIS. Therefore, in the next step, incorrect line segments are removed based on the characteristics of the road lane.
Because the car always operates between two road lanes, left and right road lane markings appear like two sides of a trapezoid in the image as shown in Figure 13. Therefore, only the left and right road lanes that satisfy the angle condition are maintained, regardless of their location [6]. In detail, we separate the ROI into two areas of left and right-side ROIs based on the middle position in the horizontal direction of ROI. That is, we decide that all line segments whose starting point has an

Detecting Correct Lane Boundaries by Eliminating Invalid Line Segments Based on Angle and Vanishing Point
As shown in Figure 13b,d, there are still incorrect line segments after adaptively adjusting the parameters by the output of FIS. Therefore, in the next step, incorrect line segments are removed based on the characteristics of the road lane.
Because the car always operates between two road lanes, left and right road lane markings appear like two sides of a trapezoid in the image as shown in Figure 13. Therefore, only the left and right road lanes that satisfy the angle condition are maintained, regardless of their location [6]. In detail, we separate the ROI into two areas of left and right-side ROIs based on the middle position in the horizontal direction of ROI. That is, we decide that all line segments whose starting point has an x-coordinate of the range [0, W ROI 2 − 1] belong to the left side-ROI; whereas, all the others belong to the right-side ROI. Here, W ROI is the width of the ROI. Then, we define empirically the range of angle of the road lane for left side-ROI and right-side ROI as θ le f t 25 respectively. Any line segments whose angle does not belong to these ranges (θ le f t 25 • − 75 • and θ right 105 • − 155 • ) are removed. As shown in Figure 14, incorrect line segments are removed after using the angle condition. There are still incorrect line segments after using the angle condition as shown in Figure 15a,c,e. Therefore, we use the vanishing point condition to remove these line segments. As explained in Section 3.2, all left and right boundaries of road lane markings intersect at a point called the vanishing point. Once the vanishing point is detected, we can obtain its x and y coordinates as x vp and y vp . Next, we can calculate slope a and y-intercept b of each detected line segment, and calculate the linear equation of this straight line with x vp to get the y coordinate value. Finally, we compare the distance value between y vp and the y coordinate value by using the linear equation of a straight line with a certain threshold value as shown in Equation (15), and remove the line segments if this distance value exceeds a certain threshold value. Figure 15b,d,f shows the results by using the vanishing point condition: There are still incorrect line segments after using the angle condition as shown in Figure 15a,c,e. Therefore, we use the vanishing point condition to remove these line segments. As explained in Section 3.2, all left and right boundaries of road lane markings intersect at a point called the vanishing point. Once the vanishing point is detected, we can obtain its x and y coordinates as and . Next, we can calculate slope and y-intercept of each detected line segment, and calculate the linear equation of this straight line with to get the y coordinate value. Finally, we compare the distance value between and the y coordinate value by using the linear equation of a straight line with a certain threshold value as shown in Equation (15), and remove the line segments if this distance value exceeds a certain threshold value. Figures 15b,d,f shows the results by using the vanishing point condition: In the case of a curved lane, the angle condition is not valid. For example, in Figure 16b the angle of the right lane of the upper region is similar to that of the left lane by the curved road. Therefore, the above angle condition is applied only in the middle and lower areas of ROI. In the upper area of ROI, the line segment whose angle is much different from that of the line segment detected below the region is removed. Detailed algorithms are referenced in [6].
However, in our research, the curved lanes are not detected correctly because of the vanishing point. This problem is depicted in Figure 16b. Based on the vanishing point condition, we only keep line segments that have an extension crossing the vanishing point; thus, we cannot detect the whole There are still incorrect line segments after using the angle condition as shown in Figure 15a,c,e. Therefore, we use the vanishing point condition to remove these line segments. As explained in Section 3.2, all left and right boundaries of road lane markings intersect at a point called the vanishing point. Once the vanishing point is detected, we can obtain its x and y coordinates as and . Next, we can calculate slope and y-intercept of each detected line segment, and calculate the linear equation of this straight line with to get the y coordinate value. Finally, we compare the distance value between and the y coordinate value by using the linear equation of a straight line with a certain threshold value as shown in Equation (15), and remove the line segments if this distance value exceeds a certain threshold value. Figures 15b,d,f shows the results by using the vanishing point condition: In the case of a curved lane, the angle condition is not valid. For example, in Figure 16b the angle of the right lane of the upper region is similar to that of the left lane by the curved road. Therefore, the above angle condition is applied only in the middle and lower areas of ROI. In the upper area of ROI, the line segment whose angle is much different from that of the line segment detected below the region is removed. Detailed algorithms are referenced in [6].
However, in our research, the curved lanes are not detected correctly because of the vanishing point. This problem is depicted in Figure 16b. Based on the vanishing point condition, we only keep line segments that have an extension crossing the vanishing point; thus, we cannot detect the whole In the case of a curved lane, the angle condition is not valid. For example, in Figure 16b the angle of the right lane of the upper region is similar to that of the left lane by the curved road. Therefore, the above angle condition is applied only in the middle and lower areas of ROI. In the upper area of ROI, the line segment whose angle is much different from that of the line segment detected below the region is removed. Detailed algorithms are referenced in [6].
However, in our research, the curved lanes are not detected correctly because of the vanishing point. This problem is depicted in Figure 16b. Based on the vanishing point condition, we only keep line segments that have an extension crossing the vanishing point; thus, we cannot detect the whole curved lane marking, but the part of the curved lane (of Figure 16b) can be removed by the vanishing point condition. To solve this problem, we apply the vanishing point condition only in the lower areas (below the violet line of Figure 16b) of ROI based on the detected vanishing point.
After eliminating the line segments according to angle and vanishing point conditions, multiple groups of line segments that belong to road lane markings remain. In this final step, we use methods similar to those that were used in [6] to combine small fragmented line segments into a single line. We define a 3 • of angle difference and three of the Euclidean distance difference as the stopping conditions, which means that we concatenate any two adjacent lines that have smaller than 3 • and three pixels of angle difference and the Euclidean distance difference, respectively. After eliminating the line segments according to angle and vanishing point conditions, multiple groups of line segments that belong to road lane markings remain. In this final step, we use methods similar to those that were used in [6] to combine small fragmented line segments into a single line. We define a 3 0 of angle difference and three of the Euclidean distance difference as the stopping conditions, which means that we concatenate any two adjacent lines that have smaller than 3 0 and three pixels of angle difference and the Euclidean distance difference, respectively.

Experimental Results
We tested our proposed method with various datasets as shown in Figures 17-19. For the Caltech dataset, 1016 images were used, and the size of the image was 640 × 480 pixels [5]. For the Santiago Lanes Dataset (SLD), 1201 images with the size of 640 × 480 pixels were used [46]. In addition, the Road Marking dataset consists of various subsidiary dataset with more than 3000 frames captured under various illumination conditions, and the image size is 800 × 600 pixels [47,48]. These databases were collected at different times along the day. We performed the experiments on a desktop computer with Intel Core TM i7 3.47 GHz, 12 GB memory of RAM, and the algorithm was implemented by Visual C++ 2015 and OpenCV library (version 3.1).
The ground-truth (starting and ending) positions of road lane markings were manually marked in the images to measure the accuracy of lane detection. Because our goal is to discriminate dashed and solid lanes in addition to lane detection, we manually detect the ground-truth point, and then compare it with detected starting and ending points with a certain interdistance threshold value to determine whether the detected line is correct or not. In

Experimental Results
We tested our proposed method with various datasets as shown in Figures 17-19. For the Caltech dataset, 1016 images were used, and the size of the image was 640 × 480 pixels [5]. For the Santiago Lanes Dataset (SLD), 1201 images with the size of 640 × 480 pixels were used [46]. In addition, the Road Marking dataset consists of various subsidiary dataset with more than 3000 frames captured under various illumination conditions, and the image size is 800 × 600 pixels [47,48]. These databases were collected at different times along the day. We performed the experiments on a desktop computer with Intel Core TM i7 3.47 GHz, 12 GB memory of RAM, and the algorithm was implemented by Visual C++ 2015 and OpenCV library (version 3.1).   The ground-truth (starting and ending) positions of road lane markings were manually marked in the images to measure the accuracy of lane detection. Because our goal is to discriminate dashed and solid lanes in addition to lane detection, we manually detect the ground-truth point, and then compare it with detected starting and ending points with a certain interdistance threshold value to determine whether the detected line is correct or not.   In our method, we only consider whether the detected line segment is a lane mark or not, so negative data do not occur (i.e., ground-truth data of a non-lane), and true negative (TN) errors are 0% in our experiments. Other kinds of errors such as true positive (TP), false positive (FP), and false negative (FN) are defined and calculated to obtain precision, recall, and F-measure as shown in Equations (16)- (18) [49,50]. The number of TP, FP, and FN are represented as #TP, #FP and #FN, respectively: Tables 4-6 show the accuracies of our method with each dataset.  Figure 20 shows correct lane detection using our method with various datasets. In addition, Figure 21 shows some examples of incorrect detection results. In Figure 21a, our method incorrectly recognized non-road lane objects such as crosswalks, road-signs, text symbols, and pavement as lane markings. In those cases, there are no dynamic conditions to distinguish which one belongs to a road lane and which one belongs to non-road lane objects. In addition, Figure 21b shows the effect of shadows on our method. Although our method uses the fuzzy rule to determine the amount of shadow in the image to automatically change the lane detector parameter, it still fails in some cases where extreme illumination occurs.   Figure 20 shows correct lane detection using our method with various datasets. In addition, Figure 21 shows some examples of incorrect detection results. In Figure 21a, our method incorrectly recognized non-road lane objects such as crosswalks, road-signs, text symbols, and pavement as lane markings. In those cases, there are no dynamic conditions to distinguish which one belongs to a road lane and which one belongs to non-road lane objects. In addition, Figure 21b shows the effect of shadows on our method. Although our method uses the fuzzy rule to determine the amount of shadow in the image to automatically change the lane detector parameter, it still fails in some cases where extreme illumination occurs. In the next experiment, we compare the performance of our method with some other methods: the Hoang et al. method [6], Aly method [5], Truong method [4], Kylesf method [7] and Nan method [1]. In [6], the line segment was detected by the LSD algorithm to detect the road lane. However, in [6], the lane detection was performed within the smaller ROI compared to the ROI in our research, and the number of images including shadows is smaller than that in our research. Therefore, the accuracies of lane detection, even with the same database using the methods [6] in Table 7, are lower than those reported in [6]. Owing to the same reasons, the accuracies by the methods [4,5] reported in [6] are different from those in Table 7. In other methods, they converted the input image by IPM with HT [5,7] to detect a straight line, and the random sample consensus (RANSAC) algorithm [5] to fit lane makers. We empirically found the optimal thresholds for these methods [1,[4][5][6][7]. As shown in Table 7 and Figure 22, our method outperforms previous methods. The reason why the accuracies by [1,4,5,7] are too low is that they did not detect the left and right boundaries of road lane, and did not discriminate the dashed and solid lanes. That is, their method did not detect the starting point and ending point of road marking as well as the left and right boundaries of road lane. Although the method [6] has these two functionalities, their method is more affected by the shadows in the image, and the accuracies by [6] are lower than ours. Moreover, this method [6] uses fixed ROI for detecting road lane and does not detect the vanishing point; thus, it generates more irrelevant line segments. That is why precision by this method is lower than that by our method. As shown in Figure 22a, we included the examples with the presence of vehicles on the same road lane of the detection vehicle. These cases were already included in our experimental databases. As shown in Figure 22a and Table  7, the presence of cars on the same road lane does not affect our detection results.  In the next experiment, we compare the performance of our method with some other methods: the Hoang et al. method [6], Aly method [5], Truong method [4], Kylesf method [7] and Nan method [1]. In [6], the line segment was detected by the LSD algorithm to detect the road lane. However, in [6], the lane detection was performed within the smaller ROI compared to the ROI in our research, and the number of images including shadows is smaller than that in our research. Therefore, the accuracies of lane detection, even with the same database using the methods [6] in Table 7, are lower than those reported in [6]. Owing to the same reasons, the accuracies by the methods [4,5] reported in [6] are different from those in Table 7. In other methods, they converted the input image by IPM with HT [5,7] to detect a straight line, and the random sample consensus (RANSAC) algorithm [5] to fit lane makers. We empirically found the optimal thresholds for these methods [1,[4][5][6][7]. As shown in Table 7 and Figure 22, our method outperforms previous methods. The reason why the accuracies by [1,4,5,7] are too low is that they did not detect the left and right boundaries of road lane, and did not discriminate the dashed and solid lanes. That is, their method did not detect the starting point and ending point of road marking as well as the left and right boundaries of road lane. Although the method [6] has these two functionalities, their method is more affected by the shadows in the image, and the accuracies by [6] are lower than ours. Moreover, this method [6] uses fixed ROI for detecting road lane and does not detect the vanishing point; thus, it generates more irrelevant line segments. That is why precision by this method is lower than that by our method. As shown in Figure 22a, we included the examples with the presence of vehicles on the same road lane of the detection vehicle. These cases were already included in our experimental databases. As shown in Figure 22a and Table 7, the presence of cars on the same road lane does not affect our detection results.    As the next experiment, we measured the processing time per frame by our method as shown in Table 8. As shown in Table 8, we can confirm that our method can be operated at a fast speed (about 40.4 frames/sec (1000/24.77)). In other previous researches [51][52][53][54], they showed the high performance of road lane detection irrespective of various weather conditions, traffic, and curved lanes, etc. However, they did not discriminate the solid and dashed lanes in the detected road lanes although it is necessary for As the next experiment, we measured the processing time per frame by our method as shown in Table 8. As shown in Table 8, we can confirm that our method can be operated at a fast speed (about 40.4 frames/s (1000/24.77)). In other previous researches [51][52][53][54], they showed the high performance of road lane detection irrespective of various weather conditions, traffic, and curved lanes, etc. However, they did not discriminate the solid and dashed lanes in the detected road lanes although it is necessary for autonomous vehicle. Different from them, even the solid and dashed lanes are discriminated in the detected road lanes by our method. In addition, more severe shadows are considered in our research compared to the examples of three results in [51][52][53][54]. In other methods [55,56], they can detect the road lane in difficult environments, but the method [55] did not discriminate the solid and dashed lanes in the detected road lanes either. The method [56] discriminated the solid and dashed lanes in the detected road lanes. However, they did not detect the exact starting and ending positions of all the dashed lanes although the accurate detection of these positions are necessary for the prompt or predictive decision of the moment of crossing road lane by fast moving autonomous vehicle. Different from them, in addition to the discrimination of the solid and dashed lanes, the accurate starting and ending positions of dashed lane are also detected by our method.

Conclusions
In this study, we proposed a method to overcome severe shadows in the image, for obtaining better road lane detection results. We used two features as the inputs for FIS: HSV color difference based on local background area (feature 1) and gray difference based on global background area (feature 2) for evaluating the level of shadow in the ROI of a road image. Two features from different color and gray spaces were used for FIS for considering the characteristics of shadow in various color and gray spaces. Using FIS based on these two features, we estimated the level of shadows based on the output of FIS after the defuzzification process. We modeled the input membership functions based on the training data of two features and maximum entropy criterion for enhancing the accuracy of FIS. By adaptively changing the parameters of LSD and CannyLines detector algorithms based on the output of FIS, more accurate line detection was possible based on the fusion of the detection results by LSD and CannyLines detector algorithms, irrespective of severe shadows on the road image.
Experiments with three open databases showed that our method outperformed previous methods, irrespective of severe shadows in the images. Because tracking information in successive image frames was not used in our method, the detection of lanes by our method was not affected by the speed of the car.
However, complex traffic with the presence of cars can affect our performance when detecting vanishing points and line segments, determining shadow levels, and locating final road lanes, which is the limitation of our system. Nevertheless, our three experimental databases do not include these cases, and we could not measure the effect of the presence of cars on the performance of our system.
In future, we would collect our own database including the complex traffic with the presence of cars, and measure the effect of these cases on our performance. In addition, we plan to solve this limitation by deep learning-based lane detection. Also, we plan to use a deep neural network for discriminating dashed and solid lane markings under various illumination conditions, as well as for detecting both straight and curved lanes. In addition, we would research to combine our method with a model-based method to enhance the performance of lane detection.