BSCNNLaneNet: A Novel Bidirectional Spatial Convolution Neural Network for Lane Detection

Abstract

Accurately detecting lane lines is a hot topic in computer vision. How to effectively utilize the relationship between lane features for detection is still an open question. In this paper, we propose a novel lane detection model based on convolutional neural network (CNN), namely, the BSCNNLaneNet (Bidirectional Spatial CNN Lane Detection Network). The proposed model is based on the spatial CNN method and incorporates a bidirectional recurrent neural network (BRNN) block to learn the spatial relationships between slice features. Additionally, a convolutional block attention mechanism is introduced to gain global features, which enhance the global connection between slice features in different directions. We conduct extensive experiments on the TuSimple dataset. The results demonstrate that the proposed method surpasses the original spatial CNN method, achieving an increase in accuracy from 96.53% to 96.86%.

1. Introduction

Lane detection has gained significant attention in research and industrial communities due to its wide range of real-world applications, such as in automatic driving, intelligent transportation, and robot navigation systems. Lane detection, especially in intelligent transportation as a critical component of autonomous driving, has become a key issue. To accurately and efficiently detect lane information, researchers have proposed numerous methods [1,2,3,4,5]. One popular approach is the spatial CNN [6], which introduces slice-by-slice convolutional operation to capture spatial relationship information between rows and columns of feature maps. In this way, slender structural targets, such as lane lines, can be identified more effectively [7,8].

Although spatial CNN has achieved advanced performance, it still encounters several limitations when identifying complex or curved lanes. First, due to the fact that the slice-by-slice convolutional operation is conducted separately along the row and column directions of a feature map, it is difficult to capture spatial information effectively when the local features of a lane line are unreliable or obscured. Second, as spatial CNN only involves a unidirectional stacking operation, it is difficult to find the corresponding relationship within the features.

For these reasons, Wang et al. [9] proposed a method based on the keypoint global association network and introduce the Lane-Aware Feature Aggregator (LFA) to adaptively capture the local correlation among adjacent keypoints. Zou et al. [10] proposed a hybrid network architecture by combining the convolutional neural network (CNN) with the recurrent neural network (RNN), although the issue of the gradient explosion of RNNs adversely affects feature learning. To effectively capture local features, Qu et al. [11] proposed a two-stage approach for detecting keypoint information of lane lines. Yoo et al. [12] devise a novel module for horizontal reduction, which can more effectively extract features and compress the horizontal representation of each lane line in an end-to-end manner. Although significant advancements have been made by the aforementioned methods, they still encounter some technical challenges. In particular, effectively capturing the features of lane lines occluded by vehicles and pedestrians presents a challenge. Additionally, detecting lane lines at a distance remains difficult.

In this paper, a novel network model based on spatial CNN, the BSCNNLaneNet (Bidirectional Spatial CNN Lane Detection Network), is proposed for feature extraction in complex scenes, which is beneficial for improving the overall performance of lane detection. First, a bidirectional recurrent neural network (BRNN) is put forward to enhance the correlation between learned slice features. Second, to more efficiently extract feature information, a convolutional attention block is proposed to refine the slice features and strengthen the global relationships between slice features in different directions.

The main contributions of this paper are outlined below:

(1)

Based on the spatial CNN method, the proposed network model introduces the bidirectional recurrent neural network (BRNN) to effectively learn spatial relationships among slice features.

(2)

In addition, our method utilizes the convolutional block attention module to refine the slice features’ output by the BRNN, which can strengthen the global relationships between the features in different directions.

The remainder of this article is structured as follows: In Section 2, a brief review of lane detection methods is summarized. Then, the proposed method is described in Section 3. After that, experiments and results are shown in Section 4. Finally, Section 5 concludes our work and briefly discusses the possible future work.

2. Related Work

In this section, we briefly review the related work of lane detection. Lane detection is a computer vision task that involves identifying the boundaries of driving lanes in a video or image of a road scene. Based on different lane representations, most of the lane detection models can be sorted into three categories: segmentation-based methods, anchor-based methods, and curve-based methods.

Segmentation-based Methods: Segmentation-based methods treat lane detection as a pixel classification problem, which classifies each pixel and assigns it to a specific lane class. For this issue, Pan et al. [6] utilized slice-by-slice convolution within feature maps and allowed for message-passing between adjacent pixels. In order to reduce the computation cost and learn more information from context, Hou et al. [13] introduced a novel knowledge distillation approach, i.e., self-attention distillation (SAD), which allows the model to learn from rich contextual information through top-down and layer-wise attention distillation within the network. Li et al. [14] proposed the line proposal unit (LPU) to locate accurate traffic curves, which forced the system to learn the global feature representation of all traffic lines. For reducing extra annotation and retraining costs, Li et al. [15] proposed the multi-level domain adaptation (MLDA) framework, which can process lane detection at three complementary levels, i.e., pixel level, instance level, and category level. In order to catch subtle lane features, RESA [16] is proposed to enrich lane features after CNN feature extraction, which is able to take advantage of strong shape priors of lanes and capture the spatial relationships of pixels across rows and columns. In addition, Qiu et al. [17] proposed the PriorLane framework, which can enhance the segmentation performance of the full vision transformer by introducing low-cost local prior knowledge. In contrast to the abovementioned methods, our approach utilizes the bidirectional recurrent neural network and attention mechanism to gather spatial information and context features, respectively.

Anchor-Based Methods: These methods involve predefining a set of anchor points on the input image and then using a neural network to predict the offsets between these anchors and the actual lane markings, which convert the lane detection problem into a regression task. Qin et al. [18] proposed a novel formulation to select the locations of lanes from predefined rows in the image using global features and treated the process of lane detection as a row-based selection problem. In LaneATT [19], the authors proposed an anchor-based attention model that focuses on capturing relevant features around the anchor points to increase the detection accuracy in conditions with occlusions. In addition, Zheng et al. [20] proposed a cross-layer refinement method that leverages the high-level semantic features and low-level texture features to enhance the proposal representations, which can minimize localization errors and improve the detection results. Su et al. [21] proposed a structure-guided framework for lane detection, which can utilize a novel vanishing point guided anchoring mechanism to generate anchors and multi-level structure constraints to improve lane perception. In ObliqueConv [22], the authors proposed a new idea of oblique convolution that can break through the limitation that prevent the traditional convolution kernel from effectively extracting the lane of the inclined cross region. In order to solve the problem of fixed-shape anchors being difficult to use for modeling complex lane-line shapes, Zhang et al. [23] proposed the ECPNet (Enhanced Curve Perception Network), which can exploit high-level features to predict lanes while leveraging local-detailed features to improve localization accuracy. Although these anchor-based approaches directly introduce geometric knowledge of the lanes into the model, which enables robust prediction for heavily occluded lanes, their precision still depends on the design and placement of the anchors.

Curve-Based Methods: Curve-based methods for lane detection fit polynomial or spline curves to the lane markings by predicting the parameters of these curves using a neural network. This approach transforms lane detection into a curve fitting problem. The pioneering work [24] proposed a method to train a lane detector in an end-to-end manner, which is able to estimate lane curvature parameters by solving a weighted least-squares problem. The PolyLaneNet [25] takes as input the images from a forward-looking camera mounted in the vehicle and estimates polynomials that represent each lane marking in the image. In order to fit well with irregular lines, the authors [26] proposed a novel framework for lane detection, which decomposes the detection process into parts: curve modeling and ground height regression. Recently, LSTR [27] used transformer blocks to predict polynomials, in which the structures for lanes and the global context can be captured efficiently. Although curve-Based Methods are more robust due to the smoothness of the fitting curve, their accuracy still lags behind that of other methods.

3. Method

In this section, we describe the proposed BSCNNLaneNet network model in detail. The workflow of BSCNNLaneNet is shown in Figure 1. First, ResNet was chosen as BSCNNLaneNet’s backbone, and then a neck module was employed to capture spatial information. After that, the RNN and the CAM module were used to extract weighted features.

3.1. BRNN-Based Serialized Feature Learning Method

In order to efficiently locate continuous lane lines, Pan et al. [6] proposed the spatial CNN method to capture the relationships among features by utilizing four-directional slices and convolution stacking. However, this method only obtains basic correlation information among the slice features. In contrast, in our method, the features are extracted in both horizontal and vertical directions. In addition, instead of performing stacking convolutional operations, our method uses a bidirectional recurrent neural network (BRNN) to gain more correlations among slice features. The forward propagation RNN transfers information in the order $t=0$ to $t=k$, and the backward counterpart performs the same operations in the opposite direction. In this way, our approach is able to capture more contextual information.

In our method, the formulas used for learning the BRNN serialization feature are as follows [28,29,30]:

$$y_1\left(t\right)=U_1·f\left(\begin{array}{c}t\end{array}\right)+W_1·tanh\left(y_1\left(t-1\right)\right)$$

(1)

$$y_2\left(t\right)=U_1·f\left(\begin{array}{c}t\end{array}\right)+W_1·tanh\left(y_2\left(t-1\right)\right)$$

(2)

$$h\left(t\right)=softmax\left(V·\left[y_1\left(t\right);y_2\left(t\right)\right]\right)$$

(3)

In these formulas, $y_1\left(t\right)$ represents the hidden state of RNN forward propagation, while $y_2\left(t\right)$ represents the hidden state of RNN backward propagation. $y_1\left(t\right)$ is related to the information before the $t$th sequence, and $y_2\left(t\right)$ is the information after the $t$th sequence. $U_1$ and $W_1$ are the weights of the forward and backward fully connected layers, respectively. $f\left(t\right)$ is the input of BRNN, which represents the feature sequence of the output slice feature map. $h\left(t\right)$ is the output of BRNN at time $t$.

3.2. CBAM-Based Feature Association Method

In our method, we introduce the convolution block attention module to strengthen the global associations among slice features. In this way, the proposed method can gain attention mapping based on both channel and spatial order, which makes it possible to focus on pertinent information and disregard irrelevant information [31,32,33,34].

Thus, in our method, CBAM is adopted, which can generate a two-dimensional channel attention map and a two-dimensional space attention map. The whole attention process can be summarized as follows:

$${\mathbf{F}}_t^{\prime }={\mathbf{M}}_{\mathrm{c}}\left({\mathbf{F}}_t\right)\otimes {\mathbf{F}}_t$$

(4)

$${\mathbf{F}}_t^{″}={\mathbf{M}}_{\mathrm{s}}\left({\mathbf{F}}_t^{\prime }\right)\otimes {\mathbf{F}}_t^{\prime },$$

(5)

where $\otimes$ denotes the element-wise multiplication operation. The channel attention weight value is propagated across the spatial dimension, and the spatial attention weight value can be propagated across the channel dimension. The details of the channel attention module and spatial attention module can be described as follows.

First, for the channel attention module, an intermediate feature map ${\mathbf{F}}_t\in {\mathbb{R}}^{C\times H\times W}$ is obtained. ${\mathbf{F}}_{t,\mathrm{a}\mathrm{v}\mathrm{g}}^{\mathrm{c}}$ and ${\mathbf{F}}_{t,\mathrm{m}\mathrm{a}\mathrm{x}}^{\mathrm{c}}$ are the computing results of the average and maximum pooling operations conducted on ${\mathbf{F}}_t$. Then, ${\mathbf{F}}_{t,\mathrm{a}\mathrm{v}\mathrm{g}}^{\mathrm{c}}$ and ${\mathbf{F}}_{t,\mathrm{m}\mathrm{a}\mathrm{x}}^{\mathrm{c}}$ are input into a shared network where a multi-layer perceptron of the hidden layer is included. To minimize parameter computation overhead, the hidden activation size can be configured as ${\mathbb{R}}^{C/r\times 1\times 1}$, where r is the compression ratio. Finally, the merge and sigmoid function operations are applied to the output result of the shared network. The whole computation process can be summarized as follows:

$$\begin{array}{ll}{M}_c\left(F_t\right)& =\sigma \left(\left(MLP\left(A\nu gPool\left(F_t\right)\right)\right)+\left(MLP\left(MaxPool\left(F_t\right)\right)\right)\right)\\ & =\sigma \left(W_1\left(W_0\left(F_{t,av\mathrm{g}}^c\right)\right)+W_1\left(W_0\left(F_{t,max}^c\right)\right)\right)\end{array},$$

(6)

where $\sigma \left(·\right)$ represents the sigmoid function, and ${\mathbf{W}}_0\in {\mathbb{R}}^{C/r\times C}$ and ${\mathbf{W}}_1\in {\mathbb{R}}^{C\times C/r}$ are the weight of the multilayer perceptron.

For the spatial attention module, first, maximum pooling and average pooling operations are performed on the input $F_{\mathrm{t}}^{\prime }$. Then, the pooling feature maps are put into a convolutional layer to generate a 2D spatial attention map. Finally, the sigmoid function operation is applied to the output of the convolution layer. This computation process can be summarized as follows:

$$\begin{array}{ll}{M}_s\left(F_t\right)& =\sigma \left(f^{7\times 7}\left(\left[AvgPool\right(F_t)\right);MaxPool\left(F_t\right)]\right)\\ & =\sigma \left(f^{7\times 7}\left(\left[F_{t,a\nu g}^s;F_{t,max}^s\right]\right)\right)\end{array},$$

(7)

where $f^{7\times 7}$ denotes the size of a convolution kernel.

By applying the channel attention module and the spatial attention module on sequence splicing, the network is able to effectively identify the key features and capture the spatial information of the lane lines.

3.3. Loss Function

In our detection model, the general cross entropy loss is adopted, which is defined in Formula (8). Here, $y^{\prime }$ is the true value, and $y$ is the predicted probability value obtained from the softmax logistic regression. $L_0$ represents the background. $L_1$, $L_2$, $L_3$, and $L_4$ represent the four lane lines, respectively. To address the issue of imbalance between the positive and negative samples, we have implemented the concept of focal loss and incorporated it into our classification loss function. Specifically, we have set the ${\lambda }_0=0.3$, while setting ${\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4=1$, respectively.

$$L\left(y^{\prime },y\right)=-\sum _{i=1}^n{y}_i^{\prime }\mathrm{l}\mathrm{o}\mathrm{g}\left(y_i\right)={\lambda }_0{L}_0+{\lambda }_1{L}_1+{\lambda }_2{L}_2+{\lambda }_3{L}_3+{\lambda }_4{L}_4$$

(8)

The learning rate formula is shown in Formula (9).

$$Learning\_Rate=[Base\_Learning\_Rate\times \left(1-{\displaystyle \frac{iter}{\mathrm{m}\mathrm{a}\mathrm{x}\_iter}}\right){]}^{power}$$

(9)

4. Experiments and Results

4.1. Datasets

The TuSimple dataset [35] is commonly used in the field of lane detection. It is a challenging dataset that comprises 3626 images in the training set and 2782 images in the test set, each with a consistent image resolution of 1280 × 720. Lane lines are represented by equally spaced points, and their two-dimensional coordinates are stored in a corresponding JSON file. The JSON file for each image includes information on all four lane lines.

The TuSimple dataset provides a clear and relatively simple depiction of high-speed driving with sparse traffic flow and distinct road markings, while real-world driving scenarios can be more complex, with challenges like heavy traffic and obscured lane markings.

4.2. Evaluation Metric

In this experiment, we used the official metric for TuSimple [35] to evaluate our method, such as accuracy rate, false positive (FP) and false negative (FN) rates, the F1-measure, etc. The accuracy rate is determined by averaging the number of correctly predicted points in the image and calculated with the following Formula (10). $C_{im}$ denotes the count of accurately predicted points, and $S_{im}$ denotes the overall count of actual label points. When the coordinate difference between the predicted and actual lane point coordinates is below the threshold (which has been set to 40 pixels in this paper), the lane point prediction is correct.

$$Accuracy={\displaystyle \frac{\sum _{im}{C}_{im}}{\sum _{im}{S}_{im}}}$$

(10)

The FP and FN rates are defined as follows:

$$FP={\displaystyle \frac{F_{pred}}{N_{pred}}}$$

(11)

$$FN={\displaystyle \frac{M_{pred}}{N_{gt}}},$$

(12)

where $F_{pred}$ is the number of wrong predicted lanes, $N_{pred}$ is the total number of predicted lanes, $M_{pred}$ is the missed ground-truth lanes in the predictions, and $N_{gt}$ is the number of all ground-truth lanes.

Additionally, the F1-score, a combination metric of precision and recall, is reported as shown in previous works [6]. The F1-score is defined as follows:

$$F1-score={\displaystyle \frac{2\ast Presision\ast Recall}{Precision+Recall}},$$

(13)

where

$$Precision={\displaystyle \frac{TP}{TP+FP}},Recall={\displaystyle \frac{TP}{TP+FN}}$$

(14)

and TP (true positive) is the number of lane points correctly predicted.

4.3. Implementation Details

4.3.1. Experiment Settings

We chose ResNet18, ResNet34, and ResNet101 as our pre-trained backbones for different computational requirements: small, medium and large. In the training process, we adopted the Adam optimizer and cosine decay learning-rate strategy. The settings, as described in [6], along with the specific hyperparameter configurations used during training, are detailed in

Table 1. Hyperparameter configurations in the training process.

Hyperparameter	Specific Settings
Image Scaling Size	320 × 800
Batch Size	5
Optimizer	Adam
Momentum Parameter	0.9
Weight Decay	5 × 10−4
Base Learning Rate	0.01
Learning Strategy	Poly (power = 0.9)

. All experiments were implemented based on Tensorflow with one NVIDIA 3090 GPU.

4.3.2. Data Preprocessing

In the experiment, the data preprocessing was mainly divided into three parts:

(1)

Assign a grayscale image with one channel as the label for the TuSimple dataset. The grayscale value of the label corresponds to the sequence number of the lane line from left to right, with 1, 2, 3, and 4 representing the lane lines, respectively.

(2)

Due to the uppermost portion of the picture mostly consisting of buildings, trees, sky, and other non-lane line elements, this area should be removed, leaving only the lane line area covering the entirety of the remaining picture. This process reduces the image size and the computation cost. All the input images are resized to 320 × 800. This can lead to faster training times and lower computational resource requirements.

(3)

Data augmentation is applied to the training phase, including random scaling and random rotation.

4.4. Main Results

We compared our approach with recent state-of-the-art methods on the TuSimple benchmark. As illustrated in

Table 2. Performance of the different methods for the TuSimple dataset.

Method	Backbone	ACC	F1@50	FP	FN	FPS
APE [4]	ResNet34	96.93	-	-	-	35
SCNN [6]	VGG16	96.53	95.97	6.17	1.80	7.5
FOLOLane [11]	ERFNet	96.92	-	4.47	2.28	100
CLRNet [20]	ResNet18	96.84	97.89	2.28	1.92	119
CLRNet [20]	ResNet34	96.87	97.82	2.27	2.08	103
CLRNet [20]	ResNet101	96.83	97.62	2.37	2.38	46
LaneATT [19]	ResNet18	95.57	96.71	3.56	3.01	250
LanetATT [19]	ResNet34	95.63	96.77	3.53	2.92	171
LaneATT [19]	ResNet101	96.10	96.06	5.64	2.17	26
UFLD [18]	ResNet34	95.86	88.02	18.91	3.75	300
RESA [16]	ResNet18	96.70	-	3.95	2.83	-
RESA [16]	ResNet34	96.82	-	3.63	2.48	35
LSTR [27]	ResNet18	96.18	96.86	2.91	3.38	420
Ours	ResNet18	96.73	96.21	2.93	2.62	188
Ours	RestNet34	96.86	96.87	2.26	1.99	176
Ours	RestNet101	96.83	96.72	2.51	2.01	153

, our results show that our method achieves a comparable performance with a 96.87% F1@50 measure with the backbone ResNet34. In addition, we also analyzed the FP and FN for each method. The FP of our method is far below other algorithms, which means that BSCNNLaneNet gains higher precision on the lane detection task and contributes to achieving higher accuracy. It is worth noting that BSCNNLaneNet reaches 150+ FPS and is much faster than most other methods, which means our method is suitable for real-time applications.

Additionally, Figure 2 presents the visualization results of other approaches, in which different lane marking instances are represented by different colors. In contrast, our method demonstrates a greater ability to precisely detect distant lane lines, avoiding the common issue of missed detections in far-off regions that is seen in other approaches. (shown in the red rectangle box in the second column).

These experimental findings serve as compelling evidence of the viability of and advancements achieved by introducing bidirectional spatial information and associating slice features globally.

4.5. Ablation Study

In this section, we conducted the ablation study to evaluate the impact of the bidirectional recurrent neural network model and the CBAM model for our proposed method on ResNet18, ResNet34, and ResNet101. The same settings as described in Section 4.3.1, although not specially mentioned, were adopted. The component-level evaluation results are shown in

Table 3. Ablation study of the components in BSCNNLaneNet on TuSimple.

Method	ResNet18	ResNet34	ResNet101
SCNN	96.57	96.62	96.66
SCNN+RNN	96.59	96.65	96.70
SCNN+CAM	96.62	96.66	96.68
BSCNNLaneNet	96.73	96.86	96.83

, and the first row demonstrates the baseline results of the original SCNN with a different ResNet backbone.

The second row of Table 3 summarizes the performance of our method based on different backbones. By adding the RNN module, we found that all three network models outperform their original counterparts, respectively.

From the results in the third row of Table 3, we can see that the CAM module creates improvement by adding spatial and channel attention as compared with the baseline of the original SCNN. SCNN+CAM achieves accuracy scores of 96.62, 96.66, and 96.68 when RestNet18, Renet34, and ResNet101, respectively, are used as the backbones.

Finally, the results in the fourth row demonstrate that the proposed network model, BSCNNLaneNet, achieves better recognition accuracy. The improvement indicates that it is valid to add spatial information and attention mechanisms for the lane line detection method.

4.6. Cross-Dataset Generalization Evaluation

In order to estimate the generalization ability of our method, cross-dataset evaluation test was conducted. We utilized the model trained on the TuSimple training set to draw inferences from the CULane testing set.

Table 4. Evaluation of the generalization ability of different methods from the TuSimple training set to the CULane testing set.

Method	Backbone	ACC	FP	FN
SCNN [6]	VGG16	5.33	95.73	97.02
UFLD [18]	RestNet34	30.07	56.73	66.47
LaneATT [19]	RestNet34	28.89	55.42	58.86
CLRNet [20]	RestNet34	30.66	52.58	55.89
Ours	RestNet34	35.32	50.33	48.66

shows that the proposed method achieves a comparable result with an accuracy of 35.32%, which surpasses the SCNN method by a significant margin of nearly 30%. Thus, it confirms the stable generalization capabilities of our method.

4.7. Study on Training Dynamics

In this section, we analyzed the training dynamics of BSCNNLaneNet. To this end, we trained our module on the TuSimple dataset for 14 epochs and evaluated the performances per two epochs. As depicted in Figure 3, it is evident that the accuracy score increases as the number of rounds increases, whereas the values of FP and FN decrease as the number of rounds increases. BSCNNLaneNet converges well at about the sixth epoch, which indicates that our method achieves a fast convergence rate. In contrast, CLRNet needs about 50 training epochs. This suggests that our method not only achieves a comparable result in the test process but can also save computation costs in the training phrase.

5. Conclusions

In this paper, we present an enhanced Spatial CNN algorithm designed for detecting lane lines, which can assist in creating intelligent transportation systems for autonomous driving, auxiliary driving, etc. This model is capable of extracting more comprehensive spatial correlation information, improving the spatial correlation among slice features in multiple directions, and extending the correlation ranges among slice features, all while minimizing computational complexity. The detectors yield precise and valuable metrics for lane detection systems, including lane-line contour and area. Our experiments demonstrate that incorporating a bidirectional recurrent neural network (BRNN) and a convolutional attention module into the spatial convolutional neural network (CNN) leads to significant improvement in lane detection. Specifically, our approach based on ResNet34 achieves a high accuracy of 96.86%. Compared with the spatial CNN, the algorithm presented in this paper achieves a reduction in both the FP and FN to 2.26% and 1.99%, respectively. Though the results demonstrate the superiority of our approach compared with existing methods, there also exist some limitations, i.e., how to improve its performance in low-light conditions and further enhance its adaptability for the issue of partially missing lane markings.