An Efficient Lane Detection Network with Channel-Enhanced Coordinate Attention

Abstract

Lane detection based on semantic segmentation can achieve high accuracy, but, in recent years, it does not have a mobile-friendly cost, which is caused by the complex iteration and costly convolutions in convolutional neural networks (CNNs) and state-of-the-art (SOTA) models based on CNNs, such as spatial CNNs (SCNNs). Although the SCNN has shown its capacity to capture the spatial relationships of pixels across rows and columns of an image, the computational cost and memory requirement needed cannot be afforded with mobile lane detection. Inspired by the channel attention and self-attention machine, we propose an integrated coordinate attention (ICA) module to capture the spatial-wise relationships of pixels. Furthermore, due to the lack of enhancement in the channel dimension, we created an efficient network with a channel-enhanced coordinate attention block named CCA, composed of ICA and other channel attention modules, for all-dimension feature enhancement. As a result, by replacing many repeated or iterative convolutions with the attention mechanism, CCA reduces the computational complexity. Thus, our method achieves a balance of accuracy and speed and has better performance on two lane datasets—TuSimple and ILane. At less than a few tenths of the computational cost, our CCA achieves superior accuracy compared to the SCNN. These results show that the low cost and great performance of our design enable the use of the lane detection task in autopilot scenarios.

1. Introduction

Lane detection is closely related to the normal driving of self-driving cars and plays an important role in assisted driving systems [1,2,3]. As a perception technique for autonomous driving, lane detection is the fundament of lane departure warning (LDW) systems and lane keeping assistance (LKA) systems [4]. Through subsequent decision-making algorithms, it makes the vehicle drive in the lane correctly and complies with the traffic rules stipulated for the lane [5]. In recent years, with the development of computer vision, many methods for detecting lanes have reached this goal [6]. However, how to deal with prior knowledge of lanes, confusion with other lines and environments, and severe occlusion from other cars still remain challenges [7].

In the past several years, lane detection has been based on hand-crafted feature detectors, typically taking into account aspects such as image preprocessing, feature extraction, lane line model fitting, and straight-line tracking. In recent years, algorithms based on deep convolutional neural networks (CNNs) have developed rapidly, their advantages in image feature extraction have become more and more obvious, and their accuracy has obviously improved. Since then, CNN-based lane detection has become a popular research method. Nowadays, the popular methods are categorized into four groups, inspired, respectively, by semantic segmentation [8], object detection [9], keypoint [10], and curve-fitting methods [11]. The approach based on semantic segmentation mostly predicts every pixel of the image with a binary label [12], with an encoder to extract features and a decoder to upsample and recover them. The first method to consider the prior knowledge was the SCNN [13], and it greatly enhanced the feature extraction of ordinary convolutional neural networks (CNNs [14]). The SCNN tries to dig out spatial features deeply with a spatial CNN, moving from deep layer-by-layer convolutions to slice-by-slice convolutions within feature maps. Based on the SCNN, a novel module named the recurrent feature-shift aggregator (RESA) [15] has been developed to gather information vertically and horizontally by shifting the sliced feature map recurrently. These approaches mostly aim to figure out more and more about the intrinsic characters of lane lines, improving the accuracy of the task. However, due to the abundant floating point operations (FLOPs), the speed of the tasks performed on high-resolution inputs is far from the design requirements of the driver assistance system (DAS).

Along with the popularity of the attention mechanism and the creation of ViT [16], the networks in computer vision have become simpler and lighter as a result of the replacement of recurrence and convolutions, without sacrificing accuracy. Inspired by ViT, we designed an efficient transformer-based module with fused kinds of attention, named integrated coordinate attention (ICA). Channel-enhanced coordinate attention (CCA) divides the feature matrices into the query (Q), key (K), and value (V), like multi-head attention [10], and then has two branches in parallel. One is for catching deeper features, and the other uses ICA to gather the attentional importance between all features. The feature branch utilizes Inverted Residuals and ECANet [17] after concatenating the Q, K, and V to output a detailed feature map. In contrast, the attention branch applies the ECANet to enhance the information of Q, K, and V, then, respectively, gather the relevance of the input features along the vertical and horizontal directions via Scaled Dot-Product Attention [10]. The combination of two branches makes our encoder accurately aggregate the spatial features of traffic lanes without a high computational cost. Several major advantages of CCA are as follows: (1) CCA applies global attention to gather information on feature maps, showing better segmentation performance. (2) Because it lacks complex spatial convolutions and recurrent spatial shifts, CCA avoids the burden of high computation for pixel-wise segmentation tasks. (3) CCA is light, simple, and convenient to transfer to other designs.

To achieve outstanding performance in the balance of speed and accuracy, we designed an upsampling module. To demonstrate it, we evaluated it on the dataset TuSimple [18]. The results in Figure 1 show that the CCA we designed performs better and consumes fewer resources than existing state-of-the-art networks of lane detection by semantic segmentation.

The core contributions we focus on are as follows:

(1) We propose CCA to capture a small amount of important lane information and discard redundant background information with the help of the attention mechanism. CCA, which includes feature extraction and enhancement as well as foreground and background segmentation, is simple yet useful and thus suitable for being applied to another network to achieve better overall performance. Additionally, we propose a simple dual-channel convolutional decoder (PDecoder), a new upsampling module adapted for the CCA which can achieve better performance.

(2) Our network achieves the best performance on the TuSimple dataset with minimum latency, demonstrating the potential to serve as a lightweight and efficient lane detection module for mobile devices.

(3) We have collected a new dataset to classify complex lane situations into seven categories, containing nearly 2000 images, to validate the excellent performance of our design.

2. Materials and Methods

2.1. Related Work

In recent years, the field of image recognition has undergone a tremendous transformation from traditional methods by hand-crafted feature to deep-learning-based methods. For the lane detection task, although many cleaver methods have emerged, the requirements of the application remain challenging. How to design a network with high detection accuracy and short detection time has always been a difficult problem. Semantic segmentation and the attention mechanism have shown excellent performance in various deep-learning tasks in recent years. As a pixel-level image processing technology, semantic segmentation can efficiently extract the details of the target, while the attention mechanism focuses on weights of features and obtaining the associations between information. Based on this, the author conducts relevant analysis and launches research and improvement on lane detection task.

2.1.1. Lane Detection by Semantic Segmentation

Since FCN [19] pioneered the “sampling + aggregation” network for semantic segmentation, how to aggregate features has been the focus of research in this field. The classical aggregation methods are as follows: UNet [20] fuses high-level and low-level features in a step-by-step manner, PSPNet [21] utilizes “multi-scale image feature fusion + upsampling”, and DeepLab series [22,23,24], inspired by the idea of multi-scale fusion, introduces the Dilated Convolution to increase the receptive field of the convolution. Cutting-edge algorithms include those proposed by SegFormer [25] for the Transformer block and multi-scale fusion methods; K-Net [26] which unifies semantic segmentation, instance segmentation, and panoramic segmentation; MaskFormer [27] which only performs instance-level classification and eliminates pixel-level classification; Mask2Former [28] which introduces a cross-attention mechanism; and segmentation anything model (SAM), proposed in 2023, which can automatically recognize all objects present in the image and is an extraordinary large-scale vision model.

Classical representative methods for lane detection such as SCNN and RESA propose spatial convolution of features based on CNN and utilize spatial iteration to pass information between neighboring rows or columns of the feature map, which is time-consuming and prone to losing dependencies between distant information. Our design is inspired by past research on the self-attention mechanism and channel attention [29], to aggregate the dependencies between different information. Without the convolutional aggregator of SCNN and RESA, the design significantly saves computational resources and time costs, so is more suitable for mobile devices.

2.1.2. Feature Enhancement by Spatial Attention

In 2015, Bahdanau et al. proposed the self-attention mechanism [30], an important technique for modeling sequence data, and Vaswani et al. proposed the Transformer [31] in 2017, which has drawn widespread attention and found extensive applications. The self-attention mechanism compares each element of the input sequence with all other elements and then assigns weights based on the similarity between them. As shown in Figure 2, the author expands the input matrix to obtain Q, K, and V. V can be interpreted as a feature backup, while Q and K carry out the transformations to solve for the weights. In addition to its success in the field of natural language processing, the self-attention mechanism has also been widely used in the field of computer vision. Moreover, the ViT model applies the self-attention mechanism to the image classification task to achieve a performance that matches or even exceeds all traditional convolutional neural networks, which brings a new concept and methodology to the deep learning models in the image field. In short, regardless of the deep learning domain, the attention mechanism is an excellent tool for weight assignment.

In this thesis, the self-attention mechanism serves as the foundation of core component and is capable of effectively capturing the long-range dependencies among plane coordinates.

2.1.3. Feature Enhancement by Channel Attention

The most classic design in the field of channel attention research is SENet [29]; Jie Hu’s team from Momenta Enterprise (Momenta, Suzhou, China) pioneered squeeze-and-excitation attention, which makes channel attention a major research hotspot in mobile network design. On one hand, SENet directly sets the values on the two plane dimensions to 1, which causes significant and irreversible damage in terms of performance. Thus, ECA-Net [17] emerged through the efforts of scholars in the field of machine learning represented by Wang and others. Only a small number of parameters are added to maintain performance without dimensionality reduction. On the other hand, SENet’s neglect of location information has become an aspect to be optimized. Therefore, Hou and other scientists from the National University of Singapore proposed coordinate attention (CA block) [32], which focuses on the grasping of image location information.

These attention mechanisms are highly beneficial to the model’s performance. Considering the shape information of lane lines, we used scaled dot-product attention to obtain stronger associations between different information, as shown in Figure 3; the inspiration of the ICA module comes from SENet, CA and self-attention mechanism. By integrating channel and plane branches, an attention module for three-direction feature enhancement is created. Then, it is applied to the traditional semantic segmentation framework and can effectively capture the key features of lanes.

2.2. Method

2.2.1. Architecture Design

We train a neural network for lane detection that comprises three components: encoder, CCA, and decoder. Firstly, we select ResNet as the encoder, which is utilized for extracting the information of raw images. It downsamples the size of input images to 1/8, retains better original features, and ensures a smoother and more stable training process. Secondly, CCA is the heart of our design, which combines long-range feature embedment in the spatial dimension and information enhancement in the channel dimension. Finally, the dual-channel convolutional decoder (PDecoder) consists of a plain channel and a detail channel. The main purpose of plain channel is to restore the size of the feature map, and the other aims to reduce feature loss due to upsampling. After that are the output branches, predicting the existence and probability of each lane and pixel.

2.2.2. Channel-Enhanced Coordinate Attention

The channel-enhanced coordinate attention (CCA) block can act as a mathematical module to analyze its principles for enhancing the representation of features. The output feature tensor of encoder is shown as $X=\left[x_1,x_2,\dots ,x_c\right]\in R^{H\times W\times C}$. After the augmented representations, it can be transformed to an output tensor $Y=\left[y_1,y_2,\dots ,y_c\right]$, whose size is same as $X$. In this part, we will first revisit the self-attention machine and ECA-Net to give an articulate presentation of CCA.

1.

Revisit ECANet.

ECANet, whose full name is efficient channel attention module, was proposed in 2019. In this paper, the researchers abolished the double-FC layer in SENet, since the double-FC layer hardly obtains the weights efficiently and thus keeps away from low costs. Instead of the unnecessary FC layers, the authors proposed an adaptive way to acquire cross-dimension interaction, achieving high-quality prediction of channel weights. The key point of ECANet is how to compute the convolution kernel adaptively.

$$t=\left|{\displaystyle \frac{{\log }_{2}C+b}{\gamma }}\right|,k=\left\{\begin{array}{cc}t,\hfill & if t=even\hfill \\ t+1,\hfill & if t=odd\hfill \end{array}\right.$$

(1)

where $C$ is the channel of input feature, and $\gamma$ and $b$ are the parameters of the mapping function. After that, channel attention can be obtained by a convolution and a sigmoid function.

2.

Revisit Attention Mechanisms.

Attention mechanisms have enhanced the existing best results with low computational costs and thus play an important role in our CCA. As demonstrated in [10], we can obtain the weights with a simple softmax function when the input consists of queries, keys, and values. The attention output can be as follows:

$$Attention\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(2)

where $d_k$ is a scaling factor, $Q$ is the query matrix, $K$ is the key matrix, and $V$ is the value matrix. The multi-head attention first obtains the weights of information from representation subspaces at different positions and then concatenates the weights, instead of computing the weight after averaging the information, avoiding the discarding of detailed information.

3.

Revisit Coordinate Attention (CA Block).

As described in [32], the coordinate attention block can be decomposed into two steps: coordinate information embedding and attention generation, which are designed for channel relationship encoding and exploration of long-range dependencies with precise position. With the input $X$, the embedding step for the $c$ channel at height $h$ and weight $w$ can be formulated as follows:

$$\begin{gathered} z_c^h\left(h\right)={\displaystyle \frac{1}{W}}{\displaystyle \sum }_{i=0}^W{x}_c\left(h,i\right) \\ {z}_c^w\left(w\right)={\displaystyle \frac{1}{H}}{\displaystyle \sum }_{j=0}^H{x}_c\left(j,w\right) \end{gathered}$$

(3)

By the pair of equations above, the features from two different spatial directions are, respectively, aggregated, avoiding the loss of many details and such expensive computational costs.

The next step is to make full use of the expressive representations resulting from the embedding transformation. Refer to three criteria–mobile, ROI-highlighted, and relationships between channels: the author first concatenates and sends them to a $1\times 1$ convolutional transformation function $F1$, which is formulated as follows:

$$f=\delta \left(F_1\left[z^h,z^w\right]\right)$$

(4)

where $\left[ , \right]$ means the concatenation operation, $\delta$ denotes activation function, and $f\in R^{C/r\times \left(H+W\right)}$ is the intermediate feature map. $f$ was split into two dimension-separate tensors $f^w$ and $f^h$ which were sent to another convolutional function $F_w$ and $F_h$.

$$g^h=\sigma \left(F_h\left(f^h\right)\right),g^W=\sigma \left(F_w\left(f^w\right)\right)$$

(5)

where $\sigma$ is the sigmoid function. Finally, they multiplied inputs and weights, yielding the output of the module $Y$ depicted as follows:

$$y_c=x_c\left(i,j\right)\times g_c^h\left(i\right)\times g_c^w\left(j\right)$$

(6)

4.

CCA Block.

The block we designed consists of two types of branches: the feature branch and the weight branch, as Figure 4.

One is the feature branch, which combines the feature preprocess module and the following feature enhancement module. It is the main branch to extract more fine-detailed features. In the preprocess module, we split the feature matrix into $Q$, $K$, and $V$, facilitating the operation of the weight branch. Then, stacking the $Q$, $K$, and $V$ in channel dimension in the enhancement module, and by the following part composed of a DWConv [33], a ReLU, a PWConv [33], and a ECANet, a new feature map can be output. The DWConv aims to capture more details, the PWConv changes the number of channels, and the ECANet enhances the channel-wise information by calculating the $w$ matrix. The output of feature branch can be as follows:

$$y_f=ECA\left(F_P\left(\sigma \left(F_D\left[Q,K,V\right]\right)\right)\right)$$

(7)

where $ECA$ denotes the ECA block, $F_P$ and $F_D$ denote the two kinds of convolutional functions, $\sigma$ is hardsigmoid function, and $\left[ , \right]$ denotes the concatenation operation.

The other one is the weight branch we call ICA as shown in Figure 5, and can be used to filter important spatial information in high resolution. ICA has two parts, the coordinate attention module and the channel attention module. Identical to the feature branch, the coordinate attention module also performs feature processing based on $Q$, $K$, and $V$. The attention branch is divided into two stages—coordinate information squeezing and spatial attention generation. However, the channel attention module is relatively easy to implement. The input of it is same as the feature enhancement module of the feature branch. Its composition is an ECANet.

• Coordinate Information Squeezing.

In SENet and ECANet, global pooling was used to squeeze the spatial information. The shortcoming is obvious. Global averaging is too crude to preserve representative details. Thus, like the coordinate attention block, we apply the global pooling but squeeze the vectors in two respective spatial directions for $Q$, $K$, and $V$ in the first step. The input in channel $c$ involves three feature matrixes $X_Q$, $X_K$, and $X_V$ as shown:

$$X_Q=\left(\begin{array}{ccc}{x}_Q\left(1,1\right)& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ x_Q\left(H_Q,1\right)& \cdots & a_{mn}\end{array}\right) \in R^{1\times H_Q\times W_Q}$$

(8)

$$X_K=\left(\begin{array}{ccc}{x}_K\left(1,1\right)& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ x_K\left(H_K,1\right)& \cdots & a_{mn}\end{array}\right) \in R^{1\times H_K\times W_K}$$

(9)

$$X_V=\left(\begin{array}{ccc}{x}_V\left(1,1\right)& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ x_V\left(H_V,1\right)& \cdots & a_{mn}\end{array}\right) \in R^{1\times H_V\times W_V}$$

(10)

$$c1=c2=c3/2$$

(11)

where height and width are $h_{*}\in \left[1,H_{*}\right]$, $w_{*}\in \left[1,W_{*}\right]$, and the outputs can be depicted as similar equations:

For $Q$:

$$\begin{gathered} Q^h=Z_{c1}^h\left(h_Q\right)={\displaystyle \frac{1}{W_Q}}{\displaystyle \sum }_{i_Q=0}^{W_Q}{x}_{c1}\left(h_Q,i_Q\right) \\ {Q}^w=Z_{c1}^w\left(w_Q\right)={\displaystyle \frac{1}{H_Q}}{\displaystyle \sum }_{j_Q=0}^{H_Q}{x}_{c1}\left(j_Q,w_Q\right) \end{gathered}$$

(12)

For $K$:

$$\begin{gathered} K^{h }=Z_{c1}^h\left(h_K\right)={\displaystyle \frac{1}{W_K}}{\displaystyle \sum }_{i_K=0}^{W_K}{x}_{c1}\left(h_K,i_K\right) \\ {K}^w=Z_{c1}^w\left(w_K\right)={\displaystyle \frac{1}{H_K}}{\displaystyle \sum }_{j_K=0}^{H_K}{x}_{c1}\left(j_K,w_K\right) \end{gathered}$$

(13)

For $V$:

$$\begin{gathered} V^h=Z_{c1}^h\left(h_V\right)={\displaystyle \frac{1}{W_V}}{\displaystyle \sum }_{i_V=0}^{W_V}{x}_{c1}\left(h_V,i_V\right) \\ {V}^w=Z_{c1}^w\left(w_V\right)={\displaystyle \frac{1}{H_V}}{\displaystyle \sum }_{j_V=0}^{H_V}{x}_{c1}\left(j_V,w_V\right) \end{gathered}$$

(14)

where $Z^w$ is the output at weight and $Z^h$ is the output at height. By pairs of equations above, $H$ height-dimension and $W$ weight-dimension vectors were squeezed, outputting six vectors of information as the input of the attention part. Aiming to dig out the more interesting importance of different spatial positions over long distances, we compute the weights for value, respectively, at the height and weight. The output of attention Functions are as follows:

$$\begin{gathered} f_h=Attention\left(Q^h, K^h,V^h\right)=softmax\left({\displaystyle \frac{Q^h{\left(K^h\right)}^T}{\sqrt{d_k}}}\right)V^h \\ {f}_w=Attention\left(Q^w,K^w,V^w\right)=softmax\left({\displaystyle \frac{Q^w{\left(K^w\right)}^T}{\sqrt{d_k}}}\right)V^w \end{gathered}$$

(15)

$f_h$ and $f_w$ denote the information map with two-directions weight, embedding to the next transformation as inputs. On one hand, with the help of the attention mechanism, spatial coordinates in two different dimensions can be encoded efficiently, preserving as many representative details as possible. On the other hand, the scaled factor $d_k$ enables our module not to have such substantial computational costs to burden. By the attention, the key, query, and value were linked, yielding a feature tensor in each direction. In addition, we must stitch the coordinate information of different directions by concatenating them in the last step:

$$f=\left[f_h,f_w\right]$$

(16)

• Spatial Attention Generation.

Further on the basis of encoding global coordinate information is how to identify the interested region, so we need to clarify the relationships between coordinates. Few operations, in this part, are proposed that differ from the CA block. The first step is transforming $f$ by a convolutional transformation function $F$ and an activation function $\delta$. The output is as follows:

$$g=\delta \left(F\left(f\right)\right)$$

(17)

where $\delta$ is the t_hardsigmoid function and $g \in R^{C/r\times \left(H+W\right)}$, and r is the reduction ratio controlling the size of the CCA block. Then, after splitting along the height and width direction, the two tensors obtained are sent separately to customized convolutional functions $F_w$ and $F_h$. Thus, the output of the enhancement branch can be written as follows:

$$C_{*}=\sigma \left(F_{*}\left(g_{*}\right)\right)$$

(18)

$*$ denotes the spatial dimension height or width, $\sigma$ is a no-linear activation function, and we choose softmax in the experiment. $C_w$ and $C_h$, the results of the enhancement, act as the weights of the feature branch. So, we need the last operation, multiplying them:

$$y_{CCA}=y_f\ast C_h\ast C_w$$

(19)

2.2.3. Dual-Channel Convolutional Decoder

In this part, we need to design a decoder to upsample the feature map so that its resolution is the same as the input. The first requirement is to be lightweight, so we choose a simple and low-parameter decoder as the base, named PlainDecoder [21]. However, the 1 × 1 convolution in PlainDecoder transforms the information into an overly sketchy situation, ignoring details that should not be ignored. Aiming to overcome this drawback, the second step we take is to add another channel to focus on the details. Just a sum of the dual channel is sufficient to output a feature map. Without high cost, it can improve a lot.

In Figure 6, the UpSampleConv is a combination of a group convolution, a channel-enhanced block, and a 1 × 1 convolution. It is similar to the BottleNeck but with a reduction ratio of 2, not 6. The OutputConv and the PlainConv are both 1 × 1 convolution, and the difference between them is the number of input channels. The latter one has the same number of inputs, while the former one has one-eighth of it. We can describe the module as follows:

$$y_p=ReLU \left(OCov\left(UConv\left(UConv\left(UConv\left(y_{CCA}\right)\right)\right)\right)+PConv \left(drop\left(y_{CCA}\right)\right)\right)$$

(20)

where $ReLu$ is a no-linear activation function, $OCov$ is the OutputConv, $UCov$ is the UpSampleConv, $PCov$ is the PlainConv, and $drop$ is Dropout, a strategy for suppressing overfitting.

3. Results and Discussions

3.1. Experiment Implementation and Dataset

3.1.1. Dataset of Experiment

We conduct experiments on a widely used lane detection benchmark dataset: the Tusimple Lane detection benchmark. The dataset consists of 3626 training images and 2782 test images with stable lighting conditions in American highways. Considering the limitations of highway, we collect a new dataset, which is called Ilane, which includes several difficult roads such as country roads and environments such as night and snow. For the test dataset, we classify Ilane into a number of different categories. The detailed information is given in Figure 7.

3.1.2. Evaluation Metrics of Experiment

The evaluation metrics are $accuracy$, $FP$, and $FN$, are depicted as follows:

$$accuracy={\displaystyle \frac{{{\displaystyle \sum }}_{i=0}^n{X}_i}{{{\displaystyle \sum }}_{i=0}^n{Y}_i}}$$

(21)

$$FP={\displaystyle \frac{{{\displaystyle \sum }}_{i=0}^n{A}_i}{{{\displaystyle \sum }}_{i=0}^n{Y}_i}}$$

(22)

$$FN={\displaystyle \frac{{{\displaystyle \sum }}_{i=0}^n{B}_i}{{{\displaystyle \sum }}_{i=0}^n{Y}_i}}$$

(23)

where $i$ is the number of images,$Y$ is the number of all point in lane, and $X$ is the number of the points with right prediction. $A$ and $B$ are the numbers of the points with wrong predictions, which are divided into missing correct values, creating incorrect values.

3.1.3. Implement Settings of Experiment

Our models are modified based on the RESA, and the initial weights of the backbone are copied from ResNet. In both tasks, we train the models using standard stochastic gradient descent (SGD) with batch size 4, base learning rate 0.02, momentum 0.9, and weight decay 0.0001. The loss function we choose is the sum of binary cross entropy (BCE) and existence cross entropy (CE), but we have also made a few setting modifications to fine-tune the configuration of the loss function. The original size of the images is 720 × 1280. For the default version, the images are cropped to the fixed size of 368 × 640. For the full-resolution version, the images are kept at the original size 720 × 1280. For the scaling-resolution version, we set the size to 720 × 640. All experiments were conducted on an NVIDIA GeForce RTX 4060 GPU.

3.2. Experimental Results and Discussions

3.2.1. Main Results

In this section, we extensively evaluate the performance of the proposed network CCA on different datasets and in real scenarios.

1.

Qualitative Analysis.

In this part, we present the qualitative comparison results of CCA with two classical algorithms. We visualize the semantic segmentation results as shown in Figure 8; we can measure missed detections in terms of the density of points. In many details, such as long-distance lane markings, the RESA has a good detection effect. Although CCA is not as dense as the RESA at long-distance detection points, but it still performs better overall. Both are significantly better than the SCNN, which cannot even detect any point on the leftmost road. Due to the irregular shape of lane lines and complex road conditions, the detection performance of existing algorithms and the network is not ideal. It can be seen from the prediction results that the SCNN algorithm, which has the same detection time as CCA, has obvious limitations, with too many missing coordinate points of lane lines and sparse dots. For the better performance of the RESA, the prediction results of the RESA can segment lane lines well, but the issues of the detection time and cost need to be analyzed in the following chapters. The target edge predicted by our algorithm is very close to the real situation on the ground, showing high robustness and no false detection. Our algorithm outperforms other comparison methods in terms of detection and segmentation performance.

2.

Quantitative Analysis.

The following will show the results of our network on different datasets and the comparison of the metrics of different networks. The first one is on the TuSimple dataset. We select some classical lane line recognition methods such as LaneNet, LaneATT, the RESA, and the SCNN. In addition, we consider two sets of “aggregator-free” networks, which are composed of two kinds of common backbones and a BUSD decoder from the RESA. The result is shown in

Table 1. Comparison with state-of-the-art results on TuSimple with image resolution (1, 3, 368, and 640).

Net	Res18-BUSD	Res34-BUSD	LaneATT	LaneNet	SCNN	RESA	CCA-18 (ours)	CCA-34 (ours)
Params (M)	11.48	21.59	22.13	42.9	21.94	25.36	11.38	21.48
Accuracy	0.9269	0.9284	0.9302	0.9338	0.9653	0.9686	0.9678	0.9681
FP	0.0948	0.0918	0.0886	0.0780	0.0617	0.0395	0.0501	0.0351
FN	0.0822	0.792	0.0734	0.0224	0.0180	0.0283	0.0298	0.0314
Latency (MS)	391	543	442	414	429	746	238	392

. While the RESA outperforms all baselines in terms of accuracy, it does so at the expense of efficiency. In contrast, CCA optimizes latency to achieve the most balanced result. Under the condition that all backbones choose resnet34, the latency of our network is 392 ms, nearly half as fast as the RESA, and even much faster than Res34-BUSD without an aggregator. It has to be mentioned that, on the other hand, the accuracy rate is 96.81%, an increase of 0.28% on the basis of the SCNN and only 0.6% lower than the RESA. Taken together, the evaluation gives us a good indication of the adaptability of CCA on mobile devices.

On the dataset Ilane that we collected, we classified the images and compared the performance of the RESA, SCNN, and CCA under different adverse conditions. The result is shown in

Table 2. Comparison with state-of-the-art results on Ilane with image resolution (1, 3, 368, and 640).

	CCA		RESA		SCNN
Category	Acc	Lat	Acc	Lat	Acc	Lat
bad light	0.9693	311	0.9643	340	0.9653	308
tunnel	0.9819	292	0.9797	816	0.9599	325
normal	0.9704	136	0.9721	185	0.9643	355
dirty	0.9649	400	0.9731	436	0.9638	369
park	0.9592	221	0.9605	259	0.9584	299
snow night	0.965	236	0.9674	285	0.962	356
follow	0.9797	167	0.9693	220	0.9685	377
mean	0.9681	280	0.9649	538	0.9602	354

. In terms of the latency, our designed CCA can reach higher-rate levels. In some conditions such as curves, junctions, and old roads, the complexity of image features increases the processing difficulty and thus lengthens the latency of the task. However, it is reassuring to see that CCA still maintains a low latency. In terms of accuracy, the detection accuracy of CCA is superior to that of the SCNN in any case. In cases of bad light, tunnel, and others, the accuracy is even better than that of the RESA.

The final output of our designed network is shown in Figure 9. The input picture of CCA is located in the first row. The network performs point recognition to output point coordinates, plotted in the input image as shown in the second row of images, plotted in the black background image as shown in the third row of images. It can be seen that our network can accurately identify the two adjacent lanes of a car and output dense point coordinates, which meets the requirements of intelligent transportation and autonomous driving for lane identification. In addition, on the side road and bypass road, there may be an inaccurate detection or poor detection effect. Therefore, how to accurately identify them is one of the key points we still continue to focus on and improve.

3.2.2. Ablation Study

To elaborate on the advantages of our approach in more detail, we combine the channel-enhanced coordinate attention block and the PDecoder discussed in the Method chapter and attempt to analyze and confirm the necessity of each module in this section.

1.

Loss of Each Module.

We first studied the processing efficiency of aggregators and the decoder. As a baseline, we chose ResNet-34 as the backbone. After extraction from the backbone network, the feature maps enter the aggregator, and the image feature weights are calculated for stacking. Finally, we use the decoder to upsample the feature maps by a factor of 8. We record the processing time for each module. To compare the aggregators, we replace the aggregator with the RESA and SCNN without changing the other modules. Similarly, to compare the decoder’s time consumption, we just replace the decoder with BUSD and PlainDecoder. We summarize the performance of each aggregator module in

Table 3. Parameters and latency of three aggregators in different high resolutions of input images. Lat1 means the latency in 256 × 256, Lat2 means 512 × 512, and Lat3 means 1024 × 1024.

Aggregator	Parameters	Lat1	Lat2	Lat3
RESA	2949,120	465.7	1347.2	3931.1
SCNN	589,824	316.8	316.8	2310.7
CCA	118,800	343.7	427	548.1

, and the performance of each decoder module in

Table 4. Parameters and latency of three decoders in different high resolutions of input images. Lat1 means the latency in 256 × 256, Lat2 means 512 × 512, and Lat3 means 1024 × 1024.

Decoder	Parameters	Lat1	Lat2	Lat3
BUSD	239,008	832.8	2197	128,000.1
PlainDecoder	903	5.8	18.5	53.6
PDecoder	14,466	290.7	408.8	681

. It can be seen that both modules significantly improve the lane detection performance, which demonstrates the capability of the modules. In particular, in the aggregator part, the computation of our new design CCA is about 1/25 of the RESA and lower than 1/5 of the SCNN, while, in the decoder part, we do not simplify it blindly, so its computation is not too low but still lower than 1/20 of BUSD, which enables a balance of time loss while keeping the decoding a without loss of accuracy.

2.

Effectiveness of Each Module.

In this section, we explore the effect of different combinations of aggregators and decoders on the processing results. Theoretically, when the amount of computation increases, it means that the complexity of the operation increases, and more accurate features can be aggregated per feature map. However, this comes at the cost of processing time. The focus of the study is how to select the right combination of aggregator and decoder to achieve the perfect balance between accuracy and speed.

At the beginning of our design, we wanted to take advantages of the CCA block in terms of the time cost. However, when comparing the accuracy of the SCNN and RESA, we found that, whether it is paired with a complex BUSD or a simple PlainDecoder, it does not achieve the best result in terms of accuracy. Therefore, we designed PDecoder to simplify the operation of BUSD while retaining its idea. As shown in

Table 5. Parameters and latency of three decoders in different high resolutions of input images. Lat1 means the latency in 256 × 256, Lat2 means 512 × 512, and Lat3 means 1024 × 1024. “↑” represents the increase in different metric and “↓” represents the decrease.

Method	Acc	Paras	Lat
RESA	0.9686	25.36	0.602
SCNN	0.9653	21.94	0.429
CCA	0.9663	21.466	0.375
+BUSD	0.9672 (↑0.00009)	21.696 (↑0.23)	0.425 (↑0.05)
+PlainDecoder	0.9639 (↓0.0024)	21.47 (↑0.004)	0.383 (↑0.008)
+PDecoder	0.9681 (↑0.00018)	21.476 (↑0.01)	0.392 (↑0.017)

, although it takes a little more time compared with PlainDecoder, it shows a significant improvement in accuracy, and the increase in computational load on the entire network is insignificant. This is a trade-off between performance and computational resources. To strike a balance between the two, we choose “CCA + PDecoder” as our final choice.

4. Conclusions

In this paper, we present two key components for lane detection: the channel-enhanced coordinate attention (CCA) and the simple upsampling decoder (PDecoder). The CCA block utilizes the superposition of multiple attention mechanisms to obtain local relationships between pixels in both the vertical and horizontal directions. In the foundation of acquiring the base plane feature weights in the vertical and horizontal directions, it uses channel attention to perform feature enhancement in the third dimension, and then aggregates the information from each pixel into global information. In addition, it can be easily integrated into other networks. The simple upsampling decoder simplifies some image convolution operations and fully connected structures, saving a significant amount of computing time while combining coarse-grained and fine-grained images. The most outstanding advantage is that the decoder can be combined with CCA to achieve the best performance. Our approach is evaluated on the lane detection benchmark dataset Tusimple to achieve the best balance among various metrics. Furthermore, considering that the TuSimple dataset has a single road condition, we collected a new dataset, ILane, which covers many rural roads with different road conditions for evaluation and testing. We conducted experiments on this dataset and found that the algorithms designed by us can maintain good results in terms of both accuracy and processing speed even under a few extreme conditions. We found that, even under the conditions of low light, extreme weather, and poor vehicle conditions, our algorithm still maintains good results in terms of both accuracy and latency, and it is expected to be embedded in existing autonomous driving systems. We are actively exploring suitable application scenarios and deploying solutions to promote the development of automotive intelligence.