CSANet: Cross-Scale Axial Attention Network for Road Segmentation

Abstract

Road segmentation from remote sensing images is an important task in many applications. However, due to the high density of roads and the complex background, the roads are often occluded by trees. This makes accurate road segmentation a challenge task. Most existing road segmentation networks rely on convolutions with small kernels; however, these methods often cannot obtain satisfying results because the long-range dependencies are not captured and the intrinsic relationships between feature maps at different scales are not fully exploited. In this paper, a deep neural network based on a cross-scale axial attention mechanism is proposed to address this problem. This model enables low-resolution features to aggregate global contextual information from high-resolution features. Among them, the axial attention mechanism realizes global attention by using vertical and horizontal attention sequentially. With this strategy, the dense long-range dependencies can be captured with extremely low computational cost. The cross-scale mechanism enables the model to effectively combine the high-resolution fine-grained features and the low-resolution coarse-grained features. The proposed method enables the network to propagate the information without losing details. Our method achieves IoUs of 58.98 and 65.28 on the Massachusetts Roads dataset and DeepGlobe dataset and outperforms other methods.

1. Introduction

Semantic segmentation is a very important and challenging task in the field of computer vision. It has very important applications in many scenarios such as autonomous driving, indoor navigation, and virtual and augmented reality [1,2]. Road segmentation is one of the basic tasks of aerial imagery processing, which aims to separate pixels belonging to roads from other background pixels in remote sensing aerial images. Road segmentation plays an essential role in many applications, such as traffic management, emergency tasks, road monitoring, etc.

Compared with conventional semantic segmentation, road segmentation in aerial images relies more on global contextual information because the shape of roads is usually long and continuous. Early road-segmentation methods were mostly implemented with complex hand-crafted features, such as the morphology-based method proposed by Steger et al. [3] and the texture analysis and beam transform-based method proposed by Sghaier et al. [4]. The methods based on topology improve the accuracy by post-processing the segmentation map, but this undoubtedly brings a higher computational cost [5,6,7]. With the rapid development of deep learning, convolutional neural networks are well known for their end-to-end properties and powerful feature-modeling capabilities. It is also widely used in road image segmentation. The split depth-wise separable Graph-Convolution Network proposed by Zhou et al. [8] separates spatial features and channel features with depth-wise separable convolutions, and then the Sobel gradient operator is used to construct the adjacency matrix of the feature map to extract the road information. Lan et al. [9] introduce residual dilated convolution and ASPP modules on the basis of Unet to expand the receptive field and capture multi-scale global context information. Tao et al. [10] designed a spatial information inference structure that enables multidirectional message passing between pixels. It is thus possible to learn local visual features of roads and global spatial structure information (such as road continuity and trend). Henry et al. [11] introduce the sensitivity of the spatial tolerance rule to thin objects so that most of the roads in the test set can be identified. Shamsolmoali et al. [12] apply structured domain adaptation to synthetic image generation and road segmentation, in which a feature pyramid (FP) network is incorporated into a generative adversarial network to minimize the difference between the source and target domains. Y-Net [13] extracts features through a downsampling-to-upsampling sub-network and a sub-network without downsampling. Then, the features of these two parts are combined by the fusion module for road segmentation. However, fully convolutional networks can only capture local information through their small convolution kernels, while for long-range dependencies, they are obtained by stacking these convolution kernels. Due to the inductive bias of the fully convolutional network itself, capturing long-range dependencies requires passing through multiple deep network layers. This results in low computational efficiency, and the network is difficult to optimize. For example, when the information of two feature points in the feature map needs to be transferred to each other, it will be very difficult. Yu et al. [14] proposed dilation convolution to dilate the receptive field. Although dilation convolution can expand the receptive field while reducing the number of feature map by downsampling, for road segmentation, it can only obtain information from very limited surrounding feature points and cannot obtain long-range context awareness. Conditional random fields (CRFs) [15], which model long-range dependencies, are used to post-process predictions for semantic segmentation. However, doing so brings additional computational overhead.

Some methods in the field of natural language processing have been applied to semantic segmentation and achieved very good results, such as the self-attention-based method [16]. It calculates the response of a position by the weighted sum of the similarity of all positions in the sequence. The non-local attention is also introduced for semantic segmentation in a similar way [17]. Although long-range dependencies can be captured with these methods, the non-local network needs to generate huge attention maps. This leads to a huge time and space complexity of $O\left(N^2\right)$, as shown in Figure 1a. N refers to the total number of spatial feature points within the feature map.

For cross-scale feature aggregation, most of the existing deep-neural-network-based methods sample features of different scales to a uniform size and then combine them by pixel-wise summation or concatenation. We believe that such an aggregation method will lose some details due to the information loss in the up- and down-sampling process.

In order to simultaneously capture the global information of features and efficiently combine the information between different scales, a cross-scale axial attention-based method is proposed in this paper. To reduce the large computational cost in non-local networks, the axial attention mechanism is adopted. Specifically, we first compute the attention map on the vertical axis, followed by the attention map on the horizontal axis. Finally, the network aggregates two feature maps with different spatial resolution sizes in a non-local manner. With the proposed method, each element in the low-resolution feature map contains the global information of the high-resolution feature map, as shown in Figure 1b. We validated our model on the Massachusetts Roads dataset [18] and DeepGlobe dataset [19]. Overall, this paper makes the following contributions:

• A novel deep model is proposed for road segmentation, which adopts a global attention mechanism to exploit the long dependence.
• Our proposed method is able to capture dense contextual information and better combine features at different scales. This results in more accurate road segmentation results.
• The proposed method consumes less computing resources than most other networks.

2. Related Work

2.1. None-Local Network

Non-local networks borrowed ideas from the self-attention mechanism proposed in the field of natural language processing, which has also achieved good results in the field of computer vision [20,21]. Non-local networks are designed to build global relationships [22]. Specifically, each feature point in the feature map is composed of the weighted sum of all the feature points in the whole image. Feature points with higher similarity to this point will have higher weights and vice versa. The non-local neural network was first proposed by Wang et al. [17], inspired by non-local means to construct an operator that can capture long-distance dependencies. DANet [23] constructs a dual attention network to aggregate long-range dependencies in spatial and channel dimensions, respectively. PSANet [24] builds global dependencies by learning adaptive attention masks and designing bidirectional information-propagation paths. OCNet [25] defines pixels of the same class as each other’s target semantics and constructs an attention map by calculating the dependency between each pixel in the feature map and its corresponding target semantics. DFN [26] uses the channel attention mechanism to distinguish feature channels of different importance during the upsampling process to solve the problem of intra-class inconsistency and adds boundary constraints to increase the difference between different classes. GCNet [27] finds that the global context information obtained by different query positions through the non-local structure is almost the same. Therefore, in order to reduce unnecessary computation, a simplified non-local structure is proposed. Chi et al. [28] propose a learnable and data-adaptive bilinear attention transform (BA-Transform). A wide range of local and global attention operations can be modeled in a data-adaptive manner. Different attention operations are often applied to different feature channel groups to resolve the differences between features. Yin et al. [29] decoupled the non-local term into a pairwise term and a unary term and make them independent of each other in learning content and gradient propagation to achieve better performance.

2.2. Axial Attention

Although non-local structures can coordinate global context information, the high computational complexity makes the model very cumbersome and difficult to train. In order to solve this problem, a new axial-recurrent attention mechanism emerges. Specifically, each element in the feature map only performs self-attention to the row and column alternately. At the same time, each element in the feature map can spread the attention range from one row or one column to the whole image with circular axial attention. Axial-DeepLab [30] proposes a position-sensitive axial attention model, in which noteworthy remote dependencies can be precisely captured. CCNet [31] obtains the attention map by paying attention to the contextual information of the cross paths where its pixels are located. By incorporating channel axial attention in calculating the spatial axial attention map, CA [32] greatly relieves the conflict between the previous channel attention and the spatial attention and improves the performance. NN-UNet [33] captures 3D features by using axial attention on the horizontal, vertical, and channel axes of the feature map in the decoder part. Medical Transformer [34] proposes a gated axial attention model, which extends existing architectures by introducing additional control mechanisms in the self-attention module. Furthermore, it operates on whole images and patches to learn global and local features, respectively. CaraNet [35] proposes a context-axis-preserving attention method to improve the segmentation performance for small objects.

3. Method

3.1. Overall Network Framework

The overall architecture of our proposed model is shown in Figure 2. It consists of an initialization layer, a backbone network (CFPNet [36]), the Cross-Scale Axial Attention (CSA) module, and an upsampling layer. The CSA module can combine global attention across different scales during the propagation process. Specifically, the initialization layer initializes the input image as the input features through three consecutive convolution operations and downsamples the length and width to 1/2 of the original size for subsequent computations. Our network backbone is mainly composed of CFP modules [36]. This module captures multi-scale contextual information in the form of parallel stacked asymmetric convolutions. In the process of forward propagation, the feature resolution is reduced layer by layer to capture global information. The CSA module performs information fusion on feature maps of two different scales. Among them, high-resolution features contain more detailed information and local features, while low-resolution features contain more global features, but some details will inevitably be lost. Dense context awareness is captured by continuous vertical and horizontal attention from high-resolution features to low-resolution features. The obtained attention map is added to the original feature map so that the network backbone can capture dense long-range dependencies without losing detailed information during the propagation process. In the upsampling part, we use the deconvolution operation instead of conventional bilinear interpolation to restore the features to the input size layer by layer and output the final road segmentation result. Although this will slightly increase the network parameters, convolutional learning can learn a better fitting method than manual design.

3.2. Backbone

In order to obtain higher performance and reduce the computation burden as much as possible, we adopt the CFP module proposed by Lou et al. [36] as the backbone of the proposed method. In this paper, we stack multiple CFP modules into two CFP clusters. They are called CFP-1 and CFP-2, respectively. As the features pass through these two clusters, the resolution is downsampled to 1/4 and 1/8 of the original image, respectively. CFP-1 consists of CFP modules with expansion ratios of [2, 2]. CFP-2 consists of CFP modules with expansion ratios of [4, 4, 8, 8, 16, 16]. The CFP module is defined as follows:

$$y_i=\sum _{j=1}^{i}f\left(W_{in}x,r_j\right)$$

(1)

$$z=W_{out}\left[y_1,y_2,y_3,y_4\right]+x$$

(2)

where x is the input feature of the CFP module. We assume that the number of channels of x is C. Both $W_{in}$ and $W_{out}$ are a 1 × 1 convolution. $r_j$ is the dilation rate of the jth feature pyramid channel. f represents the mapping function of the feature pyramid channel. $[·,·]$ is the concatenation operation. We first reduce the input dimension from C to C/4 by passing x through $W_{in}$. The features are then sent to four feature pyramid channel modules with different dilation rates. As shown in the green part of Figure 3, their expansion rates are $r_1=1$, $r_2=d/4+1$, $r_3=d/2+1$, and $r_4=d+1$. d is the dilation ratio for the CFP module. Then, we start with the feature map of the second feature pyramid channel and add this feature to the previous features to obtain the final 4 multi-scale feature maps of dimension C/4, that is, $\left[y_1,y_2,y_3,y_4\right]$. Finally, we concatenate these four feature maps together to obtain feature maps of the same dimension as the input, and activate them via $W_{out}$. To prevent the introduction of asymmetric convolutions to make the network deeper and harder to train, we add the input feature x to the output using residual connections [21]. The overall structure of the CFP module is shown in Figure 3.

For the feature pyramid channel, we borrow the idea of inception [37,38,39,40]. We replace the 5 × 5 and 7 × 7 convolution kernels with two and three pairs of asymmetric convolutions, respectively. Furthermore, the number of filters of the first to third pairs of asymmetric convolutions is set to C/16, C/16, and C/8, respectively. In this way, multi-scale feature maps with different receptive fields can be constructed with less computational overhead. In order to further reduce the amount of calculation, we share the convolution kernel and extract features from each pair of asymmetric convolutions and then concatenate them together. Finally, a multi-scale feature map with dimension C/4 is obtained. The calculation process of the feature pyramid channel can be expressed as follows:

$$x_i=w_r{x}_{i-1}$$

(3)

$$f(x,r)=\left[x_1,x_2,x_3\right]$$

(4)

where $w_r$ represents a pair of asymmetric convolutions with dilation rate r. $x_0$ is the input feature and $[·,·]$ is the concatenatin operation.

3.3. CSANet

A schematic diagram of the CSA module is shown in Figure 4. Current multi-scale backbone architectures only use simple concatenation and summation when integrating information between scales. In this process, the information interaction between different scales is not exploited. The information contained in the feature maps of different scales has a different emphasis. Among them, the high-resolution feature maps contain more position and detail information such as texture, geometry, and contour. The low-resolution feature map contains more abstract semantic information, which is a grasp of the image as a whole. The effective fusion of features of different scales can effectively reduce the information loss in the process of forward propagation. Furthermore, for the topological dependence of the road segmentation problem, it is crucial to establish long-range dependencies. The CSA module addresses both issues simultaneously by generating attention maps for low-resolution features and high-resolution features. In this paper, we have established two CSA modules called CSA-1 and CSA-2. Among them, CSA-1 acts on the high-resolution features after the initialization layer and the medium-resolution features after CFP-1. CSA-2 acts on updated medium-resolution features of CSA-1 and the low-resolution features of CFP-2. Specifically, given a high-resolution feature $F_h\in R^{W_h\times H_h\times C_h}$ and a low-resolution feature $F_l\in R^{W_l\times H_l\times C_l}$, similar to self-attention, $F_l$ obtains the query vector $Q_1\in R^{W_l\times H_l\times C_l}$ through a $1\times 1$ convolution. $F_h$ is mapped to the key vector $K_1\in R^{W_h\times H_h\times C_l}$ and the value vector $V_1\in R^{W_h\times H_h\times C_l}$ through two $1\times 1$ convolutions. To align the channel dimensions for computing the attention map, we map the channel dimensions from $C_h$ to $C_l$ when computing K and V. Next, we put axial attention to the longitudinal axis of these two features. To align their widths, we upsample the horizontal axis of $Q_1$ from $W_l$ to $W_h$ by asymmetric deconvolution and obtain an updated $Q_1\in R^{W_h\times H_l\times C_l}$. $Q_1$ and $K_1$ are then multiplied and normalized by softmax to obtain the similarity matrix $S_1\in R^{W_h\times H_l\times H_h}$. Specifically, we obtain the similarity matrix as follows:

$$S_1\left(x_h,y_l,y_h\right)=\frac{e^{\left(Q_1{\left(x_h,y_l\right)}^{\top }\times K_1\left(x_h,y_h\right)\right)}}{{\sum }_{y=1}^{H_h}\left(e^{\left(Q_1{\left(x_h,y_l\right)}^{\top }\times K_1\left(x_h,y\right)\right)}\right)}$$

(5)

Any element $\left(x_h,y_l,y_h\right)$ in $S_1$ represents the similarity between the element $\left(x_h,y_l\right)$ in $F_2$ and $\left(x_h,y_h\right)$ in $F_1$. Then, multiplication of $S_1$ and $V_1$ is performed to obtain the column-dependent attention feature map $M_1\in R^{W_h\times H_l\times C_l}$ of $F_2$ for $F_1$. The specific calculation process of $M_1$ is expressed as

$$M_1\left(x_h,y_l\right)=\sum _{y=1}^{H_h}\left(S_1\left(x_h,y_l,y\right)\times V_1\left(x_h,y\right)\right)$$

(6)

At this point, each element in $F_2$ is composed of the weighted sum of the corresponding columns in $F_1$; that is, each element in $F_2$ contains the entire column context information of the corresponding column in $F_1$.

To enable $F_2$ to obtain global contextual information about $F_1$, we calculate the vertical axis attention map based on the horizontal axis attention map. We map $F_2$ to a query vector $Q_2\in R^{W_l\times H_l\times C_l}$ through a convolution of $1\times 1$ and pass $M_1$ through two $1\times 1$ convolutional layers to obtain $\left\{K_2,V_2\right\}\in R^{W_h\times H_l\times C_l}$. Then, a similar operation on the horizontal axis can be performed as the vertical axis. The similarity matrix $S_2\in R^{W_h\times W_l\times H_l}$ is obtained by multiplying $Q_2$ and $K_2$ and going through softmax. The calculation process of $S_2$ is as follows:

$$\begin{array}{c}\hfill S_2\left(y_l,x_l,x_h\right)=\frac{e^{\left(Q_2{\left(x_l,y_l\right)}^{\top }\times K_2\left(x_h,y_l\right)\right)}}{{\sum }_{x=1}^{W_h}\left(e^{\left(Q_2{\left(x_l,y_l\right)}^{\top }\times K_2\left(x,y_l\right)\right)}\right)}\end{array}$$

(7)

Wherein any point $\left(y_l,x_l,x_h\right)$ in $S_2$ represents the similarity between the element $\left(x_l,y_l\right)$ in $F_2$ and the element $\left(x_h,y_l\right)$ in $M_1$. Finally, we multiply $S_2$ and $V_2$ to obtain the global attention feature map $M_2\in R^{W_l\times H_l\times C_l}$ of $F_2$ for $F_1$. The calculation process of $M_2$ is as follows:

$$M_2\left(x_l,y_l\right)=\sum _{x=1}^{W_h}\left(S_2\left(y_l,x_l,x\right)\times V_2\left(x,y_l\right)\right)$$

(8)

Since we perform the axial attention of the vertical axis initially, each element of $F_2$ contains the context information of the corresponding column of $F_1$, as shown in Figure 4b. We then perform axial attention on the horizontal axis, as shown in Figure 4c. Therefore, each element contains contextual information of the entire row and each element in a row contains contextual information of an entire column. In this way, each element contains global contextual information. At the same time, compared with the non-local calculation method shown in Figure 1b, time and space complexity is reduced from $O(n\times N)$ to $O\left(\sqrt{nN}(\sqrt{N}+\sqrt{n})\right)$, where n and N represent the total number of elements in the $F_2$ and $F_1$ along spatial dimensions, respectively.

The proposed method captures and combines dense contextual information across scales through axial attention. Compared to convolutional networks, we have a larger receptive field and denser context awareness. Each feature point in the feature map is related to the entire feature map. This is critical for extremely topological targets such as roads. It can ensure that a connection can be established between any two points on a long and narrow road, rather than only relying on local features. For example, for roads covered by trees, we can combine distant road features to make predictions without the limitation that convolutional networks can only extract local features. Compared to non-local networks, our model has fewer parameters and requires less computation. By stacking axial attention, we reduce the computational burden effectively without losing accuracy. Additionally, our method combines information across different scales more effectively. Each element in the low-resolution feature is obtained by weighting and summation of the high-resolution features. Thus, the information loss during the downsampling process of the image is reduced.

4. Experiments

4.1. Datasets

4.1.1. Massachusetts Roads Dataset

The Massachusetts Roads dataset is provided from V. Mnih’s doctoral dissertation [18]. It is a high-quality aerial road imagery dataset, which includes urban, suburban, rural, and other areas in Massachusetts. The total sampling area is over 2600 square kilometers. The size of each image is 1500 × 1500, and the spatial resolution is 1 m. Its training and test sets contain 1108 and 63 images, respectively. We split the image into small images at equal intervals in order to use a larger batch size to improve training effectiveness. We first upsampled the training set to 1536 × 1536 size by bilinear interpolation and crop them to small images of size 512 × 512 with a stride of 256. In this way, each training image is cropped to obtain 25 small images. At the same time, the occluded images were removed from the dataset to avoid negatively impacting the network training. For the test sets, we crop them into small images of 512 × 512 with a stride of 512. This leaves no overlapping area between each two small cropped images. Each image is divided into nine small images. Finally, the training set and test set consist of 21,951 and 567 images, respectively. The dataset is available from https://www.cs.toronto.edu/~vmnih/data/ (accessed on 23 November 2022).

4.1.2. DeepGlobe Dataset

This dataset is presented by the deepglobe2018 challenge. The data cover urban and rural areas in India, Indonesia, and Thailand. The DeepGlobe dataset contains 6226 images with ground truth. The size of each image is 1024 × 1024, and each pixel contains an area of two square meters. We use 4696 of them as the training set and the remaining 1530 as the test set. Similarly, we divide the training set into 9 512 × 512 small images with stride of 256, and divide the test set into 512 × 512 small images with stride 512. In the end, we obtained 42,264 images as the training set and 6120 images as the test set. The dataset is available at https://competitions.codalab.org/competitions/18467 (accessed on 23 November 2022).

4.2. Implementation Details

All our experiments adopted the following settings. We used two RTX 3090 GPUs to train the model under the framework of pytorch. Only simple random rotation and random flip were used for data augmentation. The optimizer for training the network was Adam, and the initial learning rate was set to 0.001. We updated the learning rate using a step-learning scheduler with an interval of 30. The learning rate was reduced to 0.1 times the original for each update. The loss function was a binary cross-entropy loss. The batchsize was set to 16, and the model was trained for a total of 100 epochs.

4.3. Metrics

Since the road width in the ground truth of Massachusetts is consistent, it does not match the real scene. Therefore, we used relaxed IoU (IoU${}^r$) to evaluate the results [41]. The relaxation factor is 4. The Precision, Recall, IoU, Accuracy and F1 score were also used to evaluate the model’s performance.

5. Results

5.1. Comparison with Other Classic Methods

In this section, we quantitatively and qualitatively compare the proposed method with other classic methods. Next, we briefly introduce the comparison methods.

1.

UNet [42]: This method adopts an encoder–decoder structure and captures local information through small convolution kernels. In the decoder part, the corresponding encoder features and decoder features are concatenated and the resolution is gradually recovered through a deconvolution operation.

2.

SegNet [43]: SegNet captures local information by stacking convolutional layers, and a lot of convolution operations and unpooling are used to restore the resolution in the decoder part.

3.

PSPNet [44]: The PSPNet method is based on multiscale feature fusion and introduces more contextual features through a global mean pooling operation. It downsamples features to different scales and reduces the feature dimension through 1 × 1 convolutional layers.

4.

ENet [45]: ENet increases the receptive field by atrous convolution and reduces information loss effectively by combining downsampling and pooling.

5.

DeepLabv3 [46]: This method captures multiscale information with parallel atrous convolution at different sampling rates through the Atrous Spatial Pyramid Pooling (ASPP) module.

6.

CFPNet [36]: CFPNet expands the receptive field with a small number of parameters through parallel asymmetric atrous convolution.

5.2. Segmentation Results on Massachusetts Roads Dataset

The quantitative results for the Massachusetts Roads dataset are summarized in

Table 1. Qualitative comparison of the proposed method (CSANet) with other methods on the Massachusetts Roads dataset.

Method	Recall	Precision	IoU	IoU${}^{\mathit{r}}$	Accuracy	F1
Unet [42]	68.15	78.23	57.29	84.49	98.06	72.84
Enet [45]	66.08	80.68	57.05	84.57	97.10	72.65
DeepLabv3 [46]	66.57	75.93	54.97	83.69	97.92	70.95
CFPNet [36]	67.73	75.89	55.73	85.10	97.94	71.57
SegNet [43]	66.47	78.30	56.13	82.84	98.02	71.90
PSPNet [44]	66.19	74.37	53.90	85.14	97.84	70.04
CSANet	69.14	80.05	58.98	84.95	98.16	74.20

. All the evaluation metrics of methods such as UNet and SegNet that capture local information by stacking convolution kernels are lower than the proposed CSANet. This shows that the long-range dependence in road segmentation is difficult to accomplish through the network structure in the form of stacked convolution kernels.

All evaluation metrics of PSPNet except IoU${}^r$ are lower than the proposed method. We argue that simply concatenating features at different scales together cannot capture their intrinsic connections. However, our CSANet captures their connections through dense attention operations between scales, resulting in more accurate segmentation results.

Atrous convolution-based methods such as ENet, DeepLabv3, and CFPNet can expand the receptive field and reduce the parameters by downsampling. The performance of ENet is lower than the proposed CSANet in all metrics except Precision. All metrics of DeepLabv3 are lower than the CSANet. All metrics of CFPNet except IoU${}^r$ are lower than those of our method. These methods fail to capture dense contextual information. However, our CSANet can capture dense contextual information through a cross-scale axial attention mechanism. Overall, for the Massachusetts Roads dataset, the proposed method achieves the best performance on Recall, IoU, Accuracy, and F1. This shows that our proposed CSANet can capture global context information and exploit more information by cross-scale fusion.

For qualitative analysis, we present the segmentation results of the proposed CSANet with UNet, ENet, DeepLabv3, CFPNet, SegNet, and PSPNet of the Massachusetts Roads dataset. As shown in Figure 5, the red boxes highlight the most obvious differences between different methods. In the small-town road scene in the second row of Figure 5, some parts of large area of the road are blocked by trees. UNet, SegNet, PSPNet, and atrous convolution-based networks all fail to segment roads correctly, whereas the proposed CSANet infers the correct result by the correlation of this location with all other locations.

Overall, all methods can segment most of the roads successfully. However, the segmentation results of comparison methods have breakpoints or noise in some complex road segments. This shows that traditional convolutional networks cannot extract long-distance information of roads due to the limited receptive field of small convolution kernels. It can only identify whether a pixel block belongs to a road by some road features. This leads to the incoherent and false detection of roads. Our proposed CSANet is able to extract both global and cross-scale information. The fusion of multi-scale features is more conducive to the network to grasp the integrity and continuity of the road.

5.3. Segmentation Results on DeepGlobe Dataset

The quantitative results for the DeepGlobe dataset are shown in

Table 2. Qualitative comparison of the proposed method (CSANet) with other methods on the DeepGlobe dataset.

Method	Recall	Precision	IoU	IoU${}^{\mathit{r}}$	Accuracy	F1
Unet [42]	57.23	78.15	49.33	69.15	98.48	66.07
Enet [45]	54.50	79.09	47.64	69.35	97.43	64.53
DeepLabv3 [46]	76.08	78.65	63.06	83.14	98.09	77.34
CFPNet [36]	76.25	80.78	64.54	83.90	98.20	78.45
SegNet [43]	61.01	74.40	50.43	67.18	97.43	67.04
PSPNet [44]	69.38	77.86	57.95	77.86	97.84	73.38
CSANet	75.05	83.37	65.28	84.17	98.29	78.99

. For the DeepGlobe dataset, the conclusions are generally consistent with those of the Massachusetts Roads dataset. However, the Accuracy of UNet is slightly higher than that of our method. This is due to its strong robustness that results in improved background segmentation. Furthermore, the Precision metric of ENet is lower than that of the proposed method. The IoU${}^r$ of DeepLabv3 is lower than thta of the proposed method. This shows that the proposed CSANet can generalize to other types of data without degrading the segmentation accuracy. For the DeepGlobe dataset, the CSANet achieved the best performance on Precision, IoU, IoU${}^r$, and F1.

The visual segmentation results of the DeepGlobe dataset are shown in Figure 6. For the forest road scene in the fourth row of Figure 6, the small road in the red box at the upper right is difficult to identify because it is too narrow and blocked by trees. The proposed CSANet is able to aggregate global information across scales. Thus, details are not lost during downsampling, and the road can be segmented correctly.

5.4. Segmentation Maps

Figure 7 shows the segmentation visualization results of our proposed CSANet and CFPNet. It can be seen that CFPNet has obvious jagged lines on the edge of the road (it is best to zoom in on the picture to view). In contrast, the segmentation results of our method have smooth edges and natural transitions. This shows that our method is able to capture dense contextual information. Furthermore, our proposed attention-based cross-scale information fusion method can make the high-level features in the deep layers of the network not lose the details in the shallow layers. Although CFPNet captures global information through the channel feature pyramid, the limitation of the size of the convolution kernel causes it to only capture long-range discrete information. Our method not only achieves optimal performance on evaluation metrics but also performs well on actual segmentation maps.

5.5. Ablation Studies

An ablation study is performed to verify the effectiveness of the CSA module.

Table 3. Ablation study of the CSA module on the Massachusetts Roads dataset.

No.	CSA-1	CSA-2	Recall	Precision	IoU	F1
1			67.32	79.5	57.36	72.9
2	✓		68.67	79.96	58.59	73.89
3		✓	67.86	79.85	57.94	73.37
4	✓	✓	69.14	80.05	58.98	74.20

and

Table 4. Ablation study of the CSA module on the DeepGlobe dataset.

No.	CSA-1	CSA-2	Recall	Precision	IoU	F1
1			73.97	83.38	64.46	78.39
2	✓		74.54	83.49	64.96	78.76
3		✓	74.06	83.28	64.48	78.40
4	✓	✓	75.05	83.37	65.28	78.99

show the quantitative results on the Massachusetts Roads dataset and the DeepGlobe dataset, respectively. The baseline (No. 1) is the network built with the CFP module. No. 2 and No. 3 indicate that the CSA-1 module and CSA-2 module are added to the baseline, respectively. The results show that both CSA-1 and CSA-2 can improve the baseline network. Compared with CSA-2, CSA-1 improves the network performance more significantly. The CSA-1 module acts on the high-resolution feature maps of the first downsampling and the medium-resolution feature maps of the second downsampling. Similarly, the CSA-2 module acts on medium-resolution feature maps and low-resolution feature maps. This shows that our CSA module is more effective in the initial stage of network propagation, i.e., when the size of the feature maps is large. No. 4 means that both the CSA-1 module and the CSA-2 module are introduced. At this point, the network achieves the highest performance. The above results show that our method is able to combine high-resolution and low-resolution information effectively.

Figure 8 shows the qualitative results on the Massachusetts Roads dataset and the DeepGlobe dataset. The red box is used to highlight the difference between different segmentation models. In the second row of Figure 8, for the parking lot scene of the Massachusetts Roads dataset, our CSANet (No. 4) recognized the interior narrow passages of parking lots that were not marked in the ground truth. In contrast, No. 1 did not recognize the inner passage of the parking lot at all. No. 2 only identified some road debris. No. 3 identified a few internal passages of the parking lot. With the introduction of CSA-1 and CSA-2, the proposed method could capture and combine both high-level and low-level semantics information effectively. As a result, details that could be captured by the baseline network (No. 1) were identified. In the fourth row of Figure 8, in the highway scene of the DeepGlobe dataset, due to the occlusion of trees and shadows on both sides of the road, the pixels of the road within this range were different from other parts, which made the road difficult distinguish. Networks No. 1 and No. 2 were barely able to identify the segment. With the support of CSA-1, No. 3 successfully captured the high-level semantic features of the road section and gave the correct segmentation results. Likewise, Network No. 4 gave a more complete segmentation map with the help of CSA-1 and CSA-2. The above qualitative results demonstrate the effectiveness of the proposed CSA module.

5.6. Computational Complexity Analysis

Table 5. Computation complexity and parameter sizes of different methods.

Method	IoU	Param	FLOPs
UNet	57.29	28.2 M	238.53 G
ENet	57.05	0.36 M	2.25 G
DeepLabv3	54.97	54.7 M	82.98 G
CFPNet	55.73	0.54 M	3.89 G
SegNet	56.13	29.48 M	562.56 G
PSPNet	53.90	46.71 M	184.44 G
CSANet	58.98	0.62 M	7.55 G

summarizes the computational cost of different methods and their IoU accuracy on the Massachusetts Roads dataset. The proposed CSANet achieves the best performance with a relative small number of parameters and computation burden.

Figure 9 visually shows the relationship between the computation burden and the IoU accuracy on the Massachusetts Roads dataset. The area of a circle is proportional to the computational complexity. ENet has a small computational load due to its use of small convolution kernels throughout the process. CFPNet and CSANet greatly reduce the amount of computation by introducing depth-wise separable convolutions. Furthermore, our CSANet uses axial attention to compute the attention map. Therefore, the computation burden is reduced further. DeepLabv3 also uses depth-wise separable convolution in Xception, so it has a smaller amount of computation. SegNet has a huge computation complexity because it has a large number of convolution operations in the encoding and decoding parts.

Figure 10 visually shows the relationship between the parameter size and the IoU accuracy on the Massachusetts Roads dataset. The area of each circle is proportional to its parameter amount. UNet brings more parameters due to its wide network structure (the number of channels increases exponentially during downsampling). The SegNet uses a large number of convolution operations in the encoding and decoding part, which also brings more parameters. PSPNet has a large number of parameters because it aggregates features at multiple scales. Furthermore, due to the existence of the ASPP module in DeepLabv3, the computation of multiscale, features with different receptive fields brings additional parameters. ENet uses asymmetric convolution to reduce the number of parameters and atrous convolution to expand the receptive field. CFPNet reduces the number of parameters by stacking parallel asymmetric depthwise convolutions. The CSANet also uses asymmetric convolution to reduce the parameter size. The CSANet achieves state-of-the-art performance with a comparable number of parameters to lightweight networks such as ENet and CFPNet.

6. Conclusions

In this paper, a novel cross-scale axial attention network is proposed. This method can exploit the high-resolution fine-grained information and low-resolution coarse-grained information with a layer-by-layer mechanism, and axial attention is introduced to extract the global contextual information. This enables the proposed method to produce road segmentation maps with higher integrity and continuity. The evaluation results based on the Massachusetts Roads dataset and DeepGlobe dataset validate the fact that our method outperforms classic segmentation algorithms.