Multi-Attention-Based Semantic Segmentation Network for Land Cover Remote Sensing Images

Abstract

Semantic segmentation is a key technology for remote sensing image analysis widely used in land cover classification, natural disaster monitoring, and other fields. Unlike traditional image segmentation, there are various targets in remote sensing images, with a large feature difference between the targets. As a result, segmentation is more difficult, and the existing models retain low accuracy and inaccurate edge segmentation when used in remote sensing images. This paper proposes a multi-attention-based semantic segmentation network for remote sensing images in order to address these problems. Specifically, we choose UNet as the baseline model, using a coordinate attention-based residual network in the encoder to improve the extraction capability of the backbone network for fine-grained features. We use a content-aware reorganization module in the decoder to replace the traditional upsampling operator to improve the network information extraction capability, and, in addition, we propose a fused attention module for feature map fusion after upsampling, aiming to solve the multi-scale problem. We evaluate our proposed model on the WHDLD dataset and our self-labeled Lu County dataset. The model achieved an mIOU of 63.27% and 72.83%, and an mPA of 74.86% and 84.72%, respectively. Through comparison and confusion matrix analysis, our model outperformed commonly used benchmark models on both datasets.

1. Introduction

Semantic segmentation is a difficult task in remote sensing image analysis and processing [1,2]. As a classical computer vision problem, full-pixel semantic segmentation means that each individual pixel in an image is assigned a different class ID according to the object of interest to which it belongs. Such algorithms have been deployed in various fields, especially resource planning, and are proven to be an improvement. In recent years, along with the rapid development of related technologies and optical sensors in the field of remote sensing, the cost of acquiring remote sensing images is now lower, the resolution is higher, and the information contained in remote sensing images is richer [3], which creates better conditions for remote sensing image information extraction, in conjunction with the demand of researchers to extract detailed information from the region of interest in remote sensing images gradually increasing, and the combination of computer vision and remote sensing fields becoming more and more intense [4]. Land use cover is gradually becoming one of the hot spots [5], or LUCC for short, and is currently at the center of global environmental change assessments [6].

For the realization of remote sensing image land use classification, the target remote sensing image must be segmented at the pixel level. The most accurate method to get land feature classification information is manual annotation; however, this method is costly and inefficient, and cannot be applied to a large amount of data [7]. Some intuitive methods have been used earlier to reduce the workload. Semantic segmentation, for example, is usually based on low-level image features such as gray level and texture information, threshold-based segmentation, edge-based segmentation, region-based image segmentation [8], and specific theory-based image segmentation, which use the low-order visual information of images themselves, such as grayscale information and edge information, but often only extract the low-level features of the image, and it is difficult to cope with data with richer feature information and more image classes. Therefore, traditional segmentation methods are only applicable to specific tasks and rely too much on expertise for setting hyperparameters, which are not very generalizable and not very efficient [9].

In modern times, deep convolutional neural networks (CNNs) [10] have been widely used in remote sensing image analysis tasks and show good performances in segmentation tasks due to their powerful recognition ability [11]. Advances in remote sensing technology enable us to obtain higher quality images, creating sufficient conditions for the integration of this field with deep learning [12,13]. Still, unlike the semantic segmentation of traditional images, the land cover classification based on deep learning is still a challenging task [14], and the main reasons are as follows:

(1)

Unlike ordinary images, the composition of remote sensing images is more complex, in which there are many kinds of targets and the features between different targets that vary greatly, so it is more difficult to carry out segmentation compared with ordinary scenes [15].

(2)

The large differences in imaging conditions can easily cause objects with the same semantic classification ID to show different features at different spatial locations, which can also easily produce more noise when conducting imaging. For example, highways with regular shapes and colors and rural roads have different features, but they all belong to the same semantic concept [16].

(3)

The network model with a high classification accuracy usually has a complex structure, high model complexity, and slow inference speed, which are not conducive to application in practical scenarios, so it is necessary to trade off between algorithm complexity and segmentation accuracy [17].

In summary, a powerful segmentation model should not only be able to accurately identify the semantic information of the target, but also enhance the edge extraction of adjacent targets, which puts higher demands on the feature extraction effect.

The powerful feature representation and learning capability of convolutional networks are more suitable for more centralized processing of large-scale data, saving labor and improving recognition efficiency. Deep neural networks have strong spatial context information mining abilities and better feature fusion effects compared with traditional methods, so more and more researchers use deep learning as the preferred method.

FCN is a framework for image semantic segmentation proposed by Jonathan Long et al. in 2015 [18]. FCN replaces all the fully connected layers of CNN with convolutional layers, the output of the network is a heat map, and the size of the input image is not limited. SegNet [19] is also based on FCN and modifies the VGG-16 network to obtain a semantic segmentation network, which is more accurate than the traditional model. DeepLab v3+ [20] is a classical architecture, which features the elimination of repeated upsampling and proposes a multi-scale structure based on extended convolution; dilated convolution is introduced to balance model accuracy and time consumption. Ronnerberge et al. propose U-Net [21], a U-type network based on an encoder and decoder, using skpi-connections to recover lost information during downsampling. The PSPNet [22] proposed based on a pyramid pooling module for scene parsing is also commonly used. This network can aggregate contextual information from different regions to mine global contextual information.

In order to investigate the application of deep learning in the task of high-resolution remote sensing image segmentation and to gain a comprehensive understanding of the latest developments in this field, we conducted a search on the Google Scholar, WoS, and Scopus databases using the keywords “high-resolution remote sensing image”, “deep learning”, “semantic segmentation”, “network architecture”, and other related terms. By combining the search results of these databases, we collected a significant number of methods based on deep learning for the segmentation of high-resolution remote sensing images. UNet++ [23] appears in which densely nested skpi-connections are used to connect the encoder and decode, and is supplemented with a deep supervision mechanism to accelerate network training convergence, but its use of large intermediate convolution leads to computational costs and dense skip-connections introduce some redundant information. FarSeg [24], proposed by Zheng et al. in 2020, addresses the problem of false positives and foreground-background imbalance using a relationship-based and optimization-based foreground modeling approach, where the relationship-based approach is somewhat similar to the self-attentive mechanism, and the approach is worthwhile. Chen proposed Trans-UNet [25] in 2021 and, through the global-based self-attention mechanism and skip connection, it can effectively obtain the overall information, so that the model can focus on more detailed features to identify fine spatial information. Attention-UNet [26] focuses more on the focused regions of the image, but lacks attention to semantic concepts, causing the encoder and decoder a large semantic gap between the feature maps, which may not capture different levels of features during decoding. SegFormer [27] is an improved approach based on Transformer, which enables semantic segmentation of images by dividing them into a series of blocks, each of which is processed by a set of Transformer encoders and decoders. The model uses a self-attentive mechanism to handle spatial relationships in images. In remote sensing image segmentation, the frequently used single-scale convolutional kernels can limit the range of feature extraction, so some researchers proposed the multi-scale full convolutional network (MSFCN) [28], which has multi-scale convolutional kernels, a channel attention block (CAB), and a global pooling module (GPM), thus improving the performance and stability of the model. The DIResUNet [29] model is based on UNet and consists of an initial module, an optimized residual block, and a dense global spatial pyramid pooling (DGSPP) module. The model further improves the performance and stability of the model by extracting multi-level features and extracting global information in parallel, which is an efficient overall approach. DPPNet [30] is an efficient image segmentation model which consists of a deep pyramid pool (DPP) block and a dense block of deep multi-expansion residual connections. The model takes full advantage of the dense pyramid structure, while considering multiple levels of features in prediction.

Based on the aforementioned discussion, because the high-resolution remote sensing images are richer in feature types and details compared with ordinary images, there is an unbalanced target distribution. In addition, some targets with small sizes or small sample sizes are easily misclassified, which increases the classification difficulty of the network. The features in remote sensing images have complex structures and rich texture information, and there are some cases with the same target class but large feature differences and some with different target classes but similar features, which put forward higher requirements on the feature extraction and abstraction ability of the network. In this study, we adopt UNet as the benchmark model and combine various attention mechanisms, and the attention fusion module for feature fusion at skip-connection, to improve the generalization ability of the model and to better cope with these problems. Due to the rich contextual information of remote sensing images, the algorithm requires upsampling of non-uniformly sampled data. Therefore, in this paper, we adopt a different upsampling operator from the traditional method, CARAFE (Content-Aware Reassembly of Features), which can adaptively adjust the size and shape of the convolution kernel, and, by learning the interrelationship of features, can better adapt to non-uniformly sampled data and improve the segmentation performance of the network with less computation to optimize the accuracy of target edge segmentation.

(1)

Based on the coordinated attention mechanism, the residual network was used in the backbone network to alleviate the problems of gradient dispersion and gradient explosion as the network deepened. At the same time, the generalization performance and fine-grained feature extraction ability of the backbone network were improved without increasing the amount of calculations.

(2)

We propose an attention fusion module for skip-connection to improve the network feature fusion capability.

(3)

We use a content-aware reorganization module CARAFE [31] in the decoder instead of the traditional upsampling method to improve the contextual information aggregation capability without increasing the computational effort.

2. Methods

This model utilizes the UNet architecture as the baseline for its structure design. The UNet model is symmetrical in structure and follows the classical encoder-decoder structure. After the encoder performs successive convolution and downsampling, a feature map with a small resolution but condensed high-dimensional semantic representation is generated, and the upsampling part is called expanding path, which actually localizes the target object and is symmetric with the downsampling part. Additionally, each layer uses the copy and crop method to complete the skip-connection operation, and the segmentation result with the same size as the original image is obtained after continuous upsampling. The combination of high-dimensional and low-dimensional feature maps makes the information extracted by the model more comprehensive, enabling the model to be more sensitive to detailed information. The skip-connection operation directly concatenates the more accurate gradient, point, line, and other information in the encoder of the same layer into the decoder of the same layer.

When applying UNet to remote sensing image segmentation, several drawbacks arise. Firstly, its architecture, based on a full convolutional neural network and multiple upsampling and downsampling operations, incurs a significant amount of memory and computational overhead. The resolution degradation caused by scaling is not conducive to improving the segmentation effect. Secondly, convolution and pooling operations tend to cause the loss of boundary information. Finally, the baseline model lacks corresponding measures to cope with the noise in the image. This task is challenging when applied to remote sensing images with a large number of targets with complex features.

To address these problems, this study introduces multiple attention mechanisms applied at different locations in the network. Additionally, an efficient upsampling method is embedded to propose a UNet based on CARAFE and multiple attention mechanisms, which we call CA-UNet.

2.1. Structure of CA-UNet

In CA-UNet, we use the residual block and coordinate attention module to construct the encoder to improve the fine-grained extraction ability of the target in the backbone network, and to reduce the detail loss caused by the pooling layer during downsampling. For the downsampling operation, this model uses a 3 × 3, step 2 convolution for the downsampling operation to improve the feature extraction capability of the model. Instead of the traditional upsampling method, CA-UNet uses the CARAFE operator in the decoder. Attention fusion modules are also constructed to combine spatial and channel attention fusion features in different feature layers. The overall network structure is shown in Figure 1. The functionality and construction method of each module are described below.

2.2. Residual Encoder Based on Coordinate Attention

Deep networks have a higher error convergence rate compared to shallow networks, which can lead to degradation problems. Hence, adding more layers to the model may sometimes result in reduced performance. The residual block [32] solves these problems without increasing the number of parameters and the computational complexity of the model. Part of the input in this structure is not passed through the convolutional network to the output, which retains part of the shallow information and avoids the loss of feature information due to the deepening of the feature extraction network; previous experiments also show that the identity shortcut connection structure can achieve better results. The specific process is shown in Figure 2. Batch normalization is a popular regularization technique used in deep neural networks to normalize inputs in each batch, resulting in better fitting of the activation function of the network, reduced gradient disappearance or explosion, accelerated model training, and improved model accuracy. In residual networks, BN is often combined with the ReLU activation function to address the problem of gradient disappearance caused by ReLU’s gradient of 0 in the negative part. Furthermore, BN can mitigate model instability that may arise when the gradient of ReLU activation becomes large due to large input, and limit the input range to a suitable range, thereby improving model robustness and generalizability.

However, only utilizing the residual module in the backbone does not effectively improve the performance of the encoder. When we consider the computational effort and the network performance, the attention module is an essential part used to tell a model “what” and “where” to attend and it has been widely used in various deep learning models to improve network performance; the most commonly used attention modules [33] are squeeze-and-excitation (SE) attention [34] and CBAM [35]. SE block has been extensively applied in remote sensing image segmentation tasks in recent years. However, the SE block design only considers the influence of channel relationships on features, ignoring spatial location information. In remote sensing image segmentation tasks, spatial location information is crucial for generating spatially selective attention maps. The lack of spatial location information modeling may lead to the inability of attention maps to accurately represent the spatial relationships between pixels, thus affecting the segmentation accuracy. Although CBAM considers information in both spatial and channel dimensions, it is computationally intensive and increases the model’s complexity and computational cost, making it less suitable for scenes requiring high computational resources. Additionally, CBAM performance is highly dependent on the convolutional layer input, and for some special scenes or irregular remote sensing image inputs, it may not be able to extract features accurately or make effective adjustments, resulting in degraded image segmentation performances. We introduce a lightweight coordinate attention module [36], which is different from the previous mechanism. Coordinate attention is an attention mechanism that can focus on specific locations in remote sensing images that require special attention. It has been shown to improve the recognition of multi-scale target information and enhance model robustness by adjusting the weights for each location and channel to enhance the variability of features. Additionally, it does not add a significant computational cost, making it a practical choice for various applications. By using coordinate attention, a model can make more comprehensive use of image information.

As shown in Figure 3, for the coordinate information embedding part, given any input tensor, the module uses two pooling kernels (H, 1) and (1, W) to encode and calculate each channel along the horizontal and vertical coordinates, and aggregates features in the spatial direction through the transformation of the two directions to generate a pair of feature maps. For each pixel point of the feature map, a coordinate vector is generated to encode its position within the feature map. This coordinate vector is then embedded into the feature vector using a fully connected layer to enhance the location information’s influence on the feature representation. The resulting feature vector and the location-embedded coordinate vector are then concatenated and embedded into a vector of dimension C using another fully connected layer. This vector represents the pixel point’s weight on different channels. The channel-embedded vector is then normalized using a function to obtain the weight of the pixel on all channels, which is subsequently multiplied with the feature vector to obtain the weighted feature vector of the pixel. This process enables the network to learn which locations require special attention during computation. The resulting weighted feature maps are globally pooled and averaged over the channel dimensions to obtain the average of each channel. Next, the weights of each channel are obtained through two fully connected layers and the sigmoid function. Finally, these weights are multiplied with the feature maps to obtain the weighted feature maps, allowing the network to learn which channels require special attention during computation. The resulting weighted feature maps are then concatenated in the channel dimension to obtain the final feature maps. This approach captures the dependence of one spatial direction while retaining the precise information of the other, which helps the model to locate the desired area of attention. In the coordinated attention generation stage, the second transformation is used to encode location information using the globalized receptive field, which satisfies the following three conditions. Firstly, this transformation should be as simple as possible to ensure that it does not increase the computational burden; secondly, to improve the information extraction ability and conversion efficiency, the information captured in the previous stage should be fully utilized; finally, the relationship between the channels should not be ignored. This encoding process allows us to coordinate attention to more accurately locate the exact position of the target object, thus facilitating better recognition throughout the model.

2.3. Feature Fusion Decoder Based on Attention Mechanism and CARAFE

During the down-sampling process, the resolution of the image is gradually reduced to obtain image information of different scales. This allows the network to extract features at different levels of abstraction, starting from low-level information, such as points, lines, and gradients in the underlying features, and gradually moving towards more abstract information such as contours and shapes. The entire network combines these “fine to coarse” features to obtain a comprehensive representation of the image. In the decoding stage, the upsampling operator is used to restore the feature map to its original size, enabling the network to generate precise predictions. Feature upsampling is a crucial component of modern convolutional network architecture and is particularly important in tasks such as instance segmentation. The methods commonly used in previous studies for upsampling, namely bilinear interpolation, deconvolution, and transposed convolution, have limitations when applied to feature-rich and detailed images such as remote sensing images. Bilinear interpolation uses a weighted average of neighboring pixels, but this method fails to capture the intricate texture information in the image, resulting in a less-detailed image after upsampling. Furthermore, bilinear interpolation is a global upsampling method that applies zoom to all pixels of the image, potentially introducing noise or artifacts that can degrade the accuracy of remote sensing image segmentation. Deconvolution, while capable of producing high-quality upscaled images, requires many reverse convolution operations and is computationally expensive for larger input images used in remote sensing image segmentation. Moreover, the upsampling process may cause pixel misalignment, further degrading image quality. Transposed convolution also suffers from computational intensity and produces a tessellation effect or high-frequency noise in the reconstructed high-resolution images. Therefore, alternative methods, such as sub-pixel convolution, nearest-neighbor interpolation, and pixel shuffle, have been proposed to address these issues in remote sensing image segmentation. In this model, the lightweight CARAFE [31] is used as the upsampling operator. Compared to traditional upsampling methods, CARAFE can effectively suppress the tessellation effect and noise. By introducing an asymmetric convolution kernel and deformable convolution operation, it can better preserve the spatial feature information of the input feature map and improve the quality of the reconstructed image. Instead of manually specifying the output size, the output size is adaptively adjusted according to the input feature map, making it more flexible for different remote sensing image segmentation tasks of varying sizes. Additionally, it boasts a higher computational efficiency.

As shown in Figure 4, given a feature mapping χ of size C × H × W and an upsampling rate σ, CARAFE will generate a new feature mapping of size C × σH × σW for the new feature mapping ${\chi }^{\prime }$. For each target position, ${\chi }^{\prime }$, of the output, $l^{\prime }=\left(i^{\prime },j^{\prime }\right)$, there is a corresponding position, $l=\left(i,j\right)$, at the input, where $i=\left[i^{\prime }/\sigma \right]$ and $j=\left[j^{\prime }/\sigma \right]$, and here $N\left({\chi }_l,k\right)$ represents the neighborhood of k × k size centered at position $l$. The kernel prediction module predicts an associated position, ${\omega }_{l^{\prime }}$, for each position, $l^{\prime }$, based on the neighbors of ${\chi }_l$, as Equations (1) and (2) show:

$${\omega }_{l^{\prime }}=\psi \left(N\left({\chi }_l,k_{encoder}\right)\right)$$

(1)

$${\chi }_{l^{\prime }}^{\prime }=\mathsf{\varphi }\left(N\left({\chi }_l,k_{up}\right),{\omega }_{l^{\prime }}\right)$$

(2)

Each location on χ corresponds to ${\sigma }^2$ target locations on ${\chi }^{\prime }$. Each target location requires a recombination kernel of size $k_{up}{\times k}_{up}$. The content encoder takes the compressed feature mapping as the input and encodes the content to generate the reorganization kernel; the kernel normalization applies the softmax function to each reorganization kernel. For each reorganization kernel, ${\omega }_{l^{\prime }}$, the content-aware reganization module reassembles the features within the local region by means of the ϕ function, which is simply a weighting operator. For the target location of $l^{\prime }$ and the corresponding square region centered at $l=(i,j)$, $N\left({\chi }_l,k_{up}\right)$ the reorganization process is shown in the following Equation (3):

$$r=\left[k_{up}/2\right],{\chi }_{l^{\prime }}^{\prime }={\sum }_{n=-r}^r{\sum }_{m=-r}^r{\omega }_{l^{\prime }\left(n,m\right)}\cdot {\chi }_{(i+n,j+m)}$$

(3)

In the reorganized kernel, $N\left({\chi }_l,k_{up}\right)$, each pixel in the region affects the upsampled pixels, $l^{\prime }$, differently according to the feature content rather than the location distance. Since the relevant regions can get more attention, the semantic expression ability of the reorganized feature map is stronger. The parameters for the three submodules in the Kernel Prediction Module are set as follows. The Channel Compressor module uses a 1 × 1 convolutional layer to compress the input feature channels. The Content Encoder module sets the encoder parameters as $k_{encoder}\times k_{encoder}\times C_m\times C_{up}$, where increasing the kencoder expands the perceptual field of the encoder. However, the computational complexity increases exponentially with the increase in kernel size, so it was decided to use the researcher’s experience and set $k_{encoder}=k_{up}-2$ to achieve a trade-off between performance and efficiency. Finally, in the Kernel Normalizer module, each $k_{up}\times k_{up}$ recombination kernel is spatially normalized with a softmax function before being applied to the input feature map to achieve true mapping.

The skip-connection of UNet is an effective way to retain the feature information at different scales; however, the simple stitching method has some disadvantages, which may result in the loss of critical feature information, and may not handle the relationship between feature maps of different resolutions effectively, leading to the loss of detailed information. Remote sensing images have a higher resolution than traditional images, containing more feature scales, and, thus, cross-layer connection methods need be selected and optimized carefully when being applied to remote sensing image segmentation to address the limitations of the simple stitching method. This paper designs an attention fusion module that combines different attention modules to enhance the recognition capability, strengthen the recognition of the distinction between the targets at different scales, refine the features between the different classes of targets, and solve the multi-scale problem.

As shown in Figure 5. This module takes feature maps extracted from the encoder and decoder as inputs. The final feature map is then restored to the same spatial resolution as the input image, and the output of the attention mechanism is re-connected with the feature map after the corresponding operation. All three convolutional kernels are 1 × 1 in size, and the information from multiple channels is fused through channel conversion to enhance the network’s information extraction capability.

For feature maps with different dimensions and different features, an appropriate attention mechanism should be selected to enable the network to locate the regions of interest for refinement more easily. In U-shaped networks, shallow feature maps are characterized by having large resolutions, where the spatial feature distribution has a significant impact on fusion. To address this issue, we have employed a spatial attention-based fusion module in CA-UNet. First, we reduce the number of channels by passing the input feature map through a 1 × 1 convolutional layer to reduce the computational effort. Next, we compute the Query (the output of the previous layer), Key (indicating the feature associated with Query), and Value (indicating the object to be weighted average, representing the intermediate feature map of the current layer) using three different 1 × 1 convolutional layers. We then compute the Attention Score based on the Query and Key, with Attention Score = Query × Key. The attention score is normalized by softmax to obtain the Attention Map, whose value ranges from 0 to 1, indicating the weight of each position under the attention mechanism. We weigh and fuse the Attention Map with Value to obtain the final output feature map. The fusion is performed in the channel or spatial dimension, and we use a 1 × 1 convolutional layer in this paper.

High-dimensional features are typically compressed based on their channels before being fused using the channel attention module for the smaller-scaled feature maps. In this study, we propose an enhanced channel attention mechanism, as traditional channel attention modules that capture the dependencies of all channels are not always necessary or efficient. This model uses ECANet [37], which removes the fully connected layer of the original channel attention module and learns directly by a 1D convolution on the globally averaged pooled features. The size of the convolution kernel affects the number of channels to be considered for the calculation of each weight of the attention mechanism, the coverage of cross-channel interactions. The process is shown in Figure 6, in which two convolutional layers are used, a 1 × 1 convolutional layer for feature map compression and a 3 × 3 convolutional layer for generating the attention map. In addition, the padding of the convolution kernel is set to 1 in order to keep the size of the input and output feature maps the same. The weights of each position in the attention map are mapped by the Sigmoid activation function.

3. Experiments and Results

This section introduces the two datasets and experimental settings used in our study. We also compare the performance of current commonly used models on these two datasets to verify the effectiveness of CA-UNet. To fairly validate the effectiveness of our model, we conducted experiments on the WHDLD public dataset and the multi-classified land cover dataset we produced for Lu’s County.

3.1. Datasets

3.1.1. WHDLD Datasets

The WHDLD dataset [12] is a widely used public dataset in remote sensing image segmentation experiments. The Gaofen-1 satellite captures images of Wuhan City using panchromatic mode, which involves imaging with a single wavelength band typically within the visible or near-infrared range to achieve high spatial resolution. These images are then combined with multi-spectral images captured at multiple wavelength bands to obtain RGB images with high resolution and color accuracy through image fusion and image resampling technologies. The spatial sampling step for the panchromatic images is 2 m per pixel, with each image eventually cropped into 6 categories labeled 256 × 256. The data are bare soil, building, road, water, pavement, and vegetation. The total number of data is 4940. Figure 7 displays the sample images and their corresponding labels.

3.1.2. Lushi County Datasets

The multiclassification land cover dataset of Lushi County is produced by the high-resolution remote sensing image processing and labeling of Lushi County. The dataset is also an RGB dataset. The original remote sensing image example is shown in Figure 8, the image resolution is 2 m × 2 m, pre-processed with arcgis software, and labelme is used for the data labeling.

The multi-category land cover remote sensing image dataset of Lushi County was labeled by us according to the Third Land Survey Manual combined with the high-resolution remote sensing image of Lushi County with a spatial resolution of 2 m/pixel, and the image was segmented into 256 × 256 by cropping, labeled into 5 categories: trees, road, building, water, and field, with a total of 5100 datapoints. Figure 9 displays the sample images and their corresponding labels.

From the above two pictures, we can see that the remote sensing image multiclassification land cover dataset is different from the dataset used for general semantic segmentation, its scene is more complex with more kinds of recognition, and there are big differences between the different targets, such as farmland, road, and field features. In addition, the same targets also have huge feature differences, such as highways and countryside dirt roads, which brings huge difficulties to segmentation, and how to cope with the chaotic distribution between the targets and big feature differences becomes an important problem.

3.2. Evaluation Metrics

In this experiment, four commonly used semantic segmentation evaluation metrics are selected for evaluating the performance of the classification algorithm and comparing it with other advanced models. The calculation method of each metric is described below.

In the problem of semantic segmentation, the intersection and union ratio represent the intersection and union of the true label and the predicted value of a class. The mean intersection over union (mIOU) is defined as the ratio of the intersection of the model predictions and the true values of each category. In other words, it is the average of the IOU values for each class on the dataset. In this experiment, mIOU has been selected as the main metric, which reflects the overall accuracy and consistency degree. Equation (4) illustrates the calculation method for mIOU. Specifically, TP represents that the true value is positive and that the prediction is positive. FN represents that the true value is positive and that the predicted value is negative. FP represents that the true value is negative and the that prediction is positive. TN represents that the true value is negative and that the prediction is negative.

$$mIOU=\frac{1}{k+1}{\sum }_{i=0}^k\frac{TP}{FN+FP+TP}$$

(4)

The mean pixel accuracy, abbreviated as MPA, calculates the percentage of pixels correctly classified for each category separately for each category, then sums and averages these values, as in Equation (5).

$$MPA=\frac{1}{k+1}{\sum }_{i=0}^k\frac{TP+TN}{FN+FP+TP+TN}$$

(5)

Precision, also known as accuracy, indicates the percentage of samples identified by the model as positive, and recall, also known as completeness, indicates the ratio of the number of samples correctly identified by the model as positive to the total number of positive samples, as shown in Equation (6).

$$P=\frac{TP}{TP+FP}$$

(6)

$$R=\frac{TP}{TP+FN}$$

(7)

3.3. Implementation Details

In this experiment, all models were tested using a computer configured with CUDA 10.0, PyTorch 1.8.0, and Python 3.6 experimental environments. To ensure the fairness of the experimental conditions, uniform data augmentation, including random cropping and rotation, was performed on the experimental data, and the final input model size was 256 × 256 × 3 for both datasets. All experiments were run on a server equipped with a Tesla P100 GPU with a 16G video memory size and Windows 10 as the operating system. The adaptive learning rate optimizer, Adam, has the advantage of low memory requirements for high computational efficiency. Therefore, it was used for adaptive network updates and parameter optimization. The initial learning rate during training was 0.001 with 8 randomly selected batches and 150 epochs. The cross-validation training method was used.

3.4. Results and Analysis

3.4.1. Experimental Results Obtained on the WHDLD Dataset and Lushi County Dataset

CA-UNet was used to conduct experiments on two high-resolution remote sensing image datasets under the same conditions, and both showed good performances; our model obtained an mIOU of 63.27 on the WHDLD dataset, a result that was 4.03 higher than the baseline model. The experimental metrics for each categorical category in the WHDLD dataset are shown in

Table 1. Results of WHDLD dataset.

	IOU	MPA	P	R
road	85.38	89.51	92.49	91.74
vegetation	56.59	71.62	90.87	73.75
water	95.35	94.76	97.95	97.29
pavement	63.75	79.36	79.13	76.64
building	74.96	85.03	84.47	86.94
bare soil	66.85	78.48	79.65	80.62

P and R stand for precision and recall, respectively; all values are reported as percentages (%).

.

The experimental metrics for each category in the Lushi County dataset are shown in

Table 2. Results of Lushi County dataset.

	IOU	MPA	P	R
road	68.26	75.51	76.40	70.91
forest	84.31	89.92	90.53	87.67
water	95.14	97.77	98.02	98.13
farmland	66.30	79.48	78.05	77.10
building	50.11	61.56	64.52	67.75

P and R stand for precision and recall, respectively; all values are reported as percentages (%).

.

3.4.2. Comparison of Predicted Results

In order to further compare the experimental results of our method with the baseline UNet, we used the training results of the two models on the WHDLD dataset and Lu County dataset to predict some complex scenes to achieve the purpose of visualizing the experimental results, and the prediction results are shown in Figure 10 and Figure 11.

The segmentation results for Lushi County are shown in Figure 11, and, according to the experimental results, we can see that the segmentation effect of the present model was significantly improved. In the Lu’s County dataset, as shown in the black box, the original model classifies large areas of water and the land boundary proximity into the wrong category, which is well mitigated in the improved model, and the segmentation results of the original model also misclassify trees into agricultural land. In addition, the improved model performs better in classifying target edges, as shown in the second row of Figure 10 where the forest boundary is handled better, and the misclassified area is smaller.

The segmentation results on the WHDLD dataset are shown in Figure 11. The classification results on slender targets are significantly better than U-Net; as shown in the black box in the first row, the vegetation segmentation near the edge of the image is better, while UNet directly loses the targets. In addition, the edge segmentation results of buildings are also more accurate. In addition, in the second row of the small target recognition results, our model can extract the contours of small targets more accurately, while U-Net does not recognize these targets. In the third row of roads and buildings and the bare soil adjacent to each other, our model performs better and not only correctly identifies the targets, but also more accurately depicts the adjacent edges.

To further analyze the improvements made by our model, we evaluated its performance on the more convincing WHDLD public dataset in comparison to the baseline models. A confusion matrix was used to display the performance of each pixel when classified, enabling the evaluation of the segmentation accuracy of the whole image. By analyzing the categories that were more commonly misclassified by the model, we were able to target specific areas for improvement and enhance the overall performance of the model. The experimental results of the baseline models UNet and CA-UNet on the WHDLD dataset are presented in

Table 3. Confusion matrix obtained from UNet’s experiments on the WHDLD dataset.

	bg	Vegetation	Water	Road	Pavement	Building	Bare Soil
vegetation	445	2,699,432	10,511	327,809	362,515	18,029	1187
water	294	6389	1,156,906	69,189	143,168	14,691	3438
road	925	364,557	124,375	2,142,061	602,204	100,887	3570
pavement	319	326,270	127,097	793,271	14,211,726	136,487	95,768
building	78	36,925	23,157	123,030	266,001	886,706	20,780
bare soil	48	1487	5639	2402	296,287	21,735	6,846,989

bg is shorthand for background.

and

Table 4. Result obtained on the WHDLD datasets.

	bg	Vegetation	Water	Road	Pavement	Building	Bare Soil
vegetation	229	2,757,283	8516	338,425	292,811	21,946	718
water	74	6153	1,212,076	66,220	92,918	13,869	2765
road	297	313,049	64,541	2,462,053	430,014	63,720	4905
pavement	161	300,475	117,836	521,161	14,484,436	158,983	117,886
building	119	34,083	18,795	63,993	180,175	1,039,717	19,795
bare soil	13	858	3107	2071	172,749	15,776	6,980,013

bg is shorthand for background.

, respectively.

After analysis, we observed that the percentage of pixels correctly classified for all target types was improved by 1.69%, 3.96%, 9.59%, 1.75%, 11.29%, and 1.85%, respectively, with significant improvements in the accuracy of small targets such as roads and buildings. Moreover, the percentage of pixels misclassified by the model decreased.

3.4.3. Comparison with Other Methods

To further validate the effectiveness of our model, we selected several classical models and recently proposed methods by researchers for semantic segmentation of remote sensing images to conduct a comparative analysis. The experimental results are presented in

Table 5. Result obtained on the WHDLD datasets.

	mIOU	MPA	P	R
UNet	59.24	71.57	73.51	70.73
UNet++	60.81	72.96	73.40	71.25
DeepLabv3+	61.01	72.49	73.87	72.01
SegNet	55.67	67.92	69.70	67.25
PsPNet	58.46	72.62	73.19	70.57
SegFormer	61.5	73.02	74.25	71.36
MSFCN	60.37	72.54	73.62	72.18
DPPNet	60.92	72.81	74.03	72.66
ResUNet	55.35	67.57	69.41	68.39
CA-UNet	63.27	74.86	76.57	74.95

and

Table 6. Result obtained on the Lushi datasets.

	mIOU	MPA	P	R
UNet	67.19	81.49	84.91	81.42
UNet++	69.85	82.72	82.63	71.25
DeepLabv3+	68.23	81.60	83.37	81.28
SegNet	66.47	79.29	80.70	78.25
PsPNet	67.64	82.38	82.27	81.65
SegFormer	70.17	81.11	83.55	80.93
MSFCN	70.31	81.90	84.24	80.77
DPPNet	67.98	79.12	81.94	78.41
ResUNet	67.72	80.46	82.62	82.54
CA-UNet	72.83	84.72	86.79	84.95

, respectively.

The U-Net is used as the baseline model, and the encoder continuously reduces the resolution in the process of downsampling to obtain image information of different scales; the information of the image gradually changes from points, lines, and gradients in the bottom information to contours and more abstract information in the top information. The skip-connection structure fuses the same scale information, which is equivalent to adding more detailed information and is beneficial for the information-rich task of the semantic segmentation of remote sensing images, which is the reason for its good performance. With improved networks after that, they make the following contributions, respectively, where PsPNet has a hierarchical global priority, contains modules with different scale information between different sub-regions, fuses four different pyramid scale features, and improves the contextual inference ability of the model, and SegNet’s innovation lies in the way the decoder upsamples the low-feature map to eliminate the process of upsampling learning in the input image to obtain the full context pixels. The lost context is extrapolated through the mirrored input image, so, for remote sensing images with more features, UNet’s UNet++ indirectly fuses multiple features at different levels through operations such as short joins and up/down sampling, and achieves better results on both datasets. UNet++, although indirectly fusing features of different sensory fields, only fuses the information of the next layer without fusing the upper layer, causing its decoder part to be inadequately fine-grained, enabling the potential loss of marginal information in the segmentation results. For a remote-sensing dataset that is rich in marginal information, this can be considered a major flaw, which causes the fine granularity of its decoder part to still not be fine enough. This involves using the spatial pyramid to capture rich contextual information by pooling operations at different resolutions and using the encoder-decoder structure to gradually obtain a clear image. An improved SegFormer approach based on Transformer, which uses a self-attentive mechanism to process spatial relationships in images, achieved mIOUs of 60.37 and 70.17 on 2 datasets. The multi-scale fully convolutional network MSFCN obtained mIOUs of 60.37 and 70.31, respectively. The residual structure-based ResUNet obtained mIOUs of 55.35 and 67.72. DPPNet, consisting of deep pyramidal pool (DPP) blocks and dense blocks connected by deep multi-extended residuals, obtained mIOUs of 60.92 and 67.98. In DeepLabv3+, ASPP can effectively capture multi-scale information thanks to the parallel way of multiple atrous rate convolution layers, which makes the model perform better on multi-scale objects; the segmentation results of this network are better than those of SegNet and PsPNet. Our proposed CA-Unet obtained optimal results on both datasets, where mIOUs reached 63.27 and 72.83, respectively.

3.4.4. Ablation Experiments

To verify the validity of the different components of our proposed model, we performed ablation experiments on two datasets. The results obtained are shown in

Table 7. Results of ablation experiments.

	Exp1	Exp2	Exp3	Exp4
Coordinate Attention + Residual	×	√	√	√
CARAFE	×	×	√	√
Attention Mechanism	×	×	×	√
mIOU (%)	59.24/67.19	60.58/69.46	62.72/71.35	63.27/72.83
Parameters (M)	19.43	19.85	18.09	18.53

, where we use VGG16 as the backbone network of the baseline model, commonly used in various previous models.

We evaluate the performance of each module on two multi-class land cover datasets by adding the improved module while keeping the other experimental conditions unchanged. Experiment 1 represents the results using the baseline UNet. Experiment 2 is based on the results obtained after the Coordinate Attention’s residual structure is added to the encoder. The mIOU is improved by 1.34% and 2.27% over UNet, respectively. Experiment 3 is based on Experiment 2 by replacing the upsampling operator in the decoder with the lightweight operator CARAFE. CARAFE gives better results with a significantly lower number of parameters, and the mIOU is improved by 2.14% and 1.89%, respectively. Experiment 4 introduces the attention fusion module in the decoder based on Experiment 3, and the mIOU is improved by 0.55% and 1.48%, respectively. The ablation experiments show that, by adding multiple attention modules and using more efficient upsampling operators, CA-UNet achieves a significant improvement in the segmentation performance for multi-classified land cover datasets.

4. Discussion

We provide a segmentation method for the land cover classification of high-resolution remote sensing images. In remote sensing images, the wide variety of target types inherently put the network feature extraction capability to a great test; in addition, the high reflectance variation of pixels in the same category makes it more difficult for the model to recognize objects of interest, so contextual information extraction is crucial to enhance the feature recognition capability. This is why we introduce CARAFE, which not only reduces the number of parameters and eases the computational burden, but also improves the feature integration capability of the model, weighing the number of parameters and feature extraction capability. Additionally, our model includes several attention modules, such as coordinate attention and a fused attention module based on spatial and channel attention, which effectively improve the feature reproduction ability of the model, as supported by experimental and prediction results. Through comparison and ablation experiments, we demonstrate that our model outperforms the baseline model UNet, effectively alleviating the problem of the inaccurate segmentation of small targets in remote sensing images and improving the accuracy of adjacent target boundary segmentation.

5. Conclusions

We propose a U-shaped network (CA-UNet) based on a multi-attention mechanism and CARAFE for the semantic segmentation of high-resolution remote sensing images. We construct a residual encoder based on the coordinate attention mechanism in the backbone. The residual block enhances the nonlinear fitting ability of the network, combines with the coordinate attention to enhance the focus on spatial location information, and more accurately captures key features. This helps to identify small targets and boundaries between different targets in remote sensing segmentation tasks and improves the multi-scale information feature extraction ability of the model.

The skip-connection is an important structure of UNet, and we embed an attention fusion module in this position to combine spatial attention and channel attention. The comprehensive use of features of different scales is crucial when segmenting remote sensing images, as the size and shape of different targets are different, and this module can further improve the performance of the model. Finally, we use the CARAFE operator to replace the traditional upsampling module. This operator is less computationally intensive and obtains more accurate contextual information through adaptive feature reorganization. It alleviates the information loss problem when upsampling and is very effective for segmenting targets of different scales and sizes of remote sensing images.

Our method achieved an mIOU of 72.83% on the Lu County land cover dataset, which is much higher than the baseline model UNet of 67.19%, and also has advantages over other commonly used models in the experiments. In the experiments conducted on the WHDLD dataset, our model achieved an mIOU of 63.27%, which is higher than the 59.24% of UNet and also achieved better results compared to other commonly used models in the experiments. Our model performs well on multi-classified remote sensing image datasets, especially multi-classified land cover datasets, with a high segmentation accuracy and strong generalizability. However, it also has some limitations, such as a larger image resolution and finer and denser target recognition problems. In addition, the segmentation of remote sensing images with large noise is also a difficult problem.

We plan to conduct further research and focus on more efficient algorithm designs to make it easier to deploy and combine with radar, multi-modal data fusion such as spectral data to improve segmentation accuracy and robustness. Finally, we aim to combine with other algorithms such as path planning to realize end-to-end applications.

Previous research has been widely used in the medical, military, and autonomous driving fields. The segmentation of high-resolution remote sensing images based on deep learning is also gaining more and more attention from researchers [38]. With wide practical significance, it has been widely used in agriculture [39,40,41,42], environmental monitoring, urban planning, etc.