Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF

Abstract

The semantic segmentation of remote sensing images is a significant research direction in digital image processing. The complex background environment, irregular size and shape of objects, and similar appearance of different categories of remote sensing images have brought great challenges to remote sensing image segmentation tasks. Traditional convolutional-neural-network-based models often ignore spatial information in the feature extraction stage and pay less attention to global context information. However, spatial context information is important in complex remote sensing images, which means that the segmentation effect of traditional models needs to be improved. In addition, neural networks with a superior segmentation performance often suffer from the problem of high computational resource consumption. To address the above issues, this paper proposes a combination model of a modified multiscale deformable convolutional neural network (mmsDCNN) and dense conditional random field (DenseCRF). Firstly, we designed a lightweight multiscale deformable convolutional network (mmsDCNN) with a large receptive field to generate a preliminary prediction probability map at each pixel. The output of the mmsDCNN model is a coarse segmentation result map, which has the same size as the input image. In addition, the preliminary segmentation result map contains rich multiscale features. Then, the multi-level DenseCRF model based on the superpixel level and the pixel level is proposed, which can make full use of the context information of the image at different levels and further optimize the rough segmentation result of mmsDCNN. To be specific, we converted the pixel-level preliminary probability map into a superpixel-level predicted probability map according to the simple linear iterative clustering (SILC) algorithm and defined the potential function of the DenseCRF model based on this. Furthermore, we added the pixel-level potential function constraint term to the superpixel-based Gaussian potential function to obtain a combined Gaussian potential function, which enabled our model to consider the features of various scales and prevent poor superpixel segmentation results from affecting the final result. To restore the contour of the object more clearly, we utilized the Sketch token edge detection algorithm to extract the edge contour features of the image and fused them into the potential function of the DenseCRF model. Finally, extensive experiments on the Potsdam and Vaihingen datasets demonstrated that the proposed model exhibited significant advantages compared to the current state-of-the-art models.

1. Introduction

At present, image semantic segmentation (ISS) is one of the most-significant areas of research in the field of digital image processing and computer vision. Compared with traditional image segmentation, ISS adds semantic information to the target and foreground of the image on this basis and can obtain the information that the image itself needs to express according to the texture, color, and other high-level semantic features of the image, which is more practical [1]. The difference between remote sensing image semantic segmentation and ordinary image semantic segmentation lies in the different processing objects. Specifically, the semantic segmentation of remote sensing images refers to the analysis of the spectrum, color, shape, and spatial information of various ground objects in remote sensing images, divides the feature space into independent subspaces, and finally, assigns each pixel in the image predetermined semantic tags. Remote sensing images contain richer ground object information and vary in size, color, and orientation, which leads to the emergence of inter-class similarities and intra-class variability [2,3,4]. For instance, “stadium”, “church”, and “baseballfield” may appear in “school”, and there may be large differences in different “school” scenes, as shown in Figure 1. Moreover, remote sensing images often have a complex background environment, irregular object shapes, similar appearances in different categories, and other factors that are not conducive to image segmentation. In particular, compared with natural images that only contain the three channels of RGB, high-resolution multispectral remote sensing images contain more channels, which contain richer ground features and a more complex spatial distribution. On account of the above-mentioned factors, the task of the semantic segmentation of remote sensing images is far more complicated than that of natural images. Furthermore, many natural image semantic segmentation models do not perform satisfactorily on remote sensing images. Recently, there has been increasing research on the semantic segmentation of remote sensing images. However, due to the above problems, the semantic segmentation of remote sensing images is still worth further study [5,6,7,8].

Currently, several new segmentation tasks have emerged, including instance segmentation and panoptic segmentation [9,10]. Different from basic semantic segmentation, instance segmentation needs to label different individual aspects of the same object [11,12,13]. On the basis of instance segmentation, panoptic segmentation needs to detect and segment all objects in the image, including the background [14,15,16]. It can be found that the instance segmentation and panoptic segmentation tasks are more complicated. At present, there are still many problems in the research of instance segmentation and panoptic segmentation, such as object occlusion and image degradation [10,11]. Although there are many new research branches of semantic segmentation, this does not mean that the basic semantic segmentation research is worthless. The semantic segmentation of remote sensing images plays an important role in many applications such as resource surveys, disaster detection, and urban planning [17]. However, due to the complexity of remote sensing images, there are many problems to be solved in basic semantic segmentation. For example, how to improve the accuracy and execution speed of remote sensing image semantic segmentation is still a problem worth studying. When the performance of the basic remote sensing semantic segmentation model is satisfactory, it is reasonable to further consider the instance segmentation and panoptic segmentation of remote sensing images.

The results of remote sensing image segmentation are mainly determined by multiscale feature extraction, spatial context information, and boundary information [18]. First of all, rich features are conducive to determining the object categories in images in complex environments. In particular, feature extraction at different scales can effectively alleviate the segmentation impact caused by the differences between object sizes in the image. Second, the global spatial context information can help determine the category of adjacent pixels, because it is not easy to obtain satisfactory segmentation results using only local information. Finally, since the satellite is not stationary when taking remote sensing images, the boundaries of objects in the image are often unclear, so more attention should be paid to the edge details of objects during image segmentation.

Traditional remote sensing image segmentation models generally use manual feature extractors for feature extraction and then use traditional machine learning classification models for the final pixel classification operations. Traditional feature extraction models include oriented FAST and rotated brief (ORB) [19], local binary pattern (LBP) [20], speeded up robust features (SURF) [21], etc. Traditional classification methods consist of support vector machine [22], logistic regression [23], etc. However, due to the complex background environment in remote sensing images, the performance of traditional remote sensing image segmentation models is often unsatisfactory. Additionally, the abilities of traditional feature extraction models cannot meet the needs of practical applications.

Deep learning has been widely used in image recognition [24,25], image classification [26,27], video prediction [28,29], and other fields. In particular, deep convolutional neural networks (DCNNs) with strong feature extraction capabilities are popular in computer vision [30,31,32,33,34,35,36,37]. For instance, Reference [38] designed a YOLOv3 model with four scale detection layers for pavement crack detection, using a multiscale fusion structure and an efficient cross-linking (EIoU) loss function. Recently, DCNNs have been gradually used in image semantic segmentation tasks. For example, Reference [39] built a segmentation model called U-Net, which fuses the features of different scales to obtain segmentation results. However, the U-net model does not consider the spatial context information in the image. The PSPNet [40] model obtains contextual information through a pyramid pooling module, which improves the segmentation performance. However, it has high computational complexity and poor efficiency. Reference [41] designed a high-resolution network (HRNet), which fuses structural features and high-level semantic information at the output end, improving the spatial information extraction ability of images, but ignoring local information. For the problem of HRNet, Reference [42] used the attention mechanism to improve the recognition performance of the model for local regions. However, this model is very resource-intensive and has a large amount of redundant information. Additionally, Reference [18] proposed a high-resolution context extraction network (HRCNet) based on a high-resolution network for the semantic segmentation of remote sensing images. Reference [43] built a multi-attention network (MANet) to extract contextual dependencies in images through multiple efficient attention modules. However, the multiscale context fusion effect of HRCNet and MANet needs to be improved, and both have the problem of high computational complexity. In summary, during the semantic segmentation task of remote sensing images in complex environments, the current CNN-based model does not pay enough attention to global context information, and its performance is not completely convincing [44]. In addition, networks with a superior segmentation performance often have the problem of excessive computing resource consumption.

Considering the main factors affecting the semantic segmentation of remote sensing images and the problems of deep learning models in the semantic segmentation of remote sensing images, this paper proposes the mmsDCNN-DenseCRF model, in which a modified multiscale deformable convolutional neural network (mmsDCNN) is used for multiscale feature extraction, while DenseCRF is utilized to capture semantic information and global context dependency and optimize rough segmentation results such as edge refinement. First, a lightweight multiscale deformable convolution neural network is proposed, which is based on the modified multiscale convolutional neural network (mmsCNN) designed in our previous work [45]. The mmsDCNN model adds an offset to the sampling position of mmsCNN convolution, allowing for the convolutional kernel to adaptively choose the receptive field size with only a slight increase in computing resources. In addition, the mmsDCNN can achieve a balance between strong multiscale feature extraction performance and low computational complexity. We used the mmsDCNN to extract rich multiscale features and obtain preliminary prediction results. Subsequently, a multi-level DenseCRF based on the superpixel and pixel level is proposed as a back-end optimization module, which can make full use of spatial context information at different granularities. The multi-level DenseCRF model considers the relationship between each pixel and all other pixels, establishes a dependency between all pixel pairs in the image, and uses the interdependencies between pixels to introduce the global information of the image, which is suitable for the semantic segmentation task of remote sensing images in complex environments [46]. Although DenseCRF improves the robustness of processing images with semantic segmentation, it can only determine the region position and approximate shape contour of the object of interest in the image marking results. As a result, the real boundary of the segmented region cannot be accurately obtained, since there are still fuzzy edge categories or segmentation mistakes. To solve the above problems, we used the Sketch token edge detection algorithm [47] to extract the edge contour features of the image and fuse the edge contour features into the potential function of the DenseCRF model. In summary, this paper proposes a new idea and specific implementation for the problems existing in the existing models based on convolutional neural networks in remote sensing image segmentation tasks.

The main contributions are as follows:

• A framework of the mmsDCNN-DenseCRF combined model for the semantic segmentation of remote sensing images is proposed.
• We designed a lightweight mmsDCNN model, which incorporates deformable convolution into the mmsCNN proposed in our previous work [45]. Notably, the mmsDCNN adds an offset to the sampling position of the mmsCNN convolution, which enables the convolutional kernel to adaptively determine the receptive field size. Compared with the mmsCNN, the mmsDCNN can achieve a satisfactory performance improvement with only a tiny increase in the computational complexity.
• The multi-level DenseCRF model based on the superpixel level and pixel level is proposed. We combined the pixel-level potential function with the superpixel-based potential function to obtain the final Gaussian potential function, which enables our model to consider features of various scales and the context information of the image and prevents poor superpixel segmentation results from affecting the final result.
• To solve the problem of blurring edge categories or segmentation errors in the semantic segmentation task of DenseCRF, we utilized a Sketch token edge detection algorithm to extract the edge contour features of the image and integrated them into the Gaussian potential function of the DenseCRF model.

The remainder of the article is organized as follows. Section 2 introduces the modified multiscale deformable convolutional neural network (mmsDCNN), and the multi-level DenseCRF model. In Section 3, we conduct some comparative experiments on the public dataset, i.e., the International Society for Photogrammetry and Remote Sensing (ISPRS) [48]. Finally, Section 5 provides a summary of the article.

2. Methodology

The pipeline of the proposed model is shown in Figure 2. The following two parts are included: the modified multiscale deformable convolutional neural network (mmsDCNN) and the multi-level DenseCRF model.

2.1. mmsDCNN

At present, multiscale CNNs can be divided into two types: the multiscale image and the multiscale convolutional kernel. The CNN based on multiscale images feeds images of different resolutions into the neural network with the same size of the convolutional kernel to obtain multiscale features. This method consumes many memory resources due to the need to process multiple images at the same time. The CNN based on the multiscale convolutional kernel feeds the image into several convolutional kernels of different sizes to obtain multiscale features. Due to the introduction of the multiscale convolutional kernels in this structure, the parameter amount of the network is greatly increased, and gradient vanishing is prone to occur during model training. To address the above problems, we proposed a modified multiscale convolutional neural network (mmsCNN) in [45]. In this paper, we combined the previous work and improved it to propose the modified multiscale deformable convolutional neural network (mmsDCNN) model, as shown in Figure 3.

Recently, Reference [49] provided a new perspective: compared with the CNN with a deep large-scale convolutional kernel, the CNN with a shallow large-scale convolutional kernel has a larger receptive field, which is more similar to the human perception mode. In addition, compared with the deep network, the shallow network has fewer parameters, and the problem of gradient disappearance is less likely to occur. Based on this, we integrated the mode of shortcut connections in the residual neural network into the msCNN based on a multiscale convolutional kernel. The proposed model consists of several various deformable convolutional kernel CNNs, i.e., $DefConv\left[\left(1,1\right),0,1\right]$, $DefConv\left[\left(3,3\right),1,1\right]$, $DefConv\left[\left(5,5\right),2,1\right]$, and $DefConv\left[\left(7,7\right),3,1\right]$. Concretely, $DefConv\left[\left(m,n\right),i,j\right]$ denotes the CNN with deformable convolutional kernel size = $m\times n$, padding = i, and stride = j. Then, we invalidated the first 1, 2, and 3 layers of $DefConv\left[\left(3,3\right),1,1\right]$, $DefConv\left[\left(5,5\right),2,1\right]$, and $DefConv\left[\left(7,7\right),3,1\right]$, respectively, which are shown in the yellow box in Figure 3. We refer to the network layer in the yellow box as the invalid layer, which is where the shortcut connections are performed. Subsequently, we flexibly set the pooling step size of the network layer after the invalid layer, which ensures that the final output scales of different convolutional networks are consistent. To be specific, we set the pooling layers to $Pooling\left[\left(4,4\right),0,4\right]$, $Pooling\left[\left(8,8\right),0,8\right]$, $Pooling\left[\left(16,16\right),0,16\right]$ according to the number of invalid layers, as shown in Figure 3. $Pooling\left[\left(m,n\right),i,j\right]$ represents the pooling layers with filtersize = $m\times n$, padding = i, and stride = j.

For pixel-level image segmentation tasks, we fused deformable convolution with the mmsCNN model, which can further expand the model’s receptive field. The deformable convolution adds an offset to each convolutional sampling point, which is equivalent to the scalable change in each block of the convolutional kernel, thereby changing the range of the receptive field [50]. Figure 4 compares standard convolutional kernels and deformable convolutional kernels. In the training phase, the convolutional kernel for generating the output features and the convolutional kernel for generating the offsets are learned synchronously. The experiments proved that the mmsDCNN model can consume fewer resources, obtain better experimental results, and effectively avoid gradient disappearance.

Since the feature map gradually becomes smaller during the convolution process, to obtain a feature map with the same size as the original image, we needed to upsample the feature map. An intuitive idea is to perform bilinear interpolation, which is easily implemented with a fixed kernel via deconvolution. Therefore, at the output of the mmsDCNN, we replaced the fully connected network of the mmsCNN model with a deconvolutional network. In addition, before deconvolution, we needed to fuse the multiscale features and then utilize a simple convolution to make the number of channels of the output feature map the same as the predetermined number of categories of the pixels. Notably, the output of the mmsDCNN model is a feature map of the same size as the input image, which will be further processed by DenseCRF to obtain the final prediction map.

Table 1. Structure parameters of our mmsDCNN model.

Type	Number	Filter Size	Pad	Stride
Conv1 + ReLU	32	$1\times 1$	0	1
Max Pooling	−	$2\times 2$	0	2
Conv2 + ReLU	64	$(1/3)\times (1/3)$	$0/1$	1
Max Pooling	−	$(2/4)\times (2/4)$	0	$2/4$
Conv3 + ReLU	128	$(1/3/5)\times (1/3/5)$	$0/1/2$	1
Max Pooling	−	$(2/4/8)\times (2/4/8)$	0	$2/4/8$
Conv4 + ReLU	256	$(1/3/5/7)\times (1/3/5/7)$	$0/1/2/3$	1
Max Pooling	−	$(2/4/8/16)\times (2/4/8/16)$	0	$2/4/8/16$
Conv5 + ReLU	6	$1\times 1$	0	1
Deconv Layer	6	$32\times 32$	8	16

shows the parameter settings of our mmsDCNN model.

2.2. The Multi-Level DenseCRF Model

The conditional random field (CRF) model is a probabilistic undirected graph model, which has recently made great progress in the application of remote sensing image semantic segmentation. In this section, we attempt to utilize DenseCRF to further refine the rough segmentation results output by the mmsDCNN model.

2.2.1. Edge Constraint

Compared with the pairwise CRF (PCRF) model, which only considers local neighborhood relations, the DenseCRF model is more suitable for image semantic segmentation [46]. It further considers the relationship between each pixel and all other pixels, establishes the dependency on all pixel pairs in the image, and uses the interdependence between pixels to introduce the global information of the image. Although DenseCRF improves the performance of processing image semantic segmentation, it can only determine the region position and approximate shape contour of the object of interest in the image marking results, so there will still be fuzzy edge categories, making it difficult to obtain the real boundary of the segmented region. In order to improve the accuracy of the image segmentation edge, we utilized the Sketch token edge detection algorithm to extract the edge contour features of the image and fused the edge contour features into the potential function of DenseCRF model. The Sketch token edge detection algorithm is a method for learning and detecting image boundary information for mid-level features. It designs Sketch tokens to represent edge structure information in image patches. The input image is first divided into image patches with a size of $35\times 35$ pixels according to certain rules; then, the Sketch token is used to obtain the channel index features and self-similar features contained in the image patches, and they are finally input into the Structured Random Forest [51] to classify the pixels. Figure 5 shows the edge detection results of some images in the Vaihingen dataset using this algorithm. The Sketch token edge detection algorithm can achieve a similar performance to that of state-of-the-art algorithms, and the global probability of boundary (GPB) [52], sparse code gradients (SCGs) [53], and running speed are greatly improved. For the image to be segmented, let $S=\left\{1,2,\cdots ,N\right\}$ represent the set of image pixels and $N_s$ represent the neighborhood area of the sth pixel. We define the edge energy function as

$$\begin{array}{cc}\hfill E& ={\omega }_p\sum _s\sum _{r\in N_s}{\psi }_{sr}\left(b_s,b_r\right)\hfill \\ \hfill & ={\omega }_p\sum _s\sum _{r\in N_s}exp\left(-\alpha {\left|b_s-b_r\right|}^2\right),\hfill \end{array}$$

(1)

where ${\omega }_p$ is the edge prior parameter, ${\psi }_{sr}\left(b_s,b_r\right)$ represents the edge constraint item, and $b_s$ and $b_r$ represent the image edge features obtained by the Sketch token edge detection algorithm. For any two pixels, if both are points on the edge or not on the edge, then the penalty of the edge constraint item is 0 to ensure that the segmentation result is not over-segmented or over-smoothed. It is beneficial to maintain the edge constraint of the image; otherwise, we added a marginal penalty to the point. The edge constraint item can increase the constraint on the target segmentation boundary, thereby improving the accuracy of target segmentation.

2.2.2. Combined Multilevel Potential Function

In Section 2.1, we used the mmsDCNN to perform feature extraction on the original image to obtain a preliminary probability map of the same size. In this section, a multi-level DenseCRF based on the superpixel level and pixel level is proposed, which takes the feature map of the mmsDCNN as the input and further utilizes the contextual information of the image at different levels. Notably, the potential energy function of multi-level DenseCRF contains two parts: superpixel-level potential energy function and pixel-level potential energy function.

Firstly, we performed superpixel segmentation on the original image according to the simple linear iterative clustering (SILC) algorithm [54]. In particular, we considered the label of a superpixel to be the category label of most of the pixels in the superpixel. Then, we converted the pixel-level preliminary probability map (feature map) output by the mmsDCNN into a superpixel-level prediction probability map according to the superpixel segmentation map. To be specific, we have

$$\begin{array}{c}\hfill P_{superpixel}\left(x_s\right)=\frac{1}{N}\sum _{n=1}^N{P}_{pixel}\left(x_n\right),\end{array}$$

(2)

where $P_{superpixel}\left(x_s\right)$ is the probability distribution of the superpixel, $x_s$ is a superpixel block, $P_{pixel}\left(x_n\right)$ is the probability distribution of the nth pixel output by the mmsDCNN model, and N represents the number of most pixels of the same category in a superpixel block. Next, we defined the superpixel-level unary potential function of the DenseCRF model according to the superpixel-level prediction probability map:

$$\begin{array}{c}\hfill {\psi }_{su}\left(x_s\right)=-log{P}_{superpixel}\left(x_s\right).\end{array}$$

(3)

Subsequently, we extracted the color feature ${\mathit{I}}_{\mathit{s}}$, spatial feature ${\mathit{P}}_{\mathit{s}}$, and texture feature ${\mathit{T}}_{\mathit{s}}$ of the superpixels according to the superpixel segmentation map. The color feature ${\mathit{I}}_{\mathit{s}}$ is the average vector of pixels in the CIE-LAB space; the spatial feature ${\mathit{P}}_{\mathit{s}}$ is the average vector of the spatial features of all pixels in the superpixel; the texture feature ${\mathit{T}}_{\mathit{s}}$ is extracted from the superpixel using the local binary pattern [55]. Then, we defined the superpixel-level binary potential function of the DenseCRF model according to the above superpixel features:

$$\begin{array}{c}\hfill {\psi }_{sb}\left(x_{si},x_{sj}\right)=\mu \left(x_{si},x_{sj}\right)\sum _{m=1}^K{\omega }_{sm}exp\left({\beta }_m{∥{\mathit{f}}_{si}-{\mathit{f}}_{sj}∥}^2\right),\end{array}$$

(4)

where $x_{si}$ and $x_{sj}$ are two different superpixel blocks, ${\omega }_{sm}$ is the weight parameter on each kernel function, and $\mathit{f}=({\mathit{I}}_{\mathit{s}},{\mathit{P}}_{\mathit{s}},{\mathit{T}}_{\mathit{s}})$ denotes the superpixel-level feature vectors. The label compatibility function is defined as follows:

$$\mu \left(x_{si},x_{sj}\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{x}_{si}=x_{sj}\hfill \\ 1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

To be more exact, the superpixel-level binary potential function of the DenseCRF model can be expanded into

$$\begin{array}{c}{\psi }_{sb}\left(x_{si},x_{sj}\right)=\mu \left(x_{si},x_{sj}\right)\left[{\omega }_{s1}exp\left(-\frac{{\left|{\mathit{P}}_{\mathit{si}}-{\mathit{P}}_{\mathit{sj}}\right|}^2}{2{\theta }_{s1}^2}\right)\right.\\ +{\omega }_{s2}exp\left(-\frac{{\left|{\mathit{P}}_{\mathit{si}}-{\mathit{P}}_{\mathit{sj}}\right|}^2}{2{\theta }_{s2}^2}-\frac{{\left|{\mathit{I}}_{\mathit{si}}-{\mathit{I}}_{\mathit{sj}}\right|}^2}{2{\theta }_{s3}^2}\right)\\ \left.+{\omega }_{s3}exp\left(-\frac{{\left|{\mathit{P}}_{\mathit{si}}-{\mathit{P}}_{\mathit{sj}}\right|}^2}{2{\theta }_{s4}^2}-\frac{{\left|{\mathit{T}}_{\mathit{si}}-{\mathit{T}}_{\mathit{sj}}\right|}^2}{2{\theta }_{s5}^2}\right)\right],\end{array}$$

where ${\theta }_s$ is the parameter corresponding to the feature.

Superpixel segmentation can not only obtain local feature information, but also better fit the edge contour of the objects. However, the CRF model based on superpixels usually enforces the consistency of the classification labels of all pixels in a superpixel, meaning that the segmentation results are seriously dependent on the quality of the clustering algorithm. To handle this problem, we added the pixel-level potential function constraint term to the superpixel-based Gaussian potential function to obtain a combined Gaussian potential function, which enabled our model to consider features of different granularities and prevent poor superpixel segmentation results from making the output of the model worse.

We defined a pixel-level unary potential function based on the pixel-level probability prediction map that was output by the mmsDCNN model:

$$\begin{array}{c}\hfill {\psi }_{pu}\left(x_i\right)=-log{P}_{pixel}\left(x_i\right),\end{array}$$

(5)

where $P_{pixel}\left(x_i\right)$ is the probability distribution of pixel $x_i$. Only the characteristics of a single independent pixel are considered in the unary potential energy term, so the result of image segmentation is often not smooth and contains noise. Therefore, we extracted pixel-level color feature $\mathit{I}$ and spatial feature $\mathit{P}$ from the original image and then constructed a pixel-level binary potential function. Similarly, the pixel-level binary potential function can be written as

$$\begin{array}{c}{\psi }_{pb}\left(x_i,x_j\right)=\mu \left(x_i,x_j\right)\left[{\omega }_{p}exp\left(-\frac{{\left|{\mathit{b}}_{\mathit{i}}-{\mathit{b}}_{\mathit{j}}\right|}^2}{2{\theta }_p^2}\right)+{\omega }_{p1}exp\left(-\frac{{\left|{\mathit{P}}_{\mathit{i}}-{\mathit{P}}_{\mathit{j}}\right|}^2}{2{\theta }_{p1}^2}\right)\right.\\ \left.+{\omega }_{p2}exp\left(-\frac{{\left|{\mathit{P}}_{\mathit{i}}-{\mathit{P}}_{\mathit{j}}\right|}^2}{2{\theta }_{p2}^2}-\frac{{\left|{\mathit{I}}_{\mathit{i}}-{\mathit{I}}_{\mathit{j}}\right|}^2}{2{\theta }_{p3}^2}\right)\right],\end{array}$$

where ${\theta }_p$ is the parameter corresponding to the feature and ${\omega }_{p}exp\left(-\frac{{\left|{\mathit{b}}_{\mathit{i}}-{\mathit{b}}_{\mathbf{j}}\right|}^2}{2{\theta }_p^2}\right)$ is the edge energy function defined in Equation (1).

To more accurately model the high-dimensional complex features of images, we introduced a high-order potential energy term based on the robust $P^n$ Potts model [56]:

$${\psi }_{pn}\left(x_c\right)=\left\{\begin{array}{cc}{N}_i\left(x_c\right)\frac{1}{Q}{\gamma }_{max},\hfill & \mathrm{if}{N}_i\left(x_c\right)\le Q\hfill \\ {\gamma }_{max},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

where

$$N_i\left(x_c\right)=\underset{l\in L}{min}\left(\left|c\right|-n_l\left(x_c\right)\right)$$

is the number of pixels that do not take the main part of the label in subcluster c, $\left|c\right|$ indicates the number of pixels in subcluster c, $n_l\left(x_c\right)$ is the number of pixels in subcluster c with class value l:

$${\gamma }_{max}={\left|c\right|}^{{\theta }_{\alpha }}\left({\theta }_1+{\theta }_{2}G\left(c\right)\right),$$

$G\left(c\right)$ represents the segmentation quality of subcluster c, $\theta$ is the model parameter, and Q is the truncation parameter.

Finally, we obtained the multi-level DenseCRF based on the superpixel level and pixel level, and its potential function can be expressed as

$$\begin{array}{cc}\hfill E\left(x\right)& =\sum _{i\in V}{\psi }_{su}\left(x_{si}\right)+\sum _{i\in V,j\in N_i}{\psi }_{sb}\left(x_{si},x_{sj}\right)\hfill \\ \hfill & +\sum _{i\in S}{\psi }_{pu}\left(x_i\right)+\sum _{i\in S,j\in N_i}{\psi }_{pb}\left(x_i,x_j\right)+\sum _{c\in C}{\psi }_{pn}\left(x_c\right),\hfill \end{array}$$

(6)

where V is the set of all superpixels in the image, $N_i$ represents the neighborhood area of the ith pixel, S is the set of all pixels in the image, and C is a set of all subclusters c in the image. The multi-level DenseCRF can consider features of different scales for segmentation, increase the effective constraints on the image target boundary, and improve the accuracy of image segmentation edges. Ultimately, DenseCRF optimizes the rough segmentation result of the mmsDCNN, such as edge thinning and precision machining, to obtain a refined segmentation image.

3. Experiment

To demonstrate the performance of the proposed model, we conducted a series of experiments using the publicly available Potsdam and Vaihingen datasets. In this section, we first provide a brief introduction to the Potsdam and Vaihingen datasets. Next, we describe the experimental parameter settings and evaluation criteria in detail. Finally, we compare the performance with several state-of-the-art models according to the evaluation criteria.

3.1. Datasets

The International Society for Photogrammetry and Remote Sensing (ISPRS) [48] dataset provides two state-of-the-art aerial image datasets for the urban classification and 3D building reconstruction test projects. This dataset employs digital surface models (DSMs) generated from high-resolution orthophotos and corresponding dense image matching techniques. Both datasets cover urban scenes, and each dataset is manually classified into six common land cover classes: impervious surfaces, buildings, low vegetation, trees, cars, and backgrounds. The background class includes water and objects distinct from other defined classes (e.g., containers, tennis courts, swimming pools), which usually belong to uninteresting semantic objects in urban scenes. The Vaihingen dataset contains 33 remote sensing images of different sizes; each image was extracted from a larger, top-level orthophoto image with a spatial resolution of 9 cm. The Potsdam dataset contains 38 image regions of the same size with a spatial resolution of 5 cm. We present several images from the Potsdam and Vaihingen datasets and their corresponding ground truth images in Figure 6, respectively. Due to the memory limitations of the experimental equipment, it was not possible to directly train and infer on the entire image. This paper used the technique in [57] to cut the original data into small pieces before use, keeping the structure of the objects in the images unchanged. We cut the image into a size of $384\times 384$ with an overlap of 72 and 192 pixels in the training and testing datasets to prevent the loss of information.

3.2. Setting of the Experiments and Evaluation Metrics

For the training and testing setting, 19 images in the Potsdam dataset were selected as the training set, 5 images as the validation set, and 14 images as the testing set. In addition, 13 images were chosen in the Vaihingen dataset as the training set, 3 images as the validation set, and the remaining 17 images as the testing set. We obtained the classification results of 10 groups of parameters on the training set and the validation set and selected the group with the best effect as the final parameters of our model.

Table 2. Parameters of the experimental dataset.

Item	Potsdam Training	Potsdam Testing	Vaihingen Training	Vaihingen Testing
size (pixel)	$384\times 384$	$384\times 384$	$384\times 384$	$384\times 384$
number	6931	13454	654	2219
overlap pixels	72	192	72	192
p pixels	72	192	72	192

reports the parameters of experimental datasets. The experimental environment is shown in

Table 3. The experimental environment.

The Experimental Environment	Experimental Configuration
Processor	$AMDRyzen7-4800H$
GPU	$TitanX12G$
Memory	$32G$
Operating system	$ubantu16.04.4$
Compiler	$Pycharm$

.

For the parameter settings, the learning rate was initialized to 0.001. The batch size was set to 8. The kernel parameters of multi-level DenseCRF and the weight parameters corresponding to each kernel function were obtained by the grid search method [46].

To more accurately and objectively evaluate the performance of different models, the overall accuracy (OA), precision, recall, F1 [58], and mean intersection over union (mIoU) were selected as our evaluation criteria. The above evaluation parameters can be calculated by the following formulas:

$$\begin{array}{c}\hfill OA=\frac{TP+TN}{FP+FN+TP+TN}\end{array}$$

(7)

$$\begin{array}{c}\hfill precision=\frac{TP}{FP+TP}\end{array}$$

(8)

$$\begin{array}{c}\hfill recall=\frac{TP}{FN+TP}\end{array}$$

(9)

$$\begin{array}{c}\hfill F1=2\times \frac{precision\times recall}{precision+recall}\end{array}$$

(10)

$$\begin{array}{c}\hfill IoU=\frac{TP}{FN+FP+TP}\end{array}$$

(11)

where $TP$, $TN$, $FP$, and $FN$ are the number of true positives, true negatives, false positives, and false negatives of pixels in the model output image, respectively.

3.3. Experimental Results

3.3.1. Experimental Results on the Potsdam Dataset

First of all, the experiments were conducted on the Potsdam dataset. The methods FCN [59], PSPNet [40], FPN [60], UNet [39], DeepLabv3 [61], DANet [62], LWRefineNet [63], and HRCNet [18] were chosen for comparison.

Table 4. Results for the Potsdam dataset using the above models.

Model	Recall (%)	Precision (%)	F1 Score (%)	OA (%)
FCN [59]	$86.07$	$85.70$	$85.75$	$85.64$
PSPNet [40]	$90.13$	$88.95$	$89.45$	$88.78$
FPN [60]	$88.59$	$89.19$	$88.72$	$88.27$
UNet [39]	$89.42$	$88.13$	$88.67$	$87.76$
DeepLabv3 [61]	$90.29$	$89.23$	$89.66$	$88.97$
DANet [62]	$90.13$	$88.80$	$89.37$	$88.82$
LWRefineNet [63]	$89.10$	$89.01$	$89.02$	$88.43$
HRCNet-W48 [18]	$90.69$	$89.90$	$90.20$	$89.50$
The proposed model	$\mathbf{91.51}$	$\mathbf{90.65}$	$\mathbf{90.92}$	$\mathbf{90.42}$

reports their overall accuracies (OA), precision, recall, and F1. In particular, the bold font represents the best indicator.

It is clear from the results our proposed model showed the best performance on the Potsdam dataset. Compared with the HRCNet-W48 model with excellent performance, the model proposed in this paper improved by 0.82% the recall index, 0.75% the precision index, 0.72% the F1 scores, and 0.92% the overall accuracies (OAs). The experimental results verified that the combined model of the multiscale deformable convolution network and multi-level DenseCRF proposed by us can accurately extract low-level details and high-level semantic features from complex remote sensing images and obtain excellent segmentation results.

Table 5. The mean intersection over union (%) on the Potsdam dataset with the above models.

Model	ImSurface	Building	Low Vegetation	Tree	Car	mIoU (%)
FCN [59]	$81.67$	$88.99$	$71.24$	$72.80$	$79.91$	$78.92$
PSPNet [40]	$82.71$	$90.21$	$72.62$	$74.22$	$81.11$	$80.18$
FPN [60]	$81.63$	$89.22$	$71.37$	$73.05$	$79.09$	$78.87$
UNet [39]	$82.58$	$90.08$	$72.58$	$74.22$	$81.38$	$80.17$
DeepLabv3 [61]	$82.81$	$89.95$	$72.22$	$74.11$	$82.16$	$80.25$
DANet [62]	$82.27$	$89.15$	$71.77$	$73.70$	$81.72$	$79.72$
LWRefineNet [63]	$80.20$	$87.15$	$71.02$	$71.54$	$75.14$	$77.01$
HRCNet-W48 [18]	$83.58$	$91.15$	$73.07$	$74.88$	$83.32$	$81.20$
The proposed model	$\mathbf{84.62}$	$\mathbf{91.83}$	$\mathbf{73.92}$	$\mathbf{75.47}$	$\mathbf{84.15}$	$\mathbf{82.00}$

shows the IoU results for each category for all experimental models on the Potsdam dataset. In particular, the bold font represents the best indicator. Based on the HRCNet-W48 model, the mmsDCNN-DenseCRF model proposed in this paper had an IoU that improved by 1.04% for the “Impervious Surfaces” category, for the “Building” category by 0.68%, for the “Low Vegetation” category by 0.85%, for the “Tree” category by 0.59%, and for the “Car”; category by 0.83%. In sum, the mean IoU (mIoU) of our model was 0.8% higher than that of the HRCNet-W48 model, proving the excellent performance of the proposed model.

Figure 7 shows the experimental results of all experimental models in the Potsdam dataset. Compared with the ground truth given in the dataset, the mmsDCNN-DenseCRF model proposed in this paper not only considers the edge information of large objects in the graph, but can also distinguish the details of small objects, which has an indelible relationship with the edge constraint added in the multi-level DenseCRF.

3.3.2. Experimental Results on the Vaihingen Dataset

In the second part of the experiment, we replaced the dataset with the Vaihingen dataset in the ISPRS. Analogously, the models FCN [59], PSPNet [40], FPN [60], UNet [39], DeepLabv3 [61], DANet [62], LWRefineNet [63], and HRCNet [18] were selected for comparison.

Table 6. Results for the Vaihingen dataset with the above models.

Model	Recall (%)	Precision (%)	F1 Score (%)	OA (%)
FCN [59]	$80.89$	$83.14$	$81.46$	$87.11$
PSPNet [40]	$84.10$	$84.86$	$84.15$	$87.17$
FPN [60]	$82.21$	$84.50$	$82.94$	$86.70$
UNet [39]	$85.01$	$84.46$	$84.44$	$87.14$
DeepLabv3 [61]	$84.38$	$85.09$	$84.42$	$87.32$
DANet [62]	$83.89$	$84.65$	$83.95$	$87.07$
LWRefineNet [63]	$84.45$	$85.32$	$84.81$	$87.23$
HRCNet-W48 [18]	$86.53$	$85.96$	$85.97$	$88.33$
The proposed model	$\mathbf{87.45}$	$\mathbf{87.03}$	$\mathbf{86.82}$	$\mathbf{89.12}$

summarizes the overall accuracies (OAs), precision, recall, and F1 of those models. In particular, bold font represents the best indicator.

Similarly, the data in the Table 6 make it clear that the model designed in this paper performed best on the Vaihingen dataset. To be specific, compared with the HRCNet-W48 model with a splendid performance, the model proposed in this paper improved by 0.92% the recall index, 1.07% the precision index, 0.85% the F1 scores, and 0.79% the overall accuracies (OAs). The experimental results showed that the multiscale deformable convolutional network model (mmsDCNN) proposed by us can extract rich multiscale features from remote sensing images, and the multi-level DenseCRF further considers the spatial context information of images and, finally, obtained a satisfactory segmentation image.

Table 7. The mean intersection over union (%) on the Vaihingen dataset with the above models.

Model	ImSurface	Building	Low Vegetation	Tree	Car	mIoU (%)
FCN [59]	$77.68$	$82.33$	$63.94$	$74.35$	$51.83$	$70.02$
PSPNet [40]	$78.90$	$84.26$	$65.34$	$75.14$	$56.18$	$71.96$
FPN [60]	$77.65$	$82.72$	$64.34$	$74.43$	$52.11$	$70.25$
UNet [39]	$79.02$	$84.46$	$65.23$	$75.13$	$57.07$	$72.18$
DeepLabv3 [61]	$79.23$	$83.70$	$64.88$	$75.14$	$57.54$	$72.10$
DANet [62]	$78.55$	$82.10$	$64.18$	$74.80$	$56.36$	$71.20$
LWRefineNet [63]	$79.21$	$85.13$	$65.32$	$76.51$	$53.38$	$71.91$
HRCNet-W48 [18]	$81.05$	$86.65$	$66.91$	$76.63$	$59.31$	$74.11$
The proposed model	$\mathbf{82.12}$	$\mathbf{87.53}$	$\mathbf{67.82}$	$\mathbf{77.45}$	$\mathbf{60.24}$	$\mathbf{75.03}$

reports the IoU results for each category for all experimental models on the Vaihingen dataset. In particular, the bold font represents the best indicator. Based on the HRCNet-W48 model, the mmsDCNN-DenseCRF model proposed in this paper showed an IoU that improved by 1.07% for the “Impervious Surfaces” category, for the “Building” category by 0.88%, for the “Low Vegetation” category by 0.91%, for the “Tree” category by 0.82%, and for the “Car” category by 0.93%. In summary, the mean IoU (mIoU) of our model was 0.92% higher than that of the HRCNet-W48 model, which fully verified that our model can also perform image segmentation tasks in Vaihingen dataset.

Figure 8 shows the experimental results of all experimental models on the Vaihingen dataset. Compared with all the experimental models, the proposed mmsDCNN-DenseCRF model not only focused on the overall contour of large objects in the image, but can also consider the fine-grained information of small objects, which showed that the mmsDCNN and multi-level DenseCRF can extract multiscale features and grasp the context of the information in the image. In addition, Figure 9 compares the initial segmentation map of the mmsDCNN and the semantic segmentation map of multi-level DenseCRF, which further verified the importance of the multi-level DenseCRF.

3.4. Robustness Verification

To further evaluate the stability of the proposed model, the robustness verification experiment was performed on the Potsdam dataset. According to the method proposed in [35], salt-and-pepper noise and Gaussian noise were randomly added to the testing image, where the salt-and-pepper noise ratio was 0.05 and the mean value and variance of the Gaussian noise were 0 and 0.05.

Table 8. Results on the damaged Potsdam testing set.

Model	mIoU (%)	Precision (%)	F1 Score (%)	OA (%)
Proposed model	$82.00$	$90.65$	$90.92$	$90.42$
Proposed model (salt-and-pepper noise)	$79.65$	$88.33$	$88.82$	$88.25$
Proposed model (Gaussian noise)	$79.78$	$88.72$	$89.03$	$88.49$

reports the experimental results for the damaged Potsdam testing set. According to the experimental results, although the performance of the proposed model declined, it was still acceptable. As a consequence, the robustness of the proposed model was verified by this experiment, and the importance of image denoising algorithm research was also illustrated.

4. Discussion

4.1. Complexity Analysis

In this part, we compared the number of parameters of all models in the experiment. In addition, we used the floating point operations per seconds (FLOPs) to compare the complexity of all experimental models.

Table 9. Complexity comparison on the Potsdam dataset.

Methods	OA	Parameter Quantities	FLOPs
FCN [59]	$85.64$	$\mathbf{14.6}$ M	$45.3$ G
PSPNet [40]	$88.78$	$46.5$ M	$104.0$ G
FPN [60]	$88.27$	$26.4$ M	$25.7$ G
UNet [39]	$87.76$	27 M	$70.0$ G
DeepLabv3 [61]	$88.97$	226 M	$96.4$ G
DANet [62]	$88.82$	$47.3$ M	$115.6$ G
LWRefineNet [63]	$88.43$	176 M	$\mathbf{2.1}$ G
HRCNet-W48 [18]	$89.50$	$59.8$ M	$52.8$ G
Proposed	$\mathbf{90.42}$	29 M	$11.2$ G

reports the number of parameters and FLOPs of all experimental models on the Potsdam dataset. Particularly, the bold fonts indicate the best indicator.

According to Table 9, although the parameter quantities of the model proposed in this paper were not the lowest, they were superior to most of the models. We had 14.4 M, 2.6 M, and 2 M more parameters than FCN [59], FPN [60], and UNet [39], respectively, but the overall accuracy (OA) of the proposed model was the highest. Additionally, compared with DeepLabv3 [61], LWRefineNet [63], and HRCNet-W48 [18] the parameter quantities of the proposed model were reduced. Compared with PSPNet [40] and DANet [62], the parameter quantities of our model were nearly 17 M less. As shown in Table 9, the FLOPs of the proposed model were superior to most of the experimental methods. In conclusion, the mmsDCNN-DenseCRF can achieve a satisfactory balance between the segmentation performance and the complexity of the model.

4.2. Improvements and Future Work

Considering the problems of deep learning models in the semantic segmentation of remote sensing images, this paper proposed the mmsDCNN-DenseCRF model. The proposed model mainly has the following contributions and improvements. First of all, the modified multiscale deformable convolutional neural network (mmsDCNN) was proposed for multiscale feature extraction. The mmsDCNN model can achieve a balance between strong multiscale feature extraction performance and low computational complexity. Secondly, a multi-level DenseCRF model integrating edge information was designed to complete the optimization task of complex remote sensing image semantic segmentation. Specifically, it can further consider the global context information in the remote sensing image at different granularities, which makes up for the shortcomings of the neural network. Besides, the DenseCRF model incorporating edge information can more accurately restore the outline of objects. The mmsDCNN-DenseCRF model has a clear division of labor and fully considers the main factors that affect the semantic segmentation effect of remote sensing images, such as multiscale feature extraction, spatial context information, and boundary information. Finally, this paper provided a new idea for the semantic segmentation of remote sensing images. Concretely, in the semantic segmentation task of remote sensing images, the learning and reasoning capabilities in the probabilistic graphical model (PGM) can be fully utilized to reduce the burden of deep neural networks and improve the performance of the segmentation model. The numerical experiments on the Potsdam and Vaihingen datasets verified that the segmentation accuracy of the proposed model was about 1% higher than that of the most-advanced model.

In addition to realizing and demonstrating this effect using a specific model, this article hopes to provide a way to solve the problem. In fact, if a feature extractor with better performance appears in the future, the experimental effect may be better, but the idea proposed in this paper is still applicable at that time. Therefore, our next step is to improve and integrate the feature extraction model with a stronger multiscale feature extraction capability and less computational complexity with the semantic segmentation model used in this paper, such as the new modified YOLOv3 model with four scale detection layers [38], which shows an excellent multiscale feature extraction performance. Furthermore, in future experiments, we will further extend the model experiments to explore multi-spectral remote sensing data.

5. Conclusions

Considering the main factors affecting the semantic segmentation of remote sensing images and the problems of deep learning models in the semantic segmentation of remote sensing images, this paper proposed the mmsDCNN-DenseCRF model. First of all, a lightweight multiscale deformable convolution network (mmsDCNN) was designed to generate a preliminary prediction probability map. The mmsDCNN model was improved on the basis of the CNN and can achieve a balance between strong multiscale feature extraction capabilities and less computational complexity. Then, a multi-level DenseCRF based on the superpixel level and pixel level was proposed as an optimization module, which can make full use of the image context information at different granularities in the decoding process to obtain more precise semantic segmentation results. In addition, to better recover the contour of the object, the Sketch token edge detection algorithm was utilized to extract the edge contour features of the image, and the edge contour features were fused into the Gaussian edge potential function of the DenseCRF model. In sum, the mmsDCNN-DenseCRF model comprehensively considers several main factors that affect the semantic segmentation effect of remote sensing images, including multiscale feature extraction, spatial context information, and boundary information. In particular, the proposed model can strike a balance between satisfactory segmentation performance and low complexity. Finally, the numerical experiments on the Potsdam and Vaihingen datasets verified that the proposed model is superior to the most-advanced model.