Multi-Feature Lightweight DeeplabV3+ Network for Polarimetric SAR Image Classification with Attention Mechanism

Abstract

Polarimetric Synthetic Aperture Radar (PolSAR) is an advanced remote sensing technology that provides rich polarimetric information. Deep learning methods have been proved an effective tool for PolSAR image classification. However, relying solely on source data input makes it challenging to effectively classify all land cover targets, especially heterogeneous targets with significant scattering variations, such as urban areas and forests. Besides, multiple features can provide more complementary information, while feature selection is crucial for classification. To address these issues, we propose a novel attention mechanism-based multi-feature lightweight DeeplabV3+ network for PolSAR image classification. The proposed method integrates feature extraction, learning, selection, and classification into an end-to-end network framework. Initially, three kinds of complementary features are extracted to serve as inputs to the network, including polarimetric original data, statistical and scattering features, textural and contour features. Subsequently, a lightweight DeeplabV3+ network is designed to conduct multi-scale feature learning on the extracted multidimensional features. Finally, an attention mechanism-based feature selection module is integrated into the network model, adaptively learning weights for multi-scale features. This enhances discriminative features but suppresses redundant or confusing features. Experiments are conducted on five real PolSAR data sets, and experimental results demonstrate the proposed method can achieve more precise boundaries and smoother regions than the state-of-the-art algorithms. In this paper, we develop a novel multi-feature learning framework, achieving a fast and effective classification network for PolSAR images.

1. Introduction

Polarimetric synthetic aperture radar (PolSAR) is an active microwave remote sensing detection technology, which is capable of obtaining multi-channel back-scattered echoes from the Earth’s surface. It possesses the capability of imaging in all weather and day-night. In comparison to Synthetic Aperture Radar (SAR) system [1], PolSAR system can capture a broader range of polarimetric characteristics and target scattering information, which has found widespread applications in various fields, including land cover classification [2], target identification [3], and ship detection [4] etc. The classification of PolSAR image is a crucial prerequisite in comprehending and interpreting PolSAR data, thus attracting increasing attention from researchers in the field of remote sensing.

Over the decades, numerous classification methods for PolSAR images have been proposed by researchers. The conventional approaches predominantly exploit shallow features to classify PolSAR images, which can be broadly categorized into three types. The first category involves statistical characteristic-based classification methods, notably including Wishart distribution [5], KummerU distribution [6], as well as the Wishart mixture distribution model [7] etc. These methods are effective in modeling the statistical distribution of PolSAR data. The second category comprises classification methods based on target decomposition [8], such as model-based polarimetric decomposition [9], polarimetric covariance eigenvalues [10], dominant scattering mechanism identification [11] and polarimetric roll-invariant features [12] etc. These methods utilize various scattering features to describe land covers from different views. In addition, machine learning approaches have been widely applied to PolSAR image classification. Widely adopted methods include support vector machine [13] and random forest [14], etc. These techniques aim to take advantages of potent polarimetric features for effective classification. However, these methods are limited to learning shallow characteristics. They encounter challenges in accurately classifying heterogeneous land covers due to the absence of high-level semantic features.

Recently, deep learning has witnessed rapid development in the realm of PolSAR image classification. In contrast to traditional methods, deep learning eliminates the need of manual selection of image features by autonomously extracting deeper features. Typical deep learning models include deep boltzmann machines [15], stacked auto-encoder [16], deep belief network [17], convolutional neural network (CNN) [18] and graph convolutional network [19], among others. To be specific, Zhang et al. [20] introduced a convolutional neural network with complex-valued coding (CV-CNN), enhancing feature extraction capability specifically tailored for PolSAR images. Guo et al. [21] presented an adaptive fuzzy super-pixel algorithm based on polarimetric scattering information for PolSAR image classification. In addition, Zhang et al. [22] introduced transfer learning and unsupervised deep learning networks to improve the classification results. Ai et al. [23] proposed a convolutional auto-encoder that incorporated textural feature fusion for land cover classification in PolSAR image.

With the evolution of deep learning models, various neural networks focusing on semantic segmentation have emerged successively. Notable networks in this domain include FCN [24], Unet [25], and DeepLab series (V1, V2, V3, V3+) [26,27,28,29]. Among them, the DeepLab series have achieved remarkable success in the field of semantic segmentation. Concurrently, in recent years, many PolSAR image classification algorithms based on the DeepLab series have been proposed. For instance, Zhang et al. [30] employed wavelet fusion and PCA dimensional reduction to suppress redundant features, combined with DeepLabV3 network for the classification of PolSAR images. In addition, Zhang et al. [31] incorporated the concept of SKNet units into DeepLabV3+, learning multi-scale polarimetric feature effectively, and thereby better applying to classification tasks. The DeepLabV3+ network [29] not only utilizes improved Xception network to integrate multi-dimensional information effectively, but also design spatial pyramid pooling (ASPP) module to learn multi-scale features. Therefore, DeepLabV3+ method can adaptive learn multi-scale features and semantic information, thereby exhibiting better performance for image classification. Here, we opt for the DeepLabv3+ network as the backbone network for PolSAR image classification. However, with too many parameters, DeepLabv3+ presents a complex network structure with a lot of time cost. To alleviate this problem, we design a lightweight DeepLabv3+ network tailored for PolSAR image classification, which can efficiently learn multi-scale semantic features for PolSAR image.

The aforementioned deep learning approaches empower the network to autonomously learn high-level features of targets, contributing to the enhanced accuracy of PolSAR image classification. However, these methods don’t fully exploit multiple scattering characteristics, which makes it challenging to effectively classify all land cover targets, particularly for heterogeneous ones such as urban areas and forests. It can be observed that significant scattering variations exists even within the interior of a heterogeneous ground object, making it difficult for the original data to categorize them into semantically consistent class. Actually, PolSAR data contain rich scattering features, target decomposition features, and texture features, which can provide complementary information for effective image classification [32,33]. For example, Wang et al. [34] introduced a PolSAR image classification method that jointly considered polarimetric features and adjacency information to gain classification advantage. Shi et al. [35] proposed a topic model for multi-feature joint learning, producing enhanced classification accuracy. Shang et al. [36] presented a CNN model based on spatial features, improving classification performance through a dual-branch network learning spatial features. Xu et al. [37] combined deep CNN with the Freeman-Durden decomposition algorithm to extract polarimetric features from PolSAR data, yielding more accurate result of building extraction. Wu et al. [38] employed a hybrid generative/discriminative network to learn the diversity of PolSAR echo statistical features and image spatial features, combined with variational Bayesian theory to integrate multiple features for PolSAR image classification. Collectively, these methods underscore the importance of leveraging multiple polarimetric features rather than relying solely on the original data.

However, existing deep learning classification methods have not fully exploited and utilized the diverse polarimetric information present in PolSAR images. In addition, it is widely acknowledged that an increased number of scattering features may not necessarily lead to better performance. Therefore, there are still two challenges for existing deep learning methods in PolSAR image classification. (1) How to maximize the utilization of the rich features embedded in PolSAR data to enhance the generalization and robustness of deep learning models. (2) How to adaptively select discriminative features from the multitude of available features, while simultaneously suppressing redundant or confusing features. Addressing these challenges is crucial for enhancing the overall classification performance of PolSAR images.

Fortunately, many feature selection methods have been proposed, such as correlation measurement [39], feature ensemble algorithm [40], dimensionality reduction [41], etc. Besides, the attention mechanism [42] has emerged as a widely used method for feature selection. It can automatically select interest regions and assign different weights for diverse regions, which has found extensive applications in target detection [43], image fusion [44] and classification [45]. However, it’s worth noting that these methods primarily operate on shallow feature selection, limiting their effectiveness in classifying complex terrain objects.

To alleviate these challenges, in this paper, we introduces a novel multi-feature lightweight deeplabV3+ network classification model, referred to as MLDnet. The model is an end-to-end deep network that integrates feature learning, feature selection, and classification together. Initially, we extract multiple scattering features from PolSAR images to provide a comprehensively representation of land cover information, encompassing original data, target decomposition features, textural and contour features. Subsequently, we design a lightweight L-DeeplabV3+ to train the network, facilitating the integration of multiple features and learning multi-scale high-level features, which significantly reduce network parameters. Additionally, an attention mechanism module is embedded after the L-DeeplabV3+ network. This module adaptively assigns weights to multidimensional features, enhancing relevant features for classification while suppressing redundant and confusing ones. Experimental results conducted on five real PolSAR data sets indicate that the proposed MFDAnet achieves higher classification accuracy compared to the state-of-the-art algorithms. The main contributions of the proposed MFDAnet method can be summarized into three key aspects.

(1)

A novel multi-feature deep attention network is proposed, which fully exploits diverse types of polarimetric features as the input, and automatically integrate the multi-feature learning, selection and classification into a unified framework. The multiple features include PolSAR original data, scattering features and image features, providing complementary information from different perspectives.

(2)

A lightweight DeeplabV3+ is developed as the backbone network of the MFDAnet. This architecture facilitates the learning of both multi-scale and multi-feature information, designing a lightweight scheme tailed for PolSAR images, ensuring a fast and effective network.

(3)

Attention mechanism-based feature selection module is embedded in the proposed MLDnet, adaptively learning weights of multi-dimensional features. This module enhances valuable features and suppresses redundant ones, thereby improving the classification performance.

The remaining article is organized as follows. Section 2 provides a detailed explanation of the proposed approach. Section 3 presents the experimental results on five real PolSAR datasets. Section 4 illustrates the discussion about parameter analysis and ablation study. Conclusion and further work are given in Section 5.

2. Proposed Method

In this paper, a novel multi-feature deep attention network (MLDnet) is proposed for PolSAR image classification. It can learn multi-scale and multiple scattering features adaptively by designing a lightweight multi-feature DeeplabV3+ network. Additionally, it selects effective features using the attention mechanism, so as to finally realize the the effective classification of different land cover types.

The framework of our proposed multi-feature deep attention network is shown in Figure 1, comprising three key components: multi-feature extraction, lightweight DeeplabV3+ network design and attention-based feature fusion and classification. Firstly, in contrast to traditional network using 9-dimension PolSAR data as the input, we extract multiple scattering features as the network input. They include original data, scattering mechanism-based and image analysis-based features, fostering enhanced information complement; Secondly, we design a lightweight DeeplabV3+ network to automatically learn features of different scales from multiple features; Finally, the multi-scale multi-features are automatically selected by the channel attention module embedded in the network. This process enhances valuable discriminating features with larger weights while disregarding useless features with smaller weights. The classification layer is then employed to determine the probability of the target, yielding the final classification result. Further details are introduced as follows.

2.1. Multi-Feature Extraction Module

Traditional CNN model usually converts the PolSAR covariance matrix into a 9-dimension real vector as the network input, creating a cube with 9 channels for each pixel. However, 9-dimension vector is not adequate to represent the complexity of PolSAR image for each pixel. Additional information, such as scattering characteristics [46,47], has been proved effective for identifying ground objects. Moreover, the contextual features, including textural and contour features, play a significant role in describing image contextual information. Numerous studies highlight the value of multiple features in enhancing classification performance. Therefore, we propose a multi-feature data representation as the network input. Before multi-feature extraction, a refined Lee filter [48] is applied to the original PolSAR image. Compare with other advanced filter methods [4,49], the refined Lee filter is a classical filtering method to reduce speckle noises fastly and effectively, which can be easily conducted upon the PolSARPro software. Specifically, three types of features are extracted to formulate a 57-dimensional feature vector as PolSAR data expression, which is extracted from PolSAR original data, target decomposition features, as well as textural and contour features respectively. The detailed feature extraction procedure is outlined below.

(1) PolSAR original data (16-dimension): It consists of scattering matrix $\mathbf{S}$, coherency matrix $\mathbf{T}$ and SPAN image. The original PolSAR data is scattering matrix $\mathbf{S}$, defined as:

$$S=\left[\begin{array}{cc}{S}_{hh}& S_{hv}\\ S_{vh}& S_{vv}\end{array}\right]$$

(1)

After multi-look processing, the most commonly used coherency matrix $\mathbf{T}$ is derived by Equation (2), which is shown on the top of the next page.

$$〈T〉={\displaystyle \frac{1}{2}}\left[\begin{array}{ccc}〈{\left|S_{hh}+S_{vv}\right|}^2〉& 〈\left(S_{hh}+S_{vv}\right){\left(S_{hh}-S_{vv}\right)}^{*}〉& 2〈\left(S_{hh}+S_{vv}\right)S_{hv}^{*}〉\\ 〈\left(S_{hh}-S_{vv}\right){\left(S_{hh}+S_{vv}\right)}^{*}〉& 〈{\left|S_{hh}-S_{vv}\right|}^2〉& 2〈\left(S_{hh}-S_{vv}\right)S_{hv}^{*}〉\\ 2〈S_{hv}{\left(S_{hh}+S_{vv}\right)}^{*}〉& 2〈S_{hv}{\left(S_{hh}-S_{vv}\right)}^{*}〉& 4〈{\left|S_{hv}\right|}^2〉\end{array}\right]$$

(2)

In Equation (2), ∗ represents the complex conjugate operation. Since the complex matrix cannot be fed to the network directly, similar to other networks, we convert the coherency matrix $\mathbf{T}$ to a 9-dimension vector, defined as

$$\begin{array}{cc}\hfill I\left(T\right)=& [T_{11},T_{22},T_{33},T_{12real},T_{13real},T_{23real},T_{21imag},T_{31imag},T_{32imag},]\hfill \end{array}$$

(3)

where $T_{12real}$ is the real part of $T_{12}$, and $T_{12imag}$ is the imaginary part of $T_{12}$. In addition, the Polarimetric-Total-Power (SPAN) value is the power image, which is calculated by summarizing the diagonal elements of $\mathbf{T}$ matrix, defined by

$$SPAN={\left|S_{hh}\right|}^2+2{\left|S_{hv}\right|}^2+{\left|S_{vv}\right|}^2$$

(4)

(2) Target decomposition-based features (17-dimension): It contains three target decomposition-based parameters and two polarimetric parameters, which can describe target’s scattering mechanism from various perspectives. Three types of target decompositions are Cloude and Pottier decomposition [50], Freeman decomposition [51] and Huynen decomposition [52]. These decomposition methods have been proven effective for PolSAR image classification, serving as valuable features for distinguishing different ground objects. Furthermore, the two polarimetric parameters are the co-polarization ratio [53] and the cross-polarization ratio [54]. The former is used to measure the difference between $S_{hh}$ and $S_{vv}$, and the latter is highly sensitive to volume scattering. All these scattering features represent independent attributes of each resolution unit, providing distinct descriptions for each pixel.

(3) Image features (24-dimension): In addition to the original data and scattering features, each pixel, acting as a unique scattering unit, also has a close relationship with neighborhoods to formulate a PolSAR image, that is contextual features such as texture and contour. These features play a crucial role in identifying the ground objects. To capture the contextual relationship, we extract Gray-level Co-occurrence Matrix (GLCM) features [55], a widely used technique for textual feature extraction in natural images. It encompasses four types of features, including contrast, energy, entropy and relativity. Additionally, we extract the edge and line energies using filters with 18 directions and 4 scales, providing information on the contour and strength of image edges [56]. All 57-dimension features are summarized into

Table 1. 57-dimension features list.

Feature	Feature Name	Feature Parameter	Number
Original data features	Scattering matrix elements	$real\left(S_{hh}\right),img\left(S_{hh}\right),real\left(S_{hv}\right),img\left(S_{hv}\right)$, $real\left(S_{vv}\right),img\left(S_{hv}\right)$	6
	Coherency matrix elements	$T_{11},T_{22},T_{33},real\left(T_{12}\right),img\left(T_{12}\right),real\left(T_{13}\right)$, $img\left(T_{13}\right),real\left(T_{23}\right),img\left(T_{23}\right)$	9
	SPAN	$span={\left|S_{hh}\right|}^2+{\left|S_{hv}\right|}^2+{\left|S_{vv}\right|}^2$	1
Target decomposition features	Cloud and Pottier decomposition	H, A, $\alpha$	3
	Freeman decomposition	the surface, double-bounce and volume scattering power	3
	Huynen decomposition	$A_0$, $B_0$, $B,C,D,E,F,G,H$	9
	Co-polarization ratio	$r_o={\displaystyle \frac{〈S_{vv}{S}_{vv}^{*}〉}{〈S_{hh}{S}_{hh}^{*}〉}}$	1
	Cross-polarization ratio	$r_x={\displaystyle \frac{〈S_{hv}{S}_{hv}^{*}〉}{〈S_{hh}{S}_{hh}^{*}〉}}$	1
Textural and contour features	GLCM features	$Contrast:con={\displaystyle \sum _i}{\displaystyle \sum _j}{(i-j)}^{2}P\left(i,j\right)$	4
		$Energy:Asm={\displaystyle \sum _i}{\displaystyle \sum _j}P{\left(i,j\right)}^2$	4
		$Entropy:Ent=-{\displaystyle \sum _i}{\displaystyle \sum _j}P\left(i,j\right)logP\left(i,j\right)$	4
		$Relativity:Corr=\raisebox{1ex}{$\left[{\displaystyle \sum _i}{\displaystyle \sum _j}\left((i,j)p(i,j)\right)-{\mu }_x{\mu }_y\right]$}\!\left/ \!\raisebox{-1ex}{${\sigma }_x{\sigma }_y$}\right.$	4
	Edge-line energy features	$E_{edge}=$$\raisebox{1ex}{$\left(\frac{1}{n}{\displaystyle \sum _{i=1}^n}{w}_i{x}_i\right)$}\!\left/ \!\raisebox{-1ex}{$\left(\frac{1}{m}{\displaystyle \sum _{j=1}^m}{w}_j{x}_j\right)$}\right.$	4
		$E_{line}=\min \{E_{\mathit{edge}}^{\mathit{ij}}$, $E_{\mathit{edge}}^{\mathit{jk}}\}$	4
Total			57

.

2.2. Lightweight DeepLabV3+ Network

The DeepLab series [26,27,28,29], a collection of semantic segmentation algorithms developed by the Google team, has demonstrated notable advancements in semantic learning. Leveraging depth-wise separable convolution technology, DeeplabV3+ efficiently integrates features in the depth direction, requiring fewer parameters. In addition, the DeepLabV3+ network introduces the Atrous Spatial Pyramid Pooling (ASPP) technique to integrate multi-scale information. This technique utilizes convolution layers with different dilation factors, increasing the receptive field and reducing information loss during down sampling. Therefore, we select the DeeplabV3+ network as the backbone of the proposed method, since it can integrate multi-dimensional features and learn multi-scale semantic information effectively.

However, the original backbone network used by DeeplabV3+ network is an improved Xception network, as depicted in Figure 2, featuring a complex network structure. Furthermore, in semantic segmentation tasks, the input unit is typically an image, while for PolSAR image classification, the input unit is a small image block, with a smaller input feature block and a larger data amount. So, considering these differences between semantic segmentation and pixel-wise classification, a revised DeeplabV3+ network should be tailored for PolSAR image classification.

In this paper, we develop a lightweight DeeplabV3+ network, named L-DeeplabV3+, as an alternative. Taking a PolSAR image with $100\times 100$ pixels as an example,

Table 2. Parameter comparison between DeeplabV3+ and L-DeeplabV3+ networks.

Network	DeeplabV3+	L-DeeplabV3+
Input feature map	$100\times 100$	$16\times 16$
Backbone network	Xception network	DWConv $5\times 5$ + Maxpooling + DWConv $3\times 3$
ASPP	Conv $3\times 3$, r = 6, 12, 18	Conv $3\times 3$, r = 2, 4, 6

provides the comparison of network parameters between the L-DeeplabV3+ network used in this article and the original DeeplabV3+network. In addition, Figure 3 also illustrates the structure differences between the proposed L-DeeplabV3+ and original DeeplabV3+ networks, which can be summarized into three key aspects.

(1) Input: First of all, as mentioned earlier, the whole image serves as the network input in the semantic segmentation task in Figure 3a. However, for PolSAR image classification task, the input image block is sampled by a square window centered on each pixel, as shown in Figure 3b. It also can be found in Table 2 that the size of the input feature map are $100\times 100$ (image size) and $16\times 16$ (window size) for DeeplabV3+ and L-DeeplabV3+ networks respectively.

(2) Backbone network: For the backbone network, the DeeplabV3+ network employs an improved Xception network, as shown in Figure 2, adept at handling larger input features. However, it may lead to overfitting when dealing with small input blocks. As depicted in Table 2, the proposed L-DeeplabV3+ network adopts two depth-wise separable convolutional layers(shorted by “DWConv”) as the backbone network. This design allows for the extraction of shallow features without significantly reducing the feature dimension. In addition, we introduce a max-pooling layer between the two DWConv layers to further reduce the number of model parameters and mitigate the risk of model overfitting.

(3) Multi-scale feature learning: In addition, as is well known, with the increasing dilated rate, the receptive field will also become larger. Therefore, in the original DeeplabV3+, a relatively large dilation rate is set to achieve the segmentation of large objects. However, for a PolSAR image block, an excessively large dilation rate can lead to the loss of local detailed information, thereby affecting the final classification results. In L-DeeplabV3+, we address this issue by significantly reducing the dilation rate in the ASPP module. This adjustment allows the network to capture multi-scale contextual information while preserving local detailed information. The detailed parameter comparison can be seen in Table 2.

In summary, the proposed multi-feature-based L-DeeplabV3+ network achieves multi-scale feature learning for the multi-dimensional features of PolSAR images. This is accomplished while maintaining the original DeeplabV3+ network framework and further reducing network parameters and unnecessary time overhead.

2.3. Attention-Based Multi-Feature Fusion and Classification

After processing through L-DeeplabV3+ network, we attain multi-scale multi-feature information in channel dimension, which can provide rich contextual features for classification. However, recognizing that not all scattering features contribute equally to classification, the selective extraction of valuable features while suppressing less relevant ones remains a challenging issue. In this paper, we ensemble an attention-based multi-feature fusion module with the L-deeplabV3+ network. This module dynamically learns different weights for diverse scattering features in channel direction, highlighting important features and diminishing less useful ones. Ultimately, we achieve multi-scale multi-feature fusion in channel dimension.

The multi-feature fusion module utilizes the attention mechanism [42] to adaptive weight the channel features, illustrated in Figure 4. Channel attention pays attention on exploring “what” is the meaningful features for the input image on the channel dimension. It begins by separately aggregating multi-scale spatial features through max-pooling and average-pooling layers. This process generates two distinct feature maps, each encapsulating spatial contextual information. Subsequently, these two feature maps are input to a shared fully connection layer, producing their respective channel attention feature maps. The next step involves summing these two attention feature maps, followed by the application of an activation function. This results in the aggregation of their information, ultimately yielding channel attention weights.

Assuming that the size of the output feature map X by L-deeplabV3+ network is $M\times N\times C$, the obtained channel attention weight is W, with the size of $1\times 1\times C$, through the attention mechanism. The procedure of the channel attention module can be described as:

$$\begin{array}{cc}\hfill W=& sigmoid(W_2·Relu\left(W_1·\left(AvgPool\left(X\right)\right)\right)\hfill \\ \hfill & +W_2·Relu\left(W_1\left(MaxPool\left(X\right)\right)\right))\hfill \end{array}$$

(5)

where X is the learned feature map from L-DeeplabV3+, $W_1$ and $W_2$ are the parameters of two fully connecting layers and $ReLu$ is the non-linear translation. $AvgPool\left(·\right)$ and $MaxPool\left(·\right)$ are average and max pooling layers respectively. $Sigmoid$ is the activation function.

The feature map $X{\in }^{H\times W\times C}$ is then multiplied by the weight $W{\in }^{1\times 1\times C}$ to obtain the channel attention fusion features. W is firstly padded to $H\times W\times C$ by $pad(.)$ function, and then X is element-wise multiplied by W to obtain weighted features, defined as:

$$\mathrm{F}=\mathrm{X}\otimes pad\left(\mathrm{W}\right)$$

(6)

where ⊗ represents element-wise multiplication. After obtained the fused feature F, a $softmax$ classifier is applied to achieve the final classification map.

The whole algorithm procedure of the proposed MLDnet is given in Algorithm 1. The algorithm begins by applying the refined Lee filter to mitigate speckle noise. Subsequently, multiple features are extracted from PolSAR images, including original PolSAR data, target decomposition features, textual and contour features. These multi-features are then input into the L-DeeplabV3+ network to learn the multi-scale high-level features. Furthermore, an attention mechanism is applied to the multi-scale features, enabling the automatic selection and fusion of multiple features by learning different weights. This ensures that valuable features are assigned larger weights while redundant features receive smaller weights. Finally, the output of the MLDnet passes through a $softmax$ classifier to obtain the final classification result.

Algorithm 1 Procedure of the proposed MLDnet method.

Input: PolSAR original data $\mathbf{S}$, PolSAR original image $\mathbf{P}$, class label map $\mathbf{L}$ and class number $\mathbf{C}$. Step 1: Apply a refined Lee filter to PolSAR original image $\mathbf{P}$ to obtain the filtered PolSAR image $P_L$. Step 2: Extract 57-dimensional features F from PolSAR image $P_L$ and PolSAR original data $\mathbf{S}$ based on Table 1. Step 3: Using a fixed-size square window $h\times h$ to sample the extracted 57-dimensional features F pixel by pixel, obtaining N data with a shape of $h\times h\times 57$, where N is the total number of image pixels. Step 4: According to a certain ratio, the N data obtained in step 3 and the label map L are divided into training set $\mathbf{R}$ and test set $\mathbf{T}$. Step 5: Import the training set $\mathbf{R}$ into the MLDnet network for training until reaching the iteration number, saving the training model, training loss, and model parameters. Step 6: Using the trained model to predict and classify the test set $\mathbf{T}$. Output: class label estimation map $\mathbf{Y}$ and various evaluation indicators.

3. Experimental Results

3.1. Experimental Data and Settings

In order to verify classification performance of the proposed MLDnet, extensive experiments are conducted on five real PolSAR data sets. These data sets cover Xi’an, Oberpfaffenhofen, San Francisco and Flevoland areas respectively. Detailed information is given as follows:

(A) Xi’an data set: The first data set is C-band full polarimetric SAR data collected from Xi’an in the western region of China using RADARSAT-2 system. The image size is $512\times 512$ pixels, with a resolution of $8\times 8$ m. It encompasses three types of land cover classes: building, grass, and water. The Pauli-RGB pseudo-color image and the corresponding ground truth image are presented in Figure 5a.

(B) Oberpfaffenhofen data set: The second data set covers the Oberpfaffenhofen area in German and consists of an 8-look L-band fully PolSAR image acquired by the E-SAR sensor at the German Aerospace Center. This polarimetric radar image has the dimension of $1300\times 1200$ pixels with a resolution of $3\times 2.2$ m. It includes five distinct land cover types: bare ground, forest, farmland, road and building. Figure 5b illustrates the Pauli RGB image of the Oberpfaffenhofen data set along with the corresponding ground truth map.

(C) San Francisco data set: The third data set is the San Francisco data set, which is a C-band fully PolSAR image collected by the RADARSAT-2 system over the San Francisco area in the United States. The image has a resolution of $10\times 5$ m and a size of $1800\times 1380$ pixels. Its Pauli RGB image and ground truth map are displayed in Figure 5c, which contains five different land cover types: ocean, vegetation, low density urban, high density urban, and developed.

(D) Flevoland1 data set: The fourth data set is C-band fully polarimetric SAR data collected by the RADARSET-2 sensor in the Flevoland region of the Netherlands. The image size is $1400\times 1200$ pixels, with a resolution of $12\times 8$ m. The PolSAR image encompasses four types of land cover, including water, urban, woodland, and cropland. We name this data set the Flevoland1 data set. Figure 5d exhibits the Pauli-RGB image along with the corresponding ground truth map of the Flevoland1 data set.

(E) Flevoland2 data set: The last data set comes from the Flevoland region and consists of full-polarization L-band SAR data acquired by the AIRSAR system. The spatial resolution is $12.1\times 6.7$ m, and the image size is $750\times 1024$ pixels. The Pauli RGB image and its ground truth are shown in Figure 5e. This image includes 15 types of crops: stem beans, peas, forest, lucerne, beets, wheat, potatoes, bare soil, grasses, rapeseed, barley, wheat2, wheat3, water and buildings. We name it the Flevoland2 data set.

The quantitative metrics employed in our experiments include average accuracy (AA), overall accuracy (OA), and Kappa coefficient. Experiments are conducted on Windows 10 operating system, using a 64 GB NVIDIA GeForce GTX 3070 PC. The algorithm is implemented with Python 3.7 and Pytorch GPU 1.9.0. In addition, we utilize the Adam optimizer with a learning rate of 0.0001. The batch size is defined as $16\times 16$, and the number of training iteration is 50. The multi-class cross-entropy loss is employed. During the experimental process, 10% of the training samples are selected for each class of land cover in the image, with the remaining 90% serving as test samples. It should be noted that the white regions in the ground truth maps are unlabeled pixels in Figure 5. During classification, all pixels in PolSAR images can obtain classification result, while unlabeled pixels are not considered into calculating quantitative metrics. Therefore, we just give the classification map with labeled pixels, and unlabeled pixels are colored by white in the following experimental results.

To assess the effectiveness of the proposed MLDnet, we compare it with seven state-of-the-art classification algorithms: Super-RF [14], CNN [18], CV-CNN [20], 3D-CNN [57], PolMPCNN [58], CEGCN [59] and SGCN-CNN [60]. Specifically, the first compared method combines the random forest algorithm with superpixels, utilizing $G^0$ statistical texture features to mitigate the interference from background targets, such as forests, on farmland classification, referred to as Super-RF. The second method, referred to as CNN, converts PolSAR data into normalized 9-D real feature vectors and learns hierarchical polarimetric spatial features automatically through two cascaded convolutional layers. The third method is a complex-valued convolutional neural network, named by CV-CNN. This approach transforms the PolSAR complex matrix into a complex vector, effectively leveraging both the amplitude and phase information presented in PolSAR images. The fourth method is based on 3-D convolutional neural network, shorted by 3D-CNN. It utilizes 3-D CNN to extract deep channel-spatial combined features, adapting well to the 3D data structure for classification. The fifth method combines polarimetric rotation kernels, employs a multi-path structure and dual-scale sampling to adaptively learn the polarimetric rotation angles of different land cover types, noted by PolMPCNN. The sixth method employs a graph encoder and decoder to facilitate collaboration between CNN and GCN within a single network, shorted by CEGCN. It enables feature learning in small-scale regular and large-scale irregular regions, respectively. The last approach similarly integrates super-pixel level and pixel-level networks into a single framework, named SGCN-CNN for short. It can obtain both global features and local features by defining the correlation matrix for feature transformation between superpixel and pixel.

3.2. Experimental Results on Xi’an Data Set

Classification results on the Xi’an data set are displayed in Figure 6b–i, illustrating the classification maps generated by Super-RF, CNN, CV-CNN, 3D-CNN, PolMPCNN, CEGCN, SGCN-CNN and the proposed MLDnet methods, respectively. Specifically, Figure 6b displays the output of the Super-RF method, revealing numerous misclassified pixels, primarily concentrated around the water class. At the same time, these misclassifications form distributed blocks due to superpixels. In contrast, the CNN method performs exceptionally well in distinguishing the water class, but exhibits misclassifications in the building class due to the lack of global information. It erroneously classifies some building as grass class. Similar to the CNN method, the CV-CNN exhibits noticeable misclassifications in the building and grass classes due to pixel-wise local features. 3D-CNN can adapt to the 3D structure of input data for classification, thus achieving better regional consistency. However, it struggles to adaptively handle distinct individual features, limiting its classification effectiveness at the boundaries of different land covers. In comparison, PolMPCNN excels in adaptively learning the polarimetric characteristics of various land covers, greatly improving overall classification effect. However, the feature learning procedures of different terrains are independent, resulting in a small number of misclassified pixels still existing at the land covers boundaries. Compared to PolMPCNN, the CEGCN method utilizes two branches, CNN and GCN, to generate complementary contextual features at the pixel and super-pixel levels, significantly improving classification accuracy at boundaries of diverse land covers. However, the CEGCN method cannot handle the inherent speckle noise in PolSAR images well, so there are some noise points in the image. The SGCN-CNN method, similar to the CEGCN method, also struggles to effectively handle the inherent speckle noise in images, resulting in the generation of a significant number of noise points. The proposed MLDnet combines the L-DeeplabV3+ network to deeply explore multi-dimensional features of PolSAR image, and select effective features through the incorporation of a channel attention module. Compared to six contrastive methods, the proposed MLDnet method in (h) demonstrates substantial improvements in both regional consistency and edge details.

Table 3. Classification accuracy of different methods on Xi’an Data Set (%).

Class	Super-RF [14]	CNN [18]	CV-CNN [20]	3D-CNN [57]	PolMPCNN [58]	CEGCN [59]	SGCN-CNN [60]	MLDnet
Water	70.91	93.12	97.26	90.27	95.52	94.47	86.57	94.10
Grass	94.97	92.82	85.97	93.60	90.95	96.50	93.33	97.72
Building	90.94	85.31	91.43	93.91	97.68	96.54	90.12	98.28
OA	89.94	90.21	89.59	93.21	94.01	96.21	91.18	97.38
AA	85.61	90.42	91.55	92.60	94.71	95.83	90.01	96.70
Kappa	83.02	83.83	83.18	88.77	90.25	93.74	85.33	95.67

The highest accuracy in each line is highlighted in bold.

lists the classification accuracy of all methods in the Xi’an data set. It can be seen that the Super-RF method without deep features has lower accuracy than other methods. The main reason is that the classification accuracy of water class is only 70.91%, which is relatively consistent with the classification result in Figure 6b. In comparison, CNN has greatly improved its accuracy in water class, but its accuracy in the building class is relatively lower, resulting in the low overall accuracy. It only improved by 0.27% compared to Super-RF. CV-CNN further improves the classification accuracy of the water class, with the highest accuracy among all methods. However, its classification accuracy in the grass class is the lowest among all methods, which also results in its OA value being slightly lower than CNN. Compared to the previous three methods, 3D-CNN has achieved a significant improvement in OA value, and PolMPCNN has further improve the performance, especially in the building class, with the accuracy of 97.68%, which is significant higher than other compared methods. Analogously, due to better distinguishing grass class, CEGCN has further improved the overall classification accuracy. The SGCN-CNN method experiences a decline in overall classification metrics due to a significant number of misclassifications within grass class. The proposed MLDnet achieves the best classification accuracy in both two land cover types and three overall evaluation indicators. Specifically, the OA values are improved by 7.44%, 7.17%, 7.79%, 4.17%, 3.37%, 1.17% and 6.20% compared to other methods.

Figure 7 shows the confusion matrix of the Xi’an data set. From the figure, it can be seen that the proposed method has the less misclassified pixels in both grass and building classes than other compared methods. The main confusion of the proposed method lies on misclassifying 2030 (about 5.70%) pixels of water class as grass, while misclassifying 1654 (about 1.40%) pixels of grass as water class.

3.3. Experimental Results on Oberpfaffenhofen Data Set

The classification maps produced by compared and proposed methods are illustrated in Figure 8b–i, respectively. As shown in Figure 8b, the Super-RF method effectively distinguishes bare ground and farmland classes due to large number of samples. However, for classes with fewer samples such as building, forest, and road classes, misclassification is more noticeable. The CNN method shown in (c) has enhanced classification performance for building and road classes. However, its effectiveness in differentiating the farmland class remains suboptimal. In comparison, the CV-CNN method in (d) shows a marginal improvement in mitigating misclassification issues related to the farmland class. Nevertheless, it exacerbates misclassification in the bare ground class in the upper left corner, and overall regional consistency remains relatively poor. The 3D-CNN method in (e) exhibits similar to the Super-RF method, displaying a notable number of misclassified pixels in the road and farmland classes. Despite this, there is an enhancement in building class. Compared to the preceding four methods, the PolMPCNN method in (f) demonstrates a substantial improvement in regional consistency. However, this method almost entirely misclassifies the road class in the lower half of the images as the building class. Additionally, the boundaries between different land covers appear blurry. CEGCN has further enhanced the classification performance of the road class, building upon the foundation laid by PolMPCNN. However, the effectiveness is still suboptimal. The SGCN-CNN method in (h) likewise demonstrates subpar performance when dealing with classes with fewer samples, such as road and farmland. The proposed MLDnet method, as exhibited in (i), demonstrates a remarkable improvement in the classification performance of all five land covers compared to other contrastive methods. Furthermore, it effectively preserves the boundaries among different land covers.

Table 4. Classification accuracy of different methods on Oberpfaffenhofen Data Set (%).

Class	Super-RF [14]	CNN [18]	CV-CNN [20]	3D-CNN [57]	PolMPCNN [58]	CEGCN [59]	SGCN-CNN [60]	MLDnet
Bare ground	91.76	85.95	68.86	91.76	86.92	93.14	92.94	96.97
Forest	85.26	83.22	81.16	84.59	82.51	90.13	81.96	96.17
Building	59.55	83.50	87.52	83.91	85.82	94.46	87.60	95.77
Farmland	57.28	53.44	70.49	65.33	65.65	80.70	32.02	95.50
Road	49.19	67.41	76.20	50.34	10.38	66.14	14.67	93.78
OA	78.40	80.87	74.88	82.82	75.82	88.93	77.29	96.19
AA	68.59	74.70	76.85	75.19	66.26	84.91	61.84	95.64
Kappa	67.42	72.03	65.73	74.28	63.46	83.66	64.76	94.44

The highest accuracy in each line is highlighted in bold.

compares the classification accuracy of different methods in the Oberpfaffenhofen area. From the table, it can be visually observed that the seven compared methods have relatively low OA values on the Oberpfaffenhofen data set, with the highest value being only 88.93% (CEGCN). Moreover, among the five land cover types, bare ground class exhibits relatively high accuracy in discrimination. The four comparative methods (Super-RF, 3D-CNN, CEGCN and SGCN-CNN) all achieve accuracy rates exceeding 90%. Meanwhile, all seven comparative methods have an accuracy rate surpassing 81% for forest class. However, for the remaining three classes, particularly farmland and road classes, none of the seven comparative methods achieve satisfactory performance. The highest discrimination rates are 80.70% (CEGCN) for farmland class and 76.20% (CVCNN) for road class. Notably, for the road class in PolMPCNN, the correct discrimination rate is only 10.38%, consistent with the classification result displayed in Figure 8f. This indicates that for data sets with unbalanced samples, such as Oberpfaffenhofen, the classification performance of the six comparison methods is constrained. However, The proposed MLDnet method excels in achieving the highest classification accuracy across all evaluation metrics, with values consistently exceeding 93%. Furthermore, in comparison to the seven comparative algorithms, the OA value sees an increase of 17.79%, 15.32%, 21.31%, 13.37%, 20.37%, 7.26% and 18.90%, respectively. This highlights the superior capability of the proposed method in effectively handling complex scenes of PolSAR images and enhancing classification accuracy.

Additionally, the confusion matrices for the Oberpfaffenhofen data set are presented in Figure 9. Our method demonstrates the highest number of correctly classified pixels in the dialog elements, indicating superior classification performance across all classes. The CV-CNN shows the most misclassifications in the bare ground category. The Super-RF frequently misclassifies pixels from buildings as bare ground. Furthermore, the SGCN-CNN misclassifies a significant number of pixels from farmland and road into the bare ground class, which aligns with the classification map shown in Figure 8h. It is evident that the primary confusion across all methods occurs between the bare ground and road classes, where our method outperforms the others.

3.4. Experimental Results on San Francisco Data Set

The experimental results of seven contrastive methods and MLDnet algorithms on the San Francisco data set can be seen in Figure 10b–i, respectively. Figure 10a displays the ground truth map of the San Francisco area. Upon comparing the ground truth map, it is evident that super-RF produces many misclassifications in high density urban (shorted by “HD urban”) class with significant surface scattering differences. It also exhibits a portion of confusion between vegetation and developed classes. The CNN method also faces challenges in distinguishing high density urban class from developed class. In comparison to the previous two methods, the classification performance of CV-CNN has been improved in high density urban and developed classes, while there are still a small number of misclassified pixels. Simultaneously, the misclassification phenomenon in low density urban (shorted by “LD urban”) class is also quite evident. The 3D-CNN method demonstrates improved classification performance on high-density urban, low-density urban, and developed classes but exhibits more misclassified pixels in the vegetation class. PolMPCNN improves classification performance in various classes. However, its classification map still contains some visually apparent misclassified pixels. CEGCN and SGCN-CNN almost perfectly distinguish various land covers; however, upon closer observation, it is not difficult to notice that they have some misclassified pixels along land cover boundaries. The proposed MLDnet method also effectively distinguishes various land covers, meanwhile, it preserves the boundaries of different land cover types well. This suggests that the MLDnet method is effective in achieving good classification results even for large-scale PolSAR data sets.

In addition,

Table 5. Classification accuracy of different methods on San Francisco Data Set (%).

Class	Super-RF [14]	CNN [18]	CV-CNN [20]	3D-CNN [57]	PolMPCNN [58]	CEGCN [59]	SGCN-CNN [60]	MLDnet
Ocean	99.98	99.59	99.99	100	99.58	99.99	100	100
Vegetation	93.89	93.51	96.42	94.80	95.63	97.87	97.40	99.89
LD urban	97.31	94.47	94.51	97.96	99.39	99.58	99.48	99.97
HD urban	77.76	92.57	96.37	97.61	98.57	99.49	99.48	99.98
Developed	81.00	90.31	95.92	96.25	96.81	99.88	99.95	100
OA	94.33	96.28	97.71	98.38	98.93	99.55	99.47	99.97
AA	89.99	94.09	96.64	97.32	98.08	99.36	99.26	99.97
Kappa	91.81	94.65	96.70	97.66	98.46	99.35	99.24	99.96

The highest accuracy in each line is highlighted in bold.

provides the classification accuracy of all methods for various land covers in the San Francisco data set, along with the corresponding OA, AA, and Kappa. Based on the table, it can be observed that, in comparison to the other three land cover types, both the Super-RF and CNN methods exhibit relatively low accuracy in classifying high density urban and developed classes. This is especially evident for the Super-RF method, with precision rates of 77.76% and 81.00%, respectively. CV-CNN exhibits a relatively low accuracy rate in the low density urban class. In comparison, the classification accuracy by 3D-CNN and PolMPCNN methods is higher, with OA values both above 98%. The CEGCN and SGCN-CNN methods both exhibit excellent performance across five different land cover types, with accuracy exceeding 99% for all categories except vegetation class. In contrast, the proposed MLDnet method achieves the highest class accuracy for all five land cover types, demonstrating superior classification performance. In terms of overall metrics, using OA metric as an example, compared to the seven other comparative algorithms, MLDnet improves OA by 5.64%, 3.69%, 2.26%, 1.59%, 1.04%, 0.42%, and 0.50% respectively.

The confusion matrices for the San Francisco data set are illustrated in Figure 11. Our method shows the least misclassification across all categories. The CV-CNN exhibits the highest misclassifications, particularly misclassifying ocean as vegetation. The Super-RF frequently misclassifies pixels from high density urban as low density urban. Additionally, the Super-RF, CNN, and CV-CNN methods show significant misclassifications within the develop class. It is clear that all methods struggle with misclassifications in heterogeneous regions, while our method demonstrates the best performance.

3.5. Experimental Results on Flevoland1 Data Set

The experimental results of various methods on the Flevoland1 data set in the Netherlands are depicted in Figure 12. Specifically, the Super_RF in (b) is a low-level feature learning method that exhibits limitation in effectively classifying urban class. In contrast, the CNN method in (c) demonstrates some enhancement in the classification of land cover boundaries; however, the overall classification performance remains inadequate, marked by numerous noisy points in the image. The CV-CNN method in (d) and the 3D-CNN method in (e) have enhanced regional consistency to some extent, but the improvement effect is not significant. There are still relatively noticeable misclassification pixels in the classification maps. PolMPCNN in (f) further enhances the regional consistency, effectively distinguishing between water and woodland classes, with only a small amount of misclassified pixels in cropland and urban classes. CEGCN in (g) and SGCN-CNN in (h) further enhance the ability to distinguish various land cover types, with only a few misclassifications occurring at the boundary between cropland and urban classes. The proposed MLDnet in (i), leveraging a 57-dimensional multi-feature and the L-DeeplabV3+ deep learning network, exhibits a notable improvement compared to previous methods. There are almost no obvious misclassification phenomena in the image. The consistency of regions and the preservation of boundaries have been significantly enhanced.

In addition, to quantitatively assess the classification performance, the classification accuracies of all methods on the Flevoland1 data set are presented in

Table 6. Classification accuracy of different methods on Flevoland1 Data Set (%).

Class	Super-RF [14]	CNN [18]	CV-CNN [20]	3D-CNN [57]	PolMPCNN [58]	CEGCN [59]	SGCN-CNN [60]	MLDnet
Urban	81.84	88.87	96.26	94.74	96.29	99.45	99.73	99.98
Water	98.69	99.68	99.85	98.87	99.14	99.65	99.77	99.99
Woodland	94.92	95.59	96.48	96.03	98.77	98.96	99.58	99.89
Cropland	94.16	93.16	93.94	96.75	98.66	99.47	99.63	99.92
OA	93.88	95.01	96.57	96.84	98.49	99.37	99.67	99.94
AA	92.40	94.33	96.63	96.60	98.21	99.38	99.68	99.95
Kappa	91.61	93.18	95.33	95.69	97.94	99.14	99.55	99.92

The highest accuracy in each line is highlighted in bold.

. It is evident that Super-RF attains the lowest correct classification rate of 81.84% in the urban class. CNN method achieves the lowest classification accuracy in the cropland class, at 93.16%. CV-CNN demonstrates great improvements, especially for the water class, with a correct discrimination rate second only to the proposed method MLDnet, reaching 99.85%. 3D-CNN method improves the classification accuracy in the cropland class compared to the previous three methods, with increases of 2.59%, 3.59%, and 2.81% respectively. PolMPCNN method further improves the classification metrics for various classes in the Flevoland1 data set, achieving an OA value of 98.49%. The CEGCN and SGCN-CNN further improve the classification accuracy of various types of land cover, with numerical values consistently above 98%. Additionally, it is noticeable that multiple comparison algorithms exhibit relatively lower correct classification rates for the urban class. This suggests that the classification of the urban class is more difficult. However, it is obvious that the proposed MLDnet excels in classifying all types of land covers and in overall classification performance. Especially for the urban class, the accuracy has also reached 99.98%. This demonstrates the effectiveness of the proposed method.

The confusion matrices for the Flevoland1 region are presented in Figure 13. It is clear that the proposed method exhibits the least misclassification across all four classes. Notably, the highest misclassification for our method occurs with 236 pixels misclassified from woodland to cropland, yet this is still the lowest count compared to other methods. Specifically, for the urban and water classes, our method shows only around 30 misclassified pixels, while other methods report misclassification numbers in the hundreds or even thousands.

3.6. Experimental Results on Flevoland2 Data Set

The experimental results using the proposed MLDnet method and seven comparison methods on the Flevoland2 data set are shown in Figure 14. By comparing these results with the label map in (a), it is evident that the superpixel-level classification method, Super-RF, and the pixel-level classification method, CNN, exhibit numerous patchy misclassified areas and scattered misclassification noise across multiple classes. In contrast, the CV-CNN method achieves better classification results, though it still shows noticeable misclassified pixels in the rapeseed class. Furthermore, due to the lack of global information, the 3D-CNN method generates spotty noise in multiple classes such as stem beans, wheat, and rapeseed. PolMPCNN effectively learns various land cover features through a multi-path architecture but still has limitations in distinguishing similar land covers such as barley, wheat2, wheat3, and rapeseed. CEGCN combines pixel-level and superpixel-level features effectively, resulting in good differentiation of various land covers, with only a few misclassifications in the rapeseed and wheat classes. The SGCN-CNN method exhibits a small number of misclassifications in the wheat2, rapeseed, and grasses classes. Additionally, the analysis reveals that all seven comparison methods exhibit varying degrees of misclassified pixels in the rapeseed class. However, the proposed MLDnet method achieves excellent classification performance across all land covers, including the rapeseed class. This success is contributed to the proposed MLDnet method, which comprehensively considers different land cover characteristics and extracts various features such as texture and contour features, thereby enhancing the network’s ability to distinguish similar crops.

Meanwhile,

Table 7. Classification accuracy of different methods on Flevoland2 Data Set (%).

Class	Super-RF [14]	CNN [18]	CV-CNN [20]	3D-CNN [57]	PolMPCNN [58]	CEGCN [59]	SGCN-CNN [60]	MLDnet
Stem beans	96.77	99.58	99.72	88.45	99.79	98.54	95.41	99.96
Peas	98.64	97.95	99.99	85.90	99.00	85.70	99.63	100
Forest	95.88	97.36	99.82	99.85	99.97	99.95	99.89	99.98
Lucerne	96.63	88.95	98.17	96.26	98.45	100	99.84	99.98
Beets	99.05	93.86	98.66	99.21	95.52	99.49	98.91	99.97
Wheat	95.70	97.43	99.22	98.57	98.00	98.54	97.60	99.69
Potatoes	96.04	94.98	99.14	98.14	98.76	99.40	99.56	99.96
Bare soil	94.57	99.09	100	99.71	99.84	100	100	99.89
Grasses	84.03	76.51	99.94	76.50	95.71	88.32	90.96	99.68
Rapeseed	53.13	83.90	94.18	94.83	93.55	99.30	97.79	99.98
Barley	100	81.75	99.58	75.36	40.25	100	91.74	100
Wheat2	79.93	95.26	99.41	97.04	95.97	99.95	99.85	100
Wheat3	99.39	98.47	99.37	99.77	93.88	99.97	99.78	99.97
Water	100	99.81	99.99	100	91.27	100	99.96	100
Buildings	0	86.01	100	50.63	96.01	80.88	96.22	100
OA	92.18	93.96	98.96	95.28	93.82	98.32	98.50	99.95
AA	85.98	92.73	99.15	90.68	93.06	96.67	97.81	99.94
Kappa	91.44	93.40	98.87	94.84	93.27	98.16	98.36	99.95

The highest accuracy in each line is highlighted in bold.

provides the per-class accuracy, overall accuracy (OA), average accuracy (AA), and Kappa coefficient for the above methods and the proposed model. The data show that the Super-RF method achieves excellent classification performance in the barley and water classes, with an accuracy of 100%, but completely misclassified the buildings class, thus lowering the average accuracy. The CNN method shows a slight improvement in OA, but not reach 90% accuracy in several classes, such as grasses and barley. In contrast, CV-CNN significantly improves both class differentiation and overall evaluation metrics, achieving an OA of 98.96%. Similar to Super-RF, 3D-CNN performs well in the water class but fails in buildings class. PolMPCNN, on the other hand, differentiates the buildings class well but has the lowest class accuracy in the barley class, with only 40.25%. CEGCN achieves over 99% class accuracy in multiple classes, with main misclassifications occurring in the peas, grasses, and buildings classes. The SGCN-CNN method shows a slight improvement over CEGCN, with all metrics exceeding 90%. However, the proposed MLDnet achieves the highest class accuracy in all 13 land cover types, with an overall accuracy of 99.95%. These results fully validate the effectiveness and reliability of the proposed method.

The confusion matrix for the Flevoland2 region is shown in Figure 15. From this figure, it is evident that the proposed method misclassifies fewer pixels than the other methods across all classes, with the highest number of misclassified pixels being only 86. Notably, even in the buildings class, which contains fewer pixels, the proposed method still outperforms the others. This highlights the method’s exceptional ability to accurately classify complex PolSAR datasets with diverse land cover types.

4. Discussion

4.1. Effect of Each Submodule

In this paper, we formulate 57-dimensional multi-features as the input, with L-DeeplabV3+ as the main network, combined with the channel attention module to form the proposed MLDnet. Therefore, in order to verify the effectiveness of the three modules mentioned above, we conducts the ablation experiments. Firstly, to assess the effectiveness of the proposed 57-dimension multi-features, we replace the 57-dimension multi-features by the original 9-dimensional PolSAR data as the input without changing other modules (denoted by “wihtout MF”). Secondly, to validate the effectiveness of proposed L-DeepLabV3+ network, we replace it by a traditional CNN model, while keeping other components the same as the proposed method, noted by “without L-deeplabV3+”. Finally, to validate the channel attention module, we remove the channel attention module from the proposed MLDnet, denoted by “without Attention”.

Table 8. Classification accuracy of different submodule networks on five experimental data sets (%).

Data Set	Xi’an		Oberpfaffenhofen		Flevoland1		Flevoland2		San Francisco
Accuracy	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa
wihtout MF	96.86	94.83	93.82	91.04	99.19	98.89	98.92	98.82	99.80	99.72
without L-deeplabV3+	96.73	94.62	95.50	93.47	99.83	99.78	99.77	99.75	99.96	99.95
without Attention	96.29	93.93	96.52	94.93	99.86	99.81	99.55	99.50	99.75	99.64
MLDnet	97.38	95.67	96.76	95.29	99.94	99.92	99.95	99.95	99.97	99.96

The highest accuracy in each line is highlighted in bold.

presents the classification performance of three ablation experiments and the proposed MLDnet on five experimental data sets, evaluated using OA value and Kappa coefficient. From the Table, it is evident that for the three sets of ablation experiments, each group is able to achieve satisfactory classification performance. The proposed MLDnet integrates the advantages of all three methods, thus achieving higher statistical classification accuracy. For example, compared with “without MF” method, MLDnet improves the OA value by 0.52%, 2.94%, 0.75%, 1.03% and 0.17% on five data sets respectively, demonstrating the effectiveness of the proposed multi-feature module. The same conclusion can be found in other modules, which prove the effectiveness of the proposed method.

4.2. Validity Analysis of L-DeeplabV3+ Network

In order to verify the performance of L-deeplabV3+, the multi-dimensional features of the extracted Flevoland2 data set are put into Deeplabv3+ and L-deeplabV3+ networks respectively for comparison, as shown in

Table 9. Classification accuracy of Deeplabv3+ and L-deeplabV3+ networks on Flevoland2 data set (%).

Class	DeeplabV3+ [31]	L-DeeplabV3+
Stem beans	95.61	99.96
Peas	70.36	100
Forest	89.49	99.97
Lucerne	97.51	99.87
Beets	94.72	100
Wheat	84.74	99.25
Potatoes	92.93	99.40
Bare soil	97.23	100
Grasses	96.04	99.96
Rapeseed	95.84	99.38
Barley	96.71	100
Wheat2	92.58	96.52
Wheat3	84.41	99.84
Water	90.43	100
Buildings	98.40	90.90
OA	97.59	99.55
Kappa	92.87	99.50

The highest accuracy in each line is highlighted in bold.

. From the table data, it can be seen that the proposed L-deeplabV3+ network achieves better classification performance by fully considering the characteristics of PolSAR data. Among the fifteen different land cover types, fourteen achieve the highest accuracy. Additionally, in terms of overall evaluation metrics, the L-deeplabV3+ network improved by 1.96% and 6.63% respectively compared to the original DeeplabV3+ network.

4.3. Validity Analysis of Multi-Features

To verify the effectiveness of the proposed 57-dimensional multi-feature input in the MLDnet method, we conduct ablation experiments on the Xi’an data set. Specifically, experiments are performed using the original data features (16 dimensions), the target decomposition features (17 dimensions), and the texture and contour features (24 dimensions) within the proposed network framework. For clarity, we name these three features as feature1, feature2, and feature3, respectively. The experimental results are shown in Figure 16. It can be seen that classification with only original data (feature1) cannot classify the heterogenous regions well in Figure 16b (such as the region in yellow circle), sine there are great scattering variations within heterogenous objects. In addition, compared with feature1, the method with feature2 can improve classification performance with multiple target decomposition-based features, while edges of urban areas, such as the region in the black circle, still cannot be well identified. The feature3 is contour and textual features, which can identify the urban and grass areas well, but confuse some pixels in water area. After utilizing multi-features, the proposed method can obtain superior performance in both heterogenous and water areas.

In addition, to quantitatively evaluate the effectiveness of multi-features, the classification accuracy of different features are given in

Table 10. Classification accuracy of different types of feature input on Xi’an data set (%).

Class	Feature 1	Feature 2	Feature 3	Multi-Features
Water	91.25	93.79	89.24	94.10
Grass	95.31	89.57	97.15	97.72
Building	95.31	89.57	97.15	98.28
OA	91.39	93.25	95.98	97.38
AA	90.83	93.85	94.53	96.70
Kappa	85.69	89.00	93.33	95.67

The highest accuracy in each line is highlighted in bold.

. Clearly, as mentioned earlier, using only the original data features is insufficient for effective classification of heterogeneous targets. For instance, in distinguishing building class, feature1 has the lowest accuracy at only 85.95%, whereas feature2 and feature3 both achieve accuracy above 97%. Additionally, for the relatively homogeneous grass class, feature3, which uses texture and contour features, demonstrates a clear advantage. For water class with intricate boundaries, feature2, which uses target decomposition features, achieves higher class accuracy. The 57-dimensional multi-feature input used in this paper comprehensively considers various terrain characteristics and provides multiple complementary features, thereby achieving the best classification performance. In terms of overall accuracy (OA), compared to the three individual feature sets, the proposed method shows improvements of 5.99%, 4.13%, and 1.4%, respectively.

4.4. Validity Analysis of Channel Attention Module

Feature selection is one of the key factors influencing classification performance. To verify the effectiveness of the channel attention module in our network framework, we conduct experimental comparative analyses of the position and number of attention modules on the Xi’an data set, as shown in

Table 11. Classification accuracy of different positions and numbers of channel attention modules on Xi’an data set (%).

Class	Front1	Front2	Front3	Behind1	Behind2	Behind3
Water	86.95	77.91	79.65	94.10	80.48	79.99
Grass	92.27	96.00	94.06	97.72	97.57	92.03
Building	94.90	92.34	93.90	98.28	88.78	94.43
OA	92.40	91.99	91.85	97.38	91.91	91.07
AA	91.37	88.75	89.20	96.70	88.94	88.82
Kappa	87.50	86.58	86.40	95.67	86.38	85.20

The highest accuracy in each line is highlighted in bold.

. Specifically, considering that different types of features may be cross-fused under convolution operations, we introduce varying numbers of channel attention modules before and after the L-deeplabV3+ network in three sets of experiments. For the experiments with attention modules placed before the L-deeplabV3+ network, we name them front1, front2, and front3, with the numbers indicating the number of added channel attention modules. Similarly, for those placed after the L-deeplabV3+ network, we name them behind1, behind2, and behind3. From the data in the table, it is evident that adding multiple channel attention modules, whether before or after the convolution operations, can lead to a decrease of varying degrees in classification performance. Additionally, it increases the model’s parameter, causing unnecessary computational burden. What’s more, comparing front1 and behind1, it is clear that behind1 significantly outperforms the former in both class accuracy and overall classification metrics, with class accuracy improvements of 7.15%, 5.45%, and 3.38%, respectively. Therefore, in the proposed MLDnet method, the optimal strategy is to introduce one channel attention module after the L-deeplabV3+ network (behind1), thereby significantly enhancing classification performance and maintaining a low computational burden.

In summary, the experimental results show that introducing an appropriate number of channel attention modules in the network, especially when placed after the network, can effectively improve the classification performance of PolSAR images. At the same time, avoiding the excessive addition of attention modules prevents a significant increase in model complexity and computational cost, thus achieving a balance between performance and efficiency.

4.5. Effect of Sampling Window Size

Figure 17 further summarizes the change curve of the overall classification accuracy on sampling window size. The size of the sampling square window changes from $8\times 8$ to $24\times 24$. From the figure, it can be observed that overall tendency of the change curve is gradually rising with increased sampling window size for most of data sets. However, there is a slightly decrease for Xi’an data set when the window size is larger than 16. The reason is analysed as that too small window is difficult to effectively capture specific contextual information, while too large window may involve a large amount of irrelevant neighborhood information, leading to misclassification. Moreover, it is evident that when the window size reaches $16\times 16$, the OA values of the five data sets do not significantly increase. Especially for the Flevoland1 and San Francisco data sets, as the window size varies from $16\times 16$ to $24\times 24$, the OA value only increases by 0.04% and 0.01%. However, during this process, the network’s complexity is significantly increased. Therefore, considering the classification performance and time complexity comprehensively, we select the sampling window size as $16\times 16$.

4.6. Effect of Training Sample Ratio

The proportion of training samples is a crucial factor in influencing the classification performance. To analyze its impact on classification accuracy by the proposed MLDnet, we set the training sample proportions from 1%, 5%, 10%, 15% to 20%, and experiments are conducted on the five data sets above. The effect of the training sample ratio on classification accuracy is given in Figure 18. It is obvious that for the Oberpfaffenhofen, Flevoland1, Flevoland2 and San Francisco data sets, when the training sample ratio reaches 10%, the model becomes stable, and the OA value no longer experiences a significant increase. However, for the Xi’an data set, it reaches stable when the training sample ratio is in the range of 10–15%. Nevertheless, when the training sample ratio is in the range of 15–20%, the OA value shows a downward trend. This may be attributed to the continuous increase in the training sample ratio exacerbating sample imbalance in the Xi’an data set, leading to the model overfitting. Therefore, considering the four experimental data sets overall, we chooses 10% as the training sample ratio for experiments.

4.7. Analysis of Running Time

Table 12. Running time of different methods on Xi’an data set (s).

	Super-RF	CNN	CV-CNN	3D-CNN	PolMPCNN	CEGCN	SGCN-CNN	DeeplabV3+	MLDnet
Training	59.22	150.59	3463.20	121.84	21,600.35	129.43	30.59	1543.21	511.10
Testing	1.85	10.56	38.43	22.80	327.53	4.45	3.00	53.50	23.48

summarizes the running time of each method on the Xi’an data set. From the table, it can be visually seen that, compared to first five deep learning methods, the traditional method Super-RF consumes the least time in both training and testing. Although it requires slightly longer time than CNN, CEGCN, SGCN-CNN and 3D-CNN methods, MLDnet can significantly reduce training and testing time compared with CV-CNN and PolMPCNN methods. In addition, compared with DeeplabV3+ method, the proposed MLDnet cost less time in both training and testing stages, which proves the effectiveness of the proposed lightweight deeplabV3+ network. Besides, the proposed MLDnet achieves the highest classification accuracy. This indicates that the proposed method can significantly improve the classification accuracy of PolSAR images without increasing too much time, thereby achieving dual improvements in time and performance.

5. Conclusions

In this paper, we propose a novel multi-feature deep attention network for PolSAR image classification. Unlike conventional networks using the original data as input, our proposed MLDnet extracts a variety of complementary features, including the original data, scattering features, and image features. Subsequently, a lightweight DeeplabV3+ network is designed to learn multi-scale high-level features, significantly reducing parameters and improving computation efficiency. Finally, a channel attention fusion module is developed for feature selection, enhancing valuable features and suppressing useless ones to improve classification performance. Experiments were conducted on five real PolSAR data sets with different bands and sensors. The results demonstrate that the proposed method achieves superior classification performance in both region homogeneity and edge preservation compared to state-of-the-art methods.

Furthermore, while the attention mechanism serves as an effective feature fusion method for high-dimensional multiple features, it remains a relatively straightforward feature selection mechanism. In our future work, we aim to incorporate mutual information to thoroughly exploit relationships among various features, thereby enhancing feature selection. Moreover, some few-shot learning methods for unbalanced samples should be exploited in the further work.