Dual-Branch Fusion of Convolutional Neural Network and Graph Convolutional Network for PolSAR Image Classification

Abstract

Polarimetric synthetic aperture radar (PolSAR) images contain useful information, which can lead to extensive land cover interpretation and a variety of output products. In contrast to optical imagery, there are several challenges in extracting beneficial features from PolSAR data. Deep learning (DL) methods can provide solutions to address PolSAR feature extraction challenges. The convolutional neural networks (CNNs) and graph convolutional networks (GCNs) can drive PolSAR image characteristics by deploying kernel abilities in considering neighborhood (local) information and graphs in considering long-range similarities. A novel dual-branch fusion of CNN and mini-GCN is proposed in this study for PolSAR image classification. To fully utilize the PolSAR image capacity, different spatial-based and polarimetric-based features are incorporated into CNN and mini-GCN branches of the proposed model. The performance of the proposed method is verified by comparing the classification results to multiple state-of-the-art approaches on the airborne synthetic aperture radar (AIRSAR) dataset of Flevoland and San Francisco. The proposed approach showed 1.3% and 2.7% improvements in overall accuracy compared to conventional methods with these AIRSAR datasets. Meanwhile, it enhanced its one-branch version by 0.73% and 1.82%. Analyses over Flevoland data further indicated the effectiveness of the dual-branch model using varied training sampling ratios, leading to a promising overall accuracy of 99.9% with a 10% sampling ratio.

1. Introduction

Polarimetric synthetic aperture radar (PolSAR) has become of the utmost importance in land cover discrimination methods, providing extensive information about land features using fully polarized data, which are collected in almost all weather conditions, as well as day and night. This polarimetric information mainly considers features based on scattering mechanisms and overlooks spatial information, unlike other remote sensing images (e.g., optical and hyperspectral images). Therefore, accessing both polarimetric and spatial information from PolSAR images requires the extraction of additional features.

Several PolSAR feature extraction techniques have been developed and investigated for classification methods [1,2,3]. Due to the complexity of full polarimetric SAR images, features containing the most distinctive information are needed for different applications, including classification and change detection. Statistical features [4,5], scattering information [6], and target decomposition features [7,8,9] are the three main types of extracted features from PolSAR image analysis. Traditional PolSAR image analysis solves the classification problem in a two-step process, including feature extraction and classification. Accordingly, conventional PolSAR image classification methods, such as Wishart [10,11,12], decision tree [13,14], and support vector machine (SVM) [15,16], have been applied after the feature extraction step. The main disadvantage of these methods is that the feature engineering step is a time-consuming task and decreases the classification accuracy when high-correlated features or features with low information content are extracted in this step. In contrast to conventional PolSAR image classification algorithms, deep learning (DL) methods remove the feature engineering task from the processing steps. As such, the capability of different DL algorithms, such as a convolutional neural network (CNN) [17], autoencoder (AE) [18,19], and deep belief network (DBN) [20], were recently investigated for PolSAR image classification. Among these advanced models, deep CNNs have achieved remarkable success in PolSAR image classification [21,22]. As reported in the literature, utilizing a single convolutional kernel of CNN and single-channel CNN does not extract sufficient information for PolSAR image classification [23]. Therefore, several networks have been developed based on this algorithm with versatile structures and PolSAR features [24,25]. For example, Chen and Tao [25] derived several polarimetric features, including entropy (H), mean alpha angle, anisotropy from a decomposition technique, total backscattering power (SPAN), and two null angles for supervised image classification using the deep convolutional neural network (DCNN). Zhang, et al. [26] also proposed complex-valued CNN (CV-CNN) for PolSAR data processing by extending each element of real-valued CNN (RV-CNN), including feed-forward and backpropagation process, to the complex domain, wherein coherency matrix values were considered as complex inputs for the network. In order to benefit from rich spatial and polarimetric features of PolSAR data, multibranch CNNs have been proposed [27,28,29]. These networks contain independent CNNs for distinguishing different classes. In another study, Gao, et al. [27] developed a dual-branch CNN, which utilizes six-channel polarimetric and three-channel spatial features (Pauli RGB (red, green, blue)) as two independent inputs of the proposed model. Accordingly, a multichannel fusion of CNNs based on different scattering mechanisms was designated [28], in which each CNN obtains one of the Freeman decomposition elements (odd bounce, double-bounce, and volume scattering), and, together with a fusion feature, they passed through fully connected (FC) layers.

Graph-based DL methods have attracted great attention among remote sensing researchers for image classification. Graph convolutional networks (GCNs) [30] consider the relation between samples over the whole image rather than the local kernels of CNNs. In the case of PolSAR classification, there is a limited number of studies which have used GCNs. In contrast, GCNs have been widely utilized in hyperspectral image (HSI) classification [31,32,33,34]. For instance, He, et al. [35] presented supervised dual GCN (DGCN) for HSI classification; one of the GCN extracts features in and among samples, while the other utilizes label distribution learning. One of the main drawbacks of GCNs is that they have a high computation cost by considering the relation between all samples, and only full-batch learning is permitted (unlike CNNs). To alleviate this limitation, Hong, et al. [36] introduced miniGCNs that train networks by applying minibatch fashion. Furthermore, there are several pioneering studies employing GCNs in the field of remote sensing. Cai and Wei [37] proposed a cross-attention mechanism and GCN integration approach to help the model select the most important characteristics for accurate remote sensing image classification. Du, et al. [38] used a multimodal graph network to provide a feature extraction–fusion network for multisource remote sensing data classification in an unsupervised manner and obtained satisfactory classification performance.

In order to exploit PolSAR images’ spatial and polarimetric features, a dual-branch fusion of miniGCN and CNN is proposed in this study. Accordingly, seven channels of spatial features of the PolSAR image are obtained from Pauli RGB and four-component Yamaguchi [39], which are fed into CNN, owing to its promising ability to extract spatial features. Meanwhile, six polarimetric features of PolSAR images, utilized in [16], would be the inputs of miniGCN to depict relationships among polarimetric features. In the end, both spatial and polarimetric features would be concatenated and passed through FC layers. As such, the main contributions of the current study can be highlighted as follows:

(1)

Considering different PolSAR image characteristics, we attempt to derive network-specific features by dividing them into spatial and polarimetric categories. Hence, Pauli RGB and Yamaguchi decomposition of the PolSAR image present spatial feature channels, and six roll-invariant and hidden polarimetric features are polarimetric features channels.

(2)

The novel method of supervised batchwise version of GCN, known as miniGCN, is investigated as a classifier for PolSAR image classification.

(3)

Dual-branch fusion of miniGCN and CNN is proposed as a PolSAR classifier. Thus, each miniGCN and CNN is fed by the features with specific characteristics corresponding to its structure. Particularly, miniGCN and CNN extract spatial and polarimetric features, respectively. Subsequently, their integrated features are followed by two FC layers to determine PolSAR image classes.

The remainder of this paper is organized as follows: Section 2 presents the background of CNN and miniGCN, including an overview of the basics of CNNs, GCN, and miniGCN. In Section 3, the proposed method is introduced by presenting extracted PolSAR features and the architecture of the dual-branch fusion network. A comprehensive analysis of the proposed network is conducted on different PolSAR benchmark datasets in Section 4. Finally, Section 5 concludes the paper’s main remarks.

2. Theory and Basics of CNN and miniGCN

This study proposes a novel dual-branch deep learning method based on the CNN and miniGCN architectures for PolSAR image classification. This section provides an overview of the fundamentals and theorems of the networks mentioned above.

2.1. CNNs Basics and Overview

A CNN is typically comprised of an input layer, convolutional layers, pooling layers, FC layers, and an output layer. The input layer firstly receives input features of the image. Then, convolutional layers extract input features using convolutional kernels (Figure 1). These kernels work based on considering neighboring pixels. Thus, spatially correlated pixels that are located in close range (2 × 2 in Figure 1) would be considered in the convolutional kernel. This capability improves the ability of the network to model spatially related features.

Deep CNNs utilize multiple convolutional layers, with the early layers extracting low-level data and the deeper ones detecting high-level features. The convolutional layer is usually followed by a pooling layer, which intends to reduce the output dimension of the preceding layer. Average- and maxpooling are the most common pooling algorithms that extract average and minimum pooling region values, respectively. A fully connected (FC) layer is usually employed after the last convolutional layer. The FC layer reshapes its input into a one-dimensional feature vector that can then be sent to the output layer. Finally, the retrieved features are mapped to their corresponding classes in the output layer.

2.2. Graph and miniGCN

(1) Graph convolutional network (GCN): The GCN [30] works based on defining graphs and relationships among samples in a non-Euclidean space (Figure 2). This provides a helpful tool for considering medium- and long-range relations between pixels rather than only considering short-range relations like convolutional networks. In other words, instead of considering spatial correlation, feature-based relations are considered through an adjacency matrix.

A graph can be defined as G(V, E), where V and E donate a set of vertices and edges, respectively. Vertices indicate input data samples, while edges represent similarities between any two vertices. The edges are defined based on the adjacency matrix (A). It is a symmetric N × N matrix, where N is the number of pixels (nodes) and represents the relationship between each pair of pixels by weights between 0 and 1. Radial basis function (RBF) can be used to calculate the adjacency matrix.

$$A_{i,j}=\exp (\frac{\Vert x_i-x_j{\Vert }^2}{{\sigma }^2})$$

(1)

where x_i and x_j are the feature vector corresponding to vertices v_i and v_j and σ is the control parameter of the RBF. Accordingly, the normalized graph Laplacian matrix L is represented using diagonal matrix D as follows [40]:

$$D_{i,j}={\sum }_j^{}{A}_{i,j}$$

(2)

$$L=I-D^{-1/2} A D^{-1/2}$$

(3)

where I is an identity matrix. Spectral decomposition on L can be performed on L as follows:

$$L=U \Lambda U^T$$

(4)

where $U$ and $\Lambda$ are orthogonal eigenvector matrix and diagonal eigenvalues of L, which is the basis function of Fourier transform of feature vector f of each node. The convolution on a graph G can be defined as

$$G[f \ast g_{\theta }] = U g_{\theta } U^T f$$

(5)

where g_θ(Λ) is a filter in the Fourier domain that represents the function of eigenvalues (Λ) of L considering the variable θ. The Kth order truncated expansion of Chebyshev polynomials is used to alleviate the computational cost of convolutional on a graph [41].

$$G[f \ast g_{\theta }] \approx {\sum }_{k=0}^K{\theta }_k^{\prime } T_k \left(\tilde{L}\right)f$$

(6)

where ${\theta }_k^{\prime }$ is the vector of Chebyshev coefficients and T_k donates Chebyshev polynomials. Normalized $\tilde{L}$ is scaled as $\tilde{L}=2 L /{\lambda }_{max}-I$. Eventually, Equation (6) can be simplified by considering K = 1 and λ_max = 2:

$$G[f \ast g_{\theta }] \approx \theta (I+D^{-1/2} A D^{-1/2})f.$$

(7)

Regarding Equation (7), the propagation rule for GCNs is defined as follows:

$$H^{l+1}= h ({\tilde{D}}^{-1/2} \tilde{A} {\tilde{D}}^{-1/2} H^l w^l {+b}^l)$$

(8)

where adjacency matrix A and D are renormalized as $\tilde{A}=A+I$ and ${\tilde{D}}_{i,j}$ = $\sum _j^{}{\tilde{A}}_{i,j}$. Meanwhile, w^l, b^l, and h() are weight matrix, bias matrix, and activation function. The output of lth and (l + 1)th layers are also indicated by H^l and H^l+1.

(2) miniGCN overview: Because of the size of the adjacency matrix, GCNs have a substantial computational cost for large graphs. miniGCN [36], which takes advantage of minibatch processing in a batchwise fashion (similar to CNNs), is a feasible solution to the high computational cost issue. For the construction of minibatches, a random node sampler of size M is generated from a full graph G with N nodes on the labeled set (M << N) (Figure 3). In each epoch, the sampler is first applied to the graph until all nodes are sampled, resulting in the generation of subgraphs (G_s):

$$\mathbb{G} = \{G_s = (V_s,E_s\left)\right|s = 1, \dots , \lceil N/M\rceil \}$$

(9)

where ⌈⌉ is ceiling operator. Consequently, the miniGCN update rule for one batch can be represented as follows:

$${\tilde{H}}_s^{l+1}=h ({\tilde{D}}_s^{-1/2} {\tilde{A}}_s {\tilde{D}}_s^{-1/2}{H}_s^l {\mathrm{w}}^{\mathrm{l}}+b_s^l)$$

(10)

where s denotes the sth subgraph as well as the sth batch. The final output in the (l + 1)th layer is calculated.

$$H^{l+1} =[{\tilde{H}}_1^{l+1}, \dots , {\tilde{H}}_s^{l+1}, \dots , {\tilde{H}}_{\lceil N/M\rceil }^{l+1}]$$

(11)

The primary difference between this batch procedure and that used in CNNs is that the adjacency matrix of each batch must be reformed after each sampling. The miniGCN model can consider the relationship between pixels using an adjacency matrix, regardless of their distance. While selecting random batchwise samples in each epoch decreases computational cost, it also makes the model compatible with batchwise CNN models and can be integrated with them.

3. The Proposed Method

3.1. PolSAR Feature Extraction

The acquired feature extraction techniques for feeding the proposed network’s spatial and polarimetric channels are defined in the following subsections. As spatial characteristics, Pauli RGB and four-component Yamaguchi decomposition algorithms are defined first. Then, roll-invariant and hidden polarimetric features are explained.

(1) Pauli and decomposition features: The Pauli RGB image is a decomposition approach produced based on scattering matrix S for visualizing PolSAR imagery. The scattering matrix for a full polarized SAR image is defined as follows:

$$S=\left[\begin{array}{cc}{S}_{HH}& S_{HV}\\ S_{VH}& S_{VV}\end{array}\right]$$

(12)

The scattering matrix is symmetrical (S_HV = S_VH) in case of satisfying the reciprocity theorem. It can be represented in a way to highlight specific scattering mechanisms:

$$k=(1/\sqrt{2}) [S_{HH} + S_{VV} S_{HH} - S_{VV} 2S_{HV}{]}^T = [a_1 a_2 a_3{]}^T$$

(13)

Accordingly, red, green, and blue bands of a false-color Pauli RGB image would be deemed |a₂|² (S_HH − S_VV), |a₃|² (2S_HV), and |a₁|² (S_HH + S_VV). This pseudo-colored image is more human-desirable and in close harmony with natural colors [42], making it easier to consider spatial characteristics such as other colored images. The Coherency matrix T₃ can be obtained as follows:

$$T_3=(1/L) {\sum }_{i=1}^L{k}_i{k}_i^H = [T_{11} T_{22} T_{33}]$$

(14)

where H donates the complex conjugate transpose, k_i is the ith sample of Pauli scattering vector (k), and L indicates the number of looks.

Four-component Yamaguchi decomposition technique decomposes T₃ into the four scattering powers of surface (P_s), double-bounce (P_d), volume (P_v), and helix scattering (P_h) [39]. This decomposition is valuable for characterizing urban man-made targets, owing to the helix scattering component that emerges in heterogenous areas [9].

(2) Polarimetric descriptors: To consider the polarimetric characteristics of PolSAR images, roll-invariant and hidden features [25] are deployed. As a result, total backscattering power SPAN is defined as follows:

SPAN = |S_HH|² + 2|S_HV|² + |S_VV|²

(15)

The coherency matrix can be decomposed based on eigenvalues and eigenvectors, as follows:

$$T_3 = U_3 \left[\begin{array}{ccc}{\lambda }_1& 0& 0\\ 0& {\lambda }_2& 0\\ 0& 0& {\lambda }_3\end{array}\right]U_{3^H}, {\lambda }_1 \ge {\lambda }_2 \ge {\lambda }_3$$

(16)

where λ₁, λ₂, and λ₃ are eigenvalues and U₃ comprises eigenvectors of the coherency matrix. Accordingly, Cloude–Pottier decomposition components [43], including entropy (H), mean alpha angle ($\overline{\alpha }$), and anisotropy (A) are derived.

$$P_i=\frac{{\lambda }_i}{{\sum }_{n=1}^3{\lambda }_n}, i = 1, 2, 3$$

(17)

$$H=-{\sum }_{n=1}^3{p}_i{\log }_3{p}_i$$

(18)

$$\overline{\alpha }={\sum }_{i=1}^3{p}_i{\alpha }_i$$

(19)

$$A=\frac{{\lambda }_2-{\lambda }_3}{{\lambda }_2+{\lambda }_3}$$

(20)

In order to leverage the rotation characteristics of PolSAR image, Chen, et al. [44] proposed a method for extending polarimetric features to the rotation domain along the radar line of sight and, accordingly, derived the null angle θ_null. The null angles of θ_null_Re[T₁₂] and θ_null_Im[T₁₂] are highly sensitive to various land covers, which offers a lot of potential for PolSAR classification. These two null angles are presented as follows:

$$\begin{array}{c}{\theta }_{\mathit{null}}\_\mathit{Re}[T_{12}]=-\frac{1}{2}\mathrm{Angle}\{\mathrm{Re}[T_{13}] + j \mathrm{Re}[T_{12}]\}\\ =\frac{1}{2}\mathrm{Angle}\{\mathrm{Re}[\langle (S_{\mathit{HH}}+S_{\mathit{VV}}) S_{HV}^{*}\rangle ]\\ +\mathrm{j} \frac{1}{2} (\langle {\left|S_{VV}\right|}^2-{\left|S_{HH}\right|}^2\rangle )\}\end{array}$$

(21)

$$\begin{array}{c}{\theta }_{\mathit{null}}\_\mathit{Im}[T_{12}]=-\frac{1}{2}\mathrm{Angle}\{\mathrm{Im}[T_{13}] + j \mathrm{Im}[T_{12}]\}\\ =\frac{1}{2}\mathrm{Angle}\{\mathrm{Im}[\langle (S_{\mathit{HH}}+S_{\mathit{VV}}) S_{HV}^{*}\rangle ]\\ +\mathrm{j} \mathrm{Im}[\langle S_{HH}{S}_{VV}^{*}\rangle ]\}\end{array}$$

(22)

where Re[] and Im[] get real and imaginary parts, respectively. In contrast, The Angle [] operator acquires phase in complex axis and range of [−π, π].

The six features of SPAN, H, $\overline{\alpha }$, A, θ_null_Re[T₁₂], and θ_null_Im[T₁₂] are considered for representing and modeling polarimetric characteristics of PolSAR images.

3.2. Dual-Branch FuNet Architecture

In this study, a dual-branch fusion of miniGCN and CNN networks (FuNet) is proposed for PolSAR image classification, and the framework is depicted in Figure 4. The spatial and polarimetric features of the PolSAR image are inputs for the proposed model. These two sets of PolSAR features are then fed to the CNN and miniGCN, which can extract additional information. The CNN branch leverages the contextual information extraction ability of convolutional kernels, and the miniGCN has the ability to find similarities (such as similarities in polarimetric features) with graphs.

The combination of these two models would help the network make the most of polarimetric data. Therefore, rather than utilizing only the spatial potential of convolutional networks in either one- or two-branch architectures, polarimetric features are deployed in a dual-branch architecture using miniGCN and CNN models. It should be noted that GCN models cannot be fused with CNNs due to their pixel-based structure. As a result, the so-called miniGCN is used to obtain GCN capability in a batchwise strategy.

The proposed network uses spatial and polarimetric features of the PolSAR image as inputs to the CNN and miniGCN branches, respectively. The miniGCN branch consists of a batchwise GCN block. This block is fed by batchwise pixels and an adjacency matrix, which represent nodes and edges of the graph, respectively. In other words, polarimetric features are considered nodes, and the adjacency matrix indicates the relationship between the nodes (edges).

Meanwhile, the CNN branch comprises three two-dimensional convolutional layers and an FC layer. The first two convolutions are followed by maxpooling (pooling size of 2 and stride 2). The FC layer aims to reshape the output of the last convolution to a one-dimensional vector to be concatenated with miniGCN output. The fused output will then be fed to two FC layers to further deepen the network and prepare concatenated features to pass through a Softmax classifier and perform classification. Moreover, a dropout with a 0.5 ratio is applied to the last FC layer to mitigate the overfitting effect.

4. Experiments

4.1. Data Description

The proposed method for PolSAR classification is evaluated in this section, adopting two airborne synthetic aperture radar (AIRSAR) benchmark datasets. Details of the utilized dataset are described as follows:

(1) AIRSAR Flevoland data: The Flevoland L-band full polarimetric dataset was acquired by the NASA/JPL AIRSAR platform in 1989 over Flevoland, Netherland. The image size is 750 × 1024 pixels, with 157,296 labeled pixels representing 15 different land cover categories in the ground truth (GT) retrieved from Zhang, et al. [26].

Table 1. Details of classes and training samples for AIRSAR Flevoland data (TR represents training ratio).

Class Number	Class Name	Train Number	Sample Number	TR (%)
1	Stem beans	62	6103	1.015894
2	Peas	92	9111	1.009768
3	Forest	150	14,944	1.003747
4	Lucerne	95	9477	1.002427
5	Wheat	173	17,283	1.000984
6	Beet	101	10,050	1.004975
7	Potatoes	153	15,292	1.000523
8	Bare soil	31	3078	1.007147
9	Grass	63	6269	1.004945
10	Rapeseed	127	12,690	1.000788
11	Barley	72	7156	1.006149
12	Wheat2	106	10,591	1.00085
13	Wheat3	214	21,300	1.004695
14	Water	135	13,476	1.001781
15	Buildings	5	476	1.05042
	All	1579	157,296	1.00384

lists these land cover categories’ names and sizes (for train and whole GT samples). Figure 5a,b show the Pauli RGB pseudo-color image and GT. The image has a resolution of 6.6 meters (m) in range and 12.1 m in the azimuth direction.

(2) AIRSAR San Francisco data: The San Francisco data also comprise an L-band full polarimetric image captured by the AIRSAR platform in 1989 over San Francisco, USA. The image has a resolution of 900 × 1024 pixels and 802,302 GT labeled pixels, with five land cover categories. The GT is accessible from [45]. Details of GT categories’ names and sizes (for train and whole sample) are compared in

Table 2. Details of classes and training samples for AIRSAR San Francisco data (TR represents training ratio).

Class Number	Class Name	Train Number	Sample Number	TR (%)
1	Bare soil	138	13,701	1.007226
2	Mountain	628	62,731	1.0011
3	Water	3296	329,566	1.000103
4	Urban	3428	342,795	1.000015
5	Vegetation	536	53,509	1.001701
	All	8026	802,302	1.000371

. Figure 6a,b represent the San Francisco image’s Pauli RGB and GT. The spatial resolution of the image is 10 in both directions.

As a preprocessing step, a refined Lee filter with a 7 × 7 window size is applied to both AIRSAR images to reduce the influence of speckle [46].

4.2. Experimental Design

In the current study, a patch-size of 15 × 15 is considered for CNN-based networks since this patch size has been established as the optimal quantity for 2D-CNN classification of AIRSAR datasets [27]. The training samples are randomly chosen from labeled data, with a ratio of 1%. The network’s hyperparameters are as follows: the training epochs are 300, and the batch sizes are 64. Batch normalization is used with a momentum parameter of 0.9. In addition, the base learning rate (LR) of 0.01 is considered. The LR is dynamically updated every 50 epochs by multiplying the base LR by √(1 − iteration/(maximum iteration)). In order to reduce overfitting, L2-norm regularization is obtained (e.g., 0.001). The hyperparameters are set over a grid search. The architecture of the proposed network is illustrated in

Table 3. The architecture of the dual-branch FuNet.

Layer	CNN	miniGCN
Input	15 × 15 × 7 (Spatial feature)	6 Polarimetric feature
Block 1	2 × 2 Conv	BN
	BN	Graph Conv
	2 × 2 Maxpool	BN
	ReLU	ReLU
Output size	8 × 8 × 30	120
Block 2	2 × 2 Conv	-
	BN	-
	2 × 2 Maxpool	-
	ReLU	-
Output size	4 × 4 × 60	-
Block 3	2 × 2 Conv	-
	BN	-
	ReLU	-
Output size	4 × 4 × 120	-
Fully connected	FC Encoder	-
	BN	-
	ReLU	-
Output size	120	-
Fusion	FC Encoder
	BN
	ReLU
Output size	240
Output	FC Encoder
	Softmax
Output size	Number of classes

. The input of the CNN branch is patchwise, represented by a four-dimensional matrix with the size of N × 15 × 15 × 7, where N is the size of the PolSAR image, 15 × 15 is the patch size, and 7 is the number of spatial features. The input size of miniGCN for the six polarimetric features is N × 6. In addition, a N × N adjacency matrix is considered in the graph model.

Several state-of-the-art classification approaches, including support vector machine (SVM), random forest (RF), 1D-CNN, 2D-CNN, miniGCN, and FuNet, are compared to the proposed method (dual-branch FuNet). SVM with RBF kernel function, RF with 200 decision and 1D-CNN, with two convolutional layers with filter sizes of 120 (with ReLU) and the number of classes (with Softmax), utilized classical classification methods. 2D-CNN with an almost similar structure to [25] is acquired, with the exception of additional L2-norm regularization (same as the proposed network), resulting in identical convolutional blocks with the proposed network’s CNN branch. The utilized miniGCN comprises two graph convolutional layers, the first of which has a size of 120 (similar to the miniGCN branch of the proposed network), and the second one has the same size as the number of classes, which is followed by a Softmax. The FuNet network design is similar to that of the proposed dual-branch method, with polarimetric features given to both miniGCN and CNN branches.

In order to evaluate and compare the performance of methods, overall accuracy (OA) and kappa coefficient (K) are adopted.

4.3. Parameter Setting, Adjacency Matrix

The adjacency matrix significantly impacts the quality of the proposed method’s miniGCN branch. As a result, optimum parameters for the construction of the adjacency matrix, including the number of neighbors (K) and RBF function width (σ), are required.

A grid search was conducted on Flevoland data to identify the suitable parameters using the miniGCN network (Figure 7). The overall accuracy of miniGCN varies substantially over the search range. While K = 20 and σ = 0.5 yield the highest OA, the OAs are not stable near those values. The most stable outcomes are obtained using K = 10 and σ = 1. In other words, the lowest standard deviations (SD) were 0.2% and 0.5%, at K = 10 and σ = 1, respectively. SDs for different sets of parameters, in contrast, are not less than 1% and 0.7% (for each K and σ). Eventually, (K = 10, σ = 1) is taken into consideration for the proposed network classification.

4.4. Effectiveness Evaluation

To indicate consistency of the proposed method, it was implemented five times, and the average of the results were considered for further evaluation. The processing stage was conducted using an 8 GB memory Intel Core i7-7500U processor. The training process for 300 epochs of the AIRSAR Flevoland dataset with 1% sampling ratio (1579 samples) was 1985 seconds.

The convergence of losses and accuracies of the dual-branch FuNet model for AIRSAR Flevoland are shown in Figure 8. After 150 epochs, the loss of both training and test sets become almost steady, indicating the effectiveness of the network’s training process. Eventually, the training accuracy reaches 100%, while the test set becomes approximately 98%.

4.5. Experiments on AIRSAR Datasets

The proposed dual-branch FuNet land cover classification results are compared to several state-of-the-art methods, including SVM, RF, 1D-CNN, 2D-CNN, miniGCN, and FuNet, over AIRSAR Flevoland and AIRSAR San Francisco. To establish a fair comparison, similar random samples (with 1% TR) are selected, and the average performance of each network’s five-time running is represented.

In Figure 9 and Figure 10, the proposed method’s classified maps are visually compared to the six state-of-the-art methods for the entire image and the labeled regions, demonstrating that the FuNet network and its dual-branch version produce significantly smoother and more satisfying results for both datasets. Furthermore, in the Flevoland case, classical methods and miniGCN classification maps are noisy and seem to be impacted by speckle, whereas networks using convolutional layers (including 2D-CNN, FuNet, and dual-branch FuNet) produce smoother results. Similarly, in the case of the San Francisco data, classical approaches and miniGCN yield noisy and misclassified results, particularly in the mountain and urban areas.

Per-class OA, overall OA, and kappa coefficient for the methods mentioned above are compared to the proposed one in

Table 4. Detailed classification OAs and K of different algorithms compared to the proposed method on AIRSAR Flevoland. Bold numbers indicate the highest accuracy in each row.

Classes	Models
Name	SVM	RF	1D-CNN	2D-CNN	miniGCN	FuNet	Dual-Branch FuNet
Stem beans	80.95	80.90	79.59	99.47	66.33	99.47	99.35
Peas	77.26	76.22	77.78	97.54	83.14	96.74	97.62
Forest	77.13	85.36	76.65	96.69	96.62	96.43	98.33
Lucerne	83.23	84.56	81.28	97.11	81.12	97.35	94.54
Wheat	71.07	72.36	73.70	93.12	63.76	95.20	98.85
Beet	77.80	79.85	83.08	94.34	71.39	94.18	98.05
Potatoes	72.42	72.06	76.19	93.34	49.38	97.03	97.02
Bare soil	66.26	68.00	81.29	100.00	56.58	100.00	94.58
Grass	69.34	70.38	71.87	96.46	65.19	95.97	94.25
Rapeseed	74.82	71.69	74.31	94.98	58.13	95.52	97.48
Barley	70.99	76.96	78.67	98.09	83.54	98.90	97.52
Wheat2	71.11	69.38	71.71	96.73	42.84	97.15	97.47
Wheat3	90.18	89.64	89.86	99.67	78.32	99.21	99.75
Water	96.45	96.78	92.93	99.00	99.91	99.03	98.94
Buildings	65.82	68.79	77.28	80.68	83.86	86.20	91.93
OA (%)	78.57	79.55	79.81	96.54	72.32	97.11	97.84
K (%)	76.57	77.64	77.94	96.22	69.86	96.84	97.64

and

Table 5. Detailed classification OAs and K of different algorithms compared to the proposed method on AIRSAR San Francisco. Bold numbers indicate the highest accuracy in each row.

Classes	Models
Name	SVM	RF	1D-CNN	2D-CNN	miniGCN	FuNet	Dual-Branch FuNet
Bare soil	40.87	45.60	44.21	74.09	50.66	76.21	88.76
Mountain	73.10	76.32	73.92	96.25	82.98	95.92	97.38
Water	98.98	98.90	98.95	99.39	99.06	99.49	99.40
Urban	94.88	94.46	94.92	94.93	48.36	96.80	98.68
Vegetation	55.84	57.53	57.45	78.07	63.74	78.53	89.36
OA (%)	91.33	91.57	91.57	95.39	72.95	96.27	98.09
K (%)	86.22	86.65	86.62	92.79	62.05	94.14	97.00

. It can be observed that classical methods of SVM, RF, and 1D-CNN have performed almost similarly, attaining OAs of 78–80% and approximately 91% for Flevoland and San Francisco, respectively. Meanwhile, the 2D-CNN achieved high OA and K, benefiting from its potential to extract 2D features and the use of neighboring pixels’ information. Accordingly, it reached the highest OA for bare soil and grass classes in the Flevoland dataset.

The miniGCN, which is powered by the graph’s long-range similarity detection ability, showed overall poor performance compared to the classical networks. The reason for this performance is graph edges or adjacency matrix, which connects pixels based on their similarities in input features rather than target classes. Therefore, in classes such as urban areas in the AIRSAR San Francisco dataset that contain mixed pixels (such as vegetation coverage), the output is affected by the similarities considered in the adjacency matrix. In contrast, in classes with fewer mixed pixels (such as building in Flevoland), even with a small number of training samples (five for the building class), the miniGCN performed well. It also achieved the highest OA in the water class. Furthermore, in the case of the San Francisco image, the miniGCN surpasses classical approaches (except in urban class).

The FuNet model outperforms the abovementioned methods by combining the convolutional advantage of 2D-CNN with the ability of graphs to consider the long-range relationships. Using both polarimetric and spatial features of PolSAR images in the dual-branch FuNet model further improves its performance compared to the one-branch model. Eventually, dual-branch FuNet achieved OA and K of 97.84% and 97.64% for Flevoland and 98.09% and 97% for San Francisco. Flevoland data had the foremost OAs for forest, beet, rapeseed, building, and all three wheat classes. Moreover, it is the only model that achieved above 90% OAs for all classes, with 91.93% for building as the lowest per-class OA. Meanwhile, for the San Francisco case, four of the existing five classes, including bare soil, mountain, urban, and vegetation, obtained their maximum OAs using the proposed dual-branch network.

4.6. Performance Analyses with Different Training Sampling Rates

In general, supervised classification methods work with a limited number of training samples; thus, their ability to discriminate varied land covers with restricted sample data is crucial. Accordingly, the effect of various training sample ratios (TRs) on the performance of the proposed method is compared to that of the other methods over the Flevoland data.

Table 6. Classification OAs with different training ratios on AIRSAR Flevoland data. Bold numbers indicate the highest accuracy in each row.

	Training Ratio (%)	SVM	RF	1D-CNN	2D-CNN	miniGCN	FuNet	Dual-Branch FuNet
OA (%)	1	78.57	79.55	79.81	96.54	72.32	97.11	97.84
	5	81.95	83.43	82.59	98.7	75.52	98.66	99.67
	10	83.16	84.45	83.1	99.63	76.94	99.2	99.9

depicts classification OAs of 1%, 5%, and 10% TRs. The result indicates that the proposed dual-branch FuNet achieved the highest OAs with different TRs, followed by 2D-CNN and FuNet.

To offer a clear visual comparison, Figure 11 illustrates the performance of the three superior networks. It can be observed that with an adequate sampling ratio of 10%, 2D-CNN outperforms FuNet and is slightly lower than the dual-branch network. When TR is reduced to 5%, the OAs of 2D-CNN and FuNet fall by 0.93% and 0.54%, whereas that of the dual-branch decreases marginally (0.23%). Considering 1% TR, 2D-CNN degrades significantly (2.58%), whereas the proposed dual-branch and its one-branch version of FuNet have more stable results. Overall, the dual-branch FuNet shows the least sensitivity to changes in training sample ratios and achieves the highest OAs.

4.7. Comparison with Other Studies

The proposed method is compared to some published PolSAR classification techniques to provide a more comprehensive verification. For a fair comparison, using similar benchmark data and ground-truth (GT) is recommended. Accordingly, AIRSAR Flevoland benchmark data, which has been widely deployed in previous studies, is regarded for comparison. It should be noted that some of the other papers have employed different GTs with different random training samples, which might impact the obtained result.

The PolSAR classification approaches of complex-valued CNN (CV-CNN) [26], dual-branch deep CNN (dual-branch DCNN) [27], 2D-CNN [25], multichannel fusion CNN (MCFCNN) [28], and semi-supervised multiscale evolving weighted GCN (MEWGCN) [47] are the ones compared with the proposed FuNet and dual-branch FuNet (

Table 7. Comparison of classification OAs with other studies on AIRSAR Flevoland data. Bold numbers indicate the highest accuracy in each row.

	Training Ratio %	CV-CNN	Dual-Branch	2D-CNN	MCFCNN	MEWGCN	Proposed
OA (%)	1	62	98.53 (75% TR)	97.57	95.83	-	97.84
	5	94		98.83	-	99.39	99.67
	10	96.2		99.3	-	-	99.9

).

It should be noted that several GTs with varied numbers of labeled points are utilized, with only CV-CNN sharing the same GT as the current study. With an identical GT, CV-CNN showed poor performance, with 1% TR, and inferior OA, with 5% and 10%. When using a limited amount of training data (1%), this method has even lower performance compared to the SVM and RF with proposed input features. This indicates the importance of model input parameters and how extracted spatial and polarimetric features can boost the classifier performance, particularly with limited training data. The dual-branch method in Gao, et al. [27] produces promising results (98.53% OA) with a sufficient sampling rate of 75%. However, it is still low compared to the proposed method’s 5% and 10%. The architecture of the proposed dual-branch model (using a miniGCN branch) aside from the modified input features are the main factors for improving dual-branch method classification ability with low TR. Although the polarimetric feature-driven 2D-CNN, MCFCNN, and MEWGCN produced satisfactory accuracies, they still have lower OAs compared to the proposed dual-branch FuNet.

5. Conclusions

Since PolSAR images’ characteristics vary from those of optical images, a specialized approach for extracting spatial information is required. Combining both spatial and polarimetric features allows us to fully consider the rich information of PolSAR images. Accordingly, a dual-branch network utilizing DCNN and miniGCN was developed in the current study to make the most of PolSAR features.

Deep convolutional models can consider spatial characteristics utilizing neighborhood information, for which, in the current study, Pauli RGB and four-component Yamaguchi decomposition were driven to provide spatial features of the PolSAR image. In contrast, roll-invariant and hidden polarimetric features were obtained to feed the miniGCN. miniGCN is a batchwise version of GCN that can cooperate with batchwise CNN and considers long-range similarities through an adjacency matrix, while it requires significantly lower computational cost. Eventually, the DCNN and miniGCN were fused in the proposed dual-branch network, called dual-branch FuNet, to discriminate PolSAR classes.

The proposed classifier outperforms several state-of-the-art models over AIRSAR Flevoland and San Francisco datasets, reaching above 97% OA for both cases. When compared to traditional approaches (SVM, RF, and 1D- and 2D-CNN), improvements ranged from 1.3% to 19.27% (OA). Furthermore, the dual-branch strategy outperforms the simple one-branch network by 0.73% to 1.82% in terms of OA. The investigated dual-branch network offers a robust technique to deal with different sampling rates (1 to 10%) and produces a stable result, even with the small-labeled sample, i.e., between 97.8% and 99.9% OA.