Collaborative Self-Supervised Transductive Few-Shot Learning for Remote Sensing Scene Classification

Abstract

With the advent of deep learning and the accessibility of massive data, scene classification algorithms based on deep learning have been extensively researched and have achieved exciting developments. However, the success of deep models often relies on a large amount of annotated remote sensing data. Additionally, deep models are typically trained and tested on the same set of classes, leading to compromised generalization performance when encountering new classes. This is where few-shot learning aims to enable models to quickly generalize to new classes with only a few reference samples. In this paper, we propose a novel collaborative self-supervised transductive few-shot learning (CS${}^2$TFSL) algorithm for remote sensing scene classification. In our approach, we construct two distinct self-supervised auxiliary tasks to jointly train the feature extractor, aiming to obtain a powerful representation. Subsequently, the feature extractor’s parameters are frozen, requiring no further training, and transferred to the inference stage. During testing, we employ transductive inference to enhance the associative information between the support and query sets by leveraging additional sample information in the data. Extensive comparisons with state-of-the-art few-shot scene classification algorithms on the WHU-RS19 and NWPU-RESISC45 datasets demonstrate the effectiveness of the proposed CS${}^2$TFSL. More specifically, CS${}^2$TFSL ranks first in the settings of five-way one-shot and five-way five-shot. Additionally, detailed ablation experiments are conducted to analyze the CS${}^2$TFSL. The experimental results reveal significant and promising performance improvements in few-shot scene classification through the combination of self-supervised learning and direct transductive inference.

1. Introduction

Remote sensing scene classification refers to the task of identifying and categorizing different objects or scenes in remote sensing imagery [1]. It plays a crucial role in geological exploration [2], environmental monitoring [3], urban planning [4], and disaster monitoring [5]. By performing scene classification, we can obtain vital geographic information to support decision making and resource management, thereby promoting sustainable development and the construction of smart cities [6].

With the advancement of deep learning and the widespread availability of large-scale remote sensing datasets, the accuracy of remote sensing scene classification has been significantly enhanced. Deep learning-based scene classification of remote sensing images is a technique that utilizes convolutional neural networks [7], attention mechanisms [8], recurrent neural networks [9], and other methods to automatically classify and identify objects and scenes in remote sensing images [10,11]. It effectively captures spatial and semantic information by automatically learning and extracting image features, resulting in more accurate and efficient scene classification [12,13,14]. Additionally, deep learning-based methods possess powerful feature learning and complex relationship modeling capabilities, making them suitable for classifying different scales and complex scene requirements [15]. Through training on large-scale data and transfer learning, this approach can improve the performance of classification models and provide significant support for geographic information analysis and applications [16].

However, traditional deep learning models rely on a large amount of annotated data for training in order to accurately capture the features and relationships between different categories [17]. In practical applications, acquiring a large number of annotated samples can be challenging and costly [18,19]. Additionally, traditional deep learning models use the same categories for training and testing, which leads to poor generalization when encountering new categories. This is because the model has not been exposed to annotated samples of these new categories and cannot accurately classify and recognize them [20].

Therefore, few-shot learning has become crucial in remote sensing scene classification [21]. Few-shot learning is a task that aims to learn informative and discriminative feature representations from only a limited number of labeled examples [22]. Through techniques such as transfer learning and meta-learning, the knowledge acquired can be applied to new categories for accurate classification [23]. This approach helps overcome the problem of insufficient annotated data and improves the model’s generalization ability on new categories [24]. Currently, there has been extensive research on few-shot remote sensing scene classification, which can generally be categorized into two different training paradigms: meta-learning-based [25,26,27] and transfer-learning-based approaches [28,29]. Among them, improving the metric space is a key focus of current research by scholars [30]. However, current research has found that a good feature extractor is the key to few-shot classification [30,31,32]. An effective feature extractor has a much more significant impact on the performance than a cleverly designed metric space.

In light of this indication, in this paper, we propose a novel collaborative self-supervised transductive few-shot learning (CS${}^2$TFSL) algorithm for remote sensing scene classification. In CS${}^2$TFSL, we train the feature extractor by leveraging two distinct self-supervised auxiliary tasks collaboratively to obtain a powerful feature representation. Unlike traditional few-shot learning methods that rely on labeled training data, self-supervised learning does not require any labels. Instead, it constructs pretext tasks and generates pseudo-labels for training. Afterwards, the trained feature extractor requires no further training, and its parameters are frozen, allowing for direct transfer to the inference stage. During the testing stage, we employ transductive inference, enhancing the associative information between the support and query sets by incorporating additional sample information from the dataset. Extensive comparisons with state-of-the-art (SOTA) few-shot scene classification algorithms demonstrate the effectiveness of the CS${}^2$TFSL. Additionally, we conduct detailed ablation experiments to analyze the components of CS${}^2$TFSL. The experimental results highlight significant and promising performance improvements in few-shot scene classification achieved through the combination of self-supervised learning and transductive inference.

Overall, our contributions can be summarized in the following two aspects:

(1)

We propose a collaborative self-supervised training strategy for few-shot scene classification. Based on the transfer learning paradigm, our approach allows the model to directly freeze and perform inference after self-supervised training, eliminating the need for complex episode-based training procedures in metric learning.

(2)

We introduce the simple soft k-means (SKM) classifier for the few-shot transductive inference. Compared to inductive inference, our model can benefit from more sample references for inference, and it is also simpler compared to the complex graph neural network of transductive inference.

The remaining sections of this article are structured as follows. Section 2 introduces some relevant studies on few-shot remote sensing classification and self-supervised learning. In Section 3, the proposed method is described in detail. The experimental results are reported and discussed in Section 4 and Section 5. The conclusion is presented in Section 6.

2. Related Works

2.1. Few-Shot Remote Sensing Scene Classification

According to the categorization in [28,29], current research on few-shot remote sensing scene classification can be divided into two categories: methods based on meta-learning and transfer learning. The main difference between them lies in the training paradigm. Taking the N-way K-shot scenario as an example, algorithms based on meta-learning utilize the episodic training approach. In each episode, N remote sensing categories are selected, and K images from each category are chosen as the support set for training the network. On the other hand, algorithms based on transfer learning do not use explicit support and query sets during training, instead, all categories are used as input for training. During the testing phase, both methods follow the same procedure. The testing set is divided into multiple episodes according to the N-way K-shot scenario. It is worth noting that there is no overlap in categories between the training and test set [33,34].

In terms of meta-learning-based few-shot scene classification algorithms, Zhai et al. [27] explored the application of lifelong learning techniques for scene recognition in remote sensing images. Lifelong learning refers to the ability of a machine learning model to continuously learn and adapt from new data throughout its lifetime. Cheng et al. [25] leveraged a Siamese Prototypical Network (SPNet), where two branches share weights and learn to extract features from both support and query images, with prototype self-calibration and inter-calibration. Ji et al. [26] addressed the challenges of data scarcity and domain shift. The proposed method aims to improve the performance of few-shot scene classification models by leveraging auxiliary objectives and incorporating transductive inference. Li et al. [35] combined the benefits of deep feature extraction and metric learning to improve few-shot classification performance in remote sensing images. The key components of DLA-MatchNet [35] include a deep layer aggregation backbone network and a matching network. Zeng et al. [36] incorporated an iterative process that progressively refines the feature representations and classification boundaries. It consists of two main components: an embedding network and an iterative distribution learning strategy. Huang et al. [37] proposed a task-adaptive attention component, which combined the meta-learning training mechanism and graph neural network transductive inference in few-shot scene classification.

In terms of transfer learning based few-shot scene classification algorithms, Gong et al. [28] proposed the two-path aggregation attention network with quad-patch data augmentation, which improves the few-shot scene classification performance from both the perspectives of data and network structure. Li et al. [29] proposed a multiform ensemble enhancement strategy to explore the effects of different self-supervised auxiliary tasks on the feature extraction performance.

Self-Supervised Learning

Self-supervised learning is a method that utilizes the inherent information within data for unsupervised learning, with the goal of acquiring useful representations or features [38,39,40]. It can learn from large-scale unlabeled data, reducing reliance on annotated data, and is applicable in scenarios with limited data or challenging annotation conditions. Additionally, it can be used as a form of pretraining, where learned features are fine-tuned to enhance performance on other tasks [29,41].

In the field of remote sensing, self-supervised learning has been extensively applied to various specific tasks and has achieved promising results [42]. For example, self-supervised learning has been widely explored in various domains such as hyperspectral unmixing [43], change detection [44], SAR target recognition [45], and hyperspectral classification [46].

In terms of few-shot remote sensing scene classification, Zhai et al. [47] applied the Bootstrap Your Own Latent (BYOL) [48] contrastive learning algorithm to feature extraction in the context of few-shot scene classification, with the aim of obtaining better sample representations. Li et al. [29] explored the impact of different self-supervised auxiliary tasks combined in various forms on feature extraction for few-shot scene classification. Gong et al. [43] employed the SimCLR [49] to explore the transfer relationship between the source domain and target domain in few-shot scene classification scenarios. In [50], the Meta-FSEO approach is proposed for few-shot remote sensing scene classification. Specifically, it is achieved through self-supervised embedding optimization. This method aims to train a model using a meta-learning framework that combines self-supervised learning and meta-learning concepts to improve the model’s generalization performance and adaptability by optimizing embedding representations. In terms of the current research, the exploration of self-supervised few-shot scene classification is relatively insufficient, and most studies focus on inductive inference, with only a few studies focusing on transductive inference.

3. Methodology

3.1. Overview

Few-shot classification refers to the task of training a model with very limited samples per class, in order to accurately classify new unseen samples [13,19]. It is a demanding task that enables the model to have strong generalization ability, extracting representative features and patterns from a small number of samples and applying them to classify unseen samples. Generally, the dataset in few-shot learning is divided into a base dataset and a novel dataset, and it is important to note that these two datasets have no overlapping classes. We adopt a training paradigm based on transfer learning, which means that in the training phase, the base dataset is not divided into multiple episodes, but all classes are directly trained together like traditional classification algorithms. However, during testing, the novel dataset is divided into multiple episodes for inference. To explain our point in a more straightforward manner, we take the common scenario of an N-way K-shot as an example. First, in constructing the support set and query set, N classes are selected from the novel dataset, and K labeled samples are provided for each class as the support set. The query set consists of unlabeled samples from these N classes. The goal is to classify the samples in the query set into their respective classes with the assistance of a small number of reference samples. Few-shot scene classification presents a challenge due to the scarcity of training samples available for each class. This necessitates the model to learn and generalize from a small number of samples to accurately classify the unseen samples.

We propose the CS${}^2$TFSL framework as illustrated in Figure 1. Initially, the feature extractor is trained by incorporating two collaborative self-supervised learning tasks, namely rotation and spatial contrastive learning (SCL), alongside the original semantic class prediction task. Once the training is complete, the feature extractor is fixed and directly applied during the inference phase. The features of both the support set and query set samples are first extracted, followed by using transductive inference to determine the class labels of the query samples.

3.2. Collaborative Self-Supervised Learning

In the training phase of the feature extractor, the total loss function $L_T$ can include the following two components:

$$L_T=L_{CE}+L_{SSL},$$

(1)

where $L_{CE}$ and $L_{SSL}$ represent the loss functions for semantic class prediction and self-supervised prediction, respectively.

The cross-entropy loss function, denoted as $L_{CE}$, is widely employed in classification tasks to quantify the disparity between the predicted class probabilities and the actual class labels. It serves as a measure of the model’s accuracy in assigning the correct class. Mathematically, it can be defined as follows:

$$L_{CE}=-\sum (y\ast log\left(\widehat{y}\right)),$$

(2)

where y is the true class label, and p is the predicted class probabilities. The cross-entropy loss function penalizes significant disparities between the predicted probabilities and the true label. By doing so, it motivates the model to reduce the differences and enhance its prediction accuracy. In essence, it encourages the model to better align its predictions with the true labels, leading to improved performance.

$L_{SSL}$ is a composite loss function formed by two self-supervised auxiliary tasks. Its definition is as follows:

$$L_{SSL}={\lambda }_R\ast L_R+{\lambda }_{SCL}\ast L_{SCL},$$

(3)

where $L_R$ is the loss function for the rotation pretext task. $L_{SCL}$ is the loss function for the spatial contrastive learning pretext task. ${\lambda }_R$ and ${\lambda }_{SCL}$ are weight parameters that control the importance of each task in the overall loss. Each task aims to provide additional learning signals to improve the feature extractor’s representation capabilities. The weights ${\lambda }_R$ and ${\lambda }_{SCL}$ allow for adjusting the relative impact of each task on the overall training process.

The purpose of the rotation prediction task is to determine the specific 2-D rotation transformation applied to an input image. In contrast to using semantic class labels, the supervision signal in this task is independent of any particular category, which facilitates information exchange across different classes. Additionally, using rotation as a supervisory signal assumes that the neural network has already acquired knowledge about object categories and learned about their constituent parts before being able to perform accurate rotation recognition. Through training the model to predict the precise rotation for an image, the neural network implicitly learns to extract robust and discriminative features that capture relevant information about objects and their spatial relationships. This approach promotes a deeper understanding of objects and refines the model’s ability to distinguish between different classes. We define a collection of rotation operators as $\mathsf{\Theta }$ = ${\left\{{\varphi }_{\omega }\right\}}_{\omega =1}^{\mathsf{\Omega }}$, where $x_{\omega }$ = ${\varphi }_{\omega }\left(x\right)$ represents the rotated image, and $\mathsf{\Omega }$ is the total number of rotations. Given the parameters ${\Delta }_f$ = [${\alpha }_1,{\alpha }_2,...,{\alpha }_R]$ of the rotation classifier, the probability for the input image $x_j$ is given by:

$$p({\widehat{y}}_k^{\varphi }=\omega |x_k)=\frac{exp\left({\alpha }_{\omega }^T{f}_{\theta }\left({\varphi }_{\omega }\left(x_k\right)\right)\right)}{{\sum }_{\omega =1}^{\mathsf{\Omega }}exp\left({\alpha }_{\omega }^T{f}_{\theta }\left({\varphi }_{\omega }\left(x_k\right)\right)\right)}.$$

(4)

where $f_{\theta }$ represents the feature extractor.

Based on this, the $L_R$ can be defined as follows:

$$L_R=-\sum _{k=1}^B\sum _{\omega }^{\mathsf{\Omega }}\prod (y_k^{\varphi }=\omega )log\left(p({\widehat{y}}_k^{\varphi }=\omega |x_k)\right).$$

(5)

where B represents the batchsize, and $y_k$ and ${\widehat{y}}_k$ denote the rotated ground-truth pseudo-labels and the predictions made by the model, respectively.

Spatial contrastive learning utilizes the concept of spatial similarity to evaluate the degree of similarity or agreement between two different feature representations. In simpler terms, when conducting spatial contrastive learning, we use a measure called “spatial similarity” to determine how similar or different two feature representations are from each other. This approach is integral in assessing the similarity and agreement between different features. This similarity captures the spatial relationship or alignment between the features extracted from different views or augmentations of the same image. The spatial similarity metric helps in determining the degree of correspondence or match between the extracted features, which is crucial for the contrastive loss function used in self-supervised learning. The definition of spatial similarity $sim(s_i,s_m)$ between two features is consistent with [51]. Following this lead, the spatial contrastive learning loss function can be defined as:

$$L_{SCL}=\sum _{i=1}^{2B}\frac{1}{2B_{y_i}-1}\sum _{j=1}^{2B}{\mathbb{1}}_{i\ne j}·{\mathbb{1}}_{y_i\ne y_j}·{\ell }_{ij},$$

(6)

$${\ell }_{ij}=-log\frac{exp(sim(s_i,s_j)/\tau )}{{\sum }_{m=1}^{2B}{\mathbb{1}}_{i\ne m}exp(sim(s_i,s_m)/\tau )},$$

(7)

where ${\mathbb{1}}_{condition}\in \{0,1\}$ is an indicator function, which takes a value of 1 only if the specified condition is satisfied. $B_{y_i}$ represents the total number of images within a dataset that possess the same label $y_i$. Moreover, $\tau$ denotes the scalar temperature parameter.

3.3. Transductive Inference

In few-shot transductive inference, a classifier that operates with limited training examples has access to both the support dataset (containing labeled examples) and the complete query dataset (unlabeled examples) during the prediction phase. In our approach, we utilize a straightforward soft k-means (SKM) classifier [30] for transductive inference. This classifier divides the query set into smaller subsets, each associated with a small number of labeled support sets. After the previous training is completed, we freeze the feature extractor and proceed directly to inference. Firstly, we obtain the sample feature representations $\mathcal{K}$ for the support set and query set. Next, we need to preprocess the vectors by centering them and projecting them onto a hypersphere, as shown below:

$${\mathcal{K}}_C=\frac{\mathcal{K}-\tilde{\mathcal{K}}}{\left|\right|\mathcal{K}-\tilde{\mathcal{K}}{\left|\right|}_2},$$

(8)

where the $\tilde{\mathcal{K}}$ is the average feature of the support set.

First, given that the barycenters in transductive inference require recalculation on each occasion, where t is the index of the sequence, the initialization of ${\tilde{N}}_t$ can be computed as follows:

$${\tilde{N}}_t^{u+1}=\sum _{{\mathcal{K}}_C\in {\mathcal{S}}_k\cup \mathcal{Q}}^{}\frac{w({\mathcal{K}}_C,{\tilde{N}}_t^u)}{{\sum }_{{\mathcal{K}}_C^{{}^{\prime }}\in {\mathcal{S}}_k\cup \mathcal{Q}}^{}w({\mathcal{K}}_C^{{}^{\prime }},{\tilde{N}}_t^u)}{\mathcal{K}}_C,$$

(9)

where $\mathcal{S}$ represents the support set, and $\mathcal{Q}$ denotes the query set. ${\mathcal{S}}_k$ is the feature extracted by the model in the k-th category. $w({\mathcal{K}}_C,{\tilde{N}}_t^u)$ is the weighting function on ${\mathcal{K}}_C$, which determines the probability associated with the ${\tilde{C}}_k$, and the weight is calculated based on a decreasing function ${\ell }_2$ of the distance between the data samples and the class centroids. Its definition is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill w({\mathcal{K}}_{CH},{\tilde{C}}_k^t)=\left\{\begin{array}{cc}1,\hfill & if{\mathcal{K}}_C\in {\mathcal{S}}_k\hfill \\ \frac{exp(-\beta ||{\mathcal{K}}_C-{\tilde{N}}_t^u|{|}_2^2)}{{\sum }_{u=1}^{N}exp(-\beta ||{\mathcal{K}}_C-{\tilde{N}}_t^u{\left|\right|}_2^2)},\hfill & if{\mathcal{K}}_C\in \mathcal{Q}\hfill \end{array}\right.\end{array},\end{array}$$

where $\beta$ represents the temperature value.

Based on this, all the unlabeled samples ${\mathcal{K}}_C$ in the query set can be classified as follows:

$$Class({\mathcal{K}}_C,[{\tilde{N}}_1^{\infty }...,{\tilde{N}}_N^{\infty }])=arg\underset{t}{min}\left|\right|{\mathcal{K}}_C-{\tilde{N}}_N^{\infty }{\left|\right|}_2.$$

(10)

4. Experimental Results and Analysis

4.1. Datasets

To assess the efficacy of our proposed CS${}^2$TFSL, we conducted comprehensive comparisons between the CS${}^2$TFSL and state-of-the-art methods on two widely recognized benchmark remote sensing few-shot scene classification datasets, namely the WHU-RS19 dataset [52] and the NWPU-RESISC45 dataset [53].

The WHU-RS19 dataset [52] is a widely used remote sensing dataset for scene classification. It was created by Wuhan University and consists of 19 classes of high-resolution remote sensing images. This dataset covers various land cover types, including urban areas, agricultural fields, forests, water bodies, and more. The images are captured by satellites with different sensors, such as SPOT-5 and GeoEye-1, providing a diverse range of spectral and spatial information. With a total of 2000 images, the dataset offers a balanced distribution of samples across different classes, ensuring equal representation for each category. Each image has a resolution of 600 × 600 pixels and is labeled with one of the 19 scene classes.

The NWPU-RESISC45 dataset [53] is created by Northwestern Polytechnical University and consists of 45 different scene categories. The images in the NWPU-RESISC45 dataset are captured from various regions using high-resolution satellite sensors, covering a wide range of scene types such as urban areas, farmland, forests, bodies of water, and industrial zones. Specifically, it is obtained using Google Earth satellite imagery and the Google Earth API. Each category in the dataset contains approximately 700 images, resulting in a total of around 31,500 images. This ensures an adequate number of samples and class balance, enabling effective evaluation of scene classification algorithms. Each image has a resolution of 256 × 256 pixels and is stored in JPEG format. The dataset also includes label files for each image, indicating their corresponding scene category.

In the context of few-shot remote sensing scene classification, we follow the partitioning method as described in [26,28,29,35,47]. The detailed division of the training set, validation set, and testing set for the two benchmark datasets is illustrated in Figure 2. From the information above, it can be inferred that the WHU-RS19 and NWPU-RESISC45 datasets pose comprehensive challenges to the few-shot scene classification algorithm performance in terms of the scale, resolution, and data volume.

4.2. Implementation Details

The backbone used in CS${}^2$TFSL is ResNet-12, which is commonly employed in few-shot scene classification task. In line with [26,28,29,35,47], we report the results for both five-way one-shot and five-way five-shot scenarios, with data presented within a 95% confidence interval. We conducted all our experiments using the Pytorch framework on CUDA 11.3, with a single NVIDIA RTX 3090 graphics card equipped with 24 GB. Based on the above software and hardware conditions, we trained our network with 600 epochs and a batch size of 16. We selected the SGD (Stochastic Gradient Descent) optimizer with a momentum of 0.9. In terms of comparative algorithms, we selected 14 classical and state-of-the-art few-shot classification algorithms, namely MAML [54], LLSR [27], Ji et al. [26], baseline [33], S2M2 [55], Meta-SGD [56], MatchingNet [57], ProtoNet [58], RelationNet [59], DLA-MatchNet [35], TAE-Net [37], IDLN [36], CAN+T [60], SPNet [25], and DANet [28]. For fair comparison, all comparison methods were reported from their respective papers and taken from the final experimental results.

4.3. Comparison with Other Methods

4.3.1. Results on the WHU-RS19 Dataset

Table 1. Comparing the overall performance (in %) of the proposed CS${}^2$TFSL with other few-shot scene classification methods on the WHU-RS19 dataset; we highlight the best result in bold.

Method	5-Way
	1-Shot	5-Shot
MAML [54]	$59.92\pm 0.35$	$82.30\pm 0.23$
LLSR [27]	$57.10$	$70.65$
Ji et al. [26]	$85.41\pm 0.35$	$92.28\pm 0.13$
baseline [33]	$75.57\pm 0.36$	$88.65\pm 0.18$
S2M2 [55]	$69.00\pm 0.41$	$82.14\pm 0.21$
Meta-SGD [56]	$51.54\pm 2.31$	$61.74\pm 2.02$
MatchingNet [57]	$76.14\pm 0.35$	$84.00\pm 0.20$
ProtoNet [58]	$77.00\pm 0.36$	$91.70\pm 0.15$
RelationNet [59]	$77.76\pm 0.34$	$86.84\pm 0.15$
DLA-MatchNet [35]	$70.21\pm 0.32$	$81.86\pm 0.52$
IDLN [36]	$73.89\pm 0.88$	$83.12\pm 0.56$
TAE-Net [37]	$73.67\pm 0.74$	$88.95\pm 0.52$
CAN+T [60]	$69.79\pm 0.56$	$79.71\pm 0.22$
SPNet [25]	$81.06\pm 0.60$	$88.04\pm 0.28$
DANet [28]	$75.02\pm 0.16$	$89.21\pm 0.07$
CS${}^2$TFSL (Ours)	$\mathbf{86}.\mathbf{03}\pm \mathbf{0}.\mathbf{13}$	$\mathbf{93}.\mathbf{09}\pm \mathbf{0}.\mathbf{05}$

presents a performance comparison between our proposed CS${}^2$TFSL and the counterparts on the WHU-RS19 dataset. The best results are highlighted in bold. In the five-way one-shot scenario, the top three performers achieved accuracy rates of $86.03\pm 0.13$% for CS${}^2$TFSL, $85.41\pm 0.35$% for Ji et al. [26], and $81.06\pm 0.60$% for SPNet [25]. More specifically, CS${}^2$TFSL not only outperformed the second-place method by a margin of 0.62% in terms of accuracy but also significantly surpassed the other competitors. The outstanding performance of the CS${}^2$TFSL is also evident in the five-way five-shot scenario. In more detail, CS${}^2$TFSL outperformed the second-best algorithm (DANet [28]) by 1.19% and surpassed the third-best algorithm (baseline [33]) by an additional 2.86%. In general, the proposed CS${}^2$TFSL performed well on the small-scale WHU-RS19 dataset. This is attributed to the synergistic training of two self-supervised auxiliary tasks, which effectively explore the unlabeled data in the WHU-RS19 dataset. Additionally, the combination of information from other samples in the dataset and the use of transductive inference contribute to its performance.

4.3.2. Results on the NWPU-RESISC45 Dataset

Table 2. Comparing the overall performance (in %) of the proposed CS${}^2$TFSL with other few-shot scene classification methods on the NWPU-RESISC45 dataset; we highlight the best result in bold.

Method	5-Way
	1-Shot	5-Shot
MAML [54]	$58.99\pm 0.45$	$72.67\pm 0.38$
LLSR [27]	$51.43$	$72.90$
Ji et al. [26]	$69.80\pm 0.53$	$82.03\pm 0.30$
baseline [33]	$69.02\pm 0.46$	$85.62\pm 0.25$
S2M2 [55]	$63.24\pm 0.47$	$83.23\pm 0.28$
Meta-SGD [56]	$60.63\pm 0.90$	$75.75\pm 0.65$
MatchingNet [57]	$61.57\pm 0.49$	$76.02\pm 0.34$
ProtoNet [58]	$64.52\pm 0.48$	$81.95\pm 0.30$
RelationNet [59]	$65.52\pm 0.85$	$78.38\pm 0.31$
DLA-MatchNet [35]	$71.56\pm 0.30$	$83.77\pm 0.64$
IDLN [36]	$75.25\pm 0.75$	$84.67\pm 0.23$
TAE-Net [37]	$69.13\pm 0.83$	$82.37\pm 0.52$
CAN+T [60]	$69.89\pm 0.58$	$81.04\pm 0.33$
SPNet [25]	$67.84\pm 0.87$	$83.94\pm 0.50$
DANet [28]	$74.30\pm 0.20$	$87.29\pm 0.11$
CS${}^2$TFSL (Ours)	$\mathbf{83}.\mathbf{67}\pm \mathbf{0}.\mathbf{25}$	$\mathbf{88}.\mathbf{48}\pm \mathbf{0}.\mathbf{17}$

presents a performance comparison between our proposed CS${}^2$TFSL and the counterparts on the NWPU-RESISC45 dataset. The best results are highlighted in bold. From the experimental results, it can be observed that the CS${}^2$TFSL also performed remarkably well on the large-scale NWPU-RESISC45 dataset, achieving the best results in both five-way one-shot and five-way five-shot scenarios. Specifically, in the five-way one-shot scenario, the CS${}^2$TFSL outperformed the second-best algorithm (IDLN [36]) by a significant margin of 8.42%. However, in the five-way five-shot scenario, it only surpassed the second-best algorithm (DANet [28]) by a slight margin of 1.19%. This is because the NWPU-RESISC45 dataset is a very large-scale dataset, and with only one reference sample, the limitation of inter-class similarity becomes significantly magnified. However, as the number of reference samples increases, the model’s requirement for small-sample classification decreases. This precisely demonstrates that the CS${}^2$TFSL also exhibits very stable performance in extreme scenarios.

5. Discussion

To fully demonstrate the effectiveness of the components proposed by the CS${}^2$TFSL, we conducted ablation experiments in this section, including collaborative self-supervised learning and transductive inference. Figure 3 displays the heatmap visualization of the ablation experiments of the CS${}^2$TFSL under different self-supervised auxiliary tasks. In the figure, “baseline” refers to the pretraining model of the feature extractor without any self-supervised auxiliary tasks. “rotation” and “SCL” indicate the inclusion of the rotation pretext task and spatial contrastive learning pretext task, respectively. It can be observed from the figure that the feature extractor in the baseline model lacks the ability to capture crucial objects in the scene. The network’s attention does not focus on the key information in the scene. By incorporating self-supervised auxiliary tasks during pretraining, the models show improved ability to capture the information about crucial objects in the scene. However, these two pretext tasks, rotation and SCL, have their own strengths and weaknesses, and they only perform well in a few specific scenes. For example, in the basketball court scene, both tasks do not perform well. On the other hand, in the river scene, the rotation task performs worse than the SCL task. Conversely, in the ground track field scene, the opposite effect is observed.

In addition, the quantitative analysis of different self-supervised auxiliary task ablation experiments in two benchmark datasets is shown in

Table 3. The ablation experimental results of the CS${}^2$TFSL self-supervised auxiliary task, and the comparison between the inductive inference and transductive inference experiments, where “†” represents the inductive inference. We highlight the best result and the second-best result in bold and underline, respectively.

Dataset	Method	5-Way
		1-Shot	5-Shot
WHU-RS19	baseline ${}^{†}$	$68.86\pm 0.22$	$76.74\pm 0.10$
	+Rotation ${}^{†}$	$75.18\pm 0.18$	$80.57\pm 0.09$
	+SCL ${}^{†}$	$79.69\pm 0.15$	$85.29\pm 0.09$
	CS${}^2$TFSL${}^{†}$	$\underline{85.26\pm 0.13}$	$91.42\pm 0.07$
	baseline	$70.16\pm 0.22$	$80.39\pm 0.10$
	+Rotation	$77.14\pm 0.21$	$83.18\pm 0.09$
	+SCL	$83.64\pm 0.17$	$88.83\pm 0.06$
	CS${}^2$TFSL	$\mathbf{86}.\mathbf{03}\pm \mathbf{0}.\mathbf{13}$	$\mathbf{93}.\mathbf{09}\pm \mathbf{0}.\mathbf{05}$
NWPU-RESISC45	baseline ${}^{†}$	$70.39\pm 0.29$	$80.65\pm 0.18$
	+Rotation ${}^{†}$	$74.65\pm 0.26$	$82.38\pm 0.18$
	+SCL ${}^{†}$	$78.19\pm 0.26$	$85.66\pm 0.18$
	CS${}^2$TFSL${}^{†}$	$\underline{80.37\pm 0.26}$	$\underline{87.58\pm 0.17}$
	baseline	$73.07\pm 0.29$	$81.90\pm 0.18$
	+Rotation	$76.99\pm 0.25$	$83.67\pm 0.18$
	+SCL	$80.15\pm 0.25$	$84.10\pm 0.17$
	CS${}^2$TFSL	$\mathbf{83}.\mathbf{67}\pm \mathbf{0}.\mathbf{25}$	$\mathbf{88}.\mathbf{48}\pm \mathbf{0}.\mathbf{17}$

. From the table, it can be observed that the design of the self-supervised auxiliary tasks improves the performance on few-shot remote sensing scene classification. The improvement in spatial contrastive learning is more significant compared to the effect of rotation tasks. Moreover, combining both approaches in the CS${}^2$TFSL leads to an incremental improvement in accuracy for few-shot scene classification. Furthermore, Table 3 also presents the comparison results between inductive inference and transductive inference on two benchmark few-shot classification datasets. Compared to the traditional approach of using inductive inference in most few-shot scene classification algorithms, transductive inference, which incorporates additional information from other samples, has shown better performance in terms of classification accuracy. Specifically, transductive inference has shown a significant improvement in overall accuracy, with at least a 1% increase. This improvement is quite substantial in the context of the few-shot scene classification task.

6. Conclusions

In this paper, we introduce a novel cooperative self-supervised transductive few-shot learning algorithm for remote sensing scene classification, called CS${}^2$TFSL. We address the challenge of limited labeled data by leveraging two distinct self-supervised auxiliary tasks to collaboratively train the feature extractor and obtain a powerful representation. Subsequently, the trained feature extractor requires no further training, and its parameters are frozen, enabling seamless transfer to the inference stage. During the inference stage, we employ transductive inference, combining additional sample information in the data to enhance the associative information between the support and query sets. Extensive comparisons with state-of-the-art (SOTA) few-shot scene classification algorithms demonstrate the effectiveness of the CS${}^2$TFSL. Furthermore, we conduct detailed ablation experiments to analyze the components of the CS${}^2$TFSL. The experimental results highlight significant and promising performance improvements in few-shot scene classification achieved through the combination of self-supervised learning and transductive inference.