Few-Shot Unsupervised Domain Adaptation Based on Refined Bi-Directional Prototypical Contrastive Learning for Cross-Scene Hyperspectral Image Classification

Abstract

Hyperspectral image cross-scene classification (HSICC) tasks are confronted with tremendous challenges due to spectral shift phenomena across scenes and the tough work of obtaining labels. Unsupervised domain adaptation has proven its effectiveness in tackling this issue, but it has a fatal limitation of intending to narrow the disparity between source and target domains by utilizing fully labeled source data and unlabeled target data. However, it is costly even to attain labels from source domains in many cases, rendering sufficient labeling as used in prior work impractical. In this work, we investigate an extreme and realistic scenario where unsupervised domain adaptation methods encounter sparsely labeled source data when handling HSICC tasks, namely, few-shot unsupervised domain adaptation. We propose an end-to-end refined bi-directional prototypical contrastive learning (RBPCL) framework for overcoming the HSICC problem with only a few labeled samples in the source domain. RBPCL captures category-level semantic features of hyperspectral data and performs feature alignment through in-domain refined prototypical self-supervised learning and bi-directional cross-domain prototypical contrastive learning, respectively. Furthermore, our framework introduces the class-balanced multicentric dynamic prototype strategy to generate more robust and representative prototypes. To facilitate prototype contrastive learning, we employ a Siamese-style distance metric loss function to aggregate intra-class features while increasing the discrepancy of inter-class features. Finally, extensive experiments and ablation analysis implemented on two public cross-scene data pairs and three pairs of self-collected ultralow-altitude hyperspectral datasets under different illumination conditions verify the effectiveness of our method, which will further enhance the practicality of hyperspectral intelligent sensing technology.

1. Introduction

Hyperspectral image (HSI) performs prominence research value in view of its hundreds of continuous spectral channels with abundant geospatial and spectral information [1,2,3]. The fine HSI classification (HSIC) aims at assigning each pixel to a predefined land cover category based on its inherent spectral and spatial properties [4,5,6,7]. It is one of the most fundamental and commonly addressed contents of various remote sensing applications, such as environmental monitoring, resource investigation, and smart city [8,9,10]. Over the past decade, deep learning (DL) has shown enormous superiority over conventional machine learning (ML) in the fields of natural language processing and computer vision due to its mighty representation capability [11,12,13,14]. Inspired by this, researchers have tried to conquer the issue of HSIC with deep learning algorithms and scored tremendous achievements. An impressive number of HSIC methods for the single scene have been investigated [15,16,17,18,19], which benefit from the massive annotated data and typically follow a common hypothesis, that is, the training and testing sets originate from the independent and identical distributions.

However, in practical remote sensing applications, hyperspectral images (HSIs) collected at different locations or times often violate the assumption of distribution consistency. This is because of differences in climate conditions or changes in the physical properties of the target material over time, resulting in distinct interclass similarity and intraclass variability between HSIs, which is called the spectral shift phenomenon [20]. Therefore, HSIC tasks often encounter cross-scene challenges that will yield a significant performance drop when generalizing the well-trained model to another unlabeled scene [21]. Transfer learning (TL) tries to transfer knowledge from the source domain to the target domain by relaxing the assumption that the training data and the test data must have independent and identical distribution [22]. Consequently, specific task-oriented transfer learning (TL), also known as domain adaptation (DA), has emerged, and it is in line with the real hyperspectral image cross-scene classification (HSICC) task [23,24,25,26,27,28,29,30,31,32], especially the unsupervised domain adaptation (UDA) [33,34,35]. Recent literature is elaborated in Section 3. UDA methods intend to fit the distribution gap between source and target domains to transfer knowledge from a sufficiently labeled source domain to an unlabeled target domain, which predicts the target data without leveraging any labeling information from the target samples.

Although many UDA methods could achieve high accuracy in general HSICC tasks, they all rely on utilizing adequate supervision information in the source domain, together with the unlabeled target samples for domain matching. In real-world scenarios, due to the high annotation cost and enormous annotation difficulty, even acquiring a fully annotated source domain is an extravagant expectation. Usually, in the fields of battlefield situational awareness and resource investigation, we can only obtain annotations for a few samples from the source domain. We call it few-shot unsupervised domain adaptation (FUDA), which is also required to be capable of handling the circumstances of no available labels in the target domain. Unfortunately, there is currently no literature involved in this challenging topic among the hyperspectral community. Almost all the related work focuses on the cross-domain few-shot learning setting, which assumes that the target domain also provides a few labeled data [36,37,38,39,40,41]. For daily RGB images, several FUDA methods [42,43,44,45,46] are proposed to solve the cross-domain classification problem with the scarcity of labeled data in the source domain, which are extensively discussed in Section 2. Since only a very limited number of labeled source samples could be adopted, it is arduous to learn distinguishable features in the source domain. These FUDA methods mine the information of unlabeled samples with the assistance of the self-supervised contrastive learning strategy [42,43] that possesses excellent representational ability or the few-shot learning scheme [44,45,46] so as to achieve domain alignment.

In this paper, inspired by the above works, we propose a refined bi-directional prototypical contrastive learning (RBPCL) architecture for the FUDA setting in the HSICC task that integrates unsupervised clustering learning, prototypical representation learning, and domain alignment with few-shot labeled source data. RBPCL comprises four major components to learn both domain-invariant and category-discriminative features. RBPCL firstly encodes the semantic structure of source and target data into the embedding space and performs K-means clustering according to the class-balanced multicentric dynamic (BMD) prototype strategy [47] to generate more robust and representative prototypes (i.e., representative embedding for a group of semantically similar instances), which tries to avoid the negative transfer trouble induced by class imbalance and the inability to aggregate each class of potential typical samples when employing the routine monocentric prototype strategy [48]. Second, RBPCL leverages the preliminarily encoded semantic information and clustered prototypes to conduct in-domain prototypical contrastive learning to learn better semantic structure. In the process, we design a Siamese-style distance metric loss function to aggregate intra-class features while increasing the discrepancy between inter-class features, ultimately mitigating the impact of false negatives. Third, RBPCL executes cross-domain bi-directional prototypical self-supervision following the robust instance-to-prototype matching pattern for transferring knowledge from the source to the target domain. It is more robust to cross-domain abnormal samples compared with the instance-to-instance matching manner, thus promoting faster and smoother optimization convergence. Fourth, RBPCL integrates prototypical contrastive learning with a cosine classifier, which is adaptively updated by feature vectors from source to target embedding space step by step, contributing to better classification performance on the target domain. The cosine classifier calculates the dot product between feature vectors and each weight vector of the cosine classifier and then inputs them into the softmax layer for probabilistic classification. To further alleviate the effect of cross-domain mismatching, similar to [43], we perform entropy maximization of expected network prediction and mutual-information (MI) maximization between input and network output to obtain more diversified outputs upon source and target datasets and get high-confidence predictions for each sample, respectively.

The main contributions of our proposed RBPCL can be summarized as follows:

1.

We propose a refined bi-directional prototypical contrastive learning (RBPCL) framework for the few-shot unsupervised domain adaptation (FUDA) setting in hyperspectral cross-scene classification (HSICC) tasks. So far as we know, this work is the first attempt to tackle the FUDA setting in the field of hyperspectral cross-scene classification.

2.

We leverage refined in-domain and bi-directional cross-domain prototypical contrastive learning to simultaneously realize efficient category-discriminative feature representation and cross-domain alignment in an end-to-end, unsupervised, and adaptive pattern.

3.

We employ the class-balanced multicentric dynamic (BMD) prototype strategy to facilitate the generation of more representative and robust clustering prototypes. Furthermore, we design a Siamese-style distance metric loss function to gather intra-class features while segregating inter-class features, ultimately promoting refined prototypical self-supervised learning.

4.

We carry out exhaustive crossover experiments on five data pairs with different degrees of domain shifts, including hyperspectral images of spatially disjoint geographical regions and multitemporal images, to demonstrate the effectiveness of the proposed method. It is noteworthy that the practical value of our RBPCL is demonstrated using three ultralow-altitude hyperspectral images, independently collected by an unmanned aerial vehicle (UAV) under varying geographic locations and illumination conditions.

The subsequent sections of this paper are arranged as follows. Section 2 introduces the latest development of related work. Section 3 describes the specific implementation of the proposed RBPCL model. Section 4 provides the details of experiments with corresponding results and analysis. The conclusions are given in Section 6.

2. Related Work

2.1. Unsupervised Domain Adaptation

The conventional unsupervised domain adaptation (UDA) setting aims to reduce the domain gap and utilize sufficient labeled source domain data to accomplish classification in the unlabeled target domain. There are plenty of UDA methods that show efficient performance, which are based on a Maximum Mean Discrepancy loss (MMD) to minimize the feature divergence across domains [34,49,50,51]. Besides, with the development of generative adversarial networks, adversarial training is widely used to learn a representation that is simultaneously discriminative in the source domain and domain-invariant for tackling domain shift [52,53,54,55,56]. Moreover, several methods train the classifier in both the source domain and target domain by leveraging pseudo-labels from the target domain to calculate classification loss [57,58,59,60]. In addition, self-supervised learning has been adopted for unsupervised domain adaptation recently [61,62]. For HSICC tasks, Wang et al. designed a domain adversarial broad adaptation network (DABAN), which integrates a bottleneck adaptation module and a conditional adaptation broad network [25]. Cheng et al. propose an unsupervised HSI classification method called soft instance-level domain adaptation with the virtual classifier [26]. Moreover, UDA with dense-based compaction (UDAD) and UDA with content-wise alignment (UDACA) were proposed for the HSICC task successively [28,29]. Zhao et al. recently proposed a novel UDA framework toward multilevel features and decision boundaries (ToMF-B) for the HSICC task [30]. Overall, these UDA methods expect full supervision of the source domain for domain alignment and cross-domain classification but would not work when encountering the dilemma of a scarce labeled source domain existing in the FUDA setting for HSICC tasks.

2.2. Prototypical Contrastive Learning

Self-supervised learning (SSL) [63] is a subset of unsupervised learning methods, which refers to learning strong semantic feature representations without the demand for class label supervision. Instead, it automatically builds supervision information from unlabeled data. Instance-wise contrastive learning (ICL) [63,64,65] is an extensively used self-supervised learning strategy to improve the discriminative ability with the idea of pulling different views from the same sample together and pushing negative samples far apart using instance-level contrastive loss. Some works demonstrate their superiority in classification downstream tasks compared with the massive ICL frameworks, focusing on the prototypical contrastive learning (PCL) task [66,67,68]. PCL introduces prototypes as latent variables to help search for the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. PCL [66] combines a cluster algorithm with contrastive learning, and it replaces the InfoNCE loss with the ProtoNCE loss for contrastive learning to promote embeddings closer to their assigned prototypes and far from negative prototypes. SPCL [67] employs a Siamese-style metric loss to mitigate the impact of hard false-negative samples, which match intra-prototype features while expanding the distance between inter-prototype representations. In the hyperspectral community, Cao et al. [69] apply unsupervised prototypical contrastive learning to hyperspectral classification by designing two autoencoder-based modules. Wang et al. [70] propose an adversarial prototype learning (APL) method for training an accurate HSI classification model in a uniform manner when the training set contains few, high-dimensional, and biased samples.

2.3. Few-Shot Unsupervised Domain Adaptation

Compared with UDA, few-shot unsupervised domain adaptation (FUDA) handles UDA tasks by leveraging a few labeled source domain samples. Besides, cross-domain FSL [36,37,38,39,40,41,70,71,72,73,74,75,76,77,78] is not capable of solving the issues of no available labels in the target domain and the large domain gap between the support and query sets in every task, which will be settled in the FUDA. In the field of computer vision, there have been a few attempts concerned with the FUDA setting recently. Kim et al. proposed a cross-domain self-supervised (CDS) [42] learning approach for domain adaptation, which captures apparent visual similarity with in-domain self-supervision in a domain-adaptive manner and performs cross-domain feature alignment with cross-domain self-supervised learning. However, cross-domain instance-to-instance matching in CDS is very sensitive to abnormal samples. Yue et al. proposed the prototypical cross-domain self-supervised learning (PCS) [43] framework, which performs prototype self-supervised learning in-domain and cross-domain concurrently, but the coarse-grained clustering manner could easily lead to negative transfer. In addition, Huang et al. [44] put forward the image-to-class sparse similarity encoding (IMSE) method, which utilizes local features to learn similarity patterns for cross-domain similarity measurement. Meta-FUDA [45] performs task-level transfer and domain-level transfer jointly through leveraging meta-learning-based optimization. Yu et al. [46] proposed a task-specific semantic feature learning method (TSECS) for FUDA. TSECS learns high-level semantic features for image-to-class similarity measurement and designs a cross-domain self-training strategy to fully utilize the few labeled samples from the source domain to establish the classifier in the target domain.

3. Methodology

3.1. Problem Definition

The few-shot unsupervised domain adaptation setting (FUDA) involves two domains: a source domain $X_{s^{+}}=X_s\cup X_{su}$ (including a very limited number of labeled samples for each category $X_s={\left\{\left(x_i^s, y_i^s\right)\right\}}_{i=1}^{n_s}$, as well as unlabeled samples $X_{su}={\left\{\left(x_i^{su}\right)\right\}}_{i=1}^{n_{su}}$, where $n_s$ represents the number of labeled samples from source domain, $n_{su}$ denotes the number of unlabeled samples from source domain) and a target domain $X_{tu}$ (including the unlabeled samples $X_{tu}={\left\{\left(x_i^{tu}\right)\right\}}_{i=1}^{n_{tu}}$ with different data characteristics and homogeneous label space from the source domain, where $n_{tu}$ denotes the number of unlabeled samples in the target domain.), where $x_i^s, x_i^{su}, x_i^{tu}\in {\mathbb{R}}^{H\times W\times B}$, $x_i^s$ and $x_i^{su}$ are collectively referred to as $x_i^{s^{+}}$, and $y_i^s\in {\left\{0, 1, 2,\dots , K\right\}}^{H\times W}$ (H and W denote the spatial dimensions (height and width) of the hyperspectral image, B represents the total number of spectral bands, and k is the number of ground-truth categories). Moreover, in the FUDA setting, the number of classes is known, and we represent it as $n_c$. Our goal is to train a cross-domain classification model by utilizing these three data sources ($X_s$, $X_{su}$, and $X_{tu}$) for evaluating on the target domain $X_{tu}$.

3.2. Overall Framework

The overview of our proposed RBPCL is presented in Figure 1. The base model includes a feature generator with the Double-Branch Dual-Attention Mechanism Network (DBDA) [79], a ${\mathcal{l}}_2$ normalization layer, which outputs a normalized feature vector, and a cosine similarity-based classifier. Specifically, hyperspectral image patches are utilized as input samples, which are extracted around the pixels being tackled. To this end, DBDA is employed as the shared feature generator for both domains, consisting of two distinct branches—namely, the spectral and spatial branches—that are designed to capture spectral and spatial features independently. Moreover, the channel-wise attention mechanism and spatial-wise attention mechanism are adopted to refine the feature maps. Thus, the spectral-spatial joint features are extracted for the HSICC task, benefiting from the synergy between spatial and spectral information in the course of training. Based on this, our approach achieves discriminative feature learning and domain alignment through unifying k-means clustering, intra-domain refined prototypical contrastive learning, bi-directional inter-domain prototypical contrastive learning, and adaptive prototypical classifier learning adaptively.

3.3. In-Domain Refined Prototypical Contrastive Learning

For domain adaptation, if prototypical contrastive self-supervised learning is naively applied on $X_{s^{+}}=X_s\cup X_{tu}$ would lead to potential problems. Primarily due to the domain shift, samples of different categories from different domains could be mistakenly gathered into the same cluster, and samples of the same category from different domains could be mapped into clusters that are far away from each other. To alleviate these problems, we perform prototypical contrastive learning separately in $X_{s^{+}}$ and in $X_{tu}$ to prevent the negative transfer caused by incorrect cluster generation across domains and indistinguishable feature learning.

Concretely, we maintain two memory banks, $V^S$ and $V^t$, for source and target domains, respectively:

$$V^s=[v_1^s, v_2^s,\dots v_{(n_s+n_{su})}^s], V^t=[v_1^t, v_2^t,\dots v_{n_{tu}}^t],$$

(1)

where $v_i^s$, $v_i^t$ are respectively the stored feature vectors of $x_i^{s^{+}}$ and $x_i^{tu}$, updated with a momentum $m=0.99$ after each batch and initialized with $f_i^s$, (the output of feature extractor F, $f_i^s=F\left(x_i^{s^{+}}\right), f_i^t=F\left(x_i^{tu}\right)$):

$$v_i^s\leftarrow m{v}_i^s+(1-m)f_i^s,v_i^t\leftarrow m{v}_i^t+(1-m)f_i^t.$$

(2)

For in-domain prototypical contrastive learning, k-means clustering is performed on $V^s$ and $V^t$ to obtain source and target clusters. However, the domain gaps between source and target are typically different between categories, resulting in relatively higher prediction confidence scores for those easy-transfer classes in both domains. Moreover, due to the domain gap, we consider that a rough monocentric feature prototype for each class could not effectively represent the source and target data and would also induce negative transfer, particularly for those hard-transfer samples. Therefore, we introduce a class-balanced multicentric dynamic prototype (BMD) strategy to generate more representative and robust prototypes, thereby promoting prototypical self-supervised training.

The BMD strategy consists of two main components, namely inter-class balanced prototype and intra-class multicentric prototype, which can jointly prevent the gradual dominance of easy-transfer categories on prototype generation and reduce the noisy labels for those hard-transfer samples. We compare the BMD strategy with existing methods in Figure 2. The inter-class balanced prototype and the intra-class multicentric prototype schemes are presented in Figure 3 and Figure 4, respectively.

In the inter-class balanced prototype scheme, we select M samples with the top-M scores for each category of ground objects as the potential represented samples along both domains $X_{s^{+}}$ and $X_{tu}$ for the k-th class and use spherical k-means for clustering, where $k\in [1,2,\dots ,k],K=n_c$. Since the top instances are most likely to be positive with k-th class, we can use the class-balanced feature prototypes ${\mu }_k^s$ and ${\mu }_k^t$.

In the intra-class multicentric prototype scheme, we assume the index of sampled top-M instances for k-th class in the source domain as $M_k^s$. The predefined number of multiple feature prototype is R. We represent each instance $x^{s^{+}}$ as the extracted feature $F\left(x^{s^{+}}\right)$ and finally employ the classical k-means algorithm to attain intra-class clustering. Therefore, we can iterate this process to obtain more stable and representative source prototypes as follows, which is similar to the target domain.

$$M_k^s=\underset{\begin{array}{c}{x}^s\in X_{S^{+}}\\ \left|M_k^s\right|=M\end{array}}{argmax}{\displaystyle \frac{{max}_{1\le r\le R}\left(exp\left(F\left(x^{s^{+}}\right)\times {\mu }_{k,r}^s\right)\right)}{{\sum }_{k=1}^K{max}_{1\le r\le R}\left(exp\left(F\left(x^{s^{+}}\right)\times {\mu }_{k,r}^s\right)\right)}}$$

(3)

$${\left\{{\mu }_{k,r}^s\right\}}_{r=1}^R=\underset{i\in M_k^s}{Kmeans}\left(F\left(x_i^{s^{+}}\right)\right)$$

(4)

where $M=max\left\{1,⌊{\displaystyle \frac{n_s+n_{su}}{\gamma \times K}}⌋\right\}$, $\gamma$ is a hyperparameter denoting the top-M selection ratio.

It can be inferred from above that k-means clustering is implemented on $V^s$ and $V^t$ to obtain source clusters $C^s={\left\{C_{k,r}^{\left(s\right)}\right\}}_{k\in [1,K],r\in [1,R]}$ and similarly target clusters $C^t$ with normalized source prototypes ${\left\{{\mu }_{k,r}^s\right\}}_{k\in [1,K],r\in [1,R]}$ and normalized target prototypes ${\left\{{\mu }_{k,r}^t\right\}}_{k\in [1,K],r\in [1,R]}$. Concretely, ${\mu }_{k,r}^s={\displaystyle \frac{{\mathbf{u}}_{k,r}^s}{∥{\mathbf{u}}_{k,r}^s∥}}$, where ${\mathbf{u}}_{k,r}^s={\displaystyle \frac{1}{\left|C_{k,r}^{\left(s\right)}\right|}}{\sum }_{v_i^s\in C_{k,r}^{\left(s\right)}}{v}_i^s$. We elaborate only on the source domain for succinct notation; all manipulations are conducted on the target domain equally.

We compute a feature vector $f_i^s=F\left(x_i^s\right)$ with the feature generator F during the training process. To execute in-domain prototypical contrastive learning, the similarity distribution vectors are calculated between $f_i^s$ and ${\left\{{\mu }_{k,r}^s\right\}}_{k\in [1,K],r\in [1,R]}$ as $P_i^s=[P_{i,1}^s,P_{i,2}^s,\dots ,P_{i,K}^s]$, with

$$P_{i,k}^s={\displaystyle \frac{{\sum }_{r=1}^{R}exp\left({\mu }_{k,r}^s\times f_i^s/\varphi \right)}{{\sum }_{k=1}^K{\sum }_{r=1}^{R}exp\left({\mu }_{k,r}^s\times f_i^s/\varphi \right)}}.$$

(5)

wherein $\varphi$ is a temperature value determining the degree of concentration. Then the in-domain prototypical contrastive loss can be calculated as:

$$L_{PC}=\sum _{i=1}^{n_s+n_{su}}{\mathrm{L}}_{CE}\left(P_i^s,c_s\left(i\right)\right)+\sum _{i=1}^{n_{tu}}{\mathrm{L}}_{CE}\left(P_i^t,c_t\left(i\right)\right),$$

(6)

where $c_s(·)$ and $c_t(·)$ indicate the cluster index of the sample.

In addition, we design a Siamese-style distance metric (SDM) loss function $L_{Metric}$ to aggregate intra-class features more effectively while increasing the discrepancy between inter-class features, ultimately mitigating the influence of false negatives. Concretely, for source domain, we randomly draw two feature embeddings, denoted as $\left\{(f_k^s,{\widehat{f}}_k^s\right\}$ and $\left\{(f_{k^{\prime }}^s,{\widehat{f}}_{k^{\prime }}^s\right\}$, that belong to prototype as positive and negative sample pairs, respectively.

Then the overall $L_{Metric}$ can be computed as

$${\mathrm{L}}_{\mathrm{Metric}}^s=-\sum _{\begin{array}{c}1\le k,k^{\prime }\le K\\ k\ne k^{\prime }\end{array}}\left({\mathrm{L}}_{CE}\left(dis\left(f_k^s,{\widehat{f}}_k^s\right),1\right)+{\mathrm{L}}_{CE}\left(dis\left(f_k^s,{\widehat{f}}_{k^{\prime }}^s\right),0\right)\right),$$

(7)

$${\mathrm{L}}_{\mathrm{Metric}}^t=-\sum _{\begin{array}{c}1\le k,k^{\prime }\le K\\ k\ne k^{\prime }\end{array}}\left({\mathrm{L}}_{CE}\left(dis\left(f_k^t,{\widehat{f}}_k^t\right),1\right)+{\mathrm{L}}_{CE}\left(dis\left(f_k^t,{\widehat{f}}_{k^{\prime }}^t\right),0\right)\right),$$

(8)

$${\mathrm{L}}_{\mathrm{Metric}}={\mathrm{L}}_{\mathrm{Metric}}^s+{\mathrm{L}}_{\mathrm{Metric}}^t,$$

(9)

where $dis(·)=0.5+0.5\ast cos〈f,\widehat{f}〉$, and $cos〈·〉$ represents the cosine distance.

Finally, we implement k-means on the instances A times with the number of clustered categories K and multiple feature prototype R. Thus, the ultimate loss for in-domain prototypical contrastive self-supervised learning is written as

$$L_{InDomain}={\displaystyle \frac{1}{A}}\sum _{a=1}^A{L}_{PC}^a+L_{Metric}^a.$$

(10)

3.4. Bi-Directional Cross-Domain Prototypical Contrastive Learning

We obtain the category-discriminative semantic features in each domain by the above operations. To explicitly enforce domain matching between source and target domains, we conduct bi-directional cross-domain prototypical self-supervised learning in an instance-prototype manner. For the purpose of discovering a match for an instance $x_i$, we perform entropy minimization on the similarity distribution vector between its representation and the centroids of the other domain.

Concretely, given clustering centroids ${\left\{{\mu }_{k,r}^t\right\}}_{k\in [1,K],r\in [1,R]}$ in the target domain and feature vector $f_i^s$ in the source domain, we initially calculate the similarity distribution vector $P_i^{s\to t}=[P_{i,1}^{s\to t},P_{i,2}^{s\to t},\dots ,P_{i,K}^{s\to t}]$, in which

$$P_{i,k}^{s\to t}={\displaystyle \frac{{\sum }_{r=1}^{R}exp({\mu }_{k,r}^t\times f_i^s/\tau )}{{\sum }_{k=1}^K{\sum }_{r=1}^{R}exp({\mu }_{k,r}^t\times f_i^s/\tau )}}.$$

(11)

Subsequently, we minimize the entropy of $P_i^{s\to t}$ as:

$$H\left(P_i^{s\to t}\right)=-\sum _{k=1}^K{P}_{i,k}^{s\to t}log{P}_{i,k}^{s\to t}.$$

(12)

Correspondingly, we can calculate $H\left(P_i^{t\to s}\right)$, and the ultimate loss for cross-domain prototypical contrastive learning is:

$$L_{CrossDomain}=\sum _{i=1}^{n_s+n_{su}}H\left(P_i^{s\to t}\right)+\sum _{i=1}^{n_{tu}}H\left(P_i^{t\to s}\right).$$

(13)

3.5. Adaptive Prototypical Classifier Learning

The objective of this section is to learn an even better domain-aligned, discriminative feature encoder F and, more significantly, a cosine classifier C that could achieve high classification accuracy on the target data.

The cosine classifier C consists of weight vectors $W=[w_1,w_2,\dots ,w_K]$, where $K=n_c$ denotes the total number of categories. The output of C, ${\displaystyle \frac{1}{T}}{W}^{T}f$ is input to a softmax layer $\sigma$ to acquire the final probabilistic output $p\left(x\right)=\sigma \left({\displaystyle \frac{1}{T}}{W}^{T}f\right)$, where T is a temperature parameter. With the availability of the labeled samples in $X_s$, we first train F and C with a standard cross-entropy loss for classification straightforwardly:

$$L_{cls}=E_{(x,y)\in X_s}{L}_{CE}(p\left(x\right),y),$$

(14)

However, in the FUDA setting, due to the few-shot labeled samples in $X_s$, it is almost impossible to obtain C with excellent classification performance on the target data only by leveraging $L_{cls}$ for training.

Thus, we employ adaptive prototype classifier update (APCU) to obtain C for classifying target samples accurately. The direction of updating the weight vector $w_k$ needs to be representative of features belonging to the corresponding class k. This indicates that the semantics of $w_k$ coincide with the ideal cluster prototype of class k. We employ an estimate of the ideal cluster prototypes to update W. Yet the computed $\left\{{\mu }_{k,r}^s\right\}$ and $\left\{{\mu }_{k,r}^t\right\}$ cannot be naively adopted for this purpose, not only because the correspondence between $\left\{w_k\right\}$ and $\left\{{\mu }_{k,r}\right\}$ is unclear, but also because the k-means result may contain very impure clusters, resulting in non-representative prototypes.

Therefore, we utilize the few-shot labeled data as well as samples with high-confidence predictions to estimate the prototype for each category. Formally, we define $X_s^{\left(k\right)}=\left\{x|(x,y)\in X_s,y=k,k\in \left[1,2,\dots ,K\right]\right\}$ and denote the set of unlabeled samples with high-confidence label k as $X_{su}^k$ and $X_{tu}^k$. With the assistance of $p\left(x\right)=[p{\left(x\right)}_1,p{\left(x\right)}_2,\dots ,p{\left(x\right)}_K],X_{su}^k=\left\{x|x\in X_{su},p{\left(x\right)}_k>\xi \right\}$, which is similar for $X_{tu}^k$, where $\xi$ is a confidence threshold. Thus, the estimate of $w_k$ from the source and target domains can be calculated as

$${\widehat{w}}_k^s={\displaystyle \frac{1}{\left|X_{s^{+}}^{\left(k\right)}\right|}}\sum _{x\in X_{s^{+}}^{\left(k\right)}}{V}^s\left(x\right);{\widehat{w}}_k^t={\displaystyle \frac{1}{\left|X_{tu}^{\left(k\right)}\right|}}\sum _{x\in X_{tu}^{\left(k\right)}}{V}^t\left(x\right),$$

(15)

where $X_{s^{+}}^k=X_s^{\left(k\right)}\cup X_{su}^{\left(k\right)}$, and $V_{\left(x\right)}$ is the embedding in memory bank corresponding to x.

Subsequently, we employ a domain adaptive scheme to update $w_k$, with ${\widehat{w}}_k^s$ in the early training stage and with ${\widehat{w}}_k^t$ during the later stage. Since ${\widehat{w}}_k^s$ is more robust in early training stage with the support of few labeled source samples, while ${\widehat{w}}_k^t$ would be more representative later for target domain to achieve better cross-domain adaptation performance. Specifically, we utilize $\left|X_{tu}^{\left(k\right)}\right|$ to decide whether ${\widehat{w}}_k^t$ is robust to adopt:

$$w_k=\left\{\begin{array}{cc}unit\left({\widehat{w}}_k^s\right)\hfill & \mathrm{if}\left|X_{\mathrm{tu}}^{\left(k\right)}\right|<t_w\hfill \\ unit\left({\widehat{w}}_k^t\right)\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(16)

where $unit(·)$ represents the normalization of the input vector, and $t_w$ is a threshold hyperparameter.

In addition, in order for promoting the above joint prototype-classifier learning paradigm to work well, the output predictions of the network are desired to be adequately confident. In other words, it means obtaining robust ${\widehat{w}}_k^s$ and ${\widehat{w}}_k^t$ for $k=1,2,\dots ,K$. First, we maximize the entropy of expected network prediction $H\left(E_{x\in X}\left[p\left(y\right|x;\theta )\right]\right)$ to facilitate the network to have diversified outputs over the dataset, where $\theta$ indicates learnable parameters in both F and C, and $X=X_s\cup X_{su}\cup X_{tu}$. Second, we adopt entropy minimization on the network output to obtain a high-confidence prediction for each instance, which has shown efficacy in label-scarce scenarios [80]. The prior distribution $p_0$ is given by $E_{x\in X}\left[p\left(y\right|x;\theta )\right]$.

Thus, the above two operations are equivalent to maximizing the mutual information between input and output:

$$I(y;x)=H\left(p_0\right)-E_x\left[H\left(p\left(y\right|x;\theta )\right)\right]$$

(17)

The detailed derivation is refered to in [43]. Then the loss of mutual information maximization $L_{MIM}$ can be written as

$$L_{MIM}=-I(y;x).$$

(18)

To sum up, our proposed RBPCL framework performs class-balanced multicentric dynamic prototype generation, in-domain refined prototypical contrastive learning, bi-directional cross-domain prototypical contrastive learning, and unified adaptive prototype-classifier learning. Together with APCU in Equations (15) and (16), the overall objective of model training is

$$L_{RBPCL}=L_{cls}+{\lambda }_{in}{L}_{InDomain}+{\lambda }_{cross}{L}_{CrossDomain}+{\lambda }_{mim}{L}_{MIM}.$$

(19)

In order to provide a concise yet comprehensive description, the training procedure of RBPCL, derived from the preceding elaboration, is formally outlined in Algorithm 1. Subsequently, the classification performance of the proposed framework is evaluated on target datasets using the trained feature generator F and the cosine similarity-based classifier C.

Algorithm 1: Training Procedure for RBPCL

4. Experiments

This section presents an empirical evaluation of RBPCL on five representative hyperspectral dataset pairs, each reflecting domain shifts induced by differences in geographic locations and acquisition dates. Specifically, two pairs are widely adopted benchmark datasets, whereas the others were acquired over two Chinese cities using a UAV-based hyperspectral sensor system under diverse lighting environments.

4.1. Datasets

4.1.1. Pavia Data Pair

The Pavia dataset pair comprises two distinct scenes: Pavia University (PU) and Pavia Center (PC). Both datasets were acquired by the ROSIS sensor during an airborne mission over the city of Pavia in northern Italy. After excluding several noisy spectral bands, the remaining data offer high-resolution spectral measurements spanning the wavelength interval from 0.43 to 0.86. The imagery has a spatial resolution of 1.3 m. PU initially contains 103 spectral bands, and PC contains 102 bands; to ensure consistency across the pair, the final band of PU is discarded. It is important to note that the two scenes do not share an identical set of land cover categories. Therefore, we select seven common classes for our experimental evaluations. As discussed in Yang et al. [81], classifying these images is highly challenging due to substantial intra-class variation and limited inter-class distinction. The number of labeled samples in each selected class is provided in

Table 1. Number of samples in each land cover category within Pavia data pair.

Class		Number of Samples
ID	Name	PU	PC
1	Trees	3064	7598
2	Asphalt	6631	9248
3	Bricks	3682	2685
4	Bitumen	1330	7287
5	Shadows	947	2863
6	Meadows	18,649	3090
7	Bare soil	5029	6584
Total		39,332	39,355

. The false-color composites for PU and PC are shown in Figure 5a and Figure 5b, respectively. Their corresponding ground-truth maps, containing the seven shared categories, are illustrated in Figure 5c,d, where the background regions are marked in black.

4.1.2. Houston Data Pair

The Houston dataset pair was provided by the NSF-funded National Center for Airborne Laser Mapping (NCALM), covering both the University of Houston campus and adjacent urban areas. It comprises two temporally distinct hyperspectral images, acquired in 2012 and 2017, respectively, using the ITRES-CASI 1500 hyperspectral sensor, DATA SURPASS TECHNOLOGY CO., LTD., Taichung City, Taiwan. These images were featured in the IEEE GRSS Data Fusion Contests of 2013 and 2018. The spatial resolutions differ between the two datasets: 2.5 m for Houston 2013 (HO2013) and 1 m for Houston 2018 (HO2018). Specifically, HO2013 consists of $349\times 1905$ pixels with 144 spectral bands, while HO2018 includes $601\times 2384$ pixels and 48 bands. Both datasets span a spectral range of 380–1050 nm. To align the spectral dimensionality between the two datasets, every three adjacent bands in HO2013 were averaged into one, resulting in a 48-band structure consistent with HO2018 [82]. For the cross-scene classification task, seven shared classes were selected. Prior studies by Liu et al. [21,24] have discussed spectral amplitude shifts that challenge domain adaptation on this dataset. The number of labeled samples per class is summarized in

Table 2. Number of samples in each land cover category within Houston data pair.

Class		Number of Samples
ID	Name	HO2013	HO2018
1	Healthy grass	1251	9799
2	Stressed grass	1254	32,502
3	Trees	1244	13,588
4	Water	325	266
5	Residential buildings	1268	39,762
6	Commercial buildings	1244	223,684
7	Road	1252	45,810
Total		7838	365,411

. Figure 6a,b displays the false-color representations of HO2013 and HO2018, while the corresponding ground-truth maps with the selected seven categories are shown in Figure 6c and Figure 6d, respectively.

4.1.3. Self-Collected Data Pairs

To evaluate the real-world applicability of the proposed RBPCL method under ultra-low-altitude flight conditions, we conducted aerial hyperspectral imaging using a Hexacopter UAV equipped with a SPECIM_FX17E hyperspectral camera, SPECIM Company, Oulu, Finland. Atmospheric correction was applied to the captured data. A total of three hyperspectral images were collected and organized into three distinct data pairs, each demonstrating varying levels of spectral shift. One of these images, referred to as CSSunny, was acquired in Changsha at 4:00 PM on 27 September 2021, under sunny conditions with a flight altitude of 30 m. This scene includes six categories of ground objects. The remaining two images were captured in Tianjin, each containing twelve object classes, and flown at an altitude of 60 m. The first, named TJSunny, was obtained at 3:00 PM on 21 March 2023, under sunny weather, while the second, designated TJCloudy, was recorded at 3:30 PM on 22 March 2023, in cloudy conditions. All three images share the same spatial resolution of $482\times 640$ pixels and cover the spectral range from 935 nm to 1720 nm with 224 spectral bands. As shown in Figure 7a–c, the false-color visualizations of these self-collected datasets are provided, while the corresponding ground-truth maps with eight common classes are illustrated in Figure 7d–f. Then we combine three self-collected images into three data pairs, namely TJSunny-TJCloudy, TJSunny-CSSunny, and TJCloudy-CSSunny. In particular, the categories of the HSI images in the TJSunny-CSSunny and TJCloudy-CSSunny scenes are not exactly the same, and we select six classes shared for experiments. The sample counts for each class in the three self-collected dataset pairs are detailed in

Table 3. Number of samples in each land cover category within TJSunny-TJCloudy data pair.

Class		Number of Samples
ID	Name	TJSunny	TJCloudy
1	Sand	5885	7052
2	Red plastic track	1335	1042
3	Green fake turf	4711	3687
4	White cloth	285	237
5	Gray cloth	274	230
6	Green bushes	1605	2340
7	Red bushes	1828	2019
8	Asphalt pavement	6252	3950
9	Silver-colored metal box	42	35
10	Grey floor tiles	2622	3683
11	Red floor tiles	6575	5392
12	Metal manhole cover	185	90
Total		31,599	29,757

and

Table 4. Number of samples in each land cover category within TJSunny-CSSunny and TJCloudy-CSSunny data pairs.

Class		Number of Samples
ID	Name	TJSunny	TJCloudy	CSSunny
1	Sand	5885	7052	6758
2	Red plastic track	1335	1042	3478
3	White cloth	300	271	1202
4	Gray cloth	184	239	1369
5	Asphalt pavement	6216	3182	2940
6	Silver-colored metal box	51	41	101
Total		13,971	11,827	15,848

. As illustrated in Figure 8, despite variations in illumination caused by differences in date and location, the spectral profiles of individual materials remain largely consistent in shape, while their amplitudes exhibit notable fluctuations. This suggests that different materials respond to changes in light intensity with varying degrees of sensitivity. It is worth noting that the light intensity of CSSunny is significantly stronger than that of TJSunny, both of which are collected in sunny weather. Specifically, subtle observation of Figure 7 can reveal differences in the spectral curve shapes between hyperspectral images collected on sunny and cloudy days within certain wavelength ranges.

4.2. Implementation Details

In our experiments, the proposed method was evaluated alongside seven baseline techniques. For each data pair, both images were alternately treated as the source domain to ensure comprehensive cross-domain comparison. The training phase of all experiments is conducted on three-shot source labels per class in all data pairs with the assistance of unlabeled source and target samples. Subsequently, the classification accuracy of all methods was assessed on the corresponding target domains. For input construction, we extracted HSI patches of size $9\times 9$, with the label assigned according to the class of the pixel located at the geometric center of each patch.

To facilitate a comprehensive comparison among intra-domain classification, unsupervised domain adaptation (UDA), and few-shot unsupervised domain adaptation (FUDA) approaches, we implemented a baseline using the DBDA model [79], which is trained solely on the source domain and directly applied to the target data without any adaptation. This baseline, referred to as “S-only”, serves as a non-transfer setting. The performance difference between “S-only” and other domain adaptation strategies reflects the relative effectiveness of each method in transferring knowledge across domains. In addition, four representative UDA techniques (DANN [52], MDDIA [60], DABAN [25], and SILDA-VC [26]) and four FUDA methods (CDS [42], PCS [43], PCS+BMD, and PCS+SDM) are included to assess the capability of our proposed RBPCL framework in addressing cross-domain hyperspectral image classification with limited labeled source data. These comparisons also highlight the advantages of integrating both the BMD clustering strategy and SDM loss within the in-domain and inter-domain prototypical contrastive learning process. It is worth mentioning that, aside from SILDA-VC—which maintains a graph convolutional backbone—the feature generator architecture F is kept consistent across all evaluated methods.

All experiments were conducted using the PyTorch 2.2.0 framework and Python 3.10 on a system equipped with an NVIDIA TITAN RTX GPU (24 GB RAM), NVIDIA, Santa Clara, CA, USA. To accelerate clustering operations, we employed the GPU-based implementation of k-means provided by FAISS [83]. The network was trained for 200 epochs. During self-supervised learning, both source and target domains used a mini-batch size of 32, while a batch size of 16 was applied for the classification loss. The optimization process was carried out using stochastic gradient descent (SGD) with a learning rate of 0.01 and a momentum parameter of 0.9. The learning rate ratio between the linear layer and the convolution layer is set to 1: 0.1. We use SGD with a weight decay rate of $5e^{-4}$. We adaptively set the temperature according to [66]. The temperature $\tau$ is set as 0.1 in all experiments. We choose hyper-parameters ${\lambda }_{in}=1$ and ${\lambda }_{cross}=0.5$, and the weight ${\lambda }_{mim}=0.05$. As for parameters m (momentum for memory bank update) and A (number of k-means in $L_{InDomain}$ ), we set $m=0.5$ and $A=20$. We calculate the weights for the cosine classifier only utilizing source data for the first five epochs and set $t_w$ to be around half of the average number of instances per class. Besides, for the k-means cluster, we fix the hyperparameters $\gamma$ and T to 3 and 0.05, respectively, for all data pairs. And we set $R=4$ for Pavia data pair and the Houston data pair and $R=2$ for self-collected data pairs. It is worth noting that centroids of clusters and weights of cosine classifiers are updated per epoch for both self-supervised learning and classification.

To comprehensively assess the performance of the compared methods, we employ three objective evaluation metrics: overall accuracy (OA), average accuracy (AA), and the statistical Kappa coefficient. The OA refers to the number of correctly classified samples divided by the total number of samples. AA refers to the average of the classification accuracy for each category. Kappa coefficients are adopted in consistency tests to measure whether the model predictions are consistent with the actual classification results. All classification results reported in this article are averaged over ten random conductances.

4.3. Results and Ablation Analysis

For the Pavia data pair, we randomly select three labeled samples from the PU dataset to form the training set together with unlabeled samples from the PU and PC datasets, then perform unsupervised transfer and classification tests on the PC scenario. The classification results are given in

Table 5. Cross-scene classification results with different methods on three shots per class of PU → PC.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	49.9	36.26	69.96	42.36	40.14	41.48	42.92	58.72	63.84	60.76
2	60.99	53.92	63.04	40.94	25.68	31.18	46.02	57.13	32.09	60.51
3	70.22	65.67	100	99.85	98.84	99.25	79.69	73.76	81.14	87.07
4	27.19	72.18	13.31	36.47	86.27	81.47	83.97	79.22	80.3	75.03
5	32.12	39.92	33.69	82.84	63.68	51.87	76.65	79.24	97.59	84.73
6	59	66.55	62.76	51.39	56.22	82.16	60.52	54.08	70.89	72.7
7	49.09	43.06	50.9	58.98	69.99	68.85	76.89	71.3	81.22	81.22
AA	49.79	53.84	56.24	58.98	62.98	65.18	66.67	67.63	72.44	74.57
OA	48.97	52.8	53.5	51.29	57.26	58.93	63.27	66.4	66.52	71.24
Kappa	39.15	44.24	44.98	43.08	50.08	52.02	57.01	60.18	61.3	65.96

. Correspondingly, the second experiment swaps the domain identities of PU and PC images, and the unsupervised classification results of the PU scenario are shown in

Table 6. Cross-scene classification results with different methods on three shots per class OF PC → PU.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	41.05	61.43	76.52	19.2	62.41	54.93	62.41	64.4	80.37	62.8
2	33.31	45.59	24.75	44.38	34.26	39.43	24.39	20.24	49.72	62.54
3	88.23	28.53	82.36	86.11	11.49	77.23	84.24	90.57	71.88	72.77
4	91.87	71.16	99.7	88.7	90.44	98.42	99.85	99.17	95.41	94.5
5	19.26	62.43	14.39	35.98	62.75	43.17	80.95	98.2	96.4	87.09
6	15.47	35.17	35.36	44.73	56.65	47.53	60.06	67.39	67.24	76.85
7	99.36	88.78	98.15	99.82	97.51	97.37	84.23	84.7	57.63	58.62
AA	55.51	56.16	61.6	59.85	59.36	65.44	70.88	74.95	74.09	73.6
OA	40.68	47.08	50.87	54.87	55.61	57.51	61.43	65.41	66.17	71.48
Kappa	30.96	37.18	40.64	43.77	45.05	47.35	50.87	55.81	56	61.35

.

Moreover, the comparison of experimental results on the Houston data pair and self-collected data pairs is displayed in

Table 7. Cross-scene classification results with different methods on three shots per class of HO2013 → HO2018.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	33.85	96.66	58.82	95.52	43.3	83.08	52.39	81.88	94.69	68.42
2	57.67	5.38	52.94	53.69	82.3	40.95	48.95	48.27	54.74	85.53
3	51.67	58.94	68.67	96.36	89.73	52.97	96.73	84.65	84.78	95.1
4	68.56	99.62	68.18	85.61	68.56	89.39	68.56	93.94	67.05	100
5	64.7	91.53	88.86	86.33	12.56	89.26	76.46	40.04	79.27	60.54
6	31.06	40.81	38.79	28.47	69.03	67.18	54.45	70.9	65.86	57.99
7	24.27	4.77	25.53	57.89	28.04	10.59	58.03	43.41	29.28	71.46
AA	47.4	56.82	57.4	71.98	56.22	61.92	65.08	66.15	67.95	77.01
OA	37.1	40.87	45.51	45.06	59	60.07	58.33	62.9	63.22	64.09
Kappa	18.75	22.93	29.8	32.26	34.39	40.47	43.05	42.69	45.47	49.27

,

Table 8. Cross-scene classification results with different methods on three shots per class of HO2018 → HO2013.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	34.03	53	27.7	31.95	73.18	77.02	70.14	84.63	79.18	78.38
2	58.71	84.9	82.67	80.11	53.83	79.87	59.66	47.68	67.49	72.52
3	32.53	31.72	67.71	74.24	91.22	33.09	72.06	82.29	75.52	77.05
4	11.76	80.19	87.31	68.42	82.35	87.31	85.14	77.71	93.19	70.28
5	63.19	53.63	63.11	59.4	45.1	24.25	72.35	82.78	56.56	82.78
6	31.56	25.52	21.74	14.98	12.96	51.21	41.3	19.81	42.35	48.39
7	41.6	64.16	56.8	64.64	52.64	91.44	72.72	86.16	87.36	88.56
AA	39.05	56.16	58.15	56.25	58.76	63.46	67.63	68.72	71.66	74
OA	42.36	53.37	54.74	54.84	55.94	60.6	65.58	67.71	69.11	74.48
Kappa	32.22	44.93	46.73	46.55	48.23	53.36	59.51	61.78	63.57	69.79

,

Table 9. Cross-scene classification results with different methods on three shots per class of TJSunny → TJCloudy.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	45.09	83	41.55	31.83	64.23	24.31	24.38	60.52	73.92	78.94
2	36.22	71.66	20.65	35.45	81.27	43.61	72.53	94.81	42.65	56.77
3	8.46	14.68	31.71	38.44	56.54	39.91	38.47	42.3	32.09	12.75
4	33.47	80.51	40.68	59.32	19.07	63.56	58.47	100	44.49	80.51
5	69	57.64	69	92.58	14.41	72.05	79.04	99.56	98.25	96.94
6	17.27	42.92	16.97	55.58	81.83	44.25	76.14	56.31	28.77	91.58
7	43.46	1.29	83.45	49.04	6	66.95	43.61	11.84	65.51	48.12
8	14.23	20.13	56.62	56.17	41.2	42.72	99.57	88.33	88.25	96.33
9	97.06	100	100	100	88.24	100	100	82.35	100	79.41
10	12.63	15.83	14.48	53.39	12.76	81.88	36.2	1.96	28.95	42.07
11	95.05	59.62	94.05	79.93	84.92	99.05	95.2	90.73	100	86.68
12	60.67	40.45	40.45	40.45	40.45	40.45	40.45	58.43	40.45	39.33
AA	44.38	48.98	50.8	57.68	49.24	59.9	63.67	65.59	61.95	67.45
OA	39.08	44.22	48.94	51.22	54.78	55.31	58.29	58.38	64.47	68.05
Kappa	32.29	38.58	42.79	47.3	50.41	49.74	53.39	53.05	59.31	63.23

,

Table 10. Cross-scene classification results with different methods on three shots per class of TJCloudy → TJSunny.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	36.22	63.9	80.13	100	100	94.12	96.82	68.59	99.35	99.63
2	60.19	22.36	76.24	85.69	32.13	34.93	54.72	53.52	64.32	89.06
3	37.26	7.03	29.58	44.6	28.4	15.05	37.45	54.12	13.99	97.35
4	32.39	86.57	91.9	45.17	57.28	52.11	97.18	70.42	100	100
5	51.28	97.43	49.45	12.77	75.54	99.63	98.53	100	99.27	100
6	48.32	53.03	85.85	49.72	42.87	97.07	68.45	92.52	100	98
7	20.09	22.29	60.76	39.5	92.45	47.45	65.96	59.77	74.44	98.41
8	100	76.86	77.24	93.02	92.05	94.9	84.73	97.36	100	15.17
9	100	100	100	92.34	87.49	100	100	100	100	58.54
10	39.03	66.11	36.05	69.18	45.96	94.01	22.24	57.38	95.99	100
11	36.71	62.59	22.36	9.75	18.31	25.37	45.5	53.73	28.84	46.88
12	44.02	46.45	72.28	36.22	61.62	21.74	34.24	59.78	44.02	83.15
AA	50.46	58.72	65.15	56.59	61.18	64.7	67.15	72.27	76.69	82.18
OA	50.25	53.62	55.17	60.73	59.23	62.37	63.36	68.45	68.59	70.88
Kappa	44.32	47.49	49.77	55.36	53.93	57.17	58.3	63.88	64.1	66.78

,

Table 11. Cross-scene classification results (%) with different methods on three shots per class of TJSunny → CSSunny.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	38.19	68.52	34.73	46.7	54.88	65.58	64.13	63.83	77.58	64.72
2	15.22	52.85	99.94	30.88	24.14	16.57	29.93	31.11	16.34	62.29
3	71.57	47.49	24.81	46.15	34.45	66.22	77.18	92.31	50.84	86.96
4	96.72	96.72	79.9	100	92.9	96.72	96.7	91.26	99.45	100
5	85.15	60.76	100	97.01	91.87	99.37	99.4	99.24	99.2	99.77
6	100	100	100	100	100	100	100	100	100	100
AA	67.81	71.06	73.23	70.12	66.37	74.08	77.89	79.63	73.9	85.62
OA	58.6	63.6	64.71	68.46	68.63	76.48	77.4	77.57	81.15	81.15
Kappa	42.26	48.48	55.59	55.94	56.08	64.9	66.54	66.34	70.7	71.64

,

Table 12. Cross-scene classification results (%) with different methods on three shots per class of CSSunny → TJSunny.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	34.02	55.59	53.88	55.59	65.48	64.07	60.33	48.98	67.64	70.73
2	53.67	79.46	47.71	79.46	7.12	38.26	61.14	57.05	35.61	43.4
3	98.66	53.85	95.97	53.85	64.88	71.48	52.35	86.29	56.86	50.17
4	46.45	96.72	91.76	96.72	93.99	100	95.05	76.5	80.87	93.44
5	42.7	47.29	66.17	47.28	63.57	74.72	79.11	93.39	89.59	98.1
6	100	100	100	100	100	100	100	100	100	100
AA	62.59	72.15	75.92	72.15	65.84	74.75	74.66	77.03	71.76	76.47
OA	41.55	54.84	60.32	54.84	59.54	67.1	69.19	70.86	74.41	80.55
Kappa	28.28	43.29	43.36	43.29	43.72	52.91	56.67	58.62	62.6	70.62

,

Table 13. Cross-scene classification results (%) with different methods on three shots per class of TJCloudy → CSSunny.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	29.75	30.61	11.85	32.59	51.61	26.96	28.9	58.34	68.4	62.82
2	40.55	99.97	100	40.12	100	100	100	100	100	47.74
3	34.97	2.16	34.8	97.75	21.57	37.3	48.21	22.4	42.8	69.28
4	49.71	12.79	37.06	73.68	9.43	33.85	48.61	33.11	62.72	92.69
5	98.43	79.72	100	98.74	67.95	100	100	92.58	47.12	100
6	41	96	86	97	85	100	97	82	94	97
AA	49.07	53.54	61.62	73.31	55.92	66.35	70.45	64.74	69.17	78.26
OA	47.05	51.66	51.94	55.41	59.55	58.38	61.29	69.08	69.12	69.69
Kappa	35.53	39.86	43.06	46.07	47.6	49.63	52.99	58.92	59.47	60.65

and

Table 14. Cross-scene classification results (%) with different methods on three shots per class of CSSunny → TJCloudy.

Class	S-Only	DANN	MDDIA	DABAN	SILDA-VC	CDS	PCS	PCS+SDM	PCS+BMD	RBPCL
1	16.3	44.94	18.35	33.35	32.76	54.91	62.76	73.22	50.28	71.85
2	24.02	31.03	13.26	30	22.83	28.27	52.83	58.89	70.96	87.7
3	25.93	31.85	37.04	69.3	34.44	76.58	44.44	99.26	65.43	77.78
4	100	58.82	94.12	100	100	92.83	100	99.58	32.07	76.47
5	49.8	35.33	91.29	58.12	76.18	39.25	40.18	25.5	90.47	37.69
6	100	100	100	67.5	100	100	100	80.49	100	100
AA	52.67	50.33	59.01	59.71	61.03	65.3	66.7	72.82	68.2	75.25
OA	28.18	41.3	39.76	41.85	45.19	49.75	56.26	60.27	63.06	64.38
Kappa	15.94	20.38	27.99	30.46	33.64	32.06	40.15	43.08	48.23	49.04

. It can be seen that our RBPCL method almost provides optimal results on three objective indicators in all experimental scenarios. This indicates the superiority of the proposed method in dealing with the FUDA problem for HSICC tasks. Furthermore, the classification performance of the three FUDA methods (e.g., RBPCL, PCS, and CDS) is significantly better than the UDA methods. This suggests that it is essential to research FUDA methods for HSICC tasks. Furthermore, it is worth noting that the PCS+SDM method, PCS+BMD method, and our RBPCL method achieve better classification results than PCS, which shows that the SDM loss and BMD strategy play a positive role in the process of prototype self-supervised learning and adaptive prototypical classifier learning. The classification performance of PCS+BMD overall outperforms PCS+SDM, which validates that the BMD strategy is more advantageous since it generates more robust and representative prototypes during the clustering process and then promotes intra-domain and cross-domain prototype contrastive learning.

To present visualized classification results and enhance the effectiveness of our proposed method, we presented labeled pixel classification maps displaying the Pavia University → Pavia Center in Figure 9. Moreover, we presented all-pixel classification maps of the TJSunny → CSSunny scenes in Figure 10. Since the distribution of ground object targets in PC hyperspectral images is relatively scattered and lacks strong spatial continuity, it is more distinguishable to display the classification results, excluding background pixels. Observing these classification maps in Figure 9 and Figure 10, we can conclude, consistent with the experimental results displayed in Table 10, Table 11, Table 12, Table 13 and Table 14. Except for individual categories, our proposed method has the best overall classification performance.

Besides, in order to demonstrate the advantages of the proposed RBPCL method more visually, the t-SNE visualizations of three FUDA-based methods (CDS, PCS, and RBPCL) are presented. Among them, different colors represent different categories of target objects. The t-SNE scatter of cross-scene learned features plotted on the Pavia data pair is presented in Figure 11, which contains our RBPCL method and baselines CDS and PCD. And the t-SNE visualizations of cross-scene learned features in Houston data pairs and self-collected data pairs are displayed in Figure 12, Figure 13, Figure 14 and Figure 15. It can be observed from Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15 that the feature distribution of our RBPCL method is significantly better than that of the other two algorithms, CDS and PCS, which means the proposed RBPCL method possesses the ability to learn better discriminative semantic features compared with the CDS and PCS methods. Especially, t-SNE visualization figures qualitatively reveal that RBPCL clusters instances with the same category in the feature space well, and the features learned from RBPCL are more closely aggregated than CDS and PCS. Therefore, it demonstrates that the refined prototypical contrastive self-supervised training process enhances the learning capability of our framework to some extent.

In particular, we constructed three data pairs based on self-collected datasets, representing different degrees of spectral shift. The TJCloudy-CSSunny data pair exhibits the most obvious spectral shift phenomenon due to the large span of time and space, as well as the large difference in illumination conditions, which can be confirmed by Figure 8. Moreover, our proposed method can stand out in all data pairs with varying degrees of spectral shift, which once again verifies its superiority and stability.

4.4. Impact of Different Parameter Settings

Our experiments involve some important hyperparameter settings, which directly affect the experimental results. Based on previous works [43,47], we preliminarily determined parameter schemes and adjusted them according to the characteristics of different hyperspectral datasets in our experiments and ultimately obtained the optimal experimental parameters through parameter analysis experiments. In this section, we conducted parameter analysis experiments on the number of prototypes R in the multi-prototype strategy and the weight parameters (${\lambda }_{in}$, ${\lambda }_{cross}$, and ${\lambda }_{mim}$) of the loss function. Firstly, we set R from 1 to 6 for comparative experiments, and classification results on Pavia and TJSunny-CSSunny data pairs are shown in Figure 16, Figure 17, Figure 18 and Figure 19. We can observe that the cross-domain classification performance of the RBPCL method shows a trend of first increasing and then decreasing with the increase in the multi-feature prototype R. The optimal R value on the Pavia data pair is 4, and the optimal R value on the TJSunny-CSSunny data pair is 2. This indicates that the more prototypes, the worse the classification performance will be. Reasonable R values should be set according to the characteristics of different datasets.

Then, we conducted analysis experiments on different combinations of loss weights (${\lambda }_{in}$, ${\lambda }_{cross}$, and ${\lambda }_{mim}$). Experimental results of Pavia and TJSunny-CSSunny data pairs are presented in Table 15, Table 16, Table 17 and Table 18. We can conclude that the optimal combination of loss weights is ${\lambda }_{in}=1$, ${\lambda }_{cross}=0.5$, and ${\lambda }_{mim}=0.05$. Moreover, we also obtained a consistent conclusion in parameter analysis experiments for the other three data pairs.

4.5. Effects of Losses

In this section, we analyze three loss modules described in Section 3. Specifically, we compare the performance of average accuracy (AA), overall accuracy (OA), and Kappa by fixing the weights of the loss modules ( $L_{InDomain}$, $L_{CrossDomain}$, and $L_{MIM}$) on experiments for all data pairs. Experimental results are shown in Table 19 and Table 20, and the setting of only $L_{cls}$ is equivalent to the S-only method in Table 5, Table 6, Table 7, Table 8, Table 9, Table 10, Table 11, Table 12, Table 13 and Table 14. As we can observe in Table 19 and Table 20, adding each component contributes to the final adaptation accuracy without any performance degradation, which demonstrates the effectiveness of all loss modules in our RBPCL framework.

4.6. Impact of the Labeled Samples in Source Domain

In this section, we conduct comparison experiments to demonstrate the effect of different numbers of labeled source samples on three FUDA algorithms. Concretely, 3, 5, 7, and 9 labeled samples per class in the source domain dataset are selected for the training; the cross-domain classification results are shown in Figure 20. It is obvious that the classification performance of the proposed RBPCL method shows an upward trend in all three indicators for each data pair as the labeled sample size increases. Overall, our method consistently exceeds the other two methods by a large margin in OA. Finally, the above experimental results demonstrate that the advantage of our proposed algorithm is its capability of adapting to circumstances with variations in the number of labeled source samples.

Table 15. Cross-scene classification results (%) OF PU → PC under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$, and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	72.39	72.01	63.74
0.5	1	0.05	71.3	68.65	62.59
1	0.5	0.05	74.57	71.24	65.96
0.5	0.5	0.05	69.93	69.22	61.81
1	1	0.01	72.61	70.25	64.15
0.5	1	0.01	70.84	68.38	62.07
1	0.5	0.01	73.45	70.69	64.28
0.5	0.5	0.01	68.73	69.47	61.46

Table 15. Cross-scene classification results (%) OF PU → PC under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$, and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	72.39	72.01	63.74
0.5	1	0.05	71.3	68.65	62.59
1	0.5	0.05	74.57	71.24	65.96
0.5	0.5	0.05	69.93	69.22	61.81
1	1	0.01	72.61	70.25	64.15
0.5	1	0.01	70.84	68.38	62.07
1	0.5	0.01	73.45	70.69	64.28
0.5	0.5	0.01	68.73	69.47	61.46

Table 16. Cross-scene classification results (%) OF PC → PU under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$ and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	72.47	70.69	60.24
0.5	1	0.05	71.22	69.38	59.77
1	0.5	0.05	73.6	71.48	61.35
0.5	0.5	0.05	70.86	68.56	59.14
1	1	0.01	71.67	70.84	59.58
0.5	1	0.01	70.75	67.92	58.76
1	0.5	0.01	72.51	68.89	60.43
0.5	0.5	0.01	70.83	67.15	59.06

Table 16. Cross-scene classification results (%) OF PC → PU under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$ and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	72.47	70.69	60.24
0.5	1	0.05	71.22	69.38	59.77
1	0.5	0.05	73.6	71.48	61.35
0.5	0.5	0.05	70.86	68.56	59.14
1	1	0.01	71.67	70.84	59.58
0.5	1	0.01	70.75	67.92	58.76
1	0.5	0.01	72.51	68.89	60.43
0.5	0.5	0.01	70.83	67.15	59.06

Table 17. Cross-scene classification results (%) OF TJSunny → CSSunny under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$ and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	81.77	80.04	69.81
0.5	1	0.05	80.13	78.66	70.28
1	0.5	0.05	85.62	81.15	71.64
0.5	0.5	0.05	79.78	77.59	70.94
1	1	0.01	80.37	79.41	69.36
0.5	1	0.01	80.21	77.64	69.13
1	0.5	0.01	82.31	81.86	70.02
0.5	0.5	0.01	79.55	76.97	68.87

Table 17. Cross-scene classification results (%) OF TJSunny → CSSunny under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$ and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	81.77	80.04	69.81
0.5	1	0.05	80.13	78.66	70.28
1	0.5	0.05	85.62	81.15	71.64
0.5	0.5	0.05	79.78	77.59	70.94
1	1	0.01	80.37	79.41	69.36
0.5	1	0.01	80.21	77.64	69.13
1	0.5	0.01	82.31	81.86	70.02
0.5	0.5	0.01	79.55	76.97	68.87

Table 18. Cross-scene classification results (%) OF CSSunny → TJSunny under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$ and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	72.61	76.33	65.24
0.5	1	0.05	71.06	76.87	66.42
1	0.5	0.05	76.47	80.55	70.62
0.5	0.5	0.05	72.18	77.35	67.53
1	1	0.01	71.36	75.47	64.95
0.5	1	0.01	70.11	76.04	65.88
1	0.5	0.01	74.39	78.76	68.13
0.5	0.5	0.01	70.99	74.82	66.75

Table 18. Cross-scene classification results (%) OF CSSunny → TJSunny under different lambda (${\lambda }_{in}$, ${\lambda }_{cross}$ and ${\lambda }_{mim}$)-value settings.

${\mathit{\lambda }}_{\mathbf{in}}$	${\mathit{\lambda }}_{\mathbf{cross}}$	${\mathit{\lambda }}_{\mathbf{mim}}$	AA	OA	Kappa
1	1	0.05	72.61	76.33	65.24
0.5	1	0.05	71.06	76.87	66.42
1	0.5	0.05	76.47	80.55	70.62
0.5	0.5	0.05	72.18	77.35	67.53
1	1	0.01	71.36	75.47	64.95
0.5	1	0.01	70.11	76.04	65.88
1	0.5	0.01	74.39	78.76	68.13
0.5	0.5	0.01	70.99	74.82	66.75

Table 19. Perfomance contribution (%) of each part in RBPCL framework (AA/OA/KAPPA).

Method	PU → PC	HO2013 → HO2018	TJSunny → TJCloudy	TJSunny → CSSunny	TJCloudy → CSSunny
$L_{cls}$	49.79/48.97/39.15	47.40/37.10/18.75	44.38/39.08/32.29	67.81/58.06/42.26	49.07/47.05/35.53
$+L_{InDomain}$	56.74/52.11/43.98	56.22/41.53/24.87	51.63/49.16/43.21	72.10/63.88/51.42	62.52/53.31/44.48
$+L_{CrossDomain}$	64.37/60.09/52.34	65.56/57.93/42.75	60.74/57.37/52.46	77.34/75.73/62.19	70.49/62.16/51.83
$+{\mathit{L}}_{\mathit{MIM}}+\mathit{APCU}$	74.57/71.24/65.96	77.01/64.09/49.27	67.45/68.05/63.23	85.62/81.15/71.64	78.26/69.69/60.65

Table 19. Perfomance contribution (%) of each part in RBPCL framework (AA/OA/KAPPA).

Method	PU → PC	HO2013 → HO2018	TJSunny → TJCloudy	TJSunny → CSSunny	TJCloudy → CSSunny
$L_{cls}$	49.79/48.97/39.15	47.40/37.10/18.75	44.38/39.08/32.29	67.81/58.06/42.26	49.07/47.05/35.53
$+L_{InDomain}$	56.74/52.11/43.98	56.22/41.53/24.87	51.63/49.16/43.21	72.10/63.88/51.42	62.52/53.31/44.48
$+L_{CrossDomain}$	64.37/60.09/52.34	65.56/57.93/42.75	60.74/57.37/52.46	77.34/75.73/62.19	70.49/62.16/51.83
$+{\mathit{L}}_{\mathit{MIM}}+\mathit{APCU}$	74.57/71.24/65.96	77.01/64.09/49.27	67.45/68.05/63.23	85.62/81.15/71.64	78.26/69.69/60.65

Table 20. Perfomance contribution (%) of each part in RBPCL framework (AA/OA/KAPPA).

Method	PC → PU	HO2018 → HO2013	TJCloudy → TJSunny	CSSunny → TJSunny	CSSunny → TJCloudy
$L_{cls}$	55.51/40.68/30.96	39.05/42.36/32.22	50.46/50.25/44.32	62.59/41.55/28.28	52.67/28.18/15.94
$+L_{InDomain}$	60.50/49.37/39.61	52.08/50.62/41.97	62.72/54.52/48.44	68.97/60.65/41.03	60.46/40.73/30.51
$+L_{CrossDomain}$	65.93/58.82/50.06	63.63/61.79/54.45	71.83/66.39/61.81	72.36/70.24/57.89	67.28/55.19/38.72
$+{\mathit{L}}_{\mathit{MIM}}+\mathit{APCU}$	73.60/71.48/61.35	74.00/74.48/69.79	82.18/70.88/66.78	76.47/80.55/70.62	75.25/64.38/49.04

Table 20. Perfomance contribution (%) of each part in RBPCL framework (AA/OA/KAPPA).

Method	PC → PU	HO2018 → HO2013	TJCloudy → TJSunny	CSSunny → TJSunny	CSSunny → TJCloudy
$L_{cls}$	55.51/40.68/30.96	39.05/42.36/32.22	50.46/50.25/44.32	62.59/41.55/28.28	52.67/28.18/15.94
$+L_{InDomain}$	60.50/49.37/39.61	52.08/50.62/41.97	62.72/54.52/48.44	68.97/60.65/41.03	60.46/40.73/30.51
$+L_{CrossDomain}$	65.93/58.82/50.06	63.63/61.79/54.45	71.83/66.39/61.81	72.36/70.24/57.89	67.28/55.19/38.72
$+{\mathit{L}}_{\mathit{MIM}}+\mathit{APCU}$	73.60/71.48/61.35	74.00/74.48/69.79	82.18/70.88/66.78	76.47/80.55/70.62	75.25/64.38/49.04

5. Discussion

In the future, considering that our method requires sufficient unlabeled source samples and unlabeled target samples to form the training set, due to restrictions on data privacy protection, we will further explore the applicability of few-shot learning to reduce dependence on unlabeled training samples in our subsequent studies.

6. Conclusions

In this paper, we explore breaking through the dilemma of hyperspectral cross-scene classification (HSICC), that is, the spectral shift phenomenon between different scenes, unlabeled target domains, and even label-scarce source domains. In response to this dilemma, we propose an end-to-end refined bi-directional prototypical contrastive learning (RBPCL) framework to learn more discriminative feature representation and achieve better domain alignment for the FUDA setting in HSICC tasks and finally enable efficient unsupervised classification in target hyperspectral images. The proposed RBPCL method comprises refined in-domain prototypical self-supervised learning, bi-directional cross-domain prototypical instance-to-prototype contrastive learning, and adaptive prototypical classifier learning, which are jointly optimized within an end-to-end framework and function in a complementary manner. Besides, in order to promote prototypical self-supervised learning effectively, we leverage the class-balanced multicentric dynamic (BMD) prototype strategy to facilitate the generation of more representative and robust clustering prototypes. Furthermore, we design a Siamese-style distance metric (SDM) loss function to aggregate intra-class features while separating inter-class features. Moreover, our RBPCL method leverages the Double-Branch Dual-Attention Mechanism Network (DBDA) as the feature generator, enabling joint modeling of spatial and spectral characteristics in hyperspectral data, thereby enhancing the discriminative power of the learned semantic representations. Finally, we demonstrate the effectiveness of the proposed method over several typical advanced UDA methods and excellent FUDA methods by carrying out experiments on public data pairs and self-collected ultra-low-altitude hyperspectral data pairs captured on different dates and locations. It is noteworthy that our RBPCL method shows potential regarding tackling the spectral shift problem of real-world hyperspectral images across different illumination conditions, despite encountering the scarcity of labeled source samples.