Agreement–Disagreement Guided Knowledge Transfer for Cross-Scene Hyperspectral Imaging

Highlights

What are the main findings?

• An agreement–disagreement guided framework is proposed for heterogeneous cross-scene hyperspectral image classification.
• Joint optimization and representation learning improve transfer performance by reducing gradient conflicts and preserving target-specific information.

What are the implications of the main findings?

• Combining optimization consistency with representation diversity improves knowledge transfer across heterogeneous hyperspectral scenes.
• The proposed framework provides a general solution for transfer learning with limited labeled target samples in remote sensing.

Abstract

Knowledge transfer plays a crucial role in cross-scene hyperspectral imaging (HSI). However, existing studies often overlook the challenges of gradient conflicts and dominant gradients that arise during the optimization of shared parameters. Moreover, many current approaches fail to simultaneously capture both agreement and disagreement information, relying only on a limited shared subset of target features and consequently missing the rich, diverse patterns present in the target scene. To address these issues, we propose an Agreement–Disagreement Guided Knowledge Transfer (ADGKT) framework that jointly models optimization consistency and representation diversity for heterogeneous cross-scene HSI classification. The proposed framework consists of two complementary mechanisms. The agreement mechanism stabilizes joint optimization by mitigating gradient conflicts and balancing the contributions of source and target domains during shared parameter learning. The disagreement mechanism explicitly preserves complementary target-specific representations through a dedicated disagreement branch and integrates transferable and target-critical information into a unified predictor. Unlike conventional transfer learning approaches that rely solely on feature alignment, the proposed framework simultaneously encourages transferable knowledge sharing and target-specific representation learning, thereby improving robustness under heterogeneous scene discrepancies. Extensive experiments demonstrate the effectiveness and superiority of the proposed method in achieving robust and balanced knowledge transfer across heterogeneous HSI scenes.

1. Introduction

Hyperspectral imaging (HSI) has attracted significant attention in remote sensing due to its ability to capture reflectance information across hundreds of contiguous spectral bands [1,2,3,4,5]. Compared with conventional RGB imagery, HSI provides substantially richer spectral information, enabling fine-grained discrimination of materials and land-cover categories. Benefiting from these characteristics, HSI has been widely applied in a broad range of remote sensing applications, including environmental monitoring [6], precision agriculture [7], mineral exploration [8], and disaster assessment [9,10].

Recent advances in deep learning have significantly expanded the application of hyperspectral image (HSI) analysis beyond classification. Representative studies have demonstrated remarkable performance in hyperspectral object detection using multi-scale memory networks with separation training [11], while diffusion-based approaches have recently been introduced to address hyperspectral unmixing by explicitly modeling spectral variability and class constraints [12]. Together with recent progress in denoising, anomaly detection, and image restoration, these advances highlight the versatility of deep learning for exploiting complex spectral–spatial information across diverse HSI tasks. Nevertheless, compared with these well-studied problems, heterogeneous cross-scene HSI classification remains considerably more challenging because substantial discrepancies in spectral distributions, imaging sensors, and semantic categories often lead to negative transfer. Therefore, developing robust knowledge transfer methods that effectively bridge heterogeneous domains remains an important research problem.

Despite its strong representational capability, HSI analysis remains heavily dependent on large amounts of labeled training data. However, annotating hyperspectral images is labor-intensive, time-consuming, and expensive, since pixel-level labeling often requires extensive domain expertise and field investigation [13]. Consequently, collecting sufficient labeled samples for every newly acquired scene is generally impractical. To alleviate this limitation, knowledge transfer has emerged as an effective solution by transferring discriminative knowledge learned from a source scene with abundant annotations to a target scene with limited labeled data [14]. By leveraging transferable representations, knowledge transfer reduces the reliance on large-scale target annotations and improves generalization in data-scarce scenarios.

Existing cross-scene knowledge transfer methods in HSI mainly focus on two scenarios: homogeneous, which addresses spectral shifts within the same scene [13,15,16], and heterogeneous, which involve matching shared categories before aligning spectral distributions across different scenes [17,18,19,20]. Despite progress in both homogeneous and heterogeneous knowledge transfer scenarios, two critical challenges remain. First, current methods often under-explore the impact of optimization issues in shared parameters, particularly when there are very few training samples for the target task. This oversight can degrade performance and lead to harmful transfer effects in the target scene [21,22,23]. Second, current approaches overlook the importance of incorporating both agreement and disagreement in the transfer process. Although the target scene may contain a wealth of complex and informative features, the predictive outcomes often rely solely on a limited, shared subset of target features. This reliance on restricted information prevents the model from fully capturing the rich and diverse patterns present in the target scene [24].

In this work, we consider cross-scene knowledge transfer from two perspectives: agreement and disagreement. To achieve agreement, the shared parameters for knowledge transfer may encounter conflicting and dominating gradients [23,25]. Specifically, gradients derived from the source and target domains may exhibit inconsistent optimization directions, resulting in gradient conflicts when the model simultaneously learns from both scenes. Such conflicts can hinder convergence and degrade the transferability of shared representations. Moreover, the source domain often contains substantially more training samples than the target domain, producing gradients with larger magnitudes that dominate the optimization process. Consequently, the learning procedure may become biased toward source-specific characteristics, leading to insufficient adaptation and potential underfitting in the target domain. In addition, the source task may have a larger dataset compared with the target task, leading to larger gradients that dominate the learning process, potentially resulting in underfitting for the target task. In cases of disagreement, when the model focuses on shared features between the source and target domains, which represent only a small subset of the predictive features for the target, it limits the diversity of the results. This approach risks losing important information that could be critical for the target task, especially if the data in the target scene differs significantly from the source scene.

To solve these challenges, we propose a method called agreement–disagreement guided knowledge transfer (ADGKT), which consists of two main components: agreement and disagreement mechanisms. The agreement part includes GradVac [26] and LogitNorm [27]; GradVac adjusts the direction of gradients during training to align them more closely, reducing conflicts between source and target gradients, while LogitNorm prevents dominating gradients by controlling the magnitude of the logits (pre-softmax outputs), equalizing the contributions from each task and reducing the impact of dominating gradients when updating shared parameters. The disagreement component comprises a disagreement restriction and an ensemble mechanism; the disagreement restriction (DiR) promotes learning diverse representations of instances in the target domain, and the ensemble component captures a diverse set of predictive target features, mitigating the risk of losing important target information. Therefore, ADGKT enhances the model’s ability to generalize effectively to the target scene.

The contributions of this work can be summarized as follows:

• We propose an Agreement–Disagreement Guided Knowledge Transfer (ADGKT) framework for heterogeneous cross-scene hyperspectral image classification, which jointly models optimization consistency and representation diversity for robust knowledge transfer.
• We develop complementary agreement and disagreement mechanisms that, respectively, improve optimization stability across heterogeneous scenes and preserve target-specific information during knowledge transfer.
• Extensive experiments on three benchmark hyperspectral datasets under six heterogeneous cross-scene transfer settings demonstrate that the proposed framework consistently outperforms representative knowledge transfer, domain adaptation, and ensemble learning methods.

2. Methodology

In this section, we present the proposed ADGKT framework for cross-scene hyperspectral image classification. The proposed method is designed to jointly address optimization issues in shared parameters and the loss of diverse target-specific information during knowledge transfer. Specifically, as shown in Figure 1 the framework consists of two complementary components: an agreement mechanism that alleviates gradient conflicts and dominating gradients, and a disagreement mechanism that promotes diversity and preserves critical target representations. We first introduce the problem setting, followed by the technical challenges and the detailed formulation of each component in the proposed framework.

2.1. Problem Setting

In the cross-scene hyperspectral image (HSI) knowledge transfer scenario, we define the source scene as $\mathcal{D}s=(x_i^s,y_i^s){i=1}^{N_s}$, where $x_i^s\in {\mathbb{R}}^{H_s\times W_s\times B_s}$ denotes a source HSI sample with height $H_s$, width $W_s$, and $B_s$ spectral bands. The corresponding label is represented by $y_i^s\in {\mathcal{Y}}_s$, where ${\mathcal{Y}}_s$ denotes the label space of the source scene. Similarly, the target scene is defined as $\mathcal{D}t=(x_j^t,y_j^t){j=1}^{N_t}$, where $x_j^t\in {\mathbb{R}}^{H_t\times W_t\times B_t}$ represents a target HSI sample with $H_t$, $W_t$, and $B_t$ corresponding to the spatial dimensions and spectral bands of the target scene, respectively. The associated target label is denoted by $y_j^t\in {\mathcal{Y}}_t$, where ${\mathcal{Y}}_t$ is the label space of the target scene.

During the knowledge transfer process, the source and target scenes are jointly optimized through their respective loss functions, denoted by ${\mathcal{L}}_s$ and ${\mathcal{L}}_t$. The corresponding gradients with respect to the shared parameters are represented by $g_s$ and $g_t$, respectively. These gradients play a critical role in the optimization process and may introduce gradient conflicts or domination effects during shared parameter learning.

2.2. Technical Challenges

One of the central issues in knowledge transfer between cross-scene HSIs is achieving agreement in the optimization of shared parameters, which often suffers from gradient conflicts and dominating gradients. During model training, the optimization process often encounters gradient conflicts, where the gradients derived from source and target data pull the model’s parameters in different directions, hindering the learning process. Without effectively resolving these gradient conflicts, the model can converge to suboptimal solutions, and the transferred knowledge fails to generalize well to the target scene. In addition, the source task with an abundance of training examples can dominate the update of shared network layers because it produces larger magnitudes of gradients and obtains more updates. The model prioritizes optimizing the source task over the target task. The target task with fewer samples struggles to learn equal magnitudes of logits, leading to potential underfitting.

Another issue during knowledge transfer is the failure to capture the rich and diverse information present in the target scene. The disagreement information for the target scene is often neglected, especially when the source and target scenes exhibit different data characteristics. During the agreement part, it is possible to neglect the target critical information since this information is less important in the source scene. Because of that, the model can overlook the target critical information, which is valuable for the target task. As a result, the final predictive feature for the target scene might not transfer well from the source to the target scene. How to build effective transfer from the source to the target scene and make sure the integrity of target information remains a key challenge.

2.3. Agreement–Disagreement Guided Knowledge Transfer (ADGKT)

In this work, we propose a method, called ADGKT, including agreement and disagreement mechanisms. The agreement mechanism consists of GradVac, which alleviates gradient conflicts, and LogitNorm, which mitigates dominant gradients. Additionally, the disagreement mechanism includes a disagreement restriction and an ensemble approach. We utilize the disagreement restriction to obtain distinct and independent information, thereby promoting diversity. Furthermore, we introduce an ensemble of models to capture diverse aspects of the target features.

2.4. Agreement

During the training process, the update of shared parameters may encounter gradient conflict and dominant gradients. As shown in Figure 2, to address gradient conflict, we implement the GradVac method, which adapts the gradients $g_s$ to $g_s^{\prime }$ during training to reduce gradient conflicts. Additionally, to mitigate the dominance gradient, we apply LogitNorm to obtain updated logits $\widehat{z_s}$ and $\widehat{z_t}$, controlling their magnitude to prevent potential underfitting.

Although GradVac was originally developed for multi-task learning, its underlying principle naturally extends to heterogeneous cross-scene knowledge transfer. In our framework, the source and target scenes are jointly optimized through a shared feature encoder. The source-domain objective aims to preserve transferable discriminative representations, whereas the target-domain objective focuses on adapting the shared representation to the target distribution. Owing to the substantial spectral distribution and semantic discrepancies between heterogeneous scenes, these two objectives may generate conflicting gradients when updating the shared parameters, resulting in unstable optimization and negative transfer. This optimization behavior is analogous to gradient conflicts in multi-task learning, where different objectives compete for shared model parameters. Therefore, we adapt GradVac to reconcile the optimization conflict between the source and target objectives, thereby improving optimization consistency during cross-scene knowledge transfer.

2.4.1. Alleviating Gradient Conflict Through GradVac

To alleviate gradient conflict, we introduce GradVac, which adjusts the gradients between source and target during training to ensure they are more aligned and less likely to interfere with each other. First, the cosine similarity score $\varphi =\cos \left(\theta \right)$ between gradients $g_s$ and $g_t$ can be defined as follows:

$$\cos \left(\theta \right)={\displaystyle \frac{g_s·g_t}{\parallel g_s\parallel \parallel g_t\parallel }},$$

(1)

where $\theta$ represents the angle between the two gradients. This similarity score $\varphi$ helps to quantify the degree of conflict between the gradients. When $\varphi <\alpha$, the gradient conflict becomes significant and cannot be ignored. In this case, the gradient $g_s$ is updated to a new gradient $g_s^{\prime }$ to mitigate the conflict:

$$g_s^{\prime }=g_s+\eta g_t,$$

(2)

where $\eta$ is determined by the Law of Sines, calculated as follows:

$$\eta ={\displaystyle \frac{\parallel g_s\parallel ({\varphi }^T\sqrt{1-{\varphi }^2}-\varphi \sqrt{1-{\left({\varphi }^T\right)}^2})}{\parallel g_t\parallel \sqrt{1-{\left({\varphi }^T\right)}^2}}}.$$

(3)

The threshold $\alpha$ is dynamically updated at training step t through an exponential moving average (EMA):

$${\alpha }^{\left(t\right)}=(1-\beta ){\alpha }^{(t-1)}+\beta {\varphi }^{(t-1)},$$

(4)

where ${\varphi }^{(t-1)}$ is the cosine similarity of the gradients $g_s$ and $g_t$ at training step $t-1$, $\beta$ is a hyperparameter, and ${\alpha }^{\left(0\right)}=0$.

2.4.2. Mitigating the Dominating Gradients via LogitNorm

To reduce the impact of dominating gradients, we employ the gradient magnitude similarity $\mathrm{\Phi }(g_s,g_t)$ to represent the contribution between source gradient $g_s$ and target gradient $g_t$ for shared parameters. The gradient magnitude similarity [25] can be represented as follows:

$$\mathrm{\Phi }(g_s,g_t)={\displaystyle \frac{2\parallel g_s{\parallel }_2{\parallel g_t\parallel }_2}{\parallel g_s{\parallel }_2^2+{\parallel g_t\parallel }_2^2}}.$$

(5)

The value of $\mathrm{\Phi }$ is between 0 and 1. When the two gradients $g_s$ and $g_t$ have the same magnitude ($\parallel g_s{\parallel }_2={\parallel g_t\parallel }_2$), the similarity $\mathrm{\Phi }$ equals 1, which means the source and target scenes have a positive impact on the shared parameter. When the magnitudes of two gradients are significantly different, $\mathrm{\Phi }$ equals 0, leading to imbalanced optimization for the shared feature, thereby causing sub-optimization of the shared parameter.

In addition, the cross-entropy loss L encourages the model to increase the magnitude of logits $\parallel z\parallel$ during the training process. The logit gradient of cross-entropy loss can be represented as follows:

$${\displaystyle \frac{\partial L}{\partial z_k}}={\widehat{y}}_k-y_k,$$

(6)

where ${\widehat{y}}_k$ denotes predicted probability for class k. $y_k$ represents the true label for class k. When the source scene contains more samples than the target scene, the source scene can have a high frequency to update the shared parameter for source presentation. Over time, the magnitude of ${\widehat{y}}_k$ for the dominant source task will increase; thereby, the magnitude of source logits will grow larger than the target.

To address the dominant gradient, we apply LogitNorm in Equation (7). By controlling the gradient magnitudes, LogitNorm can better balance the importance of source and target scenes during the shared parameter optimization.

$$\widehat{z}={\displaystyle \frac{z}{\tau \parallel z\parallel }},$$

(7)

where $\widehat{z}$ represents the normalized logits, $\parallel z\parallel$ is the magnitude of the logits vector, and $\tau$ denotes the temperature, which regulates the magnitude of the logits. The modified cross-entropy loss function with LogitNorm is as follows:

$$\mathcal{L}({\widehat{z}}_i,y)=-log\left({\displaystyle \frac{exp\left({\widehat{z}}_i\right)}{{\sum }_{j=1}^{C}exp\left({\widehat{z}}_j\right)}}\right),$$

(8)

Although GradVac and LogitNorm are both introduced to improve optimization stability, they address two complementary aspects of the optimization process. GradVac mitigates gradient conflicts by aligning inconsistent optimization directions between the source-domain and target-domain objectives, thereby reducing destructive interference during shared parameter updates. In contrast, LogitNorm addresses the imbalance in gradient magnitudes caused by differences in the scale of the source and target data, preventing the optimization from being dominated by the source-domain objective. Consequently, GradVac improves optimization consistency, whereas LogitNorm promotes balanced optimization. Their combination enables the proposed agreement mechanism to simultaneously resolve directional conflicts and magnitude imbalance, resulting in more stable and effective knowledge transfer.

2.5. Disagreement

When primarily focusing on achieving agreement between the source and target scenes, the model may overly emphasize shared transferable representations while overlooking the rich and diverse patterns inherent in the target scene. Such excessive alignment can suppress target-specific characteristics and reduce the model’s ability to capture complementary information necessary for robust generalization. To address this issue, we introduce a disagreement mechanism consisting of a disagreement restriction term, denoted as $E_{\mathrm{DiR}}$, together with an ensemble component, denoted as $E_{\mathrm{en}}$, to explicitly promote representation diversity in the target domain, as illustrated in Figure 3. By encouraging complementary and independent target representations, the proposed disagreement mechanism preserves critical target-specific information and enhances the model’s capability to learn more comprehensive and discriminative features for cross-scene HSI classification.

2.5.1. Promoting Diversity Through Disagreement Restriction

To enhance the predictive power of the target features during the training process, it is essential to promote diversity and independence among them. To achieve this, we employ a disagreement restriction (DiR) using partial distance correlation [28], which enforces orthogonality among features:

$$E_{\mathrm{DiR}}={\mathbb{E}}_{x\sim q}dCor(G\left(F_t\left(x\right)\right),G^{\prime }\left(F_t^{\prime }\left(x\right)\right)),$$

(9)

where $F_t^{\prime },G^{\prime }$ and $T_t^{\prime }$ denote the separate components to capture target critical information $G^{\prime }\left(F_t^{\prime }\left(x\right)\right)$, which is enforced to be orthogonal to the shared information between the source and target scenes, represented by $G\left(F_t\left(x\right)\right)$.

Unlike conventional diversity regularization, the proposed disagreement restriction is designed to preserve complementary target-specific information during knowledge transfer. The shared representation primarily captures transferable features that are common to both the source and target domains, whereas target-specific discriminative characteristics may be weakened during feature alignment. By minimizing the dependence between the shared and target-specific representations through partial distance correlation, DiR encourages the target-specific branch to learn complementary rather than redundant information. Consequently, the ensemble model can simultaneously exploit transferable shared knowledge and target-specific discriminative features, resulting in more comprehensive feature representations and improved cross-scene generalization.

By applying this disagreement restriction, each feature is encouraged to capture distinct and independent aspects of the target data. Therefore, the disagreement restriction promotes a diverse representation of the target scene, ensuring that important target-critical information is effectively learned.

2.5.2. Capturing a Diverse Set of Target Features Through Ensemble

We propose an ensemble of models to capture a diverse set of features for the target scene. This ensemble approach enables the model to integrate multiple perspectives of the target features. Additionally, we employ reverse distillation [29] to iteratively reduce the discrepancy between the teacher models (which include both agreement and disagreement mechanisms) and the ensemble model, thereby updating and refining the integrated information through the ensemble.

$$\begin{array}{cc}\hfill E_{{\mathrm{en}}_1}& ={\mathbb{E}}_{x\sim q}[D_{\mathrm{KL}}(T_{en}\left(G_{en}\left(F_t\left(x\right)\right)\right),T_t\left(G\left(F_t\left(x\right)\right)\right))\hfill \\ \hfill & +D_{\mathrm{KL}}(T_t\left(G\left(F_t\left(x\right)\right)\right),T_{en}\left(G_{en}\left(F_t\left(x\right)\right)\right))],\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill E_{{\mathrm{en}}_2}& ={\mathbb{E}}_{x\sim q}[D_{\mathrm{KL}}(T_{en}\left(G_{en}\left(F_t\left(x\right)\right)\right),T_t^{\prime }\left(G^{\prime }\left(F_t^{\prime }\left(x\right)\right)\right))\hfill \\ \hfill & +D_{\mathrm{KL}}(T_t^{\prime }\left(G^{\prime }\left(F_t^{\prime }\left(x\right)\right)\right),T_{en}\left(G_{en}\left(F_t\left(x\right)\right)\right))],\hfill \end{array}$$

(11)

where $D_{\mathrm{KL}}$ denotes KL divergence.

The knowledge distillation (KD) loss associated with the ensemble model under the agreement mechanism is denoted as $E_{\mathrm{en}1}$ in Equation (10). This component transfers the consensus knowledge learned from the agreement branch, which focuses on alleviating conflicting and dominating gradients during shared parameter optimization. Similarly, $E{\mathrm{en}}_2$ in Equation (11) represents the KD loss corresponding to the disagreement mechanism, which aims to preserve diverse and target-critical representations by encouraging complementary and independent target features.

By simultaneously distilling knowledge from both the agreement and disagreement teachers, the final ensemble model can effectively integrate shared transferable information together with diverse target-specific characteristics, as formulated as follows:

$$E_{\mathrm{en}}=E_{{\mathrm{en}}_1}+E_{{\mathrm{en}}_2}.$$

(12)

In this way, the proposed framework not only improves optimization stability during cross-scene knowledge transfer, but also mitigates the loss of important target information that may occur when relying solely on agreement-based learning. Since both loss terms share the same objective formulation and exhibit similar optimization scales throughout training, we adopt equal weighting to avoid introducing an additional hyperparameter. Consequently, the ensemble model is able to capture more comprehensive and discriminative target representations, leading to stronger generalization capability and improved classification performance in heterogeneous cross-scene HSI transfer tasks. The pseudocode for the ADGKT framework is shown in Algorithm 1.

Algorithm 1 The pseudocode for the core of ADGKT framework

1: Initialization: Set $t=0$, the input data $x_i^s$ from HSI source scene and $x_j^t$ from HSI target scene. Specifically, $g_s$ for the source scene and $g_t$ for the target scene are the gradients to their respective loss functions ${\mathcal{L}}_s$ and ${\mathcal{L}}_t$. z is the logit before the softmax. 2: for $t=0,1,2,\dots ,$do 3:     Agreement: 4:     if $\varphi <\alpha$ then 5:         $g_s^{\prime }=g_s+\eta g_t$       //alleviating gradient conflict 6:     end if 7:     if $\mathrm{\Phi }(g_s,g_t)\ne 1$ then 8:         $\widehat{z}={\displaystyle \frac{z}{\tau \parallel z\parallel }},\mathcal{L}({\widehat{z}}_i,y)=-log\left({\displaystyle \frac{exp\left({\widehat{z}}_i\right)}{{\sum }_{j=1}^{C}exp\left({\widehat{z}}_j\right)}}\right)$       //mitigating the dominating gradients 9:     end if; 10:         Disagreement: $E_{\mathrm{DiR}}={\mathbb{E}}_{x\sim q}dCor(G\left(F_t\left(x\right)\right),G^{\prime }\left(F_t^{\prime }\left(x\right)\right))$ //disagreement restriction for shared information $G\left(F_t\left(x\right)\right)$ and target-critical information $G^{\prime }\left(F_t^{\prime }\left(x\right)\right)$. $E_{\mathrm{en}}=E_{{\mathrm{en}}_1}+E_{{\mathrm{en}}_2}$ //ensemble for agreement and disagreement information; 11:     end for

2.6. Computational Complexity Analysis

Let N denote the batch size, d the feature dimension, and P the number of shared network parameters. Compared with conventional knowledge transfer frameworks, the proposed ADGKT introduces two additional computational components. The agreement mechanism performs gradient manipulation and logit normalization with complexities of $\mathcal{O}\left(P\right)$ and $\mathcal{O}\left(NC\right)$, respectively, where C is the number of classes. The disagreement mechanism introduces an additional target-specific branch together with a disagreement restriction based on partial distance correlation, resulting in a complexity of $\mathcal{O}\left(N^{2}d\right)$ for computing feature dependence. Therefore, the overall computational complexity of ADGKT can be expressed as

$$\mathcal{O}(F+P+N^{2}d+NC),$$

(13)

where $\mathcal{O}\left(F\right)$ denotes the complexity of the feature extraction and classification network. Since $\mathcal{O}\left(F\right)$ dominates the overall computation, the additional agreement and disagreement mechanisms introduce only a modest computational overhead. During inference, the gradient manipulation and auxiliary optimization losses are discarded, and only the prediction network is used, resulting in an inference complexity of the same order as the underlying classification network.

3. Experiments

In this section, we evaluate the effectiveness of the proposed ADGKT framework on multiple cross-scene hyperspectral image transfer tasks. We conduct extensive experiments under heterogeneous transfer settings to assess the capability of the proposed method in handling gradient conflicts, dominating gradients, and target feature diversity. The proposed framework is compared with different methods. Furthermore, ablation studies are performed to validate the importance of each component in the agreement and disagreement mechanisms.

3.1. Experimental Setup

We conducted experiments on three popular datasets: Indian Pines (I), Pavia (P) [30], and Houston2013 (H) [31]. There is no straightforward category correspondence for these three datasets, as shown in

Table 1. The detailed category information for Indian Pines, Pavia University and Houston 2013 datasets.

Indian Pines	Pavia University	Houston 2013
Alfalfa	Asphalt	Health grass
Corn-notill	Meadows	Stressed grass
Corn-mintill	Gravel	Synthetic grass
Corn	Trees	Tress
Grass-pasture	Painted metal sheets	Soil
Grass-trees	Bare Soil	Water
Grass-pasture-mowed	Bitumen	Residential
Hay-windrowed	Self-Blocking Bricks	Commercial
Oats	Shadows	Road
Soybean-notill		Highway
Soybean-mintill		Railway
Soybean-clean		Parking lot 1
Wheat		Parking lot 2
Woods		Tennis court
Buildings-Grass-Trees-Drives		Running track
Stone-Steel-Towers

. In addition, these three datasets were captured by distinct HSI sensors. During the knowledge transfer process, all training samples from the source scenes are utilized, while only 10 randomly selected samples for each category are used from the target scene. Furthermore, the baseline represents the results obtained by training solely on the target scene using 10 samples per category, without the assistance of knowledge transfer. We compare our method with six different knowledge transfer methods, including Adaptor [32], Finetune [33], Multi-Task Learning (MTL) [34], Domain Adaptation (DA) method–UAN [35] and ensemble methods–ONE [36], FFL [37]. For a fair comparison, we use the same backbone, Masked SST [38]. All the methods are trained with the Adam optimizer with a weight decay of $5\times {10}^{-3}$ and a momentum of 0.9. The initial learning rate is set to $5\times {10}^{-4}$. The batch size is set to 64 for all methods. During the transfer from IndianPine to Pavia, we set $\beta =0.1$ in Equation (4) and $\tau =2$ in Equation (7) under the agreement mechanism, while in the disagreement mechanism, the temperature is set to 1 in Equation (10) and $0.05$ in Equation (11). In addition, during the transfer from Houston to Pavia, we set $\beta =0.01$ in Equation (4) and $\tau =4$ in Equation (7) under the agreement mechanism, while in the disagreement mechanism, the temperature is set to 1 in Equation (10) and $0.001$ in Equation (11). All experiments were conducted on a workstation equipped with two NVIDIA RTX A5500 GPUs, an Intel(R) Xeon(R) Gold 5415+ CPU, and 128 GB of system memory.

3.2. Results

3.2.1. Compared with Knowledge Transfer Methods

In

Table 2. The performance (%) of knowledge transfer from IndianPine to Pavia (I → P) and from Houston to Pavia (H → P). ↑ indicates that higher values are better. Bold numbers represent the best results.

Class	Baseline	I → P							H → P
		MTL	DA	ONE	FFL	Adaptor	Finetune	Ours	MTL	DA	ONE	FFL	Adaptor	Finetune	Ours
Asphalt	83.68	78.66	78.11	79.70	73.65	85.42	79.74	86.45	77.62	78.78	87.06	85.03	79.17	83.82	88.59
Meadows	69.02	69.50	82.34	63.72	73.39	81.57	86.10	94.44	59.80	74.50	71.46	68.47	76.64	80.31	92.29
Gravel	57.25	73.72	66.89	73.28	71.46	87.99	68.32	76.64	74.93	73.22	71.35	67.27	73.39	63.47	75.87
Trees	90.01	81.73	82.49	84.51	91.14	86.44	87.81	84.68	87.23	86.47	90.18	88.32	94.71	82.66	84.10
Painted metal sheets	99.01	94.34	96.32	87.78	97.21	96.77	99.10	9605	96.86	95.87	98.38	97.12	98.65	90.57	93.62
Bare Soil	55.51	79.31	69.79	85.74	50.79	79.18	65.46	64.52	91.84	55.03	54.64	67.98	41.23	61.70	61.61
Bitumen	68.50	70.03	69.42	58.00	72.17	56.57	68.60	64.63	75.74	74.21	67.99	60.35	69.22	64.93	69.42
Self-Blocking Bricks	95.04	93.61	95.30	89.80	95.30	86.95	93.01	94.20	91.08	92.24	93.64	94.44	83.44	93.82	94.83
Shadows	93.08	88.55	85.41	87.42	91.57	89.43	91.19	93.71	89.94	89.43	92.33	93.71	91.45	94.34	90.57
OA ↑	74.27	76.26	80.77	73.90	74.89	82.97	82.67	87.52	73.61	76.14	76.30	75.70	75.46	79.69	82.88
AA ↑	79.01	81.05	80.67	78.88	79.63	83.37	82.15	83.93	82.78	79.97	80.78	80.30	78.66	81.09	80.29
$\kappa$↑	66.93	69.62	74.61	66.91	67.52	77.61	76.91	83.01	67.11	68.93	69.40	68.89	67.85	73.17	77.03

, we demonstrate that our method achieves the best results. For the majority of categories in the Pavia dataset, our method achieves the highest accuracy values. Existing methods fail to address the issues of agreement and disagreement in knowledge transfer. For example, MTL and UAN methods incorporating source scene perform similarly to or worse than the baseline transferring from Indian Pines to Pavia (I → P) and from Houston to Pavia (H → P). This suggests that merely attempting to align the source and target scenes without effectively managing gradient conflicts and dominating gradients in shared parameters is insufficient. For disagreement, ensemble methods such as ONE and FFL increased limited performance when transferring from Indian Pines to Pavia (I → P). While these methods aim to capture different information, they fall short in significantly enhancing the model’s ability to generalize to the target scene. This indicates that without disagreement restriction, the benefits of ensemble strategies remain limited. In addition, Adaptor and Finetune methods require extra computation time (pretraining on the source scene first) to improve performance, whereas our method achieves state-of-the-art results without the need for a pretrained model.

These results demonstrate the importance of addressing both agreement and disagreement in knowledge transfer. Our method not only overcomes the limitations of existing approaches but also provides a solution that enhances the model’s ability to generalize effectively to new scenes.

Table 2,

Table 3. The performance (%) of knowledge transfer from Pavia to Houston (P → H) and IndianPine to Houston (I → H). ↑ indicates that higher values are better. Bold numbers represent the best results.

Class	Baseline	P → H							I → H
		MTL	UAN	ONE	FFL	Adaptor	Finetune	Ours	MTL	UAN	ONE	FFL	Adaptor	Finetune	Ours
Healthy grass	74.55	70.28	68.85	63.53	70.66	74.55	76.26	78.44	75.59	71.89	76.92	76.35	69.99	73.03	80.25
Stressed grass	89.85	92.67	95.68	97.37	97.27	94.83	92.95	97.84	95.86	95.21	96.90	97.65	95.39	92.01	95.58
Synthetic grass	54.65	70.89	75.25	74.65	68.32	73.27	48.71	69.50	60.20	64.75	74.06	79.80	75.64	80.79	72.48
Trees	91.66	97.06	91.57	96.31	96.88	93.66	96.02	96.78	89.20	96.12	97.25	96.59	95.74	88.73	98.39
Soil	99.53	99.34	97.82	99.43	96.31	98.20	99.62	99.24	99.43	100.00	98.86	99.15	98.11	99.81	99.72
Water	77.62	74.13	81.82	81.82	82.52	69.23	70.63	81.82	83.22	77.62	81.12	80.42	69.23	65.03	80.42
Residential	78.82	88.25	86.75	79.48	86.01	84.24	86.38	88.62	84.24	83.77	83.02	83.86	77.71	72.20	89.09
Commercial	83.38	68.76	68.28	72.27	73.12	68.38	76.07	81.39	77.59	78.44	76.07	68.57	85.28	87.27	73.41
Road	82.34	71.01	69.78	84.23	62.04	82.91	66.57	77.24	63.36	64.78	79.23	67.61	86.87	78.94	77.05
Highway	62.36	72.49	55.21	56.56	68.82	48.84	53.47	77.12	51.74	66.51	51.74	66.12	62.93	55.50	83.98
Railway	70.97	77.80	80.74	82.07	82.45	80.36	86.81	78.93	73.72	80.83	86.15	78.08	79.32	78.56	79.89
Parking Lot 1	71.47	71.18	72.43	64.07	62.34	70.32	57.44	72.62	73.20	68.20	68.78	66.57	67.72	58.50	69.84
Parking Lot 2	90.18	87.37	88.77	86.67	82.81	78.95	81.75	90.18	84.21	84.21	85.96	89.47	72.63	84.56	92.28
Tennis Court	81.38	81.78	90.28	78.54	92.71	83.40	76.11	91.90	90.28	88.07	91.09	87.04	73.68	88.26	86.64
Running Track	69.13	64.27	67.02	75.48	78.01	57.29	84.14	79.70	65.75	71.88	86.05	79.70	65.33	54.55	73.36
OA (↑)	79.24	79.96	78.69	79.42	79.49	78.52	78.08	84.27	77.65	79.85	81.73	80.45	80.53	77.87	83.97
AA (↑)	78.53	79.15	79.35	79.50	80.02	77.23	76.86	84.09	77.84	79.49	82.21	81.13	78.37	77.18	83.49
$\kappa$ (↑)	77.52	78.26	76.88	77.67	77.76	76.69	76.21	82.92	75.77	78.15	80.18	78.80	78.84	75.98	82.61

and

Table 4. The performance (%) of knowledge transfer from Pavia to IndianPine (P → I) and from Houston to IndianPian (H → I). ↑ indicates that higher values are better. Bold numbers represent the best results.

Class	Baseline	P → I							H → I
		MTL	UAN	ONE	FFL	Adaptor	Finetune	Ours	MTL	UAN	ONE	FFL	Adaptor	Finetune	Ours
Alfalfa	66.04	59.10	56.29	61.42	66.84	68.21	58.09	62.28	55.63	48.77	62.21	58.53	61.27	64.09	77.46
Corn-notill	89.29	69.26	73.09	83.93	80.10	61.73	76.53	90.82	68.11	65.69	83.93	83.29	83.16	72.70	88.65
Corn-mintill	99.46	86.96	89.67	93.48	94.02	87.50	92.39	92.93	95.65	89.13	93.48	95.11	91.30	90.22	93.48
Corn	85.01	83.67	73.15	84.34	79.87	78.75	79.19	87.47	83.45	78.97	78.30	82.77	81.43	82.33	84.56
Grass-pasture	91.39	86.94	78.05	83.36	88.67	78.62	78.77	86.80	79.20	66.86	85.08	79.34	80.77	75.61	91.68
Grass-trees	93.62	94.76	94.76	92.48	93.62	91.11	92.26	94.99	94.31	94.08	92.71	94.08	93.17	90.66	94.99
Grass-pasture-mowed	70.26	60.46	54.58	49.89	68.52	62.96	62.31	75.16	63.40	66.56	66.56	61.33	69.93	55.12	70.48
Hay-windrowed	64.19	58.60	33.46	57.73	64.31	68.78	65.34	72.66	50.95	41.52	56.99	60.26	66.67	65.34	72.37
Oats	76.06	70.39	55.67	68.44	64.89	67.02	66.31	75.70	66.31	71.45	68.44	74.65	47.70	59.22	75.71
Soybean-notill	95.68	98.15	96.91	97.53	95.06	96.91	95.06	93.72	94.44	91.98	95.68	98.77	88.27	97.53	96.91
Soybean-mintill	93.89	94.21	92.12	81.91	90.19	84.81	87.86	97.78	81.83	83.04	81.03	93.17	83.52	91.80	93.49
Soybean-clean	87.88	64.85	72.12	80.91	74.24	87.27	78.79	97.78	77.58	77.58	78.48	81.52	83.94	72.42	84.24
Wheat	95.56	100.00	100.00	100.00	100.00	100.00	100.00	97.78	100.00	97.78	100.00	100.00	93.33	100.00	100.00
Woods	89.74	97.44	97.44	94.87	100.00	100.00	97.44	100.00	100.00	97.44	100.00	100.00	100.00	89.74	100.00
Buildings-Grass-Trees-Drives	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Stone-Steel-Towers	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
OA (↑)	78.15	71.66	62.74	70.58	75.31	73.50	72.53	80.11	67.56	63.49	71.78	73.45	73.24	72.08	81.64
AA (↑)	87.38	82.80	79.21	83.14	85.02	83.36	83.15	88.46	81.93	79.43	84.08	85.17	82.78	81.67	89.00
$\kappa$ (↑)	75.30	68.12	58.62	66.90	72.06	69.84	68.92	77.39	63.79	59.41	68.28	70.17	69.65	68.23	79.10

demonstrate that the proposed ADGKT framework consistently achieves superior performance across all six heterogeneous cross-scene transfer settings. Specifically, our method attains the highest overall accuracy (OA), average accuracy (AA), and $\kappa$ values in most experiments, while also achieving the best classification accuracies for the majority of categories in the Pavia, Houston, and IndianPine datasets. These results verify that the proposed framework can effectively transfer discriminative knowledge across heterogeneous hyperspectral scenes, even under extremely limited target supervision with only a few labeled samples available in the target domain.

Existing methods fail to address the agreement and disagreement issues. For example, for agreement, MTL and UAN methods show only slight improvements or even decreased performance when incorporating the target scene, compared with the baseline when transferring from IndianPine to Pavia (I → P) and from Houston to Pavia (H → P). For disagreement, ensemble methods, including ONE and FFL, increased performance when transferring from IndianPine to Pavia (I → P). In addition, Adaptor and Finetune methods require extra computation time (pretraining on the source scene first) to improve the performance, whereas our method achieves state-of-the-art results without the need for a pretrained model. On the other hand, when transferring from Pavia to IndianPine (P → I) and from Houston to IndianPine (H → I), the performance of the existing method decreased compared with the baseline. This is mainly because most categories in the IndianPine dataset are fine-grained and lack corresponding categories in the Houston and Pavia datasets. Because of this, it is difficult to increase the performance of IndianPine by utilizing the information from the Houston or Pavia datasets.

In contrast, our method considers the agreement and disagreement perspectives during knowledge transfer to mitigate the conflicting, dominating gradients and loss of target-critical information. By constraining redundancy among target features, the model can better preserve critical target-specific information that may otherwise be overlooked during agreement optimization. Combined with ensemble learning, the disagreement mechanism enables the proposed framework to capture diverse perspectives of the target scene, thereby significantly improving transferability and robustness under heterogeneous cross-scene conditions.

Overall, the experimental results demonstrate that jointly considering agreement and disagreement is essential for effective cross-scene HSI knowledge transfer. By simultaneously mitigating gradient conflicts, balancing dominating gradients, and preserving diverse target-critical information, the proposed ADGKT framework achieves stronger feature transferability, better optimization stability, and consistently superior classification performance across multiple heterogeneous hyperspectral datasets.

3.2.2. Ablation Studies

We conduct ablation studies to evaluate the individual contributions of the proposed agreement and disagreement mechanisms. As shown in

Table 5. Ablation studies on different components. ↑ indicates that higher values are better. Bold numbers represent the best results.

Datasets	GradVac	LogitNorm	Ensemble	DiR	OA ↑
I → P					76.26
	✓				81.62
	✓	✓			84.94
	✓	✓	✓		85.82
	✓	✓	✓	✓	87.52
H → P					73.61
	✓				80.43
	✓	✓			81.82
	✓	✓	✓		82.07
	✓	✓	✓	✓	82.88
P → H					79.24
	✓				77.13
	✓	✓			81.06
	✓	✓	✓		83.20
	✓	✓	✓	✓	84.27
I → H					79.24
	✓				78.77
	✓	✓			81.11
	✓	✓	✓		81.91
	✓	✓	✓	✓	83.97
P → I					78.15
	✓				69.20
	✓	✓			75.79
	✓	✓	✓		78.93
	✓	✓	✓	✓	80.11
H → I					78.15
	✓				67.58
	✓	✓			75.71
	✓	✓	✓		80.42
	✓	✓	✓	✓	81.64

, the framework is progressively constructed by introducing GradVac, LogitNorm, the ensemble branch, and the disagreement restriction (DiR), enabling us to analyze the role of each component in heterogeneous cross-scene knowledge transfer.

For the transfer tasks with Pavia as the target scene (I → P and H → P), introducing GradVac consistently improves the classification performance over the baseline configuration. This observation indicates that alleviating gradient conflicts between the source and target domains facilitates more stable optimization of the shared parameters. However, GradVac alone does not completely resolve the optimization imbalance caused by heterogeneous scenes. After incorporating LogitNorm, the performance is further improved, suggesting that balancing the optimization contributions from the source and target domains effectively complements gradient alignment. Together, these two components constitute the proposed agreement mechanism, which consistently enhances optimization stability during knowledge transfer.

We further evaluate the disagreement mechanism by progressively introducing the ensemble branch and the disagreement restriction. The ensemble branch alone yields only modest improvements, indicating that simply aggregating multiple predictors is insufficient for preserving complementary target information. After incorporating DiR, the performance consistently increases across both transfer tasks. This result demonstrates that explicitly encouraging complementary target-specific representations enables the ensemble model to capture richer information beyond the shared transferable features, thereby improving knowledge transfer.

A similar trend is observed when the Houston dataset is used as the target scene (P → H and I → H). Applying GradVac alone results in limited or even negative improvements, indicating that reducing gradient conflicts alone cannot adequately address the optimization challenges introduced by heterogeneous scene discrepancies. After combining GradVac with LogitNorm, the performance consistently improves, confirming that both gradient alignment and balanced optimization are necessary for effective transfer learning. Furthermore, introducing the disagreement mechanism provides additional performance gains, demonstrating that preserving complementary target-specific information improves generalization beyond conventional feature alignment.

Forthe more challenging transfer tasks with the Indian Pines dataset as the target scene (P → I and H → I), the agreement mechanism alone provides limited improvements because Indian Pines contains fine-grained categories with weak semantic correspondence to the coarse-grained categories in the source scenes. Consequently, optimization based primarily on shared transferable representations is insufficient to capture the discriminative characteristics of the target scene. By incorporating the disagreement mechanism, the performance is consistently improved, indicating that preserving complementary target-specific representations becomes particularly important when large semantic discrepancies exist between the source and target domains.

Overall, the ablation results demonstrate that the agreement and disagreement mechanisms play complementary roles in heterogeneous cross-scene HSI transfer. The agreement mechanism improves optimization consistency by alleviating gradient conflicts and balancing the optimization contributions of the source and target domains, while the disagreement mechanism preserves complementary target-specific representations that may otherwise be overlooked during feature alignment. Their combination consistently achieves the best performance across all heterogeneous transfer settings, validating the effectiveness of jointly modeling optimization consistency and representation diversity.

4. Relation to Prior Work

Knowledge transfer for cross-scene hyperspectral image (HSI) classification has attracted increasing attention in recent years due to the high annotation cost and limited labeled samples available in newly acquired scenes. The knowledge transfer in cross-scene HSI can be broadly classified as homogeneous and heterogeneous types [18,19,20,39,40,41]. For the homogeneous type, the methods mainly consider the spectral shift from the source to target areas in the same dataset [13,15,16,42]. In the heterogeneous type, the existing methods will manually match the shared categories first before aligning the distinct spectral distributions for different datasets [17,18,19,20,43]. In addition, existing works primarily focus on aligning source and target domains through feature matching or distribution alignment, aiming to mitigate domain shifts.

While cross-scene hyperspectral image (HSI) transfer has seen notable progress, significant gaps persist in optimizing shared parameters and preserving diverse target features. In cross-scene HSI scenarios, the training of shared parameters can be hindered by dominating and conflicting gradients. These gradient issues degrade performance and may lead to harmful transfer effects [21,22,23]. Approaches such as GradNorm [26] and PCGrad [25] have been proposed in multi-task learning to align gradients and prevent conflicts when updating shared parameters. However, these techniques are not specifically designed to address the unique challenges of cross-scene HSI transfer.

Another important limitation of current HSI transfer methods is the insufficient modeling of target diversity. Most existing approaches focus on extracting shared transferable features between the source and target domains, while neglecting target-specific information that may not be strongly correlated with the source scene. As a result, predictive outcomes often depend on a restricted subset of target representations, increasing the risk of losing critical target information and reducing the model’s ability to capture the rich diversity of the target domain [24]. This issue becomes particularly severe in heterogeneous transfer scenarios where the source and target scenes exhibit substantially different semantic structures and spectral characteristics.

Beyond HSI transfer, disagreement and diversity learning have recently attracted attention in transfer learning and representation learning research. Agree-to-Disagree [24] demonstrates that encouraging disagreement between predictors can improve transferability and representation diversity. Similarly, knowledge distillation methods have shown strong capability in integrating complementary information from multiple teacher models [44,45,46]. Ensemble distillation and online mutual distillation strategies enable models to capture diverse perspectives of the target data, improving robustness and generalization. However, these approaches are rarely explored in the context of cross-scene HSI transfer, particularly in combination with optimization-aware agreement mechanisms.

Deep learning has significantly advanced hyperspectral image analysis in recent years by learning powerful spatial–spectral representations for a wide range of tasks. Convolutional neural networks, transformers, and self-supervised learning methods have achieved remarkable performance in HSI classification, denoising, anomaly detection, and image restoration [47,48]. Recent representative methods include the Global Feature-Injected Blind-Spot Network (GFI-BSN) for hyperspectral image denoising [49], the Pixel-Associated Autoencoder (PAAE) for hyperspectral anomaly detection [50], and Eigen-CNN, which exploits eigenimages and eigennoise level maps for hyperspectral image restoration [51]. These methods demonstrate the versatility of deep neural networks in exploiting the spatial–spectral characteristics of HSI data across diverse image processing tasks.

In summary, although significant progress has been achieved in cross-scene HSI transfer learning, existing approaches still suffer from two major limitations. First, they insufficiently address optimization issues caused by conflicting and dominating gradients during shared parameter learning. Second, they fail to adequately preserve diverse target-specific representations during knowledge transfer.

5. Discussion and Broader Impact

In this paper, we introduce the agreement–disagreement guided knowledge transfer (ADGKT) method that integrates both agreement and disagreement mechanisms to enhance knowledge transfer between cross-scene HSIs. The agreement mechanisms, consisting of GradVac and LogitNorm, effectively address gradient conflicts and dominating gradients by aligning gradient directions and controlling the magnitude of logits, balancing shared parameter optimization. The disagreement mechanisms, incorporating a DiR and an ensemble approach, capture diverse and independent target features. These mechanisms mitigate the risk of losing critical target information that might be overlooked when focusing solely on agreement between source and target scenes. Although our method demonstrates state-of-the-art performance, we acknowledge several considerations for future exploration. First, the method involves a few hyperparameters that may require systematic tuning to optimize performance in different datasets. Second, incorporating both agreement and disagreement mechanisms, along with an ensemble strategy, can lead to moderately increased training complexity compared with simpler baselines.

The improved transferability and robustness of ADGKT may benefit a wide range of remote sensing applications, including environmental monitoring, precision agriculture, mineral exploration, ecological assessment, and disaster management. In particular, the ability to generalize across heterogeneous scenes can support rapid adaptation to newly acquired datasets collected by different sensors or under varying environmental conditions. This capability is especially valuable in time-sensitive applications such as flood monitoring, wildfire assessment, and post-disaster response, where obtaining sufficient labeled samples is often infeasible. From a methodological perspective, the proposed agreement and disagreement mechanisms may also inspire future research on optimization-aware transfer learning and representation diversity in remote sensing and related domains. The integration of gradient alignment, logit normalization, disagreement restriction, and ensemble learning provides a unified perspective for balancing shared transferable knowledge and target-specific information.

Finally, while this work focuses on hyperspectral remote sensing, the underlying principles of balancing agreement and disagreement during knowledge transfer may extend to other machine learning applications involving heterogeneous domains and limited supervision.

Due to the limited revision time, qualitative visualization results, such as classification maps, are not included in this work. We acknowledge that these visualizations would provide additional insight into the spatial consistency and target-specific representation learning achieved by the proposed framework.

6. Conclusions

This paper proposed the agreement–disagreement guided knowledge transfer (ADGKT) framework, which integrates both agreement and disagreement mechanisms to enhance knowledge transfer across cross-scene hyperspectral imaging (HSI). The agreement mechanism, consisting of GradVac and LogitNorm, effectively addresses gradient conflicts and dominating gradients by aligning gradient directions and controlling the magnitude of logits, thereby improving the optimization of shared parameters between source and target scenes. Meanwhile, the disagreement mechanism, incorporating a disagreement restriction (DiR) and an ensemble strategy, captures diverse and independent target representations. This mechanism mitigates the risk of losing critical target information that may be overlooked when focusing solely on agreement between source and target domains. Extensive experiments on multiple cross-scene HSI transfer settings demonstrate that ADGKT consistently achieves state-of-the-art performance and outperforms existing transfer learning, domain adaptation, and ensemble-based approaches. The experimental and ablation results further verify that jointly modeling agreement and disagreement provides complementary benefits, enabling more robust feature transfer and stronger generalization under limited target supervision. In particular, the proposed framework is effective not only in alleviating harmful transfer caused by conflicting optimization objectives, but also in preserving rich target-specific information that is essential for accurate classification in heterogeneous scenes.