HyperKon: A Self-Supervised Contrastive Network for Hyperspectral Image Analysis

Abstract

The use of a pretrained image classification model (trained on cats and dogs, for example) as a perceptual loss function for hyperspectral super-resolution and pansharpening tasks is surprisingly effective. However, RGB-based networks do not take full advantage of the spectral information in hyperspectral data. This inspired the creation of HyperKon, a dedicated hyperspectral Convolutional Neural Network backbone built with self-supervised contrastive representation learning. HyperKon uniquely leverages the high spectral continuity, range, and resolution of hyperspectral data through a spectral attention mechanism. We also perform a thorough ablation study on different kinds of layers, showing their performance in understanding hyperspectral layers. Notably, HyperKon achieves a remarkable 98% Top-1 retrieval accuracy and surpasses traditional RGB-trained backbones in both pansharpening and image classification tasks. These results highlight the potential of hyperspectral-native backbones and herald a paradigm shift in hyperspectral image analysis.

1. Introduction

Hyperspectral images (HSIs), with their ability to capture detailed spectral information across hundreds of contiguous bands, have rapidly advanced the capabilities of remote sensing analysis in various domains, including agriculture, mineralogy, and environmental monitoring [1,2]. These high-dimensional data offer a rich representation of scenes, enabling finer material distinctions than traditional RGB and multispectral images [3]. However, the exploitation of HSIs, particularly using deep learning techniques initially designed for RGB images, presents considerable challenges [4].

Convolutional Neural Networks (CNNs), which dominate the computer vision landscape, are predominantly trained and evaluated on RGB data [5]. These models often struggle to generalize effectively to hyperspectral data due to the vast difference in spectral resolution and the unique characteristics of HSIs [5,6,7]. Moreover, the scarcity of native hyperspectral backbones necessitates extensive fine-tuning for their application in this domain.

Hyperspectral-native CNN architectures, including 3D-CNNs [8], spatial–spectral residual CNNs [9,10], and hybrid dilated convolution networks [11], were developed to effectively process spectral–spatial information in hyperspectral data. These models have made significant progress by utilizing attention-based mechanisms that focus on the most relevant spectral and spatial features, thereby enhancing interpretability and boosting performance in various downstream tasks [12]. Hybrid CNN–Transformer architectures, which combine the strengths of local and global feature analysis through spectral–spatial and self-attention layers [13], have also emerged as powerful tools in this domain. However, despite their promising performance, these hybrid architectures still face challenges when dealing with the high-dimensional nature of hyperspectral data. In contrast, existing CNN layers such as the Squeeze and Excitation Block (SEB) [14] and Convolutional Block Attention Module (CBAM) [15] offer efficient solutions for processing high-dimensional data but have been largely unexplored in the hyperspectral domain. Our work presents an ablation study that demonstrates how these layers not only excel at handling high-dimensional hyperspectral data but also bring the benefits of attention mechanisms into traditional CNN architectures.

Recent advancements in the field have led to the development of Remote Sensing Foundation Models (RSFMs), which represent significant progress in the processing and analysis of remote sensing data, including HSIs. These models, drawing inspiration from the achievements of large language models in natural language processing, aim to provide a flexible and reliable foundation for a range of remote sensing applications. One notable example is SpectralGPT [16], which, although trained on multispectral images with 12 channels, adapts the generative pretrained transformer (GPT) architecture for spectral data analysis. SpectralGPT’s training on only 12 bands likely reflects the challenges of scaling transformer architectures to the hundreds of channels typically found in HSIs. Scaling transformers to such a large number of channels leads to a quadratic increase in the model’s number of parameters, making them computationally expensive and challenging to implement effectively in hyperspectral contexts. In contrast, modern attention mechanisms that operate within CNN architectures, such as the SEB, offer similar advantages of attention found in transformers but with a significantly smaller number of parameters.

The emergence of self-supervised learning in remote sensing, exemplified by models like SeCo [17], has opened new avenues for leveraging large amounts of unlabeled satellite imagery. These approaches enable the learning of rich, transferable representations from diverse remote sensing data sources, potentially improving the performance of downstream tasks in HSI analysis. Furthermore, the development of multimodal foundation models for remote sensing, such as the works by [18,19,20], demonstrates the potential for integrating various data sources to create more comprehensive and robust models for Earth observation tasks.

Self-supervised contrastive learning has emerged as a powerful paradigm in remote sensing, addressing the challenge of limited labeled data. Recent works have adapted contrastive learning frameworks for remote sensing applications [21], demonstrating improved performance in land cover change detection tasks. In the hyperspectral domain, approaches like [22,23] have shown promising results in learning spectral–spatial features without relying on labeled data. The integration of contrastive learning with attention mechanisms [24] has further improved the ability to capture long-range dependencies in hyperspectral data, leading to more robust models.

Despite these advances, several challenges persist in HSI analysis. The high dimensionality and distinct properties of hyperspectral data continue to pose computational challenges, particularly in resource-constrained environments like onboard satellite systems. The increasing demand for real-time remote sensing data analysis, as exemplified by missions like Intuition-1 [25], necessitates the development of efficient, lightweight models capable of processing hyperspectral data in orbit. To address the constraints and scarcity of hyperspectral-native solutions, we introduce HyperKon, a self-supervised contrastive network trained solely on HSIs from Environmental Mapping and Analysis Program (EnMAP) [26]. HyperKon, unlike generic CNN backbones and newer RSFMs, is trained on HSI with 224 channels while maintaining a compact architecture.

The main contributions of this paper are:

• HyperKon: a hyperspectral-native CNN backbone that can learn useful representations from large amounts of unlabeled data.
• EnHyperSet-1: an EnMAP dataset curated for use in precision agriculture and other deep learning projects.
• Hyperspectral perceptual loss: a novel perceptual loss function minimizing errors in the spectral domain.
• Demonstration that the representations learned by HyperKon improve performance in hyperspectral downstream tasks.

The remainder of this paper is organized as follows. Section 2 describes the materials and methodology, including the HyperKon architecture and contrastive learning approach. Section 3 presents experimental results and evaluations. Section 4 offers discussions on the findings, and Section 5 concludes the paper with future research directions.

2. Materials and Methods

2.1. Dataset

This study introduces EnHyperSet-1, a hyperspectral dataset curated from the EnMAP mission [26] for deep learning applications. The dataset comprises 800 scenes (200 Level 1B, 200 Level 1C, 400 Level 2A), each with an average pixel size of 1300 × 1200 and 224 spectral bands ranging from 420 nm to 2450 nm. Each 30 m × 30 m pixel captures spectral information at a resolution of 6.5 nm to 12 nm. EnHyperSet-1 features diverse global urban, forest, and agricultural scenes, as summarized in

Table 1. A comparison to popular hyperspectral datasets *.

Dataset	Number of Bands	Size	Spectral Range	Number of Images	Spatial Resolution	Imaging Location	Platform Type
Indian Pines	200	145 × 145	400–2500 nm	1	30 m	Indiana, USA	Airborne
Pavia Centre	102	1096 × 1096	430–860 nm	1	1.3 m	Pavia, Italy	Airborne
Salinas	204	512 × 217	360–2500 nm	1	3.7 m	Salinas Valley, CA, USA	Airborne
Harvard	31	1392 × 1040	420–720 nm	50	-	Harvard, USA	Airborne
Botswana	145	1476 × 256	400–2500 nm	1	30 m	Botswana	Airborne
Chikusei	100	2517 × 2335	263–1018 nm	1	2.5 m	Chikusei, Japan	Airborne
EnHyperSet-1	224	1300 × 1200	420–2450 nm	800	30 m	Global, on demand	Spaceborne

* Best values are in bold.

. For analysis, we extracted 160 × 160 pixel patches using a sliding window with a 5% overlap buffer, with edge patches zero-padded when necessary. The choice of 160 × 160 pixel patches was driven by the need to maintain consistency with existing networks and backbones used in hyperspectral image processing, such as hyperspectral pansharpening [27], while also balancing spatial detail and computational efficiency. Although the dataset consisted of data from different processing levels, these levels were not conflated or used as one. Instead, they were strategically leveraged for contrastive learning, enabling the network to learn more robust representations. A detailed explanation of how these different processing levels were utilized in the contrastive learning process is provided in Section 2.3.2.

HyperKon was pretrained on EnHyperSet-1 using self-supervised contrastive learning with the Normalized Temperature-Scaled Cross Entropy (NT-Xent) loss [28]. The network was trained for 1000 epochs with a batch size of 32, using the Adam optimizer (initial learning rate: 1 × 10⁻⁴) and a StepLR scheduler. This methodology enabled effective representation learning from unlabeled hyperspectral data while addressing the unique challenges of different processing levels.

2.2. Hyperspectral Backbone Architecture

The HyperKon architecture (Figure 1) is based on ResNeXt’s multibranch cardinality [29] and employs parallel paths to improve representational and computational efficiency. The ResNeXt architecture has fewer parameters, making it less sensitive to learning rates and other hyperparameters, especially when compared to its predecessor, ResNet [30].

Developing an effective feature extractor for this network required overcoming two main challenges: capturing the complex, high-dimensional spectral and spatial features of hyperspectral data and addressing the computational constraints associated with such high-dimensional data. In our model, we fine-tuned the feature maps to better reflect channel interdependencies using a specialized architecture based on the SEB [14]. This approach processed an input feature map $X\in {\mathbb{R}}^{H\times W\times C}$ to calculate channelwise statistics, allowing the model to more effectively highlight relevant spectral and spatial details within the hyperspectral data. This targeted recalibration ensured the network’s computations were directly aligned with the critical features of the data, leading to more efficient and accurate analysis.

The excitation operation then used a gating mechanism:

$$\begin{array}{cc}\hfill s_c& ={\displaystyle \frac{1}{H\times W}}{\displaystyle \sum _{i=1}^H}{\displaystyle \sum _{j=1}^W}{x}_{ijc},\hfill \\ \hfill e_c& =\sigma (\mathit{\beta }·s_c+\gamma ),\hfill \end{array}$$

(1)

where H and W represent the height and width of the input feature map, respectively. The term $x_{ijc}$ represents the value of the feature map X at spatial position $(i,j)$ in channel c. Here, $\sigma$ is the sigmoid activation function, and $\mathit{\beta }$ and $\gamma$ are trainable parameters. The recalibrated feature map ${\mathbf{X}}^{\prime }\in {\mathbb{R}}^{H\times W\times C}$ was then given by:

$$x_{ijc}^{\prime }=e_c·x_{ijc}$$

(2)

This recalibration emphasized channels with high interdependencies and suppresses others. The feature extractor provided preliminary feature representations which, while powerful, might not be fine-tuned to the specific challenges presented by hyperspectral data. To address this, we introduced a projection head. Given the deep feature representation $\mathbf{F}\in {\mathbb{R}}^D$ from the feature extractor, the projection head transformed it into a new space $\mathbf{Z}\in {\mathbb{R}}^{D^{\prime }}$:

$$\mathbf{Z}={\mathbf{W}}_2\sigma ({\mathbf{W}}_1\mathbf{F}+{\mathbf{b}}_1)+{\mathbf{b}}_2$$

(3)

where ${\mathbf{W}}_1\in {\mathbb{R}}^{D^{\prime }\times D}$, ${\mathbf{W}}_2\in {\mathbb{R}}^{D^{\prime }\times D^{\prime }}$, ${\mathbf{b}}_1\in {\mathbb{R}}^{D^{\prime }}$, and ${\mathbf{b}}_2\in {\mathbb{R}}^{D^{\prime }}$ are trainable parameters, and $\sigma$ is a non-linear activation function. This transformation aided in emphasizing discriminative features for hyperspectral data.

2.3. Contrastive Learning

2.3.1. Self-Supervised Contrastive Loss

While Triplet loss [31] and InfoNCE [32] can be effective, a careful selection of samples is especially challenging for hyperspectral data. The NT-Xent loss [28], which builds upon the InfoNCE concept with a softmax output layer, incorporates an L2 normalization of embeddings and typically functions as a symmetric loss. It encourages the network to discriminate between augmented versions (positive pairs) of the same hyperspectral sample and dissimilar samples (negative pairs) within a mini-batch. The primary motivation for selecting NT-Xent loss is its efficiency in contexts where generating explicit negative samples for HSIs is challenging. Positive pairs are typically generated through data augmentation techniques like random spectral scaling, spatial cropping, and random channel permutations [33]. While methods like MoCo [34] and SwAV [35] introduce momentum queues and memory banks to maintain a large set of negatives, NT-Xent relies solely on the current mini-batch for negative sampling. This computational efficiency is particularly beneficial for training on hyperspectral data, which often have high dimensionality due to the large number of spectral bands (≥200 bands). Furthermore, it is adaptable to varying batch sizes, uses cosine similarity for feature vector orientation, and incorporates a temperature parameter that controls the influence of the discrimination task on the learning process [36].

The NT-Xent loss is defined as:

$${\ell }_{({\mathbf{z}}_i,{\mathbf{z}}_j)}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}log{\displaystyle \frac{exp(sim({\mathbf{z}}_i,{\mathbf{z}}_{i^{\prime }})/\tau )}{{\sum }_{j=1}^{2N}exp(sim({\mathbf{z}}_i,{\mathbf{z}}_j)/\tau )}}$$

(4)

where N is the number of samples, ${\mathbf{z}}_i$ is the feature representation of sample i, $sim({\mathbf{z}}_i,{\mathbf{z}}_j)={\mathbf{z}}_i^T{\mathbf{z}}_j$ is the cosine similarity between ${\mathbf{z}}_i$ and ${\mathbf{z}}_j$, $\tau$ is a temperature parameter that controls the smoothness of the distribution, and ${\mathbf{z}}_{i^{\prime }}$ refers to the positive feature representation for the query sample i.

2.3.2. HSI Contrastive Sampling

Self-supervised contrastive learning relies heavily on data augmentation to generate informative positive and negative sample pairs for training [37]. In our approach, we leveraged the unique characteristics of hyperspectral data by utilizing different processing levels (e.g., 1B, 1C, 2A) to create naturally augmented samples before applying traditional transformations.

Let us denote ${\mathbf{x}}_i^l\in I_k^h$ as a sequence of patches $\left({\mathbf{x}}_i^l\right)$ taken from a series of HSIs $(I_k^h\forall k=0\dots S)$, where S represents the dataset size and l represents the processing level. For each anchor patch ${\mathbf{x}}_i^{l_1}$, we selected its positive pair ${\mathbf{x}}_i^{l_2}$ from the same spatial region but in a different processing level. This approach ensured that positive pairs maintained spatial content while introducing processing-level variations.

These patches then underwent a series of spectral and spatial transformations denoted by $A=a_1,a_2\dots a_n$, which were enacted on the patches ${\mathbf{x}}_i^l$ as follows:

$${{\mathbf{x}}_i^l}^{\prime }=a_n(\dots a_2\left(a_1\left({\mathbf{x}}_i^l\right)\right)\dots )$$

(5)

To further enhance the learning process, we leveraged the concept of hard negative mining during triplet selection [38]. This strategy focused on choosing negative pairs within the same batch that were most similar to the anchor patch. Selecting such “hard negatives” forced the model to learn more discriminative features that distinguish between even subtle spectral variations, which is crucial for HSI data due to their high dimensionality and potential spectral redundancy.

Mathematically, we used an encoder function f to map each patch $\mathbf{x}$ to its corresponding embedding $\mathbf{z}=f\left(\mathbf{x}\right)$. The cosine similarity function $s({\mathbf{z}}_i,{\mathbf{z}}_j)$ then measured the similarity between embeddings ${\mathbf{z}}_i$ and ${\mathbf{z}}_j$ for a batch of patches. During hard negative mining, for a given anchor patch ${\mathbf{x}}^a$ and its positive pair ${\mathbf{x}}^{+}$, the hardest negative patch ${\mathbf{x}}^{-}$ was selected as:

$${\mathbf{x}}^{-}=\underset{{\mathbf{x}}_k\in \{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_B\}\setminus \{{\mathbf{x}}^a,{\mathbf{x}}^{+}\}}{\mathrm{argmax}}s(f\left({\mathbf{x}}^a\right),f\left({\mathbf{x}}_k\right))$$

(6)

This means the patch $x_k$ that gave the highest similarity score with the anchor patch $x^a$ was chosen from all patches in the batch, with the exception of the positive pair $x^{+}$ and the anchor patch $x^a$ itself (as depicted in Figure 2). By combining the use of different processing levels with hard negative mining and traditional augmentations, our approach emphasized the importance of targeted data augmentation in HSIs. This method leveraged the advantages of self-supervised learning, improving task performance by learning detailed representations from large, unlabeled datasets while preserving the unique spectral characteristics of hyperspectral data.

2.4. Hyperspectral Perceptual Loss

The HyperSpectral Perceptual Loss (HSPL) function is designed to quantify the spectral differences between a predicted hyperspectral image $\left(\widehat{I}\right)$ and its reference $\left(I_{ref}\right)$ in the feature embedding space. Unlike traditional loss functions that operate directly on pixel values, the HSPL leveraged the rich spectral information captured by our pretrained HyperKon network. This approach allowed for a more nuanced comparison of hyperspectral data, capturing complex spectral relationships that may not have been evident in simple pixelwise comparisons.

The HSPL is computed as follows:

$${\ell }_{hspl}={\displaystyle \sum _{l=1}^N}{\displaystyle \frac{1}{C_l{H}_l{W}_l}}{\left|\left|f^h(I_{ref},l)-f^h(\widehat{I},l)\right|\right|}_F$$

(7)

where $f^h(·,l)$ represents the feature maps extracted by the lth layer of the pretrained network, N is the total number of layers considered, and $C_l$, $H_l$, and $W_l$ are the number of channels, height, and width of the feature maps at layer l, respectively. The Frobenius norm ${\parallel ·\parallel }_F$ quantifies the difference between the feature maps, effectively capturing the perceptual difference between the images in the hyperspectral domain.

Figure 3 illustrates the conceptual difference between the HSPL and traditional RGB-based perceptual losses. While RGB losses are limited to three broad spectral bands, the HSPL leverages the full spectral range of hyperspectral data, providing a more comprehensive measure of spectral similarity.

In terms of computational complexity, the HSPL does incur additional overhead compared to simpler pixelwise losses due to the need to compute feature maps across multiple layers. However, this increased complexity is offset by the improved performance and more meaningful loss calculations, especially for tasks that require accurate preservation of spectral characteristics.

3. Results

3.1. Ablation Study

We evaluated multiple versions of the model, each integrating different architectural components, to see how well each component of our HyperKon architecture performed and how it impacted the network’s capacity to learn meaningful representations from hyperspectral data.

Figure 4 presents the results of this ablation study, showing the Top-1 HSI retrieval accuracy for each model variant during the pretraining phase. We experimented with several key components: 3D convolutions to capture spatial–spectral relationships, depthwise separable convolutions (Depthwise Separable Convolutions (DSC)) for efficient feature extraction, convolutional block attention module (CBAM) to enhance feature refinement, and squeeze and excitation block (SEB) for adaptive feature recalibration. The results clearly demonstrated the impact of each component on the model’s performance. Notably, the version incorporating the SEB achieved the highest accuracy, surpassing other configurations. This suggested that the adaptive feature recalibration provided by the SEB was particularly effective in capturing the complex spectral–spatial relationships in hyperspectral data.

Band Attention

We investigated various dimensionality reduction techniques commonly applied to hyperspectral data [39,40,41], including Principal Component Analysis (PCA), manual, and full band selection. These techniques are crucial in HSI processing for streamlining computation and refining feature extraction by discarding redundant information.

As shown in Figure 5, the 3D convolution model outperformed both the SEB and CBAM models when dimensionality was reduced using PCA. While the SEB and CBAM generally excelled at selecting and prioritizing information across the entire spectrum of hyperspectral bands, their advantage diminishes when PCA was applied, potentially rendering their roles redundant. In contrast, 3D convolutions demonstrated superior performance when band selection was conducted a priori using PCA with 112 components, likely because 3D convolutions can effectively learn from both contiguous and non-contiguous spectral channels. This flexibility allowed 3D convolutions to adapt to the structure of the data, regardless of the continuity of the spectral bands.

As suggested in Figure 6, manual band selection gave 3D convolutions a distinct advantage. However, most notably, the SEB and CBAM clearly surpassed 3D convolution when no band selection was carried out. This observation supported our hypothesis that when band selection was not performed a priori, attention mechanisms such as the SEB and CBAM had the ability to focus on the most relevant spectral bands across the entire channel spectrum. This clearly demonstrated their robustness and ability to manage the high dimensionality common with hyperspectral data.

3.2. Super-Resolution

For the super-resolution evaluation, we utilized datasets such as the Pavia Center [42], Botswana [43], Chikusei [44], and EnMAP [26] datasets, each featuring different spectral bands. Specific bands from the EnHyperSet-1 dataset were selected to match the wavelength of each dataset, optimizing the training of HyperKon. The data preparation adhered to methodologies outlined in [27,45]. The results from HyperKon were benchmarked against traditional RGB-trained backbones using metrics [46,47,48] such as the Correlation Coefficient (CC), Spectral Angle Mapper (SAM), Root-Mean-Square Error (RMSE), Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS), and Peak Signal-to-Noise Ratio (PSNR).

Table 2 shows the average quantitative results for the Pavia Center dataset. HyperKon achieved superior performance across multiple metrics compared to other methods, demonstrating its effectiveness in hyperspectral super-resolution. A broader comparison of the average quantitative pansharpening results for RGB-native vs. HSI-native perceptual loss across multiple datasets is presented in Table 3. HyperKon consistently achieved the best results, indicating the advantages of a hyperspectral-native approach. Figure 7 provides visual results generated by different pansharpening algorithms, including HyperKon, for the Pavia Center, Botswana, Chikusei, and EnMAP datasets. Mean Absolute Error (MAE) heatmaps show that HyperKon had the lowest error across all spectral bands, further highlighting its superior performance.

Figure 7. Visual results generated by different pansharpening algorithms (HyperPNN [49], DARN [45], GPPNN [50], HyperTransformer [51], HyperKon (ours), and ground truth) for Pavia Center [42], Botswana [43], Chikusei [44], and EnMAP [26] datasets. MAE denotes the (normalized) Mean Absolute Error across all spectral bands.

Figure 7. Visual results generated by different pansharpening algorithms (HyperPNN [49], DARN [45], GPPNN [50], HyperTransformer [51], HyperKon (ours), and ground truth) for Pavia Center [42], Botswana [43], Chikusei [44], and EnMAP [26] datasets. MAE denotes the (normalized) Mean Absolute Error across all spectral bands.

Table 2. Average quantitative results: Pavia Center [42] *.

	Pavia Center Dataset
Method	CC↑	SAM↓	RMSE↓	ERGAS↓	PSNR↑
HySure [52]	0.966	6.13	1.8	3.77	35.91
HyperPNN [49]	0.967	6.09	1.67	3.82	36.7
PanNet [53]	0.968	6.36	1.83	3.89	35.61
Darn [45]	0.969	6.43	1.56	3.95	37.3
HyperKite [27]	0.98	5.61	1.29	2.85	38.65
SIPSA [54]	0.948	5.27	2.38	4.52	33.65
GPPNN [50]	0.963	6.52	1.91	4.05	35.36
HyperTransformer [51]	0.9881	4.1494	0.9862	0.5346	40.9525
HyperKon	0.9883	3.9551	0.9369	0.5152	41.9808

* Best values are in bold, 2nd best values are underlined. RMSE values are $\times {10}^{-2}$. ↑ Means higher value is better. ↓ Means smaller value is better.

Table 2. Average quantitative results: Pavia Center [42] *.

	Pavia Center Dataset
Method	CC↑	SAM↓	RMSE↓	ERGAS↓	PSNR↑
HySure [52]	0.966	6.13	1.8	3.77	35.91
HyperPNN [49]	0.967	6.09	1.67	3.82	36.7
PanNet [53]	0.968	6.36	1.83	3.89	35.61
Darn [45]	0.969	6.43	1.56	3.95	37.3
HyperKite [27]	0.98	5.61	1.29	2.85	38.65
SIPSA [54]	0.948	5.27	2.38	4.52	33.65
GPPNN [50]	0.963	6.52	1.91	4.05	35.36
HyperTransformer [51]	0.9881	4.1494	0.9862	0.5346	40.9525
HyperKon	0.9883	3.9551	0.9369	0.5152	41.9808

* Best values are in bold, 2nd best values are underlined. RMSE values are $\times {10}^{-2}$. ↑ Means higher value is better. ↓ Means smaller value is better.

Table 3. Comparison of the average quantitative pansharpening results for RGB-native vs. HSI-native perceptual loss *.

	Botswana Dataset		Chikusei Dataset		Pavia Dataset
Metric	RGB-Native	HS-Native	RGB-Native	HS-Native	RGB-Native	HS-Native
CC↑	0.9104	0.9411	0.9801	0.9777	0.9881	0.9883
SAM↓	3.1459	2.5798	2.2547	2.4192	4.1494	3.9551
RMSE↓	0.0233	0.0193	0.0123	0.0131	0.0098	0.0093
ERGAS↓	0.6753	0.5249	0.8662	0.9193	0.5346	0.5152
PSNR↑	27.3925	29.4128	36.8861	36.2889	40.9525	41.9808

* Best values are in bold. ↑ Means higher value is better. ↓ Means smaller value is better.

Table 3. Comparison of the average quantitative pansharpening results for RGB-native vs. HSI-native perceptual loss *.

	Botswana Dataset		Chikusei Dataset		Pavia Dataset
Metric	RGB-Native	HS-Native	RGB-Native	HS-Native	RGB-Native	HS-Native
CC↑	0.9104	0.9411	0.9801	0.9777	0.9881	0.9883
SAM↓	3.1459	2.5798	2.2547	2.4192	4.1494	3.9551
RMSE↓	0.0233	0.0193	0.0123	0.0131	0.0098	0.0093
ERGAS↓	0.6753	0.5249	0.8662	0.9193	0.5346	0.5152
PSNR↑	27.3925	29.4128	36.8861	36.2889	40.9525	41.9808

* Best values are in bold. ↑ Means higher value is better. ↓ Means smaller value is better.

3.3. Transfer Learning Capability of HyperKon

In addition to the super-resolution task, the performance of the HyperKon network was evaluated on the HSI classification task using the Indian Pines, Pavia University, and Salinas Scene datasets [55]. These datasets, covering a wide range of crops, serve as an excellent benchmark for assessing the accuracy and proficiency of the HyperKon network in classifying different crops. The frozen backbone of the HyperKon network, which had been pretrained on a variety of hyperspectral data, was utilized for this task. For these transfer learning experiments, we employed a patch size of 25 × 25 pixels, which provided a suitable balance between spatial context and computational efficiency for the downstream classification task.

The results were juxtaposed with those of established methods such as SSAN [56], SSRN [57], RvT [58], HiT [59], SSFTT [60], and QSSPN [61]. The assessment utilized overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa) metrics. As shown in

Table 4. Comparative analysis of hyperspectral image classification methods’ performance metrics and computational requirements for various hyperspectral image classification methods across three datasets *.

Datasets	Metrics	SSAN [56]	SSRN [57]	RvT [58]	HiT [59]	SSFTT [60]	QSSPN-3 [61]	HyperKon
IP	OA(%)	89.46	91.85	83.85	90.59	96.35	95.87	98.77
	AA(%)	85.99	81.51	79.67	86.71	89.99	96.40	97.82
	Kappa(%)	88.04	90.73	81.68	89.27	95.82	95.34	98.60
	Params.	148.83 K	735.88 K	10.78 M	49.60 M	148.50 K	910.50 K	5.54 M
	FLOPs	7.88 M	212.48 M	17.83 M	345.88 M	3.66 M	34.54 M	1.27 G
PU	OA(%)	99.15	99.63	97.37	99.43	99.52	99.71	99.89
	AA(%)	98.70	99.29	95.86	99.09	99.20	99.43	99.76
	Kappa(%)	98.87	99.51	96.52	99.24	99.36	99.61	99.86
	Params.	94.63 K	396.99 K	9.77 M	42.41 M	148.03 K	609.16 K	4.08 M
	FLOPs	5.57 M	108.04 M	16.83 M	190.85 M	3.66 M	10.25 M	370.59 M
SA	OA(%)	98.92	99.31	98.11	99.38	99.53	99.66	100.00
	AA(%)	99.33	99.70	98.83	99.70	99.72	99.81	100.00
	Kappa(%)	98.80	99.23	97.90	99.31	99.47	99.63	100.00
	Params.	149.71 K	750 K	10.82 M	50 M	148.50 K	926.90 K	5.62 M
	FLOPs	7.97 M	216.84	17.80 M	354.42 M	3.66 M	35.87 M	1.32 G

* Best values are in bold, 2nd best values are underlined.

, HyperKon consistently matched or surpassed other networks, highlighting its robustness and adaptability to new data. The high accuracy demonstrated how the model could use discriminative features from its self-supervised learning phase to provide predictions for HSI classification that were reliable. The qualitative outcomes of the HSI classification can be viewed in Figure 8, and Figure 9 shows a zoomed-out prediction accuracy map for Indian Pines.

3.4. Model Efficiency Analysis

As shown in Table 4, HyperKon demonstrated higher classification accuracy across the three datasets, consistently outperforming other methods in overall accuracy, average accuracy, and Kappa coefficient. However, this performance came at a computational cost. HyperKon had a relatively high parameter count (5.54 M for Indian Pines, 4.08 M for Pavia University, 5.62 M for Salinas Scene) compared to more lightweight models like SSAN [56] and SSFTT [60]. Its FLOPs were also higher, ranging from 370.59 M to 1.32 G. Despite this, as evident from

Table 5. Classwise accuracies and performance metrics for the HyperKon model on Indian Pines, Pavia University, and Salinas datasets.

Indian Pines		Pavia University		Salinas
Class	Acc. (%)	Class	Acc. (%)	Class	Acc. (%)
Alfalfa	100.00	Asphalt	100.00	Broccoli_green_weeds_1	100.00
Corn-notill	99.72	Meadows	100.00	Broccoli_green_weeds_2	100.00
Corn-mintill	99.16	Gravel	100.00	Fallow	100.00
Corn	100.00	Trees	99.87	Fallow_rough_plow	100.00
Grass-pasture	99.59	Painted metal sheets	100.00	Fallow_smooth	100.00
Grass-trees	100.00	Bare Soil	100.00	Stubble	100.00
Grass-pasture-mowed	100.00	Bitumen	99.85	Celery	100.00
Hay-windrowed	100.00	Self-blocking bricks	99.95	Grapes_untrained	100.00
Oats	95.00	Shadows	99.58	Soil_vinyard_develop	100.00
Soybean-notill	99.69			Corn_senesced_green	100.00
Soybean-mintill	99.80			Lettuce_romaine_4wk	100.00
Soybean-clean	99.49			Lettuce_romaine_5wk	100.00
Wheat	100.00			Lettuce_romaine_6wk	100.00
Woods	99.29			Lettuce_romaine_7wk	100.00
Buildings-grass-trees-drives	100.00			Vinyard_untrained	100.00
Stone-steel-towers	100.00			Vinyard_vertical	100.00
Performance Metrics
Total training time (s)	2346.18	Total training time (s)	9251.00	Total training time (s)	12,172.18
Total test time (s)	52.07	Total test time (s)	513.87	Total test time (s)	294.16
Avg. inference time (s)	0.0009	Avg. inference time (s)	0.0009	Avg. inference time (s)	0.0010
Throughput (samples/s)	1088.39	Throughput (samples/s)	1078.45	Throughput (samples/s)	1038.11

, HyperKon achieved near-perfect classification accuracy for most classes across all three datasets, with only a few classes in Indian Pines falling slightly below 100%. The model’s inference times were consistently low (around 0.0009 s per sample) across datasets, resulting in high throughput (1038–1088 samples/second). All experiments were carried out in a PyTorch [62] environment using an NVIDIA RTX 3090 GPU with 24 GB of memory.

4. Discussion

4.1. Architectural Considerations

The ablation study results (Figure 4) highlight the importance of carefully selecting architectural components for hyperspectral data processing. The superior performance of the SEB over other configurations suggests that adaptive feature recalibration is particularly effective for capturing the complex spectral–spatial relationships in HSIs.

4.2. Self-Supervised Learning for Hyperspectral Data

The success of HyperKon in learning meaningful representations from unlabeled hyperspectral data emphasizes the potential of self-supervised learning approaches in remote sensing. By achieving high Top-1 retrieval accuracy during pretraining (Figure 4), our model demonstrated its ability to capture important spectral–spatial features without the need for extensive labeled datasets. This is particularly valuable in the hyperspectral domain, where labeled data are often scarce and expensive to obtain [63,64].

4.3. Transfer Learning and Downstream Task Performance

The strong performance of HyperKon in downstream tasks, particularly in hyperspectral super-resolution (Table 2 and Table 3) and image classification (Table 4), demonstrates the transferability of the learned representations. This is a crucial finding, as it suggests that self-supervised pretraining on diverse hyperspectral data can lead to robust features that generalize well to specific tasks and datasets. The superior performance of HyperKon compared to RGB-native approaches in pansharpening tasks (Table 3) highlights the benefits of developing hyperspectral-native models. This aligns with previous hypotheses suggesting that models designed for RGB imagery may not fully exploit the rich spectral information available in hyperspectral data.

4.4. Hyperspectral Perceptual Loss

The introduction of the HSPL represents a novel contribution to the field of HSI processing. By focusing on spectral differences across all bands, rather than just pixelwise differences, the HSPL provides a more comprehensive measure of similarity for HSIs. The improved performance observed when using the HSPL (Figure 3) suggests that this approach could be valuable for a range of HSI processing tasks beyond super-resolution, such as image fusion or denoising [65].

4.5. Limitations and Future Work

While HyperKon demonstrates excellent performance across various tasks, there are several areas for potential improvement and future research. Future work could explore techniques such as network pruning or quantization to reduce computational requirements without sacrificing performance [66]. While HyperKon focuses on hyperspectral data, future research could explore integrating information from other sensor modalities, such as LiDAR or SAR, to create more comprehensive representations of Earth observation data. The current model does not explicitly account for temporal changes in hyperspectral imagery. Incorporating temporal information could enhance the model’s ability to capture dynamic processes such as vegetation phenology.

5. Conclusions

This study presented HyperKon, a self-supervised contrastive network developed for HSI analysis. HyperKon uses a unique hyperspectral-native convolutional architecture and the novel HSPL function to enhance performance in hyperspectral super-resolution and classification applications. The experimental results suggested that HyperKon outperformed traditional RGB-trained models and other state-of-the-art approaches, demonstrating its ability to preserve spectral integrity and capture complex spectral–spatial relationships. The successful application of self-supervised contrastive learning allowed for robust feature extraction from large volumes of unlabeled hyperspectral data.

The creation of the EnHyperSet-1 dataset, a comprehensive collection of high-resolution HSIs, contributed to advance research in this field. The dataset supports the development of models like HyperKon, which are capable of handling the high dimensionality and unique characteristics of hyperspectral data. Future research directions will involve integrating multi-modal data, improving model interpretability, and optimizing training and inference pipelines for resource-constrained environments. These efforts will ensure that HyperKon remains a valuable tool in the constantly evolving field of remote sensing.