SASR: Sensor-Agnostic Semantic Representation Unification for Cross-Modal RGB and Hyperspectral Aerial Scene Recognition

Highlights

What are the main findings?

• A novel neuro-inspired sensor-agnostic semantic representation framework is proposed to unify RGB and hyperspectral aerial scene recognition without requiring paired modalities or explicit feature fusion.
• The proposed model achieves strong performance with 96.4 (RGB) and 97.3% (HSI) accuracy, and cross-modality transfer of 87.2% (RGB → HSI) and 88.7% (HSI → RGB).

What are the implications of the main findings?

• Semantic representation unification enables robust cross-sensor knowledge transfer and reduces the modality gap between heterogeneous remote sensing data.
• The framework enables scalable deployment in real-world Earth observation systems where only a single sensing modality is available.

Abstract

Aerial scene recognition has progressed substantially with deep learning methods for RGB and hyperspectral imagery; however, existing approaches typically operate on single modalities or rely on explicit multimodal fusion, limiting scalability, flexibility, and deployment in heterogeneous sensing environments. To address this limitation, we propose a sensor-agnostic semantic representation learning framework that formulates multimodal learning as the unification of semantic representations rather than feature-level fusion. The proposed architecture employs modality-specific encoders and projection heads to map spatial and spectral–spatial features into a shared semantic embedding space, enabling modality-invariant representation learning while preserving discriminative characteristics of each sensing modality. A composite objective integrating cross-spectral alignment, intra-class compactness regularization, and prototype-based semantic anchoring is introduced to enforce consistent embedding geometry and improve class separability across modalities. A unified classifier operating within this shared space enables reliable inference from a single modality input without requiring paired data or explicit fusion. Extensive evaluations on multiple benchmark datasets, including Houston 2013 for cross-modality RGB–hyperspectral analysis, UC Merced for independent RGB aerial scene classification, and Indian Pines for hyperspectral land-cover recognition, demonstrate the robustness and generalization capability of the proposed framework. In Houston 2013, the method achieves 96.4% (RGB) and 97.3% (hyperspectral) overall accuracy, with cross-modality transfer performance of 87.2% (RGB → HSI) and 88.7% (HSI → RGB), further improving to 97.0% and 97.8% under joint training. On UC Merced and Indian Pines, the model attains 98.7% and 97.6% overall accuracy, respectively. These results establish semantic representation unification as a scalable and effective alternative to conventional multimodal fusion for heterogeneous remote sensing environments.

1. Introduction

Aerial scene recognition is a fundamental task in remote sensing that assigns semantic labels to overhead images. It plays an important role in a wide range of applications, including land-use mapping, environmental monitoring, precision agriculture, and urban planning. With the rapid growth of Earth observation platforms and sensing technologies, deep learning has become the dominant paradigm for aerial scene understanding. Earlier studies moved from handcrafted descriptors to deep learning-based pipelines, while more recent advances have been driven by convolutional neural networks and transformer-based architectures that learn rich spatial structure, texture, and contextual cues from high-resolution imagery [1]. At the same time, practical deployment has made computational efficiency increasingly important, motivating the development of lightweight and resource-aware aerial scene classification models that reduce cost while maintaining strong performance [2].

In parallel with these efforts, aerial scene recognition has evolved from a purely supervised learning approach to transferable and data-efficient representation learning. In particular, self-supervised masked modeling, which is capable of learning representations from unlabeled Earth observation data, has become an indispensable direction. For example, masked autoencoding for multispectral and temporal satellite imagery obtains outstanding performance with large gains on downstream tasks by using domain structure, rather than simply color appearance on the RGB stream [3]. Data efficiency and benchmark-driven evaluation are another prerequisite for fair comparison and reproducible progress, in which GEO-Bench provides a general benchmarking platform for a collection of Earth observation tasks and forms a solid evaluation ground for Earth observation representation learning [4].

Compared with standard RGB images, HSI obtains a denser spectro-analysis by taking continuous band measurements, which enables the obtainment of better material characteristics and better class distinction. However, learning with hyperspectral data is still problematic due to the following factors: high dimensions of HSI, lack of labeled samples, and sensitivity to the acquisition system (e.g., sensor characteristics). Based on these problems, self-supervised or masked modeling approaches which capture spatial–spectral information simultaneously have been actively investigated and proved to generate better representations even under the low-label scenario [5,6]. And there have been some works which specifically explore better spectral utilization in the area of hyperspectral learning: SpectralMAE by masking the spectrum for reconstruction [7] and contrastive masked autoencoders learning discriminative spectral–spatial features for hyperspectral classification [8]. More recent work has tried to further improve the representation by using multiscale hierarchical approaches or graph modeling techniques, e.g., SSHMT-MAE, self-supervised graph masked autoencoders and graph-based masked autoencoding for anomaly detection [9,10,11]. These achievements further illustrate the key role of the strong hyperspectral representation learning in enhancing remote sensing recognition under low-label or domain variant conditions.

At a broader level, remote sensing research is rapidly moving toward foundation-model-style pretraining and multimodal representation learning. Inspired by the success of large-scale models in computer vision and natural language processing, recent remote sensing foundation models aim to learn generalizable features that transfer effectively across datasets and downstream tasks. Vision–language foundation models have shown that aligning remote sensing visual features with semantic language supervision can significantly improve transferability and downstream performance [12]. Multimodal self-supervised frameworks based on contrastive learning and masked autoencoding have also demonstrated that cross-modal pretraining can yield representations that generalize across sensing conditions and datasets [13]. Spectral-specialized foundation models further highlight the importance of architectures designed explicitly for large-scale spectral–spatial modeling [14].

This trend is also reflected in geospatial foundation modeling more generally. Scaling studies show that larger vision transformer models can improve performance and data efficiency on remote sensing tasks [15]. Evaluation frameworks such as PhilEO Bench provide diverse benchmarks to test geospatial foundation models under multiple supervision settings [16]. Recent models such as Prithvi-EO-2.0 further extend this paradigm to multi-temporal and global-scale representation learning by incorporating temporal and location embeddings [17], while independent evaluations reveal both the promise and the limitations of these models under domain shift [18]. Related remote sensing foundation models, including RingMo-Sense and large-scale pretraining resources such as SatlasPretrain, similarly aim to improve robustness and downstream generalization across varied Earth observation tasks [19,20].

Despite these advances, most existing multimodal aerial scene recognition methods still rely on feature-level fusion or direct modality concatenation and usually assume that the input data are paired and co-registered. In real Earth observation scenarios, however, data are often collected by different sensors, at different times, under different spatial resolutions and spectral responses. As a result, multimodal observations are frequently unpaired, only partially overlapping, or affected by sensor mismatch. Under such conditions, conventional fusion strategies become difficult to scale and less robust. A more practical alternative is to learn sensor-agnostic semantic representations that capture high-level scene semantics independent of sensing modality. Such a formulation allows knowledge learned from one modality to transfer to another and can improve recognition performance without requiring exact cross-sensor correspondence.

This challenge has motivated growing interest in multimodal and sensor-flexible remote sensing models. A neuroplasticity-inspired multimodal foundation model, for example, uses dynamic wavelength-adaptive mechanisms to support multiple sensor types and even unseen sensors within a single framework [21]. At the same time, multimodal contrastive learning has emerged as a promising alternative to strict fusion assumptions. Modal-aware contrastive learning has been applied to hyperspectral and LiDAR classification, showing that consistency objectives can effectively extract complementary information across modalities [22]. Other multimodal contrastive learning frameworks in remote sensing similarly demonstrate that robust feature learning can be achieved even when heterogeneous inputs are not perfectly aligned [23]. These studies support the idea that semantic consistency, rather than strict pixel-level correspondence, may be the key to scalable multimodal remote sensing learning.

In addition to sensor-level multimodal learning, semantic alignment has also been explored through vision–language modeling and image–text supervision. RSGPT introduces a remote sensing vision–language model and benchmark suite aimed at richer semantic understanding [24]. RS5M and GeoRSCLIP extend this direction by constructing large-scale image–text paired datasets and adapting CLIP-style objectives for remote sensing, thereby improving cross-modal retrieval and semantic transfer [25]. GeoCLIP, although not designed for aerial scene classification, further illustrates the power of alignment-based objectives by pairing image features with continuous geographic embeddings to learn meaningful semantic structure [26]. These works reinforce the broader view that representation alignment is a powerful mechanism for building robust remote sensing models.

In addition, several complementary cross-domain enhancement and fusion strategies have been studied. For example, deep image prior-based methods are designed for unsupervised HS image super-resolution [27], lightweight dual-domain methods are presented for VIS–IR image fusion [28], and conditional variational model with adaptive normalization for difficult enhancement scenarios (e.g., underwater imaging) [29]. Although these methods primarily focus on pixel-level reconstruction, enhancement, or fusion, they further emphasize the importance of leveraging complementary information across domains. In contrast, a representation-level formulation seeks to unify modality-specific semantics without requiring explicit image-level fusion or reconstruction.

From a conceptual perspective, the idea of sensor-agnostic learning is also related to brain-inspired interpretation in remote sensing. Neuro-cognitive principles such as parallel sensory pathways, abstraction hierarchies, and modality-invariant semantics suggest that biological systems form stable concepts across heterogeneous sensory inputs without requiring exact correspondence between modalities [30]. In this sense, the same object or scene can be understood despite differences in the form of sensory input. Translating these principles into modern representation learning suggests that multimodal remote sensing systems may become more accurate, robust, and transferable when they emphasize semantic abstraction over low-level fusion. At the same time, it is important to clarify that the term “neuro-inspired” in this work refers to high-level conceptual design principles rather than a biologically faithful implementation of specific neural mechanisms.

Furthermore, recent surveys on remote sensing foundation models emphasize that cross-modal alignment and multimodal generalization across heterogeneous sensors remain open challenges for large-scale Earth observation intelligence [31,32]. This indicates that future progress depends not only on scaling models or increasing data volume, but also on designing learning objectives and architectures that can unify heterogeneous sensing streams at the representation level.

Motivated by these observations, this paper proposes a neuro-sensor-agnostic representation learning framework for aerial scene recognition across RGB and hyperspectral modalities. Rather than relying on paired samples or explicit feature fusion, the proposed framework employs modality-specific encoders to independently extract features from each modality and project them into a shared semantic embedding space that acts as a common representational cortex for unified decision-making. Neuro-inspired consistency and cross-spectral alignment objectives are introduced to encourage modality-invariant semantics while preserving inter-class discrimination and representation robustness. In this way, multimodal learning is formulated as semantic representation unification rather than direct feature concatenation. This design supports effective knowledge transfer across modalities and enables robust, transferable, and modality-invariant aerial scene recognition across heterogeneous remote sensing inputs.

Research Contributions

The main contributions of this paper are summarized as follows:

• We propose a sensor-agnostic semantic representation learning framework that formulates multimodal aerial scene recognition as semantic embedding unification rather than explicit feature fusion.
• We design a modality-specific dual-encoder architecture with shared embedding projection and a unified classifier, enabling consistent inference across RGB and hyperspectral modalities without requiring paired inputs at test time.
• We introduce a composite learning objective that integrates cross-spectral alignment, intra-class compactness regularization, and prototype-based semantic anchoring to enforce modality-invariant representation geometry.
• We conduct comprehensive evaluations including modality-specific recognition, strict cross-modality generalization without fine-tuning, joint-modality training, ablation analysis, and computational efficiency comparison across multiple benchmark datasets (Houston 2013, UC Merced, and Indian Pines).
• Experimental results demonstrate strong cross-modal transfer performance and confirm that semantic representation unification provides a scalable alternative to traditional multimodal fusion approaches.

2. Materials and Methods

The proposed neuro-sensor-agnostic aerial scene recognition framework is designed to learn modality-invariant semantic representations from different types of data, such as RGB and hyperspectral images. Unlike conventional multimodal fusion methods, the proposed framework does not require paired cross-modal samples or explicit feature fusion. Instead, it aligns modality-specific features into a shared semantic embedding space and performs classification using a single unified classifier. The overall workflow has six main stages: (1) modality-specific data preprocessing, (2) modality-dependent representation encoding, (3) shared semantic embedding formation, (4) neuro-inspired cross-spectral alignment, (5) unified classifier learning, and (6) sensor-agnostic inference.

In the first stage, RGB and hyperspectral images are preprocessed separately based on their sensor characteristics. RGB images are resized to a fixed spatial resolution, normalized, and augmented using standard methods such as random flipping and rotation to improve generalization. Hyperspectral images are spectrally normalized and, optionally, processed to reduce noise and redundant information using spectral band selection or linear projection methods. This preprocessing helps preserve important spectral and spatial information while also reducing computational cost and improving the model’s training stability.

In the second stage, each modality is processed by its own encoder to extract high-level semantic features. The RGB encoder captures spatial structure, texture, and object visual information from RGB images. The hyperspectral encoder captures spectral–spatial relations and material-specific spectral signatures from hyperspectral data. Both encoders operate independently and do not share parameters, so each can learn features tailored to its own modality’s physical properties and data characteristics.

In the third stage, modality-specific features are projected into a shared semantic embedding space using modality-dependent projection heads. Each projection head has two fully connected layers with batch normalization, ReLU activation, and ${\mathsf{\ell }}_2$ normalization, and it produces a fixed-size embedding vector. This shared embedding space serves as a common representation space, where samples from the same class are close together and those from different classes are far apart. Importantly, this projection is learned without exact pixel-level matching or paired RGB–hyperspectral samples, enabling flexible and effective cross-modality feature learning.

In the fourth stage, neuro-inspired alignment and consistency learning objectives are applied to organize the shared embedding space. A cross-spectral semantic alignment objective aligns embeddings from different modalities within the same class in embedding space so that the model can learn a modality-independent semantic representation. At the same time, discriminative learning objectives increase the distance between different classes, thereby improving classification robustness. An intra-class compactness constraint also reduces variation within the same class, so embedded clusters become more stable and well separated. Additionally, prototype-based semantic anchoring is used, where each class has a prototype acting as a semantic reference point to guide feature alignment. All these mechanisms help form stable, modality-invariant semantic representations, similar to associative memory behavior in biological perception systems.

Finally, during inference, the proposed framework works in a sensor-agnostic way. It needs only a single modality input. The input sample is fed to its modality-specific encoder and projection head to produce a shared semantic embedding. After that, this embedding is classified by the unified classifier, which works in a shared semantic space. There is no need for paired modality data, no explicit feature fusion, and no extra modality information during inference. This sensor-agnostic inference capability enables flexible deployment in real-world remote sensing applications where only a single sensor modality is available, while maintaining consistent semantic understanding across different sensors. Figure 1 shows the overall workflow of the proposed neuro-sensor-agnostic representation learning framework. All experiments were implemented using Python 3.10 and PyTorch 2.7.0.

Proposed Architecture

The proposed architecture is based on neuro-inspired dual sensory pathways and a shared semantic cortex paradigm. This paradigm shows how biological perception systems process different sensory inputs through modality-specific pathways and then combine them into a unified semantic representation at higher cortical levels. Inspired by this idea, the proposed framework uses independent modality-specific encoders to process RGB and hyperspectral inputs separately. After that, the features are projected into a shared semantic embedding space that serves as a common representational cortex. All recognition decisions are performed within this shared semantic space using a single unified classifier. This design enables sensor-agnostic inference and eliminates the need for paired RGB–hyperspectral samples or explicit feature-level fusion.

At the first stage, an RGB encoder $f_r(·;{\theta }_r)$ receives an RGB image $x_i^r$ and produces a spatial–semantic feature representation:

$$u_i^r=f_r(x_i^r;{\theta }_r),u_i^r\in {\mathbb{R}}^{d_r}.$$

(1)

The RGB encoder captures spatial structure, texture patterns, and object-level semantic information characteristic of optical imagery. In parallel, a hyperspectral encoder $f_h(·;{\theta }_h)$ processes a hyperspectral input $x_j^h$ and extracts a spectral–spatial feature representation:

$$u_j^h=f_h(x_j^h;{\theta }_h),u_j^h\in {\mathbb{R}}^{d_h}.$$

(2)

The hyperspectral encoder learns the correlation between spectral bands and also preserves spatial information, which helps represent material spectral signatures more effectively. The RGB encoder is implemented using a ResNet-50 backbone pretrained on ImageNet, followed by a global average pooling layer to generate compact spatial–semantic feature embeddings. The hyperspectral encoder is implemented using a 3D convolutional neural network that includes spectral–spatial convolutional layers, batch normalization, and ReLU activation to capture both spectral and spatial dependencies. Both encoders operate independently and do not share parameters, allowing each to learn and specialize according to the physical and statistical properties of its sensing modality. Figure 2 illustrates the modality-specific dual pathways, where the RGB encoder extracts spatial–semantic features and the hyperspectral encoder learns spectral–spatial representations independently.

To achieve modality unification, modality-specific features are mapped into a shared semantic embedding space using projection heads $g_r(·;{\varphi }_r)$ and $g_h(·;{\varphi }_h)$. Each projection head has a two-layer fully connected network with batch normalization, ReLU activation, and ${\mathsf{\ell }}_2$ normalization. This projection converts encoder-specific features into a fixed-dimensional, unified embedding space, making the semantic representation consistent across modalities. The projected embeddings are defined as:

$$z_i^r=\mathrm{norm}\left(g_r(u_i^r;{\varphi }_r)\right),$$

(3)

$$z_j^h=\mathrm{norm}\left(g_h(u_j^h;{\varphi }_h)\right),$$

(4)

where $\mathrm{norm}(·)$ denotes ${\mathsf{\ell }}_2$ normalization, which ensures that embeddings lie on the unit hypersphere. This normalization helps improve training stability and also allows cosine-based similarity comparisons. The resulting shared embedding space acts as a modality-invariant semantic representation space, where semantic concepts are encoded independently of the sensing modality. As shown in Figure 3, modality-specific features are projected into a shared semantic embedding space and ${\mathsf{\ell }}_2$-normalized to enable modality-invariant representation learning.

On top of the shared semantic embedding space, a unified classifier $q(·;\psi )$ is applied to embeddings from both modalities. The predicted class probability is computed as:

$$\widehat{p}(y=c\mid z)=\mathrm{softmax}{(Wz+b)}_c,$$

(5)

where $W\in {\mathbb{R}}^{C\times d}$ and $b\in {\mathbb{R}}^C$ are learnable parameters, and C is the number of classes. Using a single classifier, we ensure that both RGB and hyperspectral samples are classified with the same decision boundaries, which enforce semantic consistency and help with sensor-agnostic recognition.

Training of the architecture is guided by a composite objective function that learns strong, discriminative representations and improves cross-spectral semantic alignment. First, a supervised classification loss ensures that learned embeddings remain predictive of correct class labels. Second, a cross-spectral alignment loss helps embeddings from different modalities within the same class stay close in the shared embedding space, while embeddings from different classes remain far apart. Third, an intra-class compactness constraint reduces variance within each class, improving cluster stability and class separation. Finally, a prototype-based agreement mechanism introduces modality-invariant class prototypes that serve as semantic anchors, guiding both modalities toward consistent semantic representations. These combined objective functions help the model learn robust, modality-invariant semantic features. Figure 4 presents the unified classifier operating in the shared embedding space and summarizes the composite objective functions that enforce semantic consistency across modalities. The complete training procedure of the proposed neuro-sensor-agnostic representation learning framework is summarized in Algorithm 1.

Algorithm 1 Neuro-sensor-agnostic representation learning (training).

Require: RGB dataset ${\mathcal{D}}_r=\left\{(x_i^r,y_i)\right\}$, HSI dataset ${\mathcal{D}}_h=\left\{(x_j^h,y_j)\right\}$, encoders $f_r,f_h$, projection heads $g_r,g_h$, classifier q, prototypes ${\left\{p_c\right\}}_{c=1}^C$, loss weights ${\lambda }_1,{\lambda }_2,{\lambda }_3$ Ensure: Trained parameters ${\theta }_r,{\theta }_h,{\varphi }_r,{\varphi }_h,\psi$ and prototypes $\left\{p_c\right\}$ 1: Initialize network parameters and normalize prototypes 2: for epoch $=1$ to T do 3:     Sample mini-batch ${\mathcal{B}}_r$ from ${\mathcal{D}}_r$ and ${\mathcal{B}}_h$ from ${\mathcal{D}}_h$ 4:     Modality-specific encoding: 5:     $U_r\leftarrow f_r\left({\mathcal{B}}_r\right)$ 6:     $U_h\leftarrow f_h\left({\mathcal{B}}_h\right)$ 7:     Shared-space projection: 8:     $Z_r\leftarrow \mathrm{Normalize}\left(g_r\left(U_r\right)\right)$ 9:     $Z_h\leftarrow \mathrm{Normalize}\left(g_h\left(U_h\right)\right)$ 10:     Unified classification loss: 11:     ${\mathcal{L}}_{cls}\leftarrow \mathrm{CE}(q\left(Z_r\right),Y_r)+\mathrm{CE}(q\left(Z_h\right),Y_h)$ 12:     Cross-spectral alignment: 13:     ${\mathcal{L}}_{align}\leftarrow \mathrm{ClassContrastive}(Z_r,Z_h)$ 14:     Intra-class compactness: 15:     ${\mathcal{L}}_{compact}\leftarrow \mathrm{WithinClassVar}(Z_r,Z_h)$ 16:     Prototype agreement: 17:     ${\mathcal{L}}_{proto}\leftarrow \mathrm{ProtoCE}(Z_r,Z_h,\left\{p_c\right\})$ 18:     Total loss: 19:     $\mathcal{L}\leftarrow {\mathcal{L}}_{cls}+{\lambda }_1{\mathcal{L}}_{align}+{\lambda }_2{\mathcal{L}}_{compact}+{\lambda }_3{\mathcal{L}}_{proto}$ 20:     Update network parameters using backpropagation 21:     Update prototypes using exponential moving average: 22:     $p_c\leftarrow \alpha p_c+(1-\alpha ){\overline{z}}_c$ 23: end for 24: return trained model and prototypes

During inference, only one modality input is needed. The input sample is passed through its corresponding modality-specific encoder and projection head to generate a shared semantic embedding. This embedding is then classified using the unified classifier in the shared semantic space. Since classification is performed on modality-invariant features, no paired modality data, explicit feature fusion, or additional modality information is required. This sensing-agnostic inference ability allows for wide deployment in different remote sensing scenarios with consistent semantics. The overall architecture of the proposed neuro-sensor-agnostic framework is shown in Figure 5. The sensor-agnostic inference process, where a single available modality is projected into the shared semantic space for classification, is presented in Algorithm 2.

Algorithm 2 Sensor-agnostic inference.

Require: Test sample x, modality indicator $m\in \{\mathrm{RGB},\mathrm{HSI}\}$, encoders $f_r,f_h$, projection heads $g_r,g_h$, classifier q Ensure: Predicted label $\widehat{y}$ 1: Preprocess: resize and normalize $x\to x^{\prime }$ 2: if $m=$ RGB then 3:     $z\leftarrow \mathrm{Normalize}\left(g_r\left(f_r\left(x^{\prime }\right)\right)\right)$ 4: else 5:     $z\leftarrow \mathrm{Normalize}\left(g_h\left(f_h\left(x^{\prime }\right)\right)\right)$ 6: end if 7: $\widehat{y}\leftarrow argmaxq\left(z\right)$ 8: return $\widehat{y}$

3. Result

In this section, we extensively validate our proposed neuro-sensor-agnostic representation learning framework on the Houston 2013 hyperspectral dataset. To ensure rigorous cross-modality validation, two modalities are generated from the same data. The hyperspectral modality takes the whole spectral cube as input, whereas the RGB modality is generated by taking a visible band composite from the spectral bands of the hyperspectral data. Both modalities contain the same spatial scenes, and ground-truth labels are identical, such that they guarantee the same semantics, thus making evaluation of modality-invariant representation learning rigorous.

Although both modalities arise from the same acquisition, their spectral characteristics are very different, so are their statistical distributions, which make cross-modality semantic alignment and representation transfer reliable to assess. The framework is tested in four circumstances: intra-modality recognition, cross-modality generalization, joint-modality training, and ablation study. Accuracy (OA) is the mainly reported accuracy, while the average accuracy (AA) and the kappa coefficient ($\kappa$) are also given to represent the confidence of classification.

3.1. Datasets

To provide a comprehensive evaluation of the sensor-agnostic learning framework of semantic representation, we conducted experiments on three widely adopted benchmark datasets. Those datasets are Houston 2013, UC Merced Land Use and Indian Pines.

Houston 2013 Dataset: The Houston 2013 dataset is a commonly used hyperspectral image classification benchmark, providing high-resolution hyperspectral images of 15 scene classes. Both RGB and hyperspectral modalities were generated from the Houston 2013 dataset for the purpose of having precise semantic correspondences and for a strict modality-invariant representation learning evaluation. The spectral modality was built based on the whole spectral cube (noisy bands and water absorption bands are discarded); whereas the RGB modality was formed by an RGB composite of selected spectral bands in hyperspectral data. Since the two modalities were extracted from identical scenes and with the same ground-truth annotations, semantic discrepancies were excluded and robust cross-modality evaluation could be ensured.

UC Merced Land Use Dataset: UC Merced dataset is a common aerial scene classification benchmark. There are 2100 RGB images, which are labeled into 21 land-use classes (100 images per class). All images have a resolution of $256\times 256$ pixels. The image size for training was down-sampled to $224\times 224$. We used a 80/20 split as train/test ratio. We used this dataset to test the generalization ability of the proposed RGB pathway on another aerial benchmark.

Indian Pines Dataset: The Indian Pines dataset is a well-known hyperspectral land-cover classification benchmark acquired by the AVIRIS sensor. It contains 16 land-cover classes and 220 spectral bands after removing noisy and water-absorption bands. Following common practice, an 80/20 train–test split was used. Spectral normalization was applied prior to training to ensure stable optimization. This dataset was used to validate the effectiveness of the hyperspectral pathway in an independent single-modality setting.

The detailed characteristics of all datasets are summarized in

Table 1. Summary of datasets used for evaluation.

Property	Houston (HSI)	Houston (RGB)	External Datasets
Source dataset	Houston 2013	Houston 2013	UC Merced/Indian Pines
Number of classes	15	15	21 (UCM)/16 (IP)
Modality type	Hyperspectral	RGB (derived)	RGB/Hyperspectral
Channels	144 bands	3	3 (UCM)/220 (IP)
Spatial resolution	2.5 m	2.5 m	256 × 256 (UCM)/20 m (IP)
Train–test split	80/20	80/20	80/20
Purpose	Cross-modality eval.	Cross-modality eval.	Independent validation

.

3.2. Experiment 1: Modality-Specific Recognition Performance

This experiment evaluated the recognition performance of the proposed framework when trained and tested separately for each modality. In the RGB modality case, only the RGB encoder and its projection head were active. In contrast, in the hyperspectral modality case, only the hyperspectral encoder and its projection head were used. The unified classifier operated on a shared semantic embedding space in both cases, ensuring consistent decision boundaries. This experiment verified that the proposed shared embedding architecture did not reduce single-modality performance and still maintained strong discriminative capability.

Table 2. Modality-specific recognition performance on the Houston 2013 dataset.

Method	RGB OA (%)	HSI OA (%)	HSI $\mathit{\kappa }$
CNN Baseline	92.1	93.7	0.931
Transformer Baseline	94.0	95.1	0.946
Self-Supervised Baseline	95.2	96.0	0.954
Proposed Method	96.4	97.3	0.968

shows the modality-specific recognition performance on the Houston 2013 dataset. The proposed framework achieves better performance than baseline convolutional and transformer-based architectures across both modalities. Specifically, the proposed method achieves overall accuracies of 96.4% in the RGB modality and 97.3% in the hyperspectral modality. Hyperspectral modality shows a high Kappa coefficient of 0.968, indicating high reliability for classification. This implies that the shared semantic embedding space plays an effective role as a regularizer that increases feature discrimination.

In Figure 6, we have the comparison of within-modality recognition results. It indicates the proposed method obtains high accuracy for both RGB and hyperspectral modality. It shows the learning sensor-agnostic representation improves classification accuracy and also preserves good modality discrimination.

3.3. Experiment 2: Cross-Modality Generalization Performance

We evaluated the modality-invariance of the proposed framework through a cross-modality generalization experiment. Under this condition, the model was trained by samples from one sensing modality and directly tested on the other modality with no fine-tuning, parameter modification, nor domain adaptation. This was designed to reflect the raw semantic transferability, rather than implicit adaptation.

More precisely, two transfers were investigated: (1) RGB → HSI (trained on RGB samples and tested on HSI images), (2) HSI → RGB (trained on HSI samples and tested on RGB images). The classifier was uniform, and the embedding space was fixed for all tests. Since both modalities shared the same spatial location and same class annotation in the Houston 2013 dataset, we directly observed the modality-independent semantic structure encoded in the embedding space.

Table 3. Cross-modality recognition performance on the Houston 2013 dataset.

Method	RGB → HSI (%)	HSI → RGB (%)
CNN Baseline	59.4	61.0
Transformer Baseline	65.2	66.8
Feature Alignment Baseline	72.5	74.1
Contrastive Learning Baseline	80.6	82.3
Proposed Method	87.2	88.7

summarizes the results for the cross-modality recognition tasks. For RGB → HSI and HSI → RGB tasks, the proposed method achieved a global accuracy of 87.2% and 88.7% respectively. Both results substantially exceed the baseline performance and indicate that both cross-spectral alignment and prototype-based semantic anchoring are effective in closing the domain gap. Although the results slightly drop in contrast to the within-modality experiments, this is indeed foreseeable considering the inherent difference in spectra and statistics between RGB and hyperspectral data. However, the decrease in performance is rather small, which proves that the learned common embedding space predominantly represents semantics instead of modality characteristics, and thus a unification on semantic level is reasonable for cross-domain sensor environment.

Figure 7 indicates the performance comparison under cross-modality recognition tasks. The developed framework always obtains superior performances on both transferring directions, which is evidence that the modality-invariant semantic features can be learned by our model.

3.4. Experiment 3: Joint-Modality Training Performance

We evaluated the performance of the framework with the joint training on both RGB and hyperspectral modes. In this setup, both the RGB and hyperspectral data were fed to the network simultaneously, allowing for the promotion of semantic alignment and the development of more discriminative features. The common semantic embedding space was jointly optimized by the classification loss, cross-spectral alignment loss, compactness loss, and prototype agreement loss. In

Table 4. Joint-modality training performance on the Houston 2013 dataset.

Method	RGB OA (%)	HSI OA (%)
Simple Joint Training	94.5	95.6
Without Alignment Loss	95.8	96.5
Proposed Full Model	97.0	97.8

, we show the results of training on both modalities. Our framework yields an accuracy of 97.0% and 97.8% on the RGB and hyperspectral modality respectively. This demonstrates the superiority of joint modality training in improving the quality of representation and better performance on unseen samples.

Figure 8 illustrate the results of joint-modality training, in which the proposed method obtains the highest accuracy, further proving the necessity of semantic alignment and prototype learning.

3.5. Training and Validation Convergence Analysis

The training and validation accuracy and loss curves are examined to understand the optimization and convergence robustness of the framework. Figure 9 shows the training and validation accuracy as a function of training epochs. The accuracy curves increase rapidly during the early stages of training and then gradually stabilize as the model learns more discriminative semantic features. The validation accuracy is approximately the same as training accuracy, with extremely low overfitting.

Figure 10 represents the training and validation loss curves. Both loss curves decrease steadily as the number of epochs increases, which proves that optimization is stable and the learning process is efficient. It demonstrates that the use of shared semantic embedding architecture and prototype-guided representation learning strategy are effective in learning good feature representations due to their smooth convergence trends.

3.6. Confusion Matrix Analysis

In order to assess the class-wise recognition performance of the neuro-sensor-agnostic framework more precisely, the confusion matrix on the Houston 2013 dataset is presented in Figure 11. The confusion matrix shows more information in classification performance when comparing actual and predicted class label. From the figure, a majority of samples are located on the diagonal line, which indicates that the majority of samples are correctly predicted. The concentrated line shows that the proposed framework is applicable for learning highly discriminative semantic representations in the common embedding space.

The minimal off-diagonal values indicate low misclassification between different classes, confirming that the learned embedding space achieves good separation between classes. Classes with similar spectral or spatial characteristics, which are usually difficult to distinguish in hyperspectral classification, are still classified with high accuracy. This suggests that modality-specific encoders and cross-spectral alignment objectives successfully capture both spatial structure and spectral signatures while preserving class-specific semantic information. Also, the prototype-based semantic alignment helps form compact, well-separated clusters for each class, reducing confusion and improving classification robustness.

These results show that the shared semantic embedding space successfully aligns modality-specific features into a unified representation while preserving class discrimination. The unified classifier operating in this embedding space can learn consistent, reliable decision boundaries across all scene categories. The confusion matrix collectively indicates that the proposed sensor-agnostic approach delivers well-balanced, robust, and effective classification accuracy across all the classes and effectively addresses the task of aerial scene classification in heterogeneous remote sensing environment.

3.7. Experiment 4: RGB Evaluation on UC Merced Dataset

To investigate the robustness of the proposed RGB path even beyond the setting of RGB-HOUSTON, we tested our system on the UC Merced Land Use Dataset, a commonly used aerial scene classification benchmark comprising 2100 color images of 21 land-use classes. Each class has 100 images. We uses an 80/20 train/test split for our experiments. The images were rescaled to 224 × 224 and simple data augmentation like random horizontal flip and random small rotation was used. In this RGB-only experiment, only the RGB encoder and projection head were activated, and classification was done by a unified classifier working in a shared semantic embedding space.

As shown in

Table 5. RGB aerial scene recognition performance on the UC Merced Land Use Dataset (80/20 split).

Method	Overall Accuracy (%)
ResNet-50 Baseline	97.8
Proposed (Shared Embedding RGB Pathway)	98.7

, the proposed framework achieves an overall accuracy of 98.7%, which is higher than the standard ResNet-50 baseline (97.8%). These results show that a shared semantic embedding design can still preserve intense spatial feature discrimination even in the RGB-only case. The training and validation accuracy curves in Figure 12 show fast convergence in the early epochs, followed by stable performance, and validation accuracy is very close to training accuracy, confirming less overfitting and good generalization. Also, the training and validation loss curves in Figure 12 show smooth, decreasing behavior, indicating a stable optimization process. The confusion matrix in Figure 13 shows strong diagonal values with minimal inter-class confusion, which indicate reliable class-wise performance across all 21 classes. Overall, these results confirm that the proposed RGB pathway generalizes well to independent aerial datasets and maintains high classification reliability.

3.8. Experiment 5: Hyperspectral Evaluation on Indian Pines Dataset

In order to assess the robustness of our framework in a purely hyperspectral environment, we also applied it to the Indian Pines dataset, a common benchmark for hyperspectral image classification. It includes 16 land-cover classes captured by an AVIRIS sensor and uses 220 spectral bands after removing noisy and water absorption bands. An 80/20 training-testing split scheme was adopted, as it is widely applied. All hyperspectral samples were normalized spectrally before the training stage for smooth optimization.

In this hyperspectral only experiment, only the HSI encoder and the HSI projection head were switched on, and the RGB path was switched off. The classification was done using the same classifier on the shared semantic embedding space.

As indicated in

Table 6. Hyperspectral classification performance on the Indian Pines dataset (80/20 split).

Method	Overall Accuracy (%)
ResNet-style Baseline	95.8
Proposed (Shared Embedding HSI Pathway)	97.6

, the proposed framework obtained an average accuracy of 97.6%, which significantly improved over the baseline of standard ResNet-style (95.8%). This suggests that shared semantic embedding architecture maintains spectral–spatial discriminative information successfully even in single-modality hyperspectral case. From Figure 14, it is shown that the training and validation accuracy curves converge rapidly at the beginning of epochs and smoothly saturate, while the validation accuracy curves roughly match with the training ones, so that it is not easy to overfit and has good generalization ability. From Figure 14, it can be observed that the training and validation loss curves decay smoothly, indicating that the training is stable. The confusion matrix provided in Figure 15 is highly diagonal with less misclassification across each class, so that it is good at distinguishing each of the 16 land-cover types.

3.9. Ablation Study

To evaluate the contribution of individual components of the proposed neuro-sensor-agnostic framework, an ablation study was performed by removing key loss functions one by one while keeping all other architectural components unchanged. Specifically, the cross-spectral alignment loss, intra-class compactness loss, and prototype agreement loss were removed individually to assess their impact on classification performance.

Table 7. Ablation study results on Houston 2013 dataset.

Configuration	RGB OA (%)	HSI OA (%)
Full Model	97.0	97.8
Without Alignment Loss	95.2	96.1
Without Compactness Loss	95.9	96.8
Without Prototype Loss	95.5	96.4

shows the ablation results for both RGB and hyperspectral modalities. The complete model achieves the highest overall accuracy of 97.0% for RGB data and 97.8% for hyperspectral data, demonstrating the effectiveness of the complete learning objective.

When cross-spectral alignment loss is removed, performance drops significantly, and accuracy falls to 95.2% for RGB and 96.1% for the hyperspectral modality. This shows that semantic alignment between modalities is essential for learning modality-invariant features and reducing the domain gap. Removing the compactness loss also leads to a decline in performance, suggesting that the intra-class compactness helps to make the features more robust and separable. The same observation also occurs when removing the prototype agreement loss, illustrating that class prototypes provide a semantic reference to make the representations consistent and stable. From the ablation results, it can be observed that each part contributes significantly to learning a stable, discriminative, and modality-invariant shared feature space. Besides the performance drops between settings, Figure 16 also displays the magnitude of performance drops among different choices.

Figure 16 illustrates the ablation study performance comparison, showing the effectiveness of each component.

3.10. Computational Efficiency and Scalability Analysis

In addition to recognition performance, computational efficiency and scalability are also very important factors for practical deployment in large-scale remote sensing applications. The proposed neuro-sensor-agnostic framework is designed with modality-specific encoders and a shared semantic embedding space, thereby avoiding the extra computational cost of explicit multimodal fusion layers. By doing semantic alignment in a low-dimensional embedding space and using a unified classifier, the framework reduces computational complexity and memory usage while still maintaining strong classification performance.

Table 8. Computational efficiency comparison.

Method	Parameters (M)	FLOPs (G)	Inference Time (ms)
CNN Baseline	24.6	4.1	12.8
Transformer Baseline	28.3	4.5	15.6
Contrastive Learning Model	26.9	4.3	14.2
Proposed Framework	22.7	3.8	11.3

compares the computational efficiency of the proposed framework with representative baseline models in terms of the number of trainable parameters, FLOPs, and inference time per sample. Our framework also gains fewer parameters and less computational complexity than conventional CNNs, transformers and contrastive-learning-based models. It removes superfluous modality-specific classifiers and costly fusion operations, allowing for faster inference, but it does not compromise strong discriminative feature learning ability.

Due to its fewer number of parameters and FLOPs, the presented framework is efficient and friendly to deploy on limited-resource devices. Moreover, since the unified semantics embedding space reduces memory redundancy, the unique unified classifier works for all modalities. There are no extra multiple classifiers or fusion network involved.

It is also possible to scale up the proposed architecture as the number of sensing modalities increases. It just involves adding a separate modality-specific encoder for each modality and retaining the common embedding space and the final classifier. Any further sensing modality such as SAR, thermal imagery, or multi-spectral data can be introduced without altering the overall architecture. Hence, the system offers flexibility and scalability for the multimodal remote sensing tasks.

Moreover, from the analysis on the training and validation convergence, the optimization strategy is stable, and the curves of accuracy and loss converge smoothly. Prototype-based semantic alignment and intra-class compactness regularization enable robust learning of embeddings and class separability. It indicates that the proposed framework strikes a suitable trade-off between the recognition performance, computational cost, and scalability, so it can be widely applied to the sensor-agnostic aerial scene recognition of heterogeneous remote sensing systems.

3.11. Comparison with State-of-the-Art Methods

To evaluate the effectiveness of the proposed neuro-sensor-agnostic representation learning framework, its performance was compared with recent state-of-the-art hyperspectral image classification methods on the Houston 2013 dataset. Recent progress in hyperspectral classification is mainly driven by deep convolutional neural networks, transformer-based models, and spectral–spatial representation learning methods. Early deep learning methods based on CNNs showed strong ability to extract spatial features, but they had limitations in capturing long-range spectral dependencies. To solve this problem, transformer-based models such as SpectralFormer [33] introduced spectral self-attention to capture global spectral relationships, thereby significantly improving classification accuracy. Similarly, spatial–spectral transformer architectures [34] further improve performance by jointly learning spatial and spectral features. More recently, spectral–spatial Mamba-based models [35] have been proposed to improve computational efficiency while maintaining high classification accuracy by using an efficient sequence modeling approach.

Table 9. Comparison with state-of-the-art methods on the Houston 2013 dataset.

Method	Overall Accuracy (%)
3D CNN [36]	93.7
SpectralFormer [33]	95.8
Spatial–Spectral Transformer [34]	96.5
Spectral–Spatial Mamba [35]	97.1
Proposed Neuro-Sensor-Agnostic Framework	97.8

shows the quantitative comparison between the proposed framework and different state-of-the-art methods. The proposed neuro-sensor-agnostic framework achieves an overall accuracy of 97.8%, which is higher than that of CNN-based methods, transformer-based models such as SpectralFormer [33] and spatial–spectral transformers [34], and Mamba-based architectures [35]. This better performance is due to the shared semantic embedding space and cross-modality alignment mechanism, which help the model learn modality-invariant features and improve representation generalization. Different from conventional methods that mainly learn features from a single modality, the proposed framework learn modality-invariant semantic representations directly, which improves classification accuracy and also makes the model more robust. Recent self-supervised and foundation-model-based approaches are not included in this comparison due to differences in training paradigms and reliance on large-scale pretraining.

The better performance of the proposed framework shows that semantic representation unification is very effective for aerial scene recognition. By aligning features across different modalities into a shared semantic embedding space, the framework improves classification accuracy while maintaining good computational efficiency and scalability. These results show that semantic representation unification is a strong and scalable alternative compared to traditional modality-specific deep learning methods for hyperspectral and multimodal remote sensing applications.

4. Discussion

The experimental results demonstrate that the proposed neuro-sensor-agnostic representation learning framework is an efficient and scalable approach for aerial scene recognition in a variety of sensing modalities. The system learns modality-invariant semantic embeddings in order to perform the same across RGB and hyperspectral modalities, with no requirement for the explicit feature fusion or multi-class classification of modality-specific classifiers. This capability is significant for real-world remote sensing applications, where data come from different sensors with different spectral characteristics and imaging properties. The shared semantic embedding space serves as a unified representation layer, allowing both modalities to contribute to a common decision space while preserving their modality-specific feature information.

Moreover, the modality-specific recognition performance values also prove that the proposed unification of representation does not compromise single-modality performance and instead yields better performance than modality-specific baselines. Such improvement is achieved because the unified embedding space regularizes the network such that more semantic and discriminative features are learned. Also, the projection heads and prototype alignment force stable semantic representation over training samples. We observed that semantic representation unification not only increased generalization but also robustness.

The cross-modality generalization experiments further strengthen the conclusion about the success of the proposed sensor-agnostic learning approach. The generalization of the standard modality-specific methods has been limited by significant spectral and statistical dissimilarities between RGB and hyperspectral data. By aligning the semantic embeddings rather than fusing features, the proposed framework mitigates the modality gap, and the unified classifier can perform consistently on the disparate modalities. This demonstrate that the representation learning at the semantic level offers a robust and extensible way to bridge the heterogeneity in different modalities. These findings support the core thesis of this paper: uniting semantics of different sensing modalities can close the gap between them.

Although the cross-modality results and ablation study indirectly support the effectiveness of the shared embedding space, the current study does not explicitly visualize or quantify feature-distribution alignment using tools such as t-SNE or Maximum Mean Discrepancy. In addition, the present ablation mainly focuses on loss components, while architectural factors such as embedding dimensionality, projection head design, and the shared embedding/unified classifier formulation may also influence cross-modal transfer performance. The current projection heads use identical architectures with independent parameters, which provides a stable and consistent mapping into the shared space, but alternative modality-adaptive designs could also be explored. Similarly, the EMA-based prototype update offers stable alignment during training, although more advanced strategies such as online clustering or optimal transport may provide further gains. These aspects remain important directions for future investigation.

Joint-modality training further improves recognition performance in both RGB and hyperspectral modalities. This confirms that exposure to heterogeneous data improves representation quality and generalization capability. The shared embedding space benefits from complementary information across modalities, thereby improving semantic discrimination. The ablation study also confirms the importance of each component of the proposed framework. In particular, cross-spectral alignment plays an important role in ensuring modality consistency, while prototype-based representation learning improves class separability and stabilizes the embedding geometry. The intra-class compactness constraint further improves cluster coherence and enhances classification reliability.

Both training and validation curves exhibit stable training process and clear learning trends. The accuracy curves, both on training and validation data, achieve consistent improvement and early convergence as do loss curves. All these demonstrate that the framework learn valuable semantic information without overfitting. Strong class-wise performance can also be verified from confusion matrix: most images are classified correctly, and an extremely small number of misclassifications exists. All the results verify that the semantic embedding space, although shared by all classes, maintain high class-discrimination capacity.

Although the proposed framework shows strong performance, several limitations remain. First, the strict cross-modality evaluation was mainly conducted on a single benchmark setting, and broader validation across additional datasets and hyperspectral sensors with substantially different spectral resolutions is still needed to further strengthen the sensor-agnostic claim. Second, the current framework relies on supervised alignment and heuristic EMA-based prototype updates; extending it toward self-supervised or semi-supervised learning and more adaptive alignment strategies would improve scalability under limited labeled data. Third, recent foundation-model and self-supervised approaches in remote sensing often rely on large-scale pretraining and external supervision, whereas the proposed framework emphasizes semantic representation unification and efficient cross-modal transfer without such requirements. Finally, extending the framework to additional sensing modalities such as SAR or thermal imagery, together with evaluation under real deployment constraints, would further improve its robustness and practical applicability.

It is important to clarify that, in the Houston 2013 experiments, the RGB modality was derived from the hyperspectral cube, so both modalities shared the same scenes and semantic labels. This controlled setting was used to rigorously evaluate semantic alignment and cross-modality transfer under consistent ground-truth conditions. Therefore, the current experiments should not be interpreted as a fully unpaired cross-sensor training scenario. Rather, the main contribution is that the proposed framework does not require simultaneous multimodal input, explicit fusion, or paired modalities at inference time, since prediction is performed from a single modality through the shared semantic embedding space. Evaluating the framework under truly unpaired cross-sensor conditions remains an important direction for future work.

Overall, the proposed neuro-sensor-agnostic representation-learning framework offers a principled and practical alternative to traditional multimodal fusion approaches. Instead of directly combining features, the framework focuses on unifying the semantic representation, thereby enabling the learning of more meaningful and general features. Because of this, the framework can achieve scalable, robust aerial scene recognition across heterogeneous sensing modalities. This approach also opens new directions for developing unified remote sensing recognition systems that can work effectively across different sensor environments and conditions. In addition, the framework remains practical for deployment because only a single modality-specific encoder is active during inference, while the shared embedding space and unified classifier reduce redundancy in decision-making. Combined with the reported parameter count, FLOPs, and inference-time advantages over CNN and transformer baselines, this design supports applicability on resource-constrained aerial and satellite platforms.

5. Conclusions

This paper presented a neuro-sensor-agnostic representation learning framework for aerial scene recognition using RGB and hyperspectral sensing modalities in a shared semantic embedding space. The framework independently encodes modality-specific features and aligns them at a higher-level semantic representation, enabling consistent and robust recognition without the need for explicit feature fusion or modality pairing. Experimental results on the Houston 2013 dataset showed that the proposed method achieved strong performance, with an overall accuracy of 96.4% for RGB and 97.3% for hyperspectral data. Also, it achieved a high cross-modality generalization accuracy of 87.2% (RGB → HSI) and 88.7% (HSI → RGB). Joint-modality training further improved performance to 97.0% and 97.8% for the RGB and hyperspectral modalities, confirming their advantage in shared semantic learning. An ablation analysis verified the importance of semantic alignment and prototype-based representation anchoring, and the confusion matrix showed excellent class-wise discrimination for all scene categories. These results confirm that semantic representation unification provides a practical and scalable alternative to traditional multimodal fusion methods, enabling robust and generalizable aerial scene recognition across different sensing environments.