Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Abstract

Although the collaborative use of hyperspectral images (HSIs) and LiDAR data in land cover classification tasks has demonstrated significant importance and potential, several challenges remain. Notably, the heterogeneity in cross-modal information integration presents a major obstacle. Furthermore, most existing research relies heavily on category names, neglecting the rich contextual information from language descriptions. Visual-language pretraining (VLP) has achieved notable success in image recognition within natural domains by using multimodal information to enhance training efficiency and effectiveness. VLP has also shown great potential for land cover classification in remote sensing. This paper introduces a dual-sensor multimodal semantic collaborative classification network (DSMSC²N). It uses large language models (LLMs) in an instruction-driven manner to generate land cover category descriptions enriched with domain-specific knowledge in remote sensing. This approach aims to guide the model to accurately focus on and extract key features. Simultaneously, we integrate and optimize the complementary relationship between HSI and LiDAR data, enhancing the separability of land cover categories and improving classification accuracy. We conduct comprehensive experiments on benchmark datasets like Houston 2013, Trento, and MUUFL Gulfport, validating DSMSC²N’s effectiveness compared to various baseline methods.

1. Introduction

Remote sensing image classification plays an important role in land cover monitoring, urban planning, forest management, and other fields [1]. Among these, HSI stands out due to its unique advantages. HSI consists of spectra from hundreds of contiguous bands in the same area, offering the unique advantages of high spectral resolution and spectral–spatial integration. Its rich spectral information can be used to identify the composition and intrinsic structure of different land cover materials [2,3,4]. In recent years, deep learning models, such as convolutional neural network(CNN) [5], 3DCNN [6], recurrent neural network (RNN) [7], and transformer [8], have automatically learned complex feature representations in images, providing new approaches for hyperspectral image classification [9].

Recent advances in Earth observation technologies have driven the widespread adoption of multimodal remote sensing data processing. This progress has accelerated the research in semantic segmentation and multimodal data matching [10,11,12], significantly improving the accuracy of remote sensing image classification. Beyond HSI, remote sensing techniques can leverage different types of sensors to simultaneously and synergistically depict terrestrial objects in the same area and time frame [13,14,15,16]. For instance, LiDAR can generate digital surface models (DSMs) that capture the three-dimensional morphology of ground targets. The variations in elevation features derived from these models are widely used in classification tasks [17].

HSI provides not only rich spectral information but also valuable spatial data due to its unique data cube structure. Along the spectral axis, HSI displays a precise distribution of bands. Across the two spatial dimensions, it captures the geometric layout of the scene, creating a structure that integrates both the spectral and spatial information. This structure allows HSI to capture the spatial arrangement and texture of land covers, crucial for distinguishing between types with similar spectral characteristics but different spatial patterns.

When HSI’s spatial information is combined with the high-precision elevation data from LiDAR, the complementary strengths of these two data sources become clear. In complex geographic environments, integrating HSI’s spatial details with LiDAR’s vertical structural data significantly improves the accuracy of land cover classification and identification. This integration not only enhances the separability of the land cover categories but also offers a more comprehensive and detailed description of surface features, greatly improving the performance and reliability of classification tasks.

By deeply integrating the HSI and LiDAR data, we fully harness HSI’s strengths in spectral analysis and spatial positioning while effectively incorporating LiDAR’s expertise in terrain height and spatial detail. This integrated analytical strategy creates a more comprehensive and fine-grained multidimensional representation of land cover properties, significantly enhancing classification accuracy and deepening our understanding of the essential characteristics of land covers.

Despite their complementary strengths, HSI images and LiDAR data represent two distinct remote sensing modalities, exhibiting apparent heterogeneity [18,19]. How to extract richer feature representations to fully exploit the complementary information between the two data sources is one of the key challenges currently limiting the in-depth collaboration of heterogeneous remote sensing data [20].

Numerous studies have explored the comprehensive exploitation of complementary information between HSI and LiDAR. With the rapid development of deep learning techniques, novel network architectures such as ConvNeXt [21,22], InceptionNeXt [23], Vision Transformer(ViT) [24], Dual Attention Vision Transformers(DaViT) [25], and SpectFormer [26] have also been introduced to the collaborative classification task of multi-source remote sensing image data, achieving significant progress. For example, satisfactory collaborative classification results have been achieved in IP-CNN [27], hierarchical CNN and transformer [28], and CNN-transformer [29]. Mainstream multimodal models include dual encoder and fusion encoder, and many research efforts have focused on how to improve the collaborative integration of multimodal models. For example, two-branch CNN [30], EndNet [31], Hypermlp [32], Ms2canet [33], HFSNet [34], and so on. These advances highlight the potential for further improvements in the integration of multi-modal remote sensing data, driving innovation and enhancing the accuracy and reliability of land cover classification models.

Recently, visual-language pretraining (VLP) [35], contrastive language-image pre-training (CLIP) [36], bootstrapping language-image pre-training (BLIP) [37,38], and InstructBLIP [39], have achieved significant success in computer vision and natural language processing, dominating the downstream tasks such as image classification. This success demonstrates the potential of cross-modal learning in understanding and representing complex information. It further suggests that image-text joint pre-training strategies, such as RemoteCLIP [40], could greatly improve models’ ability to grasp the deep semantic content of remote sensing images. This is because remote sensing images are rich in visual information and often include detailed textual descriptions.

Despite the significant advances in classification performance achieved by large-scale image-text pre-training techniques, some inherent limitations remain [41]. For instance, the models occasionally make illogical classification decisions or produce unexpected results. CLIP may misclassify an image of a skier as “snow mountain” in certain scenarios, revealing an overreliance on background features while neglecting primary subjects. This highlights the importance of enhancing the logical coherence and interpretability of classifications.

To tackle this challenge, we propose a novel strategy that uses language as a bridge between hyperspectral image features and land cover categories. By leveraging language models to analyze the detailed characteristics of land covers, we move beyond pre-defined class labels to achieve finer-grained interpretation and feature learning. For instance, when identifying specific land cover types, the model can generate relevant linguistic descriptions based on spectral information, spatial information, and other attributes, accurately recognizing and extracting key features, thus enhancing the model’s explanatory power.

However, manually defining such features is not only time-consuming but also lacks scalability. To overcome this obstacle, extensive world knowledge from large language models (LLMs) such as GPT-3 [42,43] can be utilized. These models can automatically obtain detailed visual descriptions with remote sensing knowledge based on regional categories, thereby enhancing the interpretability and accuracy while ensuring the model’s classification performance.

Faced with the limitations and interpretability demands of current remote sensing image classification tasks, an innovative network, the dual-sensor multimodal semantic collaborative classification network (DSMSC²N) has been developed in this paper. This network initially leverages LLMs to generate high-precision spectral category textual descriptions, thereby enriching the visual features with substantial semantic information. Subsequently, the ModaUnion encoder, a core component, extracts the shared features from the bimodal data of HSI and LiDAR, facilitating effective feature fusion and cooperative analysis of the high-dimensional heterogeneous data. Additionally, the mixture of experts encoder module (MoE-EN) is incorporated to deeply explore and utilize the complementary information between different modal data, effectively addressing the significant differences and high integration difficulty between modalities. Finally, a contrastive learning strategy for land cover classification is employed, which not only optimizes the model’s classification performance but also enhances the differentiation between categories, providing a more intuitive and interpretable foundation for model decision-making. The main contributions can be summarized as follows:

(1)

Enhancing Land Cover Classification Accuracy with Instruction-driven Large Language Models: instruction-driven large language models guide the model to focus on and extract the critical features, thereby improving land cover classification accuracy.

(2)

Improving Multisource Data Feature Extraction with the ModaUnion Encoder: the ModaUnion encoder enhances multisource data feature extraction quality by implementing parameter sharing.

(3)

Addressing Multisource Heterogeneity with MoE-EN and Contrastive Learning: the MoE-EN structure and contrastive learning strategy enhance the expression of complementary information from each data source, effectively managing multisource heterogeneity.

2. Relate Work

2.1. Large Language Models

Large language models (LLMs), such as OpenAI’s GPT series [44,45], have fundamentally redefined the boundaries of human–computer interaction since their inception. They essentially represent and compress human knowledge and experience, mapping it into the language space. LLMs continuously demonstrate their powerful capabilities, excelling not only in natural language processing fields but also extending their influence to a wide range of applications, thereby empowering other domains. They can transcend limitations and solve complex problems once solvable only by specialized algorithms or domain experts, demonstrating exceptional adaptability and innovation in cross-disciplinary tasks.

For instance, in the field of mathematical reasoning, Yue et al. [46] fine-tuned LLMs through instruction-based training, enabling them to excel in mathematical tasks. In the medical domain [47], Elliot et al. [48] fine-tuned smaller, more targeted medical LLMs, achieving impressive results comparable to larger models. In the field of chemistry, Li et al. [49] introduced the concept of in-context molecule alignment (ICMA), enabling large language models to learn molecule-description alignment.

Our study in this paper explores the innovative application of LLMs in the domain of remote sensing image analysis, utilizing them to enhance the accuracy of remote sensing land cover classification.

2.2. Multi-Modal Contrastive Representation Learning

Multi-modal contrastive representation learning aims to create a unified representation space that integrates information from various sensory modalities. This framework includes multiple single-modality encoders, each tailored to a specific modality (such as image, text, or audio). These encoders are responsible for transforming the raw input of their respective modalities into a form that can be compared within a shared representation domain. These single-modal encoders are optimized through pre-training on large-scale paired data with a contrastive loss function, ensuring that semantically related inputs from different modalities are close to each other in the representation space, while unrelated inputs are far apart. This approach captures cross-modal semantic associations within the shared representation space, providing a robust foundation for subsequent multi-modal downstream tasks.

In recent years, visual-language contrastive pre-trained models have achieved remarkable success, with models such as CLIP [36] and UMG-CLIP [50] demonstrating outstanding zero-shot retrieval and classification performance, as well as significant generalization capabilities across numerous downstream task [51,52,53,54,55,56,57]. Arora et al. [58] proposed a theoretical framework to directly analyze the generalization of InfoNCE on downstream tasks. Wang et al. [59] introduced alignment and uniformity as empirical metrics to predict downstream performance. Haochen et al. [60] analyzed self-supervised contrastive learning from the perspective of matrix factorization. Huang et al. [61] primarily focused on theoretically analyzing the generalization of self-supervised contrastive learning, understanding and mathematically characterizing the role of data augmentation. Building on these theoretical foundations, this study also explores the role of contrastive learning in multi-modal remote sensing land cover classification.

3. Method

In this section, we present the technical details of DSMSC²N as illustrated in Figure 1, which comprises the following components.

(1)

Automatic generation of category descriptions to create textual data corresponding to the categories.

(2)

ModaUnion Encoder is used for extracting shared features from HSI and LiDAR.

(3)

HSI and LiDAR Encoders are used for extracting visual embedding vectors. Text Encoder is used for extracting language embedding vectors.

(4)

HSI-LiDAR bidirectional contrastive loss (HLBCL), HSI-Text bidirectional contrastive loss (HTBCL), and cross-entropy (CE) loss are used for training the entire model.

3.1. Building Descriptors

We adopted an automated approach to construct a set of hyperspectral category descriptors, as shown in Figure 2. Specifically, we achieved this by instructing LLMs (such as GPT-3) to describe the spectral features unique to each category in hyperspectral images. We guided the LLMs using input prompts like the following example:

Q: What are useful visual features for distinguishing these classes in a Hyperspectral image? To represent with the most significant description.

A: Healthy grass: High chlorophyll content and uniform canopy cover, reflecting strongly in the green and NIR regions.

LLMs possess vast and diverse global knowledge, including extensive geoscientific information in the hyperspectral image domain. By effectively utilizing these large models, researchers can accurately extract relevant knowledge about land cover categories and uncover key features hidden in hyperspectral images. This approach provides a more comprehensive and in-depth perspective for land cover classification, thereby further improving its accuracy and effectiveness.

3.2. Vision and Text Encoder

In multimodal land cover classification research, we followed the mainstream paradigm of vision-language pre-training and trained the model through targeted design and adjustments. Our goal was to fully exploit the information from HSI, LiDAR, and advanced abstract representations of land cover category descriptions provided by language models, achieving more accurate and interpretable land cover category recognition and analysis.

DSMSC²N comprises three different vision encoders and one text encoder. The vision encoders include the ModaUnion encoder, MoE-EN for HSI, and MoE-EN for LiDAR. The ModaUnion encoder extracts common features across multiple modalities, facilitating the deep collaborative processing of heterogeneous data. The MoE-EN module addresses the issue of source heterogeneity by compressing source-specific patterns into distinct encodings.

ModaUnion Encoder: The fusion encoder often struggles to fully utilize the unique information in each modality, while the dual encoder may analyze different modalities in isolation, leading to underutilization of the complementary low-frequency information from HSI and LiDAR. To address this, this study designed the ModaUnion encoder, based on the InceptionNeXt architecture, to effectively integrate information from both modalities, enhance feature learning, and better coordinate complementary information by leveraging HSI’s spatial–spectral coupling and LiDAR’s precise terrain data.

This encoder, through parameter sharing, facilitates the synergistic handling of HSI and LiDAR data. Specifically, the ModaUnion encoder consists of InceptionNeXt modules, as illustrated in Figure 3, inspired by the original Inception framework. These modules decompose large convolutional kernels into parallel smaller convolutional kernels and orthogonal strip convolutional kernels, promoting the effective extraction of information across various scales and orientations. This design not only allows for the efficient capture of low-frequency common features between the two modalities but also significantly boosts the computational efficiency and data processing capacity of the model through parallel processing. Moreover, the implementation of global response normalization (GRN) technology optimizes information transmission, achieving a profound integration of heterogeneous data.

The joint encoding $Z_p\in R^P$ of the two modalities, HSI images and LiDAR data, is achieved through the ModaUnion encoder, represented as follows:

$$Z_p=E_{Union}(X_{HSI},X_{LiDAR})$$

(1)

MoE-EN of HSI: The MoE-EN of HSI consists of a spectral–spatial perception branch and a global spectral understanding branch, as shown in Figure 1. The spectral–spatial perception branch utilizes the spatial window multi-head self-attention (SW-MHSA) mechanism and the channel group multi-head self-attention (CG-MHSA) mechanism. Combined with the global spectral understanding branch, this configuration captures local spectral–spatial features and global spectral semantic information, comprehensively enhancing the model’s capability to understand and represent HSI features.

Specifically, the SW-MHSA mechanism finely partitions the input HSI into multiple local regions, defining a bounded spatial window for each region where attention weights are computed and allocated. Within the multi-head architecture, distinct heads specialize in learning various local spatial dependency patterns. This feature enables SW-MHSA to model local spatial information more efficiently, significantly enhancing the model’s capability to capture and learn spatial contextual information.

Meanwhile, the CG-MHSA mechanism subdivides the channel dimension of the input features into multiple independent groups, performing self-attention computations within each group to learn and comprehend the interdependencies among channels. This design allows the model to focus on exploring the intrinsic connections between spectral channels, thereby deepening the understanding of inter-spectral correlations within HSI. Moreover, through the global spectral understanding branch, the model learns and disentangles dependencies among spectral features, acquiring a contextual representation of the central pixel from a global spectral perspective. This enriches the model’s understanding of contextual semantic information between spectral features.

Through the HSI encoder, this study achieves the precise handling of spatial, global spectral, and local associative information in hyperspectral images. This strengthens the model’s ability to understand spatial characteristics and fosters cooperative learning of spectral relational information, ultimately enhancing the model’s overall representational power and classification accuracy.

The joint encoding, denoted as $Z$, undergoes HSI encoding to obtain HSI embeddings $v_h\in R^N$ and is then mapped to a raw embedding $H_d\in R^D$ through a projection head, represented as follows:

$$v_h=E_{HSI}\left(Z\right)$$

(2)

$$H_d=f_h\left(v_h\right)$$

(3)

MoE-EN of LiDAR: The MoE-EN of LiDAR utilizes a frequency domain attention mechanism (spectrum former), as described in Figure 1. Because LiDAR data contain rich elevation information, such as building boundaries and vegetation shapes, this encoder fully learns LiDAR’s elevation features, significantly enhancing the collaborative classification performance.

Specifically, Spectrum-LiDAR innovatively integrates a transformer encoder architecture with a self-attention mechanism enhanced by Fourier transforms, achieving the efficient processing of LiDAR data. By mapping the data into the frequency domain and amplifying its high-frequency components, the system sharply focuses on critical topographical features essential for constructing an overall scene understanding. This process not only markedly enhances the model’s robustness, enabling it to handle data fluctuations caused by varying viewpoints and environmental changes with stability, but also ensures that even under noisy or challenging acquisition conditions, the model can generate more accurate and stable representations.

Furthermore, Spectrum-LiDAR employs the inverse Fourier transform to convert the frequency-optimized data back into the spatial domain. This step paves the way for subsequent multimodal feature integration, ensuring that information from various sensors is effectively combined.

The joint encoding, denoted as $Z$, undergoes LiDAR encoding to obtain LiDAR embeddings $v_l\in R^N$ and is then mapped to a raw embedding $L_d\in R^D$ through a projection head, represented as follows:

$$v_l=E_{LiDAR}\left(Z\right)$$

(4)

$$L_d=f_l\left(v_l\right)$$

(5)

Text Encoder: We used a text encoder (a frozen pre-trained RemoteCLIP ViT-B-32 model) to generate meaningful embeddings $t\in R^M$ for hyperspectral remote sensing data.

The RemoteCLIP model excels by combining remote sensing images with corresponding specialized textual information (such as land cover classifications and environmental descriptions on satellite images). This approach enables the model to grasp intricate connections between remote sensing visual content and professional narratives. Through this process, the model embeds profound remote sensing expertise, significantly enhancing its ability to recognize complex land cover and phenomena in remote sensing images. This includes accurately distinguishing between different vegetation communities, analyzing changes in land use patterns, and precisely identifying water body characteristics. Compared to the general-purpose CLIP model, RemoteCLIP demonstrates superior accuracy and efficiency in specialized classification and recognition tasks for remote sensing images, due to its tailored learning and understanding of the remote sensing context. We project them to $T_d\in R^D$ as follows:

$$t=E_{txt}\left(X_{Text}\right)$$

(6)

$$T_d=f_t\left(t\right)$$

(7)

where $f_t$ denotes the projection head, while $E_{txt}$ represents the text encoder. This arrangement ensures that the embedding dimension $D$ aligns with that of the visual encoder, rendering it conducive for contrastive learning.

3.3. Loss Function

In previous discussions, the heterogeneity of multi-source data has been clearly identified as a key bottleneck limiting the performance improvement in hyperspectral multimodal land cover classification. To address this issue, we introduced an innovative strategy by constructing a specially designed joint training loss function system to guide the model in deeply exploring and effectively integrating shared features from multi-source data. The optimization framework for this training phase comprised three core components. Firstly, HSI-LiDAR bidirectional contrastive loss (HLBCL) is for HSI and LiDAR data. Secondly, HSI-Text bidirectional contrastive loss (HTBCL) is for HSI and text data; and finally, the cross-entropy (CE) loss is for land cover classification. In conclusion, the overall optimization objective can be formally expressed as follows:

$${\mathcal{L}}_{all}=\lambda ({\omega }_1{\mathcal{L}}_{BCL}^{HL}+{\omega }_2{\mathcal{L}}_{BCL}^{HT})+(1-\lambda ){\mathcal{L}}_{CE}$$

(8)

$${\mathcal{L}}_{CE}=-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\sum }_{k=1}^K(y_{ik}log(p\left(x_{ik}\right)\left)\right)$$

(9)

where $\lambda$ is the hyperparameter to balance the training of the CE Loss and multiple BCL losses; $\omega$ is a learnable weight used to aggregate the contributions of different modal sources to land classification; $N$ represents the number of samples; $K$ denotes the total number of classes; $y_{ik}$ indicates whether the $i^{th}$ sample belongs to the $k^{th}$ land cover class; and $p\left(x_{ik}\right)$ denotes the probability that the model predicts the $i^{th}$ sample belongs to the $k^{th}$ land cover class.

To comprehensively elucidate the entire training process, we began by providing a detailed introduction to BCL. Given N paired instances from two different modalities, we mapped the $i^{th}$ pair to L2-normalized embeddings $x_i$ and $t_i$ via two encoders.

$${\mathcal{L}}_{i,j}^{x\to t}=log{\displaystyle \frac{e^{(\frac{sim\left(x_i,t_i\right)}{\tau })}}{{\sum }_{k=1, k\ne i}^N{e}^{(\frac{sim\left(x_i,t_k\right)}{\tau })}}}$$

(10)

$${\mathcal{L}}_{i,j}^{t\to x}=log{\displaystyle \frac{e^{(\frac{sim\left(t_i,x_i\right)}{\tau })}}{{\sum }_{k=1, k\ne i}^N{e}^{(\frac{sim\left(t_i,x_k\right)}{\tau })}}}$$

(11)

$$sim(x,t)={\displaystyle \frac{x^{T}t}{‖x‖‖t‖}}$$

(12)

$${\mathcal{L}}_{BCL}=-{\displaystyle \frac{1}{2N}}{\sum }_{i=1}^N\left[{\mathcal{L}}_{i, j}^{x\to t}+{\mathcal{L}}_{i, j}^{t\to x}\right]$$

(13)

where $sim(\cdot )$ represents the cosine similarity between the embeddings of $x_i$ and $t_i$ and $\tau$ represents a temperature parameter initialized at 0.07.

Next, we analyzed why multi-modal contrastive learning is useful for downstream land classification tasks, which can be decomposed into two parts.

$${\mathcal{L}}_{i,j}^{x\to t}={\displaystyle \frac{sim\left(x_i,t_i\right)}{\tau }}-log({\sum }_{j=1}^N{e}^{({\displaystyle \frac{sim\left(x_i,t_j\right)}{\tau }})})$$

(14)

Here, $f(u)={\displaystyle \frac{u}{‖u‖}}$. we reformulated the equation as follows:

$${\mathcal{L}}_{i,j}^{x\to t}={\displaystyle \frac{f(x_i{)}^{T}f(t_i)}{\tau }}-log({\sum }_{j=1}^N{e}^{({\displaystyle \frac{f(x_i{)}^{T}f(t_j)}{\tau }})})$$

(15)

$${\mathcal{L}}_{i,j}^{x\to t}={\mathcal{L}}_{pos}-{\mathcal{L}}_{reg}$$

(16)

The first part ${\mathcal{L}}_{pos}$ ensures that the positive samples are as close as possible in the feature space, which is also one of the goals of contrastive learning. The second part ${\mathcal{L}}_{reg}$ ensures that the centers of different categories are sufficiently far apart in the feature space, as simply bringing positive samples closer together might lead to the dimensional collapse.

As depicted in Figure 4, the samples from different modalities of the same category are drawn infinitely close to the same point, and only a minimal angular separation ${\displaystyle \frac{⟨E_{x_i},E_{t_j}⟩}{‖E_{x_i}‖‖E_{t_j}‖}}<1$ is needed to accurately distinguish different categories. This approach prevents the learned representations from collapse.

The analysis indicated that multimodal contrastive learning enhanced the clustering of the learned feature space, maintaining sufficient distance between category centers and preventing dimensional collapse. This mechanism significantly improved the accuracy of the downstream classification tasks.

Utilizing HTBCL and HLBCL, our method enabled the model to acquire more accurate and semantically rich land cover feature representations. These loss mechanisms work synergistically, driving the model to deeply explore and understand the similarities and differences between different modal data. This comprehensive utilization of multi-source data significantly enhances the performance and accuracy of land cover classification tasks.

3.4. Final Classifier

The overall training process of DSMSC²N is elaborated in Algorithm 1. Feature embeddings $H_d$ and $L_d$ extracted from HSI and LiDAR data by the MoE-EN are fed into a carefully designed multilayer perceptron (MLP) to perform the final classification task. This MLP is built on multiple hidden layers, each integrating linear layer and GReLU activation functions, aimed at enhancing the network’s nonlinear expressive power. The size of the output layer is customized according to the specific number of categories in the classification task. The Softmax function is used to normalize the outputs, ensuring that the sum of category probabilities for each sample is 1, thus constructing the posterior probability distribution of category. By identifying the maximum value in the probability distribution, precise land cover classification $Y_p$ is achieved as follows:

$$p\left(x_i\right)=MLP\left(Concat\left(H_d^i,L_d^i\right)\right)$$

(17)

$$Y_p=argmax\left(p\left(x_i\right)\right)$$

(18)

$p\left(x_i\right)$ denotes the probability that the model predicts the $i^{th}$ sample.

Algorithm 1 gives the overall network training process of DSMSC²N.

Algorithm 1 Training DSMSC2N model.

Input: HSI images: $D_{HSI}$, LiDAR data: $D_{LiDAR}$, Text data: $X_{Text}$, Training labels: $y$

Output: land cover result: $Y_p$    1. Initialize: batch size = 64, epochs = 200, the initial learning rate set to 5 × 10−4 of AdamW;    2. Patches: Divide $D_{HSI}$$\mathrm{and} D_{LiDAR}$$\mathrm{into} \mathrm{contextual} \mathrm{cubes} X_{HSI}$$\mathrm{and} \mathrm{patches} X_{LiDAR}$ respectively;    3. for i = 1 to epochs do    4.  \\ Extact Featre Embedding-ModaUnion Encoder    5.  $Z_P^i$$\leftarrow \mathrm{ModaUnion} \mathrm{Encoder} (X_{HSI}^i$$, X_{LiDAR}^i)$;    6.  $H_d^i$$\leftarrow \mathrm{MoE}\text{-}\mathrm{EN} \mathrm{of} \mathrm{HSI}\left(Z_P^i\right)$;    7.  $L_d^i$$\leftarrow \mathrm{MoE}\text{-}\mathrm{EN} \mathrm{of} \mathrm{LiDAR}\left(Z_P^i\right)$;    8.  $T_d^i$$\leftarrow \mathrm{Text} \mathrm{Encoder}\left(X_{Text}^i\right)$;    9.  Optimize Feature Representation and Update the discriminators by optimizing Equation (8);    10.    Obtain the land cover result: $Y_p$ by computing Equations (17) and (18).    11.   end for

4. Results

4.1. Datasets

Houston 2013 dataset: This dataset involves two datasets—hyperspectral images and LiDAR-derived DSM, both consisting of 349 × 1905 pixels with the same spatial resolution (2.5 m). The data were acquired by the NSF-funded Center for Airborne Laser Mapping (NCALM) over the University of Houston’s 2013 campus and the neighboring urban area. The HSI has 144 spectral bands in the 380 nm to 1050 nm region, including 15 classes. Figure 5 gives the visualization results of the Houston 2013 dataset. These data and reference classes can be obtained online from the IEEE GRSS website (http://dase.grss-ieee.org/ (accessed on 10 September 2022)). The land class details of Houston 2013 are listed in

Table 1. Houston 2013 dataset: the numbers of training and testing samples for each class.

Class	Class Name	Train Num	Test Num	Color
C1	Healthy Grass	198	1053
C2	Stressed Grass	190	1064
C3	Synthetic Grass	192	505
C4	Trees	188	1056
C5	Soil	186	1056
C6	Water	182	143
C7	Residential	196	1072
C8	Commercial	191	1053
C9	Road	193	1059
C10	Highway	191	1036
C11	Railway	181	1054
C12	Parking Lot1	192	1041
C13	Parking Lot2	184	285
C14	Tennis Court	181	247
C15	Running Track	187	473
-	Total	2832	12,197

.

Trento dataset: This dataset involves two datasets—hyperspectral images and LiDAR-derived DSM, both consisting of 600 × 166 pixels with the same spatial resolution (1 m). The data were acquired by the AISA Eagle sensor, and the LiDAR DSM was produced using first- and last-point cloud pulses obtained by the Optech ALTM 3100EAsensor. The HSI has 63 spectral bands covering the 402.89 to 989.09 nm region and includes six classes. Figure 6 gives the visualization results of the Trento dataset, and

Table 2. Trento dataset: the numbers of training and testing samples for each class.

Class	Class Name	Train Number	Test Number	Color
C1	Apples	129	3905
C2	Buildings	15	2778
C3	Ground	105	374
C4	Woods	154	8969
C5	Vineyard	184	10,317
C6	Roads	122	3252
-	Total	819	29,395

lists the number of samples of different classes and the color of each class.

MUUFL Gulfport dataset: This dataset was captured over the University of Southern Mississippi campus in November 2010 using the reflective optics system imaging spectrometer (ROSIS) sensor. It comprises a hyperspectral image (HSI) with dimensions of 325 × 220 pixels and 72 spectral bands. Additionally, the LiDAR image includes elevation data provided in two raster layers, with a total of 64 bands after the removal of 8 initial and final bands affected by noise. This dataset encompasses 11 urban land-cover classes, with 53,687 ground truth pixels annotated. There are a total of 53,687 ground truth pixels spanning 11 classes. For training, 100 pixels per class are selected, leaving 52,587 pixels for testing. Figure 7 provides a visualization of the MUUFL Gulfport dataset, and the number of samples in each class and the corresponding colors for each class is listed in

Table 3. MUUFL Gulfport dataset: the numbers of training and testing samples for each class.

Class	Class Name	Train Number	Test Number	Color
C1	Trees	100	23,146
C2	Mostly grass	100	4170
C3	Mixed ground surface	100	6782
C4	Dirt and sand	100	1726
C5	Road	100	6587
C6	Water	100	366
C7	Buildings shadow	100	2133
C8	Buildings	100	6140
C9	Sidewalk	100	1285
C10	Yellow curb	100	83
C11	Cloth panels	100	169
-	Total	1100	52,587

.

4.2. Implementation Details

Configuration: This experiment was conducted on a computational server equipped with a Tesla P100 GPU and an Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz processor running Ubuntu 18.04. The experimental environment was built using Python 3.8 and PyTorch 2.0.1 deep learning framework. Our network utilized a fixed patch size of 11 × 11 for all datasets. These patches are centered around pixels with known ground truth labels. The model was trained with a batch size of 64 for 200 epochs. The training and validation datasets were randomly split at a ratio of 8:2, ensuring no overlap between training and test samples. The AdamW optimizer and cosine learning rate scheduling strategy were employed, with an initial learning rate set to 5 × 10⁻⁴ and a weight decay coefficient of 1 × 10⁻¹. For the CG-MHSA module, the number of groups $g$ was set to 1, and for the contrastive learning loss, the hyperparameter $\tau$ was set to 0.07. Additionally, the weighting hyperparameter $\lambda$ in the final joint loss was set to 0.6.

Evaluation Matrix: We employed several widely used quantitative metrics to evaluate the performance of the proposed network and compared it with other methods. These metrics include overall accuracy (OA), average accuracy (AA), and the statistical Kappa coefficient (K). OA indicates the proportion of correctly classified test samples to all test samples, while AA represents the average accuracy across classes. The statistical Kappa coefficient (K) reflects the level of agreement between the classification maps generated by the model under consideration and the provided ground truth.

4.3. Ablation Study

To assess the impact of the proposed network architecture and the integration of LiDAR and text modal data on land classification performance, ablation experiments were conducted on the Trento dataset. Transformer was employed as the baseline network for comparative analysis. Furthermore, the impact of the joint loss function on the proposed network was examined.

Effect of Incorporating LiDAR and Text Modal: The experimental results in

Table 4. Ablation experiments of incorporating multi-modal on Trento.

Data Source			Evaluation Index
HSI	LiDAR	Text	OA (%)	AA (%)	K × 100
✓	✗	✗	95.99	94.55	94.64
✗	✓	✗	88.84	85.67	85.21
✓	✓	✗	98.75	98.13	98.33
✓	✓	✓	98.78	98.14	98.37

demonstrate a significant improvement in hyperspectral land cover classification performance with the inclusion of LiDAR and textual modalities.

We compared the classification performance of single HSI images, single LiDAR data, the combination of HSI and LiDAR, and the tri-modal combination of HSI, LiDAR, and text. The LiDAR data provide elevation information, helping to distinguish different land cover classes, while the textual modality contributes semantic descriptions of land covers (such as names, functions, and attributes), enriching the semantic context of the classification task. The tri-modal combination exhibited improvements over the baseline models using only HSI or LiDAR, with increases in OA, AA, and Kappa of 2.79%, 3.59%, and 3.73%, respectively, compared to the HSI baseline model, and of 9.94%, 2.47%, and 13.16%, respectively, compared to the LiDAR baseline model. Additionally, compared to the combined HSI-LiDAR baseline model, the tri-modal setup achieved increases in OA, AA, and Kappa of 0.03%, 0.01%, and 0.04%, respectively, providing evidence of the value of integrating LiDAR and text modalities for enhancing the hyperspectral land classification performance.

Effect of Fusion Strategies: We investigated the collaborative methods of auxiliary encoders based on LiDAR and text data and a primary encoder based on HSI in classification models, implementing three fusion strategies: early fusion, late fusion, and our proposed fusion method. Early fusion integrates the feature information extracted at various levels by the LiDAR and text auxiliary encoders with the hyperspectral primary encoder. Late fusion combines the features of the LiDAR and text auxiliary encoders with those of the HSI primary encoder only at the final layer of the model. In contrast, our proposed fusion method first uses the ModaUnion module for early fusion to extract the common features from the HSI and LiDAR modalities, then employs MoE-EN to obtain the characteristic features of the HSI and LiDAR modalities, and finally performs late fusion on the features of HSI, LiDAR, and text. The experimental results are shown in

Table 5. Ablation experiments of fusion strategies on Trento.

Network Structure	Evaluation Index
	OA (%)	AA (%)	K × 100
Transformer (EF)	96.96	94.43	95.28
MoE-EN(LF)	97.01	94.46	96.01
ModaUnion + MoE-EN (MF)	98.78	98.14	98.37

.

Compared with the early and late fusion strategies, our proposed fusion method leverages the advantages of the ModaUnion module and MoE-EN, demonstrating superior classification performance and effectively utilizing the complementary information among LiDAR, HSI, and text modalities.

Effect of MoE-EN: We analyzed the experimental results of the expert hybrid network structure, revealing its outstanding performance and advantages in remote sensing classification tasks. Regarding the spectral–spatial perception branch in the HSI encoder, the roles of the SW-MHSA and CG-MHSA mechanisms were discussed (see

Table 6. Ablation experiments of MoE-EN on Trento.

Network Structure	Evaluation Index
	OA (%)	AA (%)	K × 100
MHSA	97.41	95.98	96.54
SW-MHSA	97.96	96.79	97.27
CG-MHSA	98.33	97.42	97.78
SW-MHSA + CG-MHSA	98.78	98.14	98.37

). The experimental results indicate that using CG-MHSA alone, compared to using SW-MHSA alone, achieved increases of 0.37%, 0.63%, and 0.51% in OA, AA, and Kappa metrics, respectively. This comparison indirectly reveals the importance of the rich spectral information contained in HSI images for improving the classification performance. Specifically, in analyses dominated by spectral features, CG-MHSA effectively utilizes this information to enhance the feature learning capabilities. Moreover, the combined use of SW-MHSA and CG-MHSA, compared to using no mechanism, improved OA, AA, and Kappa metrics by 1.37%, 2.16%, and 1.83%, respectively.

When compared to using SW-MHSA alone, the combined approach increased OA, AA, and Kappa by 0.82%, 1.35%, and 1.1%, respectively. Additionally, compared to using CG-MHSA alone, the combined approach resulted in enhancements of 0.45%, 0.72%, and 0.59% in OA, AA, and Kappa, respectively. This series of experimental results strongly indicates the critical importance of simultaneously considering local spatial structural features and overall spectral information in HSI classification tasks. Particularly in complex scenarios, a single type of attention mechanism is insufficient to fully exploit all potentially beneficial information. The fusion of spatial features and spectral information becomes an effective strategy for improving recognition accuracy. This finding further validates the significant enhancement of HSI image classification performance achieved through the combined application of SW-MHSA and CG-MHSA, highlighting the value of multidimensional information integration.

Furthermore, we discussed the roles of the spectral–spatial perception branch, global spectral understanding branch, and the LiDAR encoder (see

Table 7. Ablation experiments of MoE-EN on Trento.

MoE-EN			ACC (%)						Evaluation Index
Spatial–Channel Sub-Branch	Spectral Context Sub-Branch	Spectrum Former	Apples	Buildings	Ground	Woods	Vineyard	Roads	OA (%)	AA (%)	K × 100
✓	✗	✗	97.17	98.03	99.37	99.92	99.99	87.55	98.10	97.01	97.46
✗	✓	✗	88.37	99.08	77.99	95.72	95.03	69.04	91.72	87.53	89.01
✓	✓	✗	98.35	99.59	98.74	99.99	99.97	90.47	98.35	97.42	97.81
✓	✗	✓	96.78	98.76	95.81	99.58	99.93	92.27	98.42	97.19	97.90
✗	✓	✓	86.59	95.59	77.99	97.27	91.40	75.20	91.00	87.34	88.04
✓	✓	✓	97.76	99.48	98.95	99.62	99.97	93.01	98.78	98.13	98.37

for details). By incorporating global spectral understanding, the model’s performance in handling land cover classes with complex spectral characteristics, such as roads, showed improvements of 0.25%, 0.41%, and 0.35% in OA, AA, and Kappa, respectively, compared to the model without global spectral understanding. However, if the spectral–spatial perception branch is absent, leaving only the global spectral understanding branch, there is a decrease in OA, AA, and Kappa of 6.63%, 9.89%, and 8.8%, respectively. The spectrum former in the LiDAR encoder efficiently extracts the elevation features, such as building boundaries and vegetation shapes, improving the recognition of land cover categories dependent on elevation information, such as ground and roads.

We also observed an interesting phenomenon: when the global spectral understanding branch is combined with the spectrum former, the performance unexpectedly decreases compared to using only the global spectral understanding branch. Our analysis suggests that the elevation frequency domain information extracted by spectrum former interferes with the spectral feature information captured by the global spectral understanding branch, resulting in performance degradation. However, the introduction of the spectral–spatial perception branch, through operations in the spatial dimension, helps the model better understand and utilize the relationships among the spectral, spatial, and frequency domain features, thus enhancing performance.

In the MoE-EN, HSI and LiDAR demonstrate significant advantages in capturing HSI features and extracting elevation features, respectively, collectively contributing to the precise and efficient execution of remote sensing classification tasks.

Effect of Loss Function: We also discussed the ablation study on the effect of loss functions on the Trento dataset, as illustrated in Figure 8 with the t-SNE plots of feature embeddings. Compared to using the BCE loss function alone, the combined application of HTBCL and HLBCL significantly improves the feature embedding performance. This combined approach demonstrates a more pronounced intra-class cohesion and inter-class separation, resulting in tighter clustering and clearer boundaries between categories. These findings strongly confirm that the combination of HTBCL and HLBCL can optimize the model’s feature representation capabilities, which is particularly critical for addressing the complex challenges in land cover classification.

4.4. Comparison with Other Methods

To validate the effectiveness of our proposed model for joint land cover classification, we compared it with seven state-of-the-art methods. Specifically, these methods include: two-branch [30], which utilizes a dual-branch convolutional network for land cover classification; EndNet [31], which integrates multi-modal features using a deep encoding-decoding network for land cover classification; MDL-Middle [62], which employs an intermediate layer fusion network with multi-source features for land cover classification; MAHiDFNet [63], which employs a multi-attention hierarchical dense fusion network for land cover classification; FusAtNet [64], which combines self-attention and cross-attention mechanisms to extract spatial–spectral features from HSI and LiDAR data for land cover classification; CALC [65], a classification method based on coupled adversarial learning for land cover classification; and SepG-ResNet50 [66], which introduces the SepDGConv module to automatically learn the group convolution structure within a single-stream neural network, enabling deep single-stream CNN models for land cover classification based on this module.

We meticulously followed the original documentation and open-source code default configurations for all comparative methods, optimizing their parameters accordingly. To ensure consistency, the training and testing samples used by all comparative methods were identical to those used by the proposed model. Performance metrics were derived from the average results of 10 experiments for each method, enhancing statistical reliability. Although data augmentation is a common strategy to mitigate overfitting and improve classification performance, simple image transformation-based augmentations (e.g., flipping and rotating) can be quickly learned by the model, potentially compromising the fairness of comparisons. Therefore, DSMSC²N did not employ any data augmentation strategies in its comparative evaluations.

Results on Houston 2013 dataset:

Table 8. Comparison of classification results on Houston 2013.

Class	Two- Branch	EndNet	MDL- Middle	MAHiDFNet	FusAtNet	CALC	SepG-ResNet50	DSMSC2N
Healthy grass	82.90	81.58	83.10	82.91	80.72	86.51	72.36	90.12
Stressed grass	84.31	83.65	85.06	84.68	97.46	84.59	77.35	84.59
Synthetic grass	96.44	100.00	99.60	100.00	90.69	90.50	34.85	98.81
Trees	96.59	93.09	91.57	93.37	99.72	91.86	86.84	90.91
Soil	99.62	99.91	98.86	99.43	97.92	100.00	91.38	100.00
Water	82.52	95.10	100.00	99.30	93.71	99.30	95.10	95.80
Residential	85.54	82.65	97.64	83.58	91.98	88.71	81.62	91.12
Commercial	76.64	81.29	88.13	81.96	85.19	83.19	61.73	95.38
Road	87.35	88.29	85.93	83.76	85.93	91.60	86.31	95.04
Highway	60.71	89.00	74.42	66.41	69.50	65.44	46.26	67.66
Railway	90.61	83.78	84.54	74.57	85.48	95.92	69.35	97.22
Parking Lot1	90.78	90.39	95.39	88.38	89.15	90.78	86.94	93.66
Parking Lot2	86.67	82.46	87.37	88.42	77.19	91.93	78.25	92.63
Tennis court	92.31	100.00	95.14	100.00	84.21	94.74	87.04	100.00
Running track	99.79	98.10	100.00	100.00	87.53	100.00	18.82	97.46
OA (%)	86.68	88.52	89.55	85.87	88.14	88.84	72.67	91.49
AA (%)	87.52	89.95	91.05	88.55	87.76	90.34	71.63	92.69
K × 100	85.56	87.59	87.59	84.76	87.12	87.92	70.40	90.76

presents the classification results of different algorithms on the Houston 2013 dataset. The results demonstrate that the proposed DSMSC²N outperforms other methods in terms of OA (91.49%), AA (92.69%), and Kappa (90.76%) coefficients. It achieves the highest accuracy in seven categories, namely “Healthy grass”, “Soil”, “Commercial”, “Road”, “Railway”, “Parking Lot2”, and “Tennis court”. In particular, the performance of “Soil” (100%) stands out significantly compared to other methods. However, the effectiveness of SepDGConv, which employs a deep single-stream CNN model, falls short in comparison to other methods. Although SepDGConv exhibits a clever design with good global modeling capability, the feature representation of a single-stream model is insufficient for the joint classification of the HSI and LiDAR data.

Figure 9 presents the classification result maps of all the methods included in Table 8. In comparison to other models, DSMSC²N demonstrates a reduction in classification errors, particularly for ground objects with high similarity in complex scenes, such as “Road”. This highlights the significant advantage of DSMSC²N, which leverages a cleverly designed network structure guided by text information, in effectively expressing and organizing heterogeneous information.

Results on Trento dataset:

Table 9. Comparison of classification results on Trento.

Class	Two- Branch	EndNet	MDL- Middle	MAHiDFNet	FusAtNet	CALC	SepG-ResNet50	DSMSC2N
Apples	98.61	93.95	99.93	100.00	99.45	94.55	93.28	99.33
Buildings	98.93	96.54	98.14	99.80	89.87	99.55	99.38	97.42
Ground	75.16	96.24	97.08	96.03	91.23	92.69	74.35	96.66
Woods	98.72	99.36	99.93	100.00	93.86	100.00	99.88	99.29
Vineyard	97.43	80.72	98.54	95.30	92.92	99.53	95.91	99.70
Roads	96.83	90.14	89.51	86.74	90.71	93.82	68.05	96.60
OA (%)	96.53	90.86	98.14	96.89	93.53	98.30	93.82	98.93
AA (%)	92.30	92.81	97.19	96.31	93.01	96.69	88.47	98.16
K × 100	95.38	88.01	97.52	95.87	91.51	97.74	91.79	98.57

presents the classification results of all methods on the Trento dataset, further demonstrating the effectiveness of our proposed DSMSC²N, which outperforms other methods in terms of OA with a score of 98.93%, AA with a score of 98.16%, and Kappa with a score of 98.57%. Considering the differences in size and data type, the classification accuracy on the Trento dataset is generally higher than that of the Houston 2013 and MUUFL Gulfport datasets. From Table 9, it can be observed that DSMSC²N achieves satisfactory results in each category, which can be attributed to the carefully designed ModaUnion encoder and MoE-EN modules in DSMSC²N. In particular, the performance of the “Vineyard” category (99.70%) stands out compared to other methods.

We visualize the classification results in Figure 10. It can be observed that DSMSC²N accurately delineates the edges of categories such as “Apples”, presenting clearer and smoother contours. In contrast, the boundaries obtained by other methods exhibit noticeable jagged edges and lack smoothness. This indicates that the DSMSC²N model is more powerful than other classification methods in terms of fine-grained feature representation and extraction capabilities. It can capture subtle details of complex scenes and achieve more precise and coherent semantic understanding, particularly excelling in depicting complex boundaries.

Results on MUUFL Gulfport dataset:

On the challenging MUUFL Gulfport dataset,

Table 10. Comparison of classification results on MUUFL Gulfport.

Class	Two- Branch	EndNet	MDL- Middle	MAHiDFNet	FusAtNet	CALC	SepG- ResNet50	DSMSC2N
Trees	90.29	83.55	87.31	89.87	80.36	91.97	86.78	94.23
Mostly grass	75.68	79.38	76.38	63.19	73.57	81.77	78.47	85.81
Mixed ground surface	69.71	76.28	68.33	75.85	68.24	77.56	71.20	81.97
Dirt and sand	93.97	87.08	78.74	96.18	70.74	95.19	89.98	86.65
Road	91.79	89.59	83.76	88.52	80.95	89.19	76.38	89.72
Water	99.73	95.90	88.52	85.25	81.15	100.00	99.73	99.65
Buildings shadow	91.84	88.28	92.12	9..72	89.40	95.17	91.70	94.18
Buildings	94.79	92.07	89.69	95.44	87.92	96.91	87.31	90.87
Sidewalk	72.30	76.96	77.04	75.80	77.98	69.81	69.88	79.75
Yellow curb	96.39	95.18	86.75	91.57	78.31	95.18	90.36	92.00
Cloth panels	97.63	97.63	99.41	99.41	99.41	100.00	99.41	98.82
OA (%)	87.03	84.33	83.54	86.45	79.24	89.31	82.90	91.19
AA (%)	88.56	87.45	84.37	86.80	80.73	90.25	85.56	90.87
K × 100	83.10	79.85	78.84	82.37	73.81	86.05	77.94	88.33

presents the classification results of various methods. It is encouraging to note that our approach achieved the highest OA of 98.93%, AA of 98.16%, and Kappa of 98.57%. These results demonstrate that DSMSC²N, by effectively utilizing multimodal information and leveraging language models, achieved the highest accuracy rates in categories such as “Trees”, “Soil”, “Mostly grass”, “Mixed ground surface”, and “Sidewalk”.

We visually depict the classification results in Figure 11. Upon observation, DSMSC²N exhibited a superior visual performance compared to other comparative algorithms, resulting in a closer proximity to the ground truth. Notably, DSMSC²N successfully captured subtle details in complex scenes, such as “Sidewalk”, and demonstrated enhanced coherence in the classification of “Trees”, accompanied by clear boundaries. These findings further validate the model’s exceptional performance in capturing details and accurately delineating boundaries.

4.5. Computational Complexity Analysis

In this paper, we evaluate the computational complexity of different models using two metrics, floating-point operations (FLOPs) and the number of parameters (#param), as shown in

Table 11. Comparison of FLOPs and parameters of different classification models on Houston 2013.

Methods	#Param. (M)	FLOPs (M)
Two-Branch	5.6	120.81
EndNet	0.09	0.09
MDL-Middle	0.1	5.28
MAHiDFNet	77.0	155.00
FusAtNet	36.9	3460.31
CALC	0.3	28.79
SepG-ResNet50	14.7	48.28
DSMSC2N	0.7	104.92

. FLOPs represent the number of floating-point operations required for processing a single image and completing one forward pass, reflecting the time complexity of the model. #param represents the total number of parameters in the model, determining the size of the model itself and directly impacting the memory consumption during inference, reflecting the spatial complexity of the model.

Compared to the two-branch, EndNet, MDL-Middle, MAHiDFNet, FusAtNet, CALC, and SepG-ResNet50 models, our proposed model in this paper presents a more compact and efficient architecture design, with a relatively small number of trainable parameters, ranking fourth among the eight models. It achieves good performance with a lower model space complexity. Additionally, DSMSC²N employs multiple transformer-based encoder branches, enabling more comprehensive feature extraction. However, the multi-head self-attention mechanism in transformers incurs high memory usage and frequent high-bandwidth memory (HBM) read/write operations, imposing significant pressure on computational efficiency and memory consumption. To address this challenge and improve DSMSC²N’s computational efficiency, we accelerate it using Flash Attention [67], resulting in DSMSC²N ranking fifth in terms of FLOPs among the eight models. Considering the trade-off between effectiveness and complexity, although our model requires more FLOPs, it occupies less memory space. This trade-off leads to a significant improvement in classification accuracy, achieving an optimal balance between performance and complexity.

5. Conclusions

This paper introduces a novel remote sensing multimodal semantic cooperative classification Network designed to fully leverage the semantic information from large language models and synergistically utilize multi-source remote sensing data for the precise joint classification of HSI and LiDAR data. Using an instruction-driven approach, DSMSC²N employs LLMs to generate informative descriptions of land cover categories enriched with remote sensing domain knowledge, guiding the model to focus on and exploit key features accurately. Furthermore, DSMSC²N effectively fuses multi-source remote sensing data using the ModaUnion and MoE-EN modules, mining complementary features across different modalities and mitigating the adverse effects of data heterogeneity. Comprehensive experiments on three benchmark datasets validated the superior performance of DSMSC²N compared to various baseline methods. This work demonstrates the immense potential of integrating LLMs with multimodal remote sensing learning, offering a novel and effective solution for enhancing the accuracy and robustness of remote sensing scene understanding.

In the future, we plan to explore advanced fusion strategies and further refine the design of the textual modality to achieve greater performance improvements in remote sensing tasks. Concurrently, we will also continue to refine the model architecture, focusing on in-depth adjustments to the ModaUnion and MoE-EN modules. Our plan includes implementing adaptive mechanisms that allow the network to adjust automatically according to data characteristics and task requirements, enhancing its adaptability across diverse environments and complex scenarios.