A Hierarchical Semantic Consistency Constraint Framework for Hyperspectral and LiDAR Data Joint Classification

Highlights

What are the main findings?

• A hierarchical semantic consistency constraint framework (HSCC) is proposed for joint classification of HSI and LiDAR data.
• The framework progressively strengthens cross-modal interaction through a progressive interactive fusion network (PIFNet) and introduces a semantic consistency constraint (SCC) strategy to explicitly enforce feature similarity for the same object across different modalities and levels, effectively mitigating semantic drift.

What are the implications of the main findings?

• HSCC achieves state-of-the-art classification performance on three public datasets, providing a high-accuracy solution for multi-source remote sensing data fusion in complex land-cover scenes.
• HSCC offers a hierarchical semantic alignment paradigm for heterogeneous feature fusion of multi-source remote sensing data, which can be extended to other multimodal tasks.

Abstract

Hyperspectral image (HSI) and LiDAR data fusion is valuable for land-cover classification in complex surface scenes. Existing methods typically extract features from each modality independently and then consider how to fuse them, ignoring the semantic consistency between features of different modalities and across different hierarchical levels. Moreover, fully mining and exploiting the complementary information between multimodal remote sensing data remains a critical issue. To address these challenges, this paper proposes a hierarchical semantic consistency constraint (HSCC) framework for HSI and LiDAR data joint classification. The framework is co-constructed by a progressive interactive fusion network (PIFNet) and a semantic consistency constraint (SCC) strategy. Specifically, PIFNet progressively calibrates the semantic representations of multimodal features at different abstraction levels through Cross-Modal Shared Attention and Symmetric Cross-Attention mechanisms, promoting information parity in deep interactions. The SCC strategy establishes multi-level semantic associations and employs a semantic consistency constraint loss to guide the network to autonomously maintain the consistency of the same land-cover object across heterogeneous feature representations, thereby further enhancing the discriminative power of the fused features. Experiments on three public datasets, MUUFL, Houston2013, and Augsburg, demonstrate that HSCC outperforms current state-of-the-art methods, validating its effectiveness in multi-source remote sensing data fusion classification tasks.

1. Introduction

With the rapid development of Earth observation technology, multi-source remote sensing data fusion has become a key technique for characterizing complex land surface scenes and improving land-cover interpretation accuracy [1,2]. In this context, hyperspectral imaging (HSI) and light detection and ranging (LiDAR) data have attracted sustained attention in the remote sensing community due to their unique and complementary information characteristics [3]. HSI captures fine spectral features of ground objects through dozens to hundreds of continuous narrow bands, providing crucial evidence for material identification [4] and demonstrating significant potential in tasks such as anomaly detection [5] and mixed pixel unmixing [6]. However, in highly complex urban and other surface environments, HSI data are often constrained by the phenomena of “same object with different spectra” and “different objects with same spectrum”, as well as limited spatial details [7,8,9], leading to a bottleneck in classification using only spectral information. In contrast, LiDAR can actively acquire accurate three-dimensional elevation and geometric structure information, exhibiting strong robustness to illumination and atmospheric conditions [10], thus effectively compensating for the limitations of HSI in characterizing object morphology, vertical structure, and spatial distribution [11]. Exploring efficient fusion mechanisms for HSI and LiDAR data to overcome the heterogeneity among multi-source data and fully leverage their complementary advantages has become a frontier topic in multi-source remote sensing information processing, demonstrating great application potential in urban planning [12], ecological environment monitoring [13], precision agriculture management [14], and forest resource inventory [15].

In early studies, classification primarily relied on traditional machine learning methods such as support vector machines [16] and random forests [17]. These approaches typically required manual feature design and selection based on empirical knowledge, for example, extracting morphological profiles [18] or extended multi-attribute profiles [19], followed by simple concatenation of multi-source features before feeding them into a classifier. Although these methods validated the potential of multi-source data fusion, their performance was constrained by feature engineering heavily dependent on prior knowledge, making it difficult to adaptively capture the complex and deep nonlinear correlations between HSI and LiDAR data. Consequently, when processing high-dimensional and heterogeneous remote sensing data, their classification accuracy and generalization ability faced significant bottlenecks [20,21].

With the rise of deep learning, convolutional neural networks (CNNs), leveraging their powerful local perception capability, can effectively extract spatial–spectral features from HSI as well as texture and elevation information from LiDAR data. Roy et al. [22], Feng et al. [23], Zhang et al. [24], and Li et al. [25] designed dual-branch or multi-branch CNN architectures to process heterogeneous data, aiming to learn spectral–spatial and geometric elevation features separately and then fuse them effectively. To enhance feature representation capability, Xu et al. [26] and Li et al. [27] combined traditional morphological profiles with CNN deep features to construct hybrid feature models. However, these CNN-based methods are inherently limited by the local receptive field of CNNs, making it difficult to effectively capture the long-range contextual associations widely present in remote sensing scenes, and the fused features still suffer from semantic ambiguity.

In recent years, the Transformer, with its global self-attention mechanism, has significantly enhanced the holistic understanding of complex remote sensing scenes [28]. Depending on how multimodal features interact within the Transformer architecture, the existing studies mainly follow two fusion strategies. The first adopts an early fusion approach, where Roy et al. [29], Feng et al. [30], and Yang et al. [31] concatenate or stack features from HSI and LiDAR at the input layer, forming a unified token sequence that is then fed into a Transformer encoder for joint modeling. The second is a deep fusion architecture based on cross-attention. Li et al. [32], Seneviratne et al. [33], and Zhao et al. [34] treat features from one modality as queries and features from the other modality as both keys and values, actively retrieving relevant information through cross-attention layers, thereby achieving deeper feature alignment and complementary information mining.

Although previous works have achieved notable results, two key challenges remain. On the one hand, existing methods typically lack consideration of the semantic consistency across modalities and across hierarchical levels when extracting features from HSI and LiDAR data [35,36,37]. Ideally, for the same land-cover object, its HSI and LiDAR features should exhibit high similarity at different network levels. However, due to modality heterogeneity and asynchronous learning of the network, this inherent semantic consistency is often disrupted, causing the spectral features and elevation features of the same object to deviate from each other in the feature space. This hierarchical semantic inconsistency problem hinders the follow-up fusion from fully exploiting complementary information and limits the model’s ability to mine synergistic discriminative information across modalities. On the other hand, when employing cross-attention mechanisms to facilitate modality interaction, the existing studies [38,39] still mostly adopt unidirectional cross-attention. In this setup, the modality serving as the query dominates the optimization process, while the modality serving as the key/value remains in a passive state [40]. This further exacerbates cross-modal semantic inconsistency, because the features of the passive modality are difficult for the active modality to perceive equally. Moreover, some studies [41,42] try to stack multiple cross-attention modules to deepen fusion, yet they still confine themselves to simple addition or concatenation of the outputs from each branch.

Motivated by the above considerations, we propose a hierarchical semantic consistency constraint (HSCC) framework for the joint classification of HSI and LiDAR data. Through the synergistic collaboration of a progressive interactive fusion network (PIFNet) and a semantic consistency constraint (SCC) strategy, the framework guides the network to autonomously align cross-modal and cross-level semantic representations during multi-stage feature learning, thereby producing more discriminative joint features.

The main contributions of this paper are as follows:

• We develop a HSCC framework for the joint classification of HSI and LiDAR data. By integrating the PIFNet and the SCC strategy, the framework supports progressive multimodal interaction and fusion, further improving the accuracy of HSI and LiDAR joint classification. Experimental results on three publicly available multimodal remote sensing classification benchmark datasets demonstrate that HSCC achieves optimal performance across multiple evaluation metrics, fully validating its effectiveness.
• We design a PIFNet network to gradually narrow the modality discrepancy between HSI and LiDAR at the feature level and alleviate the potential single-modality dominance issue during deep interactions. Through the Cross-Modal Shared Attention (CMSA) and Symmetric Cross-Attention (SCA) mechanisms, PIFNet progressively strengthens cross-modal information interaction layer by layer, calibrating the semantic shifts between spectral and elevation information at different levels and thereby generating more consistent multi-level feature representations. These features are finally fused into a compact and semantically clear joint feature space, effectively supporting the classification task.
• We propose a SCC strategy that constructs multiple groups of semantic associations between features of different modalities and different levels, and introduces a semantic consistency constraint loss to explicitly guide the network to maintain the intrinsic similarity of features for the same land-cover object. This strategy can effectively alleviate the semantic inconsistency between features from different modalities and enhance the network’s perception capability for semantic alignment.

2. Related Work

2.1. CNN–Transformer Networks

Hybrid networks that combine the local perception capability of CNNs with the global context modeling advantage of Transformers have become an important research direction. Sun et al. [43] adopted a multi-scale 3D–2D hybrid CNN for feature extraction and designed a lightweight attention-free Transformer encoder for information integration. Wang et al. [44] used multi-scale asymmetric depthwise convolutions to capture shallow local-global correlations, and then extracted deep joint features via a local–global Transformer. Bai et al. [45] designed a hierarchical convolution module coupled with octave convolution for initial feature extraction and fusion, and captured cross-dimensional dependencies through a triplet rotational multi-head attention mechanism. To fully integrate heterogeneous data, many studies have been devoted to designing more complex cross-modal interaction mechanisms. Zhang et al. [46] introduced a diffusion model to mine latent features of HSI, and then constructed a three-input diffusion-augmented CNN–Transformer network, aggregating multi-source information via a spectral pooling attention interaction mechanism and an inverted residual structure. Wang et al. [47] used CNN and a Transformer in parallel during the feature fusion stage to explore deep features and interaction information across modalities. Although the above hybrid networks have achieved remarkable progress, they typically set up separate feature extraction branches for the two types of remote sensing data, with inter-modal interactions occurring only at deep stages of the network. Such a design makes it difficult to ensure the consistency between spectral and elevation information of the same ground object, thereby limiting the full integration of complementary information to some extent.

2.2. Cross-Attention

The cross-attention mechanism, capable of explicitly establishing semantic correlations between features of different modalities, has become key to achieving deep fusion. Ge et al. [48] combined local convolution with global cross-attention, enhanced convergence via residual aggregation, and simultaneously captured contextual details and long-range dependencies to achieve multi-source feature fusion and relational semantic representation. Liu et al. [49] introduced a neighborhood contrastive learning mechanism to enhance the discriminative ability of feature representations through a cross-source Transformer. Guo et al. [50] addressed feature redundancy in traditional stacking fusion by adopting a dual-branch pyramid structure for multi-scale feature extraction and achieving dynamic alignment of spectral and elevation information via a bidirectional cross-attention mechanism. Li et al. [51] used HSI features as queries and concatenated the key-value pairs from both modalities separately, enabling deep interaction of dual-modal features during attention weight computation. Zhang et al. [52] incorporated a cross-modal attention fusion module into an adaptive multi-stage fusion framework, dynamically capturing cross-modal correlations between hyperspectral and LiDAR features through an adaptive gating mechanism. To more comprehensively characterize dependencies among multimodal data, Jiao et al. [53] proposed a Common-Unique Transformer integrating a cross dynamic-agent-attention mechanism, which explicitly separates and fuses shared and unique features from different data sources. The above methods have conducted valuable explorations in cross-modal interaction, but most are limited to unidirectional querying or simple concatenation/linear combination of the two branch outputs. How to achieve deeper attentional synergy while ensuring bidirectional equal interaction remains an open issue for further research.

3. Proposed Method

In this section, we first outline the overall architecture of the HSCC framework, and then introduce the PIFNet and the SCC strategy in detail.

3.1. HSCC Framework

To address the problems of cross-modal semantic inconsistency and imbalanced modality contribution during interaction in HSI and LiDAR joint classification, this paper proposes a HSCC framework. As shown in Figure 1, the framework consists of two core components: PIFNet network and SCC strategy. Specifically, PIFNet is responsible for multi-stage feature alignment and bidirectional interactive fusion, while the SCC strategy imposes consistency constraints on the features output by PIFNet by establishing multi-level semantic correspondences.

In PIFNet, a progressive alignment mechanism is adopted to gradually enhance the information interaction between HSI and LiDAR across multiple levels. First, the network performs initial feature extraction and dimension unification for both modalities using simple convolutions. Subsequently, the Cross-Modal Shared Attention (CMSA) module generates and shares the same set of spatial attention maps for HSI and LiDAR, establishing preliminary cross-modal correlations while preserving the unique attributes of each modality, thereby guiding the network to focus on discriminative regions to which both modalities respond jointly. On this basis, the SCAT module achieves equal cross-modal information integration through Symmetrical Cross-Attention mechanism, effectively suppressing the dominant bias of a single modality. After the above progressive processing, PIFNet gradually produces multi-level aligned features, including low-level spatial features, cross-modal interactive features, and high-level semantic abstraction features. Finally, these multi-level cross-modal features are fused to form a unified joint feature representation for subsequent classification tasks. Meanwhile, the SCC strategy constructs same-level and cross-level semantic association pairs between HSI and LiDAR features, thereby establishing multi-level semantic correspondences. By optimizing the similarity among these semantic associations, SCC effectively suppresses semantic deviations across different modalities and levels, ultimately improving the semantic consistency and representational quality of the fused features.

For the same ground surface area, given a hyperspectral image $\mathrm{H}\in {\mathbb{R}}^{\mathrm{m}\times \mathrm{n}\times {\mathrm{c}}_{\mathrm{h}}}$ and the corresponding LiDAR image $\mathrm{L}\in {\mathbb{R}}^{\mathrm{m}\times \mathrm{n}\times {\mathrm{c}}_{\mathrm{l}}}$, where $\mathrm{m}$ and $\mathrm{n}$ denote the height and width of the images, respectively, and ${\mathrm{c}}_{\mathrm{h}}$ and ${\mathrm{c}}_{\mathrm{l}}$ denote the number of bands of the HSI and LiDAR data, respectively. After padding the image boundaries, for each foreground pixel, a pair of image patches centered at that pixel is extracted, resulting in a HSI cube ${\mathsf{{\rm X}}}_{\mathrm{h}}\in {\mathrm{R}}^{\mathrm{p}\times \mathrm{p}\times {\mathrm{c}}_{\mathrm{h}}}$ and a LiDAR cube ${\mathsf{{\rm X}}}_{\mathrm{l}}\in {\mathrm{R}}^{\mathrm{p}\times \mathrm{p}\times {\mathrm{c}}_{\mathrm{l}}}$ as inputs, where $\mathrm{p}$ is the patch size. Our goal is to predict a class label $\mathrm{y}$ for each central pixel, where $\mathrm{y}\in \left\{1, 2, ..., \mathrm{CL}\right\}$ and $\mathrm{C}\mathrm{L}$ is the total number of classes.

Algorithm 1 presents the detailed implementation of the HSCC framework. In each iteration, PIFNet first progressively aggregates the hierarchical features of the two modalities through multi-stage interactions, and on this basis forms discriminative representations oriented toward classification. Subsequently, the SCC strategy establishes multiple groups of semantic associations within the aligned feature space to guide cross-level and cross-modal feature consistency. The two objectives are jointly constrained under a unified framework, collectively driving the update of network parameters.

Algorithm 1 Training Procedure of the HSCC Framework

Input: HSI cube ${\mathsf{{\rm X}}}_{\mathrm{h}}$, LiDAR cube ${\mathsf{{\rm X}}}_{\mathrm{l}}$, epochs $\mathrm{E}$

Output: Classification map $\mathrm{P}$

1. Initialize network parameters;

2. for each training epoch $\mathrm{e}=1$ to $\mathrm{E}$ do

3. Extract HSI shallow features ${\mathrm{h}}_{\mathrm{low}}$ and LiDAR shallow features ${\mathrm{l}}_{\mathrm{low}}$;

4. Obtain cross-modal interaction features ${\mathrm{h}}_{\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}}$ and ${\mathrm{l}}_{\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}}$ for HSI and LiDAR respectively via the CMSA module;

5. Perform bidirectional symmetric attention interaction through SCAT, output HSI and LiDAR high-level semantic features ${\mathrm{h}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}}$, ${\mathrm{l}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}}$;

6. Integrate multi-level features to compute classification loss ${\mathrm{L}}_{\mathrm{CE}}$;

7. for each feature in {${\mathrm{h}}_{\mathrm{low}}$, ${\mathrm{l}}_{\mathrm{low}}$, ${\mathrm{h}}_{\mathrm{cross}}$, ${\mathrm{l}}_{\mathrm{cross}}$, ${\mathrm{h}}_{\mathrm{high}}$, ${\mathrm{l}}_{\mathrm{high}}$} do

8. Construct same-level semantic association pairs;

9. Construct cross-level semantic association pairs;

10. Compute the similarity of each association pair to obtain the semantic consistency constraint loss ${\mathrm{L}}_{\mathrm{SCC}}$;

11. end for

12. Evaluate classification loss ${\mathrm{L}}_{\mathrm{CE}}$ and semantic consistency constraint loss ${\mathrm{L}}_{\mathrm{SCC}}$;

13. Backpropagate and update network parameters;

14. end for

3.2. PIFNet

The joint classification of HSI and LiDAR is essentially a dual task of cross-modal feature collaboration and semantic unification. However, the existing methods generally adopt a strategy of first encoding each modality independently and then fusing them, which implicitly assumes that the two modalities naturally possess semantic correspondences after being learned separately. Furthermore, when using cross-attention mechanisms for deep fusion, one modality tends to dominate, leading to an imbalance in the contribution of multimodal data. To address these issues, this paper designs the PIFNet. Through a multi-stage symmetric interaction mechanism, PIFNet alleviates modality imbalance while promoting the gradual semantic convergence of multimodal features. As shown in Figure 1a, PIFNet consists of the following four parts.

3.2.1. Feature Alignment Encoding

Considering the rich spectral information contained in HSI data, we first use a $3\times 3\times 3$ 3D convolutional layer to extract spectral sequence features, and simultaneously use a $3\times 3$ 2D convolutional layer to capture texture and structural features. The numbers of filters in the two layers are set to 4 and 64, respectively. For the LiDAR branch, we extract elevation information using two stacked $3\times 3$ convolutional layers, with the numbers of filters set to 32 and 64, respectively. In addition, to accelerate model convergence, each convolutional layer in both the HSI and LiDAR branches is sequentially followed by a batch normalization (BN) layer and a Gaussian error linear unit (GELU) activation function. After the above processing, the patches of the two modalities are respectively mapped to ${\mathrm{F}}_{\mathrm{h}}\in {\mathrm{R}}^{\mathrm{p}\times \mathrm{p}\times \mathrm{c}}$ and ${\mathrm{F}}_{\mathrm{l}}\in {\mathrm{R}}^{\mathrm{p}\times \mathrm{p}\times \mathrm{c}}$, where $\mathrm{c}=64$.

In this stage, the feature encoding of HSI and LiDAR is performed independently, with interaction occurring only after the feature dimensions are unified. This design avoids semantic confusion caused by the premature fusion of heterogeneous shallow features, while ensuring structural compatibility between the feature representations of the two modalities, thus creating favorable conditions for the subsequent progressive interaction.

3.2.2. Cross-Modal Shared Attention Module

As shown in Figure 2, the proposed mechanism generates and shares the same set of spatial attention weights for HSI and LiDAR, thereby preserving the discriminative features of each modality while guiding the network to focus on spatial regions that elicit common responses from the multi-source data.

First, to enhance the internal contextual correlations of each modality’s features, we apply a multi-head self-attention mechanism to the previously obtained HSI and LiDAR features respectively for context aggregation. This process can be expressed as:

$${\mathrm{F}}^{\mathrm{s}\mathrm{a}}=\mathrm{M}\mathrm{H}\mathrm{S}\mathrm{A}\left(\mathrm{L}\mathrm{N}\left(\widehat{\mathrm{F}}\right)\right)+\widehat{\mathrm{F}}$$

(1)

In the formula, $\widehat{\mathrm{F}}$ denotes the flattened feature of $\mathrm{F}$, $\mathrm{M}\mathrm{H}\mathrm{S}\mathrm{A}\left(·\right)$ represents the multi-head self-attention mechanism, and $\mathrm{L}\mathrm{N}\left(·\right)$ denotes the layer normalization operation. For each attention head, we use a convolutional layer instead of the linear projection in the standard Transformer to generate ${\mathrm{Q}}_{\mathrm{i}}$, ${\mathrm{K}}_{\mathrm{i}}$, and ${\mathrm{V}}_{\mathrm{i}}$, where $\mathrm{i}$ denotes the $\mathrm{i}$th head. The self-attention SA of the $\mathrm{i}$th head can be calculated using the following formula:

$${\mathrm{S}\mathrm{A}}_{\mathrm{i}}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{{\mathrm{Q}}_{\mathrm{i}}{\mathrm{K}}_{\mathrm{i}}^{\mathrm{T}}}{\sqrt{\mathrm{d}/\mathrm{n}}}}\right){\mathrm{V}}_{\mathrm{i}}$$

(2)

where $\mathrm{T}$ denotes the transpose operation, $\mathrm{d}$ denotes the channel dimension, and $\mathrm{n}$ is the number of attention heads. Finally, the outputs of all attention heads are concatenated to obtain the context-aggregated features.

Next, the context-aggregated features ${\mathrm{F}}_{\mathrm{h}}^{\mathrm{s}\mathrm{a}}$ and ${\mathrm{F}}_{\mathrm{l}}^{\mathrm{s}\mathrm{a}}$ of HSI and LiDAR are respectively projected into a common subspace. After summation and $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}$ activation, a shared spatial attention weight map $\mathrm{M}$ is generated via $\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}$ normalization. This weight map reflects the regions commonly attended to by both modalities and is applied to the original aggregated features of both modalities in the same manner to recalibrate the spatial dimension. Finally, each modality integrates the calibrated features with the original features through residual connections to obtain the outputs ${\mathrm{Z}}_{\mathrm{h}}$ and ${\mathrm{Z}}_{\mathrm{l}}$. The specific formulas are as follows:

$$\mathrm{M}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left({\mathrm{F}}_{\mathrm{h}}^{\mathrm{s}\mathrm{a}}\right)+\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left({\mathrm{F}}_{\mathrm{l}}^{\mathrm{s}\mathrm{a}}\right)\right)\right)$$

(3)

$${\mathrm{Z}}_{\mathrm{h}}=\mathrm{M}\odot {\mathrm{F}}_{\mathrm{h}}^{\mathrm{s}\mathrm{a}}+{\mathrm{F}}_{\mathrm{h}}^{\mathrm{s}\mathrm{a}}$$

(4)

$${\mathrm{Z}}_{\mathrm{l}}=\mathrm{M}\odot {\mathrm{F}}_{\mathrm{l}}^{\mathrm{s}\mathrm{a}}+{\mathrm{F}}_{\mathrm{l}}^{\mathrm{s}\mathrm{a}}$$

(5)

where $\mathrm{M}\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{p}\times 1}$, $\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left(·\right)$ denotes a fully connected layer, $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(·\right)$ is the hyperbolic tangent activation function, and $\odot$ denotes element-wise multiplication.

This design achieves semantically aware spatial guidance through the shared attention maps, establishing a spatial constraint foundation for cross-modal semantic consistency before entering deep fusion, and thereby effectively suppressing subsequent semantic shifts caused by modality dominance bias.

3.2.3. Symmetrical Cross-Attention Transformer

Attention-based fusion strategies can more flexibly capture semantic correlations between modalities, thereby obtaining more discriminative feature representations. However, the cross-attention mechanism used in existing methods tends to implicitly allow one modality to dominate during deep fusion, while the unique information of the other modality may be suppressed or diluted. This not only compromises the reliability of the fused features but also exacerbates the difficulty of cross-modal semantic alignment.

To address the above modality imbalance problem, this paper proposes a Symmetric Cross-Attention Transformer (SCAT), whose structure is shown in Figure 3. Specifically, we construct equal bidirectional interaction channels for HSI and LiDAR, where both modalities have equal opportunities to serve as the query and the value, and the attention maps generated from both directions are jointly learned and fused, thereby ensuring equal status for the two modalities during information interaction. We use lightweight convolutions instead of linear layers to generate $\mathrm{Q}$, $\mathrm{K}$, and $\mathrm{V}$ for each modality, reducing the number of parameters while preserving spatial structure awareness. Taking the HSI feature ${\mathrm{Z}}_{\mathrm{h}}$ as an example, the generation process is as follows:

$${\mathrm{Q}}_{\mathrm{h}}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}\left({\mathrm{D}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}\left(\mathrm{L}\mathrm{N}\left({\mathrm{Z}}_{\mathrm{h}}\right)\right)\right)$$

(6)

$${\mathrm{K}}_{\mathrm{h}}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}\left({\mathrm{D}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}\left(\mathrm{L}\mathrm{N}\left({\mathrm{Z}}_{\mathrm{h}}\right)\right)\right)$$

(7)

$${\mathrm{V}}_{\mathrm{h}}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}\left({\mathrm{D}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}\left(\mathrm{L}\mathrm{N}\left({\mathrm{Z}}_{\mathrm{h}}\right)\right)\right)$$

(8)

where $\mathrm{L}\mathrm{N}\left(·\right)$ denotes the layer normalization operation, $\mathrm{D}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\left(·\right)$ represents depthwise separable convolution, and the generation process of ${\mathrm{Q}}_{\mathrm{l}}$, ${\mathrm{K}}_{\mathrm{l}}$, and ${\mathrm{V}}_{\mathrm{l}}$ for the LiDAR branch is the same as this.

Subsequently, the query ${\mathrm{Q}}_{\mathrm{h}}$ of HSI features and the key ${\mathrm{K}}_{\mathrm{l}}$ of LiDAR features are used to compute the attention map ${\mathrm{A}}_{\mathrm{h}}$, while the query ${\mathrm{Q}}_{\mathrm{l}}$ of LiDAR features and the key ${\mathrm{K}}_{\mathrm{h}}$ of HSI features are used to compute the attention map ${\mathrm{A}}_{\mathrm{l}}$. To enhance the correlation between the two attention pathways, ${\mathrm{A}}_{\mathrm{h}}$ and ${\mathrm{A}}_{\mathrm{l}}$ are concatenated and passed through a lightweight projection network to extract a shared interaction pattern, generating a collaborative attention component, which is then added to the respective attention maps via residual connections to obtain ${\mathrm{A}}_{\mathrm{h}}^{\prime }$ and ${\mathrm{A}}_{\mathrm{l}}^{\prime }$. The specific calculation process is as follows:

$${\mathrm{A}}_{\mathrm{h}}={\displaystyle \frac{{\mathrm{Q}}_{\mathrm{h}}^{\mathrm{T}}{\mathrm{K}}_{\mathrm{l}}}{{\mathsf{\tau }}_{\mathrm{h}}}}$$

(9)

$${\mathrm{A}}_{\mathrm{l}}={\displaystyle \frac{{\mathrm{Q}}_{\mathrm{l}}^{\mathrm{T}}{\mathrm{K}}_{\mathrm{h}}}{{\mathsf{\tau }}_{\mathrm{l}}}}$$

(10)

$${\mathrm{A}}_{\mathrm{h}}^{\prime }=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\left(\left[{\mathrm{A}}_{\mathrm{h}},{\mathrm{A}}_{\mathrm{l}}\right]\right)+{\mathrm{A}}_{\mathrm{h}}\right)\right)$$

(11)

$${\mathrm{A}}_{\mathrm{l}}^{\prime }=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\left(\left[{\mathrm{A}}_{\mathrm{h}},{\mathrm{A}}_{\mathrm{l}}\right]\right)+{\mathrm{A}}_{\mathrm{l}}\right)\right)$$

(12)

where ${\mathsf{\tau }}_{\mathrm{h}}$ and ${\mathsf{\tau }}_{\mathrm{l}}$ are learnable scaling parameters, $\left[·\right]$ denotes the concatenation operation, and $\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(·\right)$ is used to control the generalization ability of the model with the dropout rate set to 0.1.

Finally, ${\mathrm{A}}_{\mathrm{h}}^{\prime }$ and ${\mathrm{A}}_{\mathrm{l}}^{\prime }$ are used to perform weighted summation on ${\mathrm{V}}_{\mathrm{h}}$ and ${\mathrm{V}}_{\mathrm{l}}$ respectively, and then residual connections are applied to obtain ${\mathrm{Z}}_{\mathrm{h}}^{\prime }$ and ${\mathrm{Z}}_{\mathrm{l}}^{\prime }$. The two are then concatenated and fused through a $1\times 1$ convolution for dimensionality reduction, yielding the cross-modal fusion feature ${\mathrm{Z}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}^{\mathrm{c}\mathrm{a}}$.

$${\mathrm{Z}}_{\mathrm{h}}^{\prime }={\mathrm{A}}_{\mathrm{h}}^{\prime }{\mathrm{V}}_{\mathrm{h}}+{\mathrm{V}}_{\mathrm{h}}+{\mathrm{Z}}_{\mathrm{h}}$$

(13)

$${\mathrm{Z}}_{\mathrm{l}}^{\prime }={\mathrm{A}}_{\mathrm{l}}^{\prime }{\mathrm{V}}_{\mathrm{l}}+{\mathrm{V}}_{\mathrm{l}}+{\mathrm{Z}}_{\mathrm{l}}$$

(14)

$${\mathrm{Z}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}^{\mathrm{c}\mathrm{a}}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}\left(\left[{\mathrm{Z}}_{\mathrm{h}}^{\prime },{\mathrm{Z}}_{\mathrm{l}}^{\prime }\right]\right)$$

(15)

The above symmetric fusion design ensures that HSI and LiDAR have equal semantic contribution capabilities during the fusion process, effectively mitigating modality dominance bias and providing a well-aligned cross-modal feature foundation for the semantic consistency constraint.

3.2.4. Classification and Loss Function

To effectively align multi-modal features at different levels under semantic consistency constraints, we perform projection transformations on the low-level features, cross-modal interactive features, and high-level semantic features of HSI and LiDAR, respectively. For the low-level and cross-modal interactive features, we adopt a parameter-shared multilayer perceptron (MLP) to map them into a common semantic subspace. This MLP first expands the original features to a higher dimension to enhance representational capacity, then applies batch normalization and the GELU activation function to stabilize the training process, introduces dropout to mitigate overfitting, and finally compresses the features to the number of classes. For the high-level semantic features, they are refined by another lightweight projection network with a similar structure, except that the features are first appropriately reduced in dimensionality while maintaining alignment with the shallow features. The projected features are fed into the SCC strategy for semantic consistency measurement, forcing representations that were originally separated due to modality or level differences to converge semantically.

We sum the features obtained from HSI and LiDAR at each stage to form a unified multi-level feature representation. The aggregated features are finally fed into a fully connected layer for pixel-wise classification. The overall loss function is defined as shown in Equation (16), which consists of two terms: the classification loss ${\mathcal{L}}_{\mathrm{C}\mathrm{E}}$ and the semantic consistency constraint loss ${\mathcal{L}}_{\mathrm{S}\mathrm{C}\mathrm{C}}$. The classification loss adopts the standard cross-entropy form, while the detailed design of ${\mathcal{L}}_{\mathrm{S}\mathrm{C}\mathrm{C}}$ will be elaborated in Section 3.3.

$$\mathcal{L}={\mathcal{L}}_{\mathrm{C}\mathrm{E}}+\mathsf{\mu }{\mathcal{L}}_{\mathrm{S}\mathrm{C}\mathrm{C}}$$

(16)

$${\mathcal{L}}_{\mathrm{C}\mathrm{E}}=-{\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}{\displaystyle \sum _{\mathrm{c}\mathrm{l}=1}^{\mathrm{C}\mathrm{L}}}{\mathrm{y}}_{\mathrm{i},\mathrm{c}\mathrm{l}}\log \left({\widehat{\mathrm{y}}}_{\mathrm{i},\mathrm{c}\mathrm{l}}\right)$$

(17)

where a balancing coefficient $\mathsf{\mu }$ controls the contribution of the ${\mathcal{L}}_{\mathrm{S}\mathrm{C}\mathrm{C}}$, $\mathrm{N}$ is the total number of samples, $\mathrm{C}\mathrm{L}$ is the total number of classes, ${\mathrm{y}}_{\mathrm{i},\mathrm{c}\mathrm{l}}$ is the ground-truth label indicating whether sample $\mathrm{i}$ belongs to class $\mathrm{c}\mathrm{l}$, and ${\widehat{\mathrm{y}}}_{\mathrm{i},\mathrm{c}\mathrm{l}}$ is the predicted probability that sample $\mathrm{i}$ belongs to class $\mathrm{c}\mathrm{l}$.

3.3. SCC Strategy

For HSI and LiDAR data at the same spatial location, features at the same level of abstraction should inherently be consistent. This intra-level semantic correspondence forms the foundation of cross-modal understanding. Furthermore, features from different modalities at different levels should also maintain semantic associations. Cross-level semantic consistency facilitates the full exploitation of complementary information from heterogeneous data sources. To address these requirements, this paper proposes the SCC strategy, which simultaneously establishes two types of semantic associations. By explicitly increasing the similarity between these semantically corresponding features, the proposed strategy improves the reliability of the fused representation. This process is illustrated in Figure 1b.

In implementation, SCC constructs multiple groups of semantic associations by aligning features at the same level across the two modalities and by matching the high-level semantic features of one modality with the low-level and cross-modal interaction features of the other. Since semantic alignment essentially depends on the directional consistency of feature vectors, and cosine similarity directly measures directional correlation through the cosine of the angle, is invariant to feature scaling, and has a bounded range that facilitates loss optimization, this paper adopts cosine similarity to measure the consistency of each semantic association pair and trains the model using the semantic consistency constraint loss ${\mathcal{L}}_{\mathrm{S}\mathrm{C}\mathrm{C}}$, encouraging the feature representations of the same pixel to maintain high semantic similarity across different modalities and levels. The specific formulation is as follows:

$${\mathcal{L}}_{\mathrm{S}\mathrm{C}\mathrm{C}}={\mathcal{L}}_{\mathrm{S}\mathrm{L}}+{\mathcal{L}}_{\mathrm{C}\mathrm{L}1}+{\mathcal{L}}_{\mathrm{C}\mathrm{L}2}$$

(18)

$${\mathcal{L}}_{\mathrm{S}\mathrm{L}}=\left(1-\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{h}}_{\mathrm{l}\mathrm{o}\mathrm{w}},{\mathrm{l}}_{\mathrm{l}\mathrm{o}\mathrm{w}}\right)\right)+\left(1-\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{h}}_{\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}},{\mathrm{l}}_{\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}}\right)\right)+\left(1-\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{h}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}},{\mathrm{l}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}}\right)\right)$$

(19)

$${\mathcal{L}}_{\mathrm{C}\mathrm{L}1}=\left(1-\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{h}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}},{\mathrm{l}}_{\mathrm{l}\mathrm{o}\mathrm{w}}\right)\right)+\left(1-\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{h}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}},{\mathrm{l}}_{\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}}\right)\right)$$

(20)

$${\mathcal{L}}_{\mathrm{C}\mathrm{L}2}=\left(1-\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{l}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}},{\mathrm{h}}_{\mathrm{l}\mathrm{o}\mathrm{w}}\right)\right)+\left(1-\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathrm{l}}_{\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}},{\mathrm{h}}_{\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}}\right)\right)$$

(21)

$$\mathrm{s}\mathrm{i}\mathrm{m}\left(\mathsf{\alpha },\mathsf{\beta }\right)={\displaystyle \frac{\mathsf{\alpha }·\mathsf{\beta }}{‖\mathsf{\alpha }‖‖\mathsf{\beta }‖}}$$

(22)

where ${\mathrm{h}}_{\mathrm{low}}$ and ${\mathrm{l}}_{\mathrm{low}}$ denote the low-level features extracted by the CNN encoder; ${\mathrm{h}}_{\mathrm{cross}}$ and ${\mathrm{l}}_{\mathrm{cross}}$ represent the preliminary cross-modal interaction features generated via the CMSA module; and ${\mathrm{h}}_{\mathrm{high}}$ and ${\mathrm{l}}_{\mathrm{high}}$ are the high-level semantic features produced by the SCAT module, which contain rich semantic information and deeply fused information.

4. Results

4.1. Dataset

• MUUFL dataset: The MUUFL dataset was acquired in November 2010 from the Gulf Park campus of the University of Southern Mississippi in Long Beach, Mississippi, USA. This dataset contains two data sources: hyperspectral images and LiDAR data. The hyperspectral data were collected by the ITRES CASI-1500 sensor, consisting of 325 × 220 pixels with 64 effective spectral channels, covering a wavelength range from 0.38 to 1.05 µm. The LiDAR data were simultaneously acquired by the Gemini ALTM sensor with a wavelength of 1.06 µm. The dataset contains a total of 53,687 labeled pixels corresponding to 11 different land cover categories. Figure 4 shows the false-color composite image of the HSI data, the grayscale image of the LiDAR data, and the ground-truth labels.
• Houston2013 dataset: The Houston2013 dataset was provided by the IEEE GRSS Data Fusion Contest and was acquired over the University of Houston campus and its surrounding urban area in June 2012. The HSI data were collected by the ITRES CASI-1500 sensor, containing 144 spectral bands with wavelengths ranging from 0.38 to 1.05 µm, while the simultaneously acquired LiDAR data are a single-band digital surface model. Both modalities have dimensions of 349 × 1905 pixels with a spatial resolution of 2.5 m. The dataset contains a total of 15,029 labeled pixels corresponding to 15 different land cover types. Figure 5 shows the false-color composite image of the HSI data, the grayscale image of the LiDAR data, and the ground-truth labels.
• Augsburg dataset: The Augsburg dataset was acquired over the city of Augsburg, Germany, and its surrounding areas. The HSI data were collected by the DAS-EOC HySpex sensor, covering 180 continuous spectral bands with wavelengths ranging from 0.4 µm to 2.5 µm. The LiDAR data were obtained by the DLR-3K system. The spatial dimensions of the dataset are 332 × 485 pixels, comprising 7 land cover categories. Figure 6 shows the false-color composite image of the HSI data, the grayscale image of the LiDAR data, and the ground-truth labels.

  Table 1. Number of samples per class in the three datasets.

  No.	MUUFL			Houston2013			Augsburg
  	Class Name	Train	Test	Class Name	Train	Test	Class Name	Train	Test
  1	Trees	50	23,196	Healthy grass	20	1231	Forest	270	13,237
  2	Mostly grass	50	4220	Stressed grass	20	1234	Residential area	607	29,722
  3	Mixed ground surface	50	6832	Synthetic grass	20	677	Industrial area	77	3774
  4	Dirt and sand	50	1776	Trees	20	1224	Low plants	537	26,320
  5	Road	50	6637	Soil	20	1222	Allotment	12	563
  6	Water	50	416	Water	20	305	Commercial area	33	1612
  7	Buildings shadow	50	2183	Residential	20	1248	Water	31	1499
  8	Buildings	50	6190	Commercial	20	1224
  9	Sidewalk	50	1335	Road	20	1232
  10	Yellow curb	50	133	Highway	20	1207
  11	Cloth panels	50	219	Railway	20	1215
  12				Parking lot 1	20	1213
  13				Parking lot 2	20	449
  14				Tennis Court	20	408
  15				Running Track	20	640
  	Total	550	53,137	Total	300	14,729	Total	1567	76,727

  lists the class names and the number of training and test samples for each class for the three datasets: MUUFL, Houston2013, and Augsburg.

4.2. Experimental Setup

• Evaluation metrics: To objectively evaluate the classification performance of the proposed network and other compared methods, we select three widely used quantitative evaluation metrics in the field of remote sensing image classification: Overall Accuracy (OA), Average Accuracy (AA), and Kappa coefficient (Kappa). OA represents the proportion of correctly classified samples to the total number of test samples. AA denotes the arithmetic mean of classification accuracies for each class. Kappa reflects the consistency between the classification results and the ground-truth distribution. For each metric, a higher value indicates better classification performance.
• Implementation details: We implement the proposed model using the PyTorch 2.0.1 deep learning framework. All experiments are conducted on a server equipped with a NVIDIA GeForce RTX 4090 GPU (24 GB memory). In the training setup, the number of epochs is set to 500, and the batch size is 64. We adopt the Adam optimizer to update network parameters, and the learning rate scheduler uses the StepLR strategy, decaying the learning rate to 0.9 times its previous value every 50 epochs. To monitor model performance and save the best weights, the model is evaluated on the test set every 10 epochs during training.

4.3. Classification Results and Analysis

To demonstrate the effectiveness of the proposed HSCC, we compared it with eight other state-of-the-art HSI and LiDAR joint classification methods, including IDNet [54], DSymFuser [55], mPMCL [56], FDNet [57], MICFNet [58], AFDSE [59], MCFNet [60], and PICNet [61]. All compared methods were implemented using the original code released by their respective authors, and the model parameters were strictly set according to the recommendations in the corresponding original papers. Furthermore, all methods were evaluated on the same training and test sets, where the data splits were determined by random sampling and kept consistent across all comparative experiments. Each method is described as follows:

IDNet: This method embeds Laplacian high-frequency features into a Transformer architecture, learns high-frequency features through a dynamic range-aware and similarity-based spatial–spectral feature grouping interaction mechanism, and employs differential convolution to enhance high-frequency information during the fusion stage, thereby achieving adaptive dynamic fusion.

DSymFuser: This method adopts a dual-branch Transformer framework, where each branch adaptively aggregates local and global features extracted by CNN and Transformer, and generates bidirectional cross-modal features through a single attention computation.

mPMCL: This method constructs multiple positive sample pairs between HSI and LiDAR features for contrastive training, thereby avoiding the sensitivity to negative sample selection inherent in traditional contrastive learning.

FDNet: This method utilizes a fast Fourier transform-based self-attention mechanism to capture local frequency-domain features of spatial, spectral, and elevation information in HSI and LiDAR data, and embeds deformable convolution offsets into the frequency-domain Transformer to enhance local representation and capture global semantics.

MICFNet: This method achieves spatial information interaction through cross-modal attention map fusion and similarity constraints, dynamically adjusts the learning rate of each modality to alleviate learning imbalance, and exchanges semantic pseudo-labels between modalities for mutually guided fusion.

AFDSE: This method adaptively enhances local detail information using dynamic weights and Laplacian convolution kernels, suppresses redundancy and noise through a sparse attention mechanism and sparse gating, and integrates original and frequency-domain features via a gated fusion strategy, thereby achieving deep joint classification of multi-source remote sensing data.

MCFNet: This method captures local structures in the spatial domain through convolutional networks and employs a cross-domain attention mechanism to interactively fuse frequency-domain and spatial-domain features, thereby enhancing the joint classification capability of hyperspectral and LiDAR data.

PICNet: This method introduces learnable modality prototype vectors to perform cross-attention computation with raw features, thereby leveraging global complementary information to enhance cross-modal feature alignment and fusion.

The classification performance of different methods on the MUUFL dataset is shown in

Table 2. Classification performance of different methods on the MUUFL dataset.

Class	IDNet	DSymFuser	mPMCL	FDNet	MICFNet	AFDSE	MCFNet	PICNet	Our
Trees	89.28	90.00	85.32	89.48	93.68	89.73	86.97	88.53	95.96
Mostly grass	84.50	74.12	77.63	80.92	74.38	78.06	81.54	71.94	79.12
Mixed ground surface	72.64	81.40	75.91	74.25	83.90	76.72	74.99	75.07	77.96
Dirt and sand	91.50	92.68	90.93	87.22	94.65	94.30	91.33	87.66	85.98
Road	83.46	80.13	73.54	83.28	77.72	76.16	81.56	83.73	83.22
Water	99.28	99.52	96.15	99.76	100	99.72	99.52	98.56	99.76
Buildings shadow	93.36	96.84	83.97	93.63	94.64	91.97	96.61	93.13	91.89
Buildings	90.05	91.62	91.45	93.78	93.15	90.63	92.23	93.99	93.72
Sidewalk	64.94	78.43	58.95	76.33	71.84	57.09	72.36	69.59	79.85
Yellow curb	89.47	83.46	78.20	91.73	86.47	90.27	90.98	79.70	87.97
Cloth panels	98.63	95.87	99.54	99.09	96.35	98.92	99.54	96.35	99.54
OA	85.87	86.75	82.34	86.46	88.40	85.08	85.27	85.29	89.58
AA	87.01	87.64	82.87	88.13	87.89	85.78	87.97	85.30	88.63
Kappa	81.81	82.85	77.34	82.50	84.82	80.65	81.10	80.98	86.28

Bold indicates the best or tied-best result in each row.

. For clarity, the best value in each row is shown in bold. The results indicate that our method outperforms the competing methods across all evaluation metrics. For example, compared with the best performing baseline model MICFNet, our model improves OA, AA and Kappa by 1.18%, 0.74% and 1.46% respectively. Although HSCC does not achieve the highest classification accuracy in some classes, the accuracy differences between them and the highest values are marginal. Notably, for the challenging categories “Trees” and “Sidewalk”, our method achieves accuracies of 95.96% and 79.85% respectively, which are significantly better than the suboptimal methods. The visualized classification results are shown in Figure 7. It can be observed that methods such as AFDSE and PICNet exhibit obvious noise, and most methods misclassify “Trees”. In contrast, the classification maps generated by HSCC have less noise and clearer boundaries, benefiting from its effective utilization of multimodal complementary information.

The classification performance of different methods on the Houston2013 dataset is presented in

Table 3. Classification performance of different methods on the Houston2013 dataset.

Class	IDNet	DSymFuser	mPMCL	FDNet	MICFNet	AFDSE	MCFNet	PICNet	Our
Health grass	92.77	80.34	93.01	95.85	92.45	92.83	98.73	95.29	98.05
Stressed grass	98.62	100	98.54	99.19	98.54	97.24	97.04	98.62	98.38
Synthetic grass	99.70	99.60	99.56	98.82	99.56	99.65	99.33	98.23	98.52
Trees	100	97.44	96.81	98.61	98.12	98.08	98.26	98.94	98.94
Soil	100	98.96	100	99.51	100	100	99.88	99.59	100
Water	100	100	98.69	97.70	98.03	99.61	97.12	94.43	96.72
Residential	95.27	92.54	90.38	95.19	91.51	97.36	96.08	94.71	98.40
Commercial	76.06	92.21	66.58	87.83	90.03	77.79	83.03	84.74	87.83
Road	90.02	89.90	91.40	85.88	89.61	96.37	90.68	94.24	98.21
Highway	97.35	98.07	94.20	99.25	99.50	99.12	93.48	92.29	96.11
Railway	99.84	95.52	86.17	20.82	98.93	98.35	93.83	98.10	99.84
Parking lot 1	94.47	96.73	72.22	90.02	91.84	92.63	96.47	75.60	93.16
Parking lot 2	99.55	91.58	98.44	97.77	97.77	100	100	99.33	100
Tennis court	99.51	100	99.26	100	100	100	100	99.75	100
Running track	100	93.02	100	99.84	100	100	99.77	100	100
OA	95.33	94.47	90.69	89.24	95.74	95.80	95.54	94.15	97.28
AA	96.21	95.06	92.35	91.09	96.39	96.60	96.25	94.92	97.61
Kappa	94.95	94.00	89.95	88.38	95.40	95.46	95.18	93.68	97.06

Bold indicates the best or tied-best result in each row.

. Experimental results demonstrate that our model achieves the best performance across all evaluation metrics, with an overall accuracy of 97.28%, an average accuracy of 97.61%, and a Kappa coefficient of 97.06%, significantly surpassing other compared methods. Furthermore, our method attains the highest classification accuracy in 7 out of 15 classes. Two main factors account for this superior performance. On one hand, the proposed PIFNet progressively calibrates the semantic representations of dual-modal features through cross-modal shared attention and symmetric cross-attention mechanisms. Its bidirectional symmetric interaction effectively suppresses the dominant bias of a single modality, ensuring that multi-modal information is fully exploited. On the other hand, the SCC strategy imposes both intra-level and cross-level semantic consistency constraints, tightly aligning multi-modal features at multiple abstraction levels and further enhancing the discriminative power of the fused features. Visualized results are shown in Figure 8. Due to the large spatial extent of the Houston2013 dataset, a region delineated by a red box is locally enlarged to illustrate classification details clearly. From the comparison, it is evident that the classification map generated by HSCC is closest to the ground truth. Within the enlarged region, HSCC achieves significantly better recognition results for the “Health grass” and “Road” categories than other methods, whereas the competing models generally exhibit considerable misclassifications in this area. These observations further validate the effectiveness of our method.

The classification performance of different methods on the Augsburg dataset is shown in

Table 4. Classification performance of different methods on the Augsburg dataset.

Class	IDNet	DSymFuser	mPMCL	FDNet	MICFNet	AFDSE	MCFNet	PICNet	Our
Forest	99.52	98.58	98.81	99.64	97.59	99.02	99.40	96.40	99.18
Residential area	98.59	97.95	96.07	97.89	98.28	97.47	98.55	96.56	98.79
Industrial area	84.45	84.03	81.43	78.62	88.31	87.38	76.87	72.98	88.66
Low plants	98.29	98.03	98.34	98.58	98.45	97.93	98.35	97.32	98.45
Allotment	72.65	82.34	57.55	55.06	83.66	84.34	55.24	51.33	69.63
Commercial area	43.55	44.37	55.52	34.18	38.28	45.11	28.78	57.82	60.79
Water	59.37	61.72	62.11	58.84	62.58	62.95	56.10	69.03	63.38
OA	95.84	95.48	94.80	95.06	95.67	95.53	94.95	94.27	96.54
AA	79.49	81.00	78.55	74.69	81.02	82.03	73.33	74.16	82.70
Kappa	94.01	93.52	92.58	92.91	93.78	93.60	92.72	91.79	95.03

Bold indicates the best or tied-best result in each row.

. Similar to the previous two datasets, our method achieves the best performance among the three evaluation metrics and attains the highest classification accuracy in three out of all seven classes. Taking AA as an example, HSCC outperforms IDNet, DSymFuser, mPMCL, FDNet, MICFNet, AFDSE, MCFNet, and PICNet by 3.21%, 1.70%, 4.15%, 8.01%, 1.68%, 0.67%, 9.37%, and 8.54%, respectively, representing substantial improvements. Notably, for the two challenging categories, “Industrial area” and “Commercial area”, HSCC achieves accuracies of 88.66% and 60.79%, respectively, significantly surpassing the compared methods. Figure 9 presents the classification maps of various methods. Although the quality of the classification maps produced by all methods is acceptable, the prediction results of our method are more accurate. For instance, in the classification results for the “Residential area” category, HSCC yields fewer misclassified pixels and smoother boundaries than other methods.

5. Discussion

5.1. Ablation Analysis of Different Components

To verify the effectiveness of the PIFNet and SCC strategy, we conducted ablation experiments on three datasets, and the results are shown in

Table 5. Ablation study results of different components of HSCC on three datasets.

Module			MUUFL			Houston2013			Augsburg
CMSA	SCAT	SCC	OA	AA	Kappa	OA	AA	Kappa	OA	AA	Kappa
×	×	×	87.35	80.14	83.46	94.76	95.33	94.33	95.44	76.95	93.56
√	×	×	87.52	86.99	83.70	94.82	95.37	94.40	95.54	79.16	93.60
×	√	×	88.41	88.37	84.93	96.52	97.14	96.19	96.05	81.45	94.09
×	×	√	88.06	87.12	84.40	96.48	97.04	96.19	95.64	82.06	94.19
×	√	√	88.12	87.94	84.53	96.20	96.64	95.90	96.08	81.78	94.36
√	×	√	88.08	87.72	84.47	95.87	96.53	95.54	96.25	80.18	94.72
√	√	×	88.58	88.07	85.54	96.61	97.13	96.33	95.98	81.64	94.23
√	√	√	89.58	88.63	86.28	97.28	97.61	97.06	96.54	82.70	95.03

Bold indicates the best or tied-best result in each column.

. It can be observed that introducing CMSA, SCAT, or SCC individually all improves classification performance, demonstrating that each of the three has its own independent contribution. The complete framework achieves the best performance on all datasets, while removing any component leads to varying degrees of performance degradation, confirming the necessity and synergistic effect of each module.

Specifically, when only CMSA is introduced, the OA on the MUUFL, Houston2013, and Augsburg datasets improves by 0.17%, 0.04%, and 0.10%, respectively, compared to the baseline. In contrast, when the CMSA module is removed, the OA on the three datasets decreases by 1.46%, 1.08%, and 0.46%, respectively, indicating that the CMSA module plays a significant role in establishing cross-modal associations at shallow layers and guiding the network to focus on jointly discriminative regions. Its absence causes subsequent semantic alignment to lack a reliable spatial prior. When only SCAT is introduced, the OA on the three datasets improves by 1.06%, 1.76%, and 0.61%, respectively. When standard cross-attention is used in place of the SCAT module, the OA drops by 1.50%, 1.41%, and 0.29%, respectively, demonstrating that the balanced bidirectional interaction achieved by SCAT is crucial for deep fusion, providing a more equitable feature foundation for cross-modal semantic consistency constraints. Notably, when both CMSA and SCAT are introduced together, the OA is significantly superior to using either module alone. For example, on the Augsburg dataset, it improves by 1.00% and 0.49% compared to using CMSA or SCAT individually, respectively. This indicates that shallow spatial guidance and deep bidirectional interaction can effectively collaborate, integrating cross-modal information through multi-stage progressive alignment, thereby enhancing the expressive power of fused features for classification. When only the SCC strategy is introduced, the OA improves by 0.71%, 1.72%, and 0.50%, respectively. After removing the SCC strategy, the OA decreases by 1.00%, 0.67%, and 0.56%, respectively, with similar trends observed for AA and Kappa, confirming that SCC, by explicitly constraining the semantic similarity between cross-level and cross-modal features, can significantly reduce representation discrepancies caused by heterogeneity, thus yielding higher-quality fused representations.

In summary, PIFNet progressively narrows modality differences through the spatial guidance provided by CMSA and the deep balanced interaction ensured by SCAT, while the SCC strategy further enhances feature consistency via semantic consistency constraints. Together, they synergistically support the superior performance of the HSCC framework.

5.2. Analysis of SCC Strategy

To systematically analyze the impact of different semantic associations on the combination, in addition to the full SCC strategy, we also test two other possible semantic association strategies, specifically: (1) Strategy a: constructing only same-level semantic association pairs; (2) Strategy b: constructing only cross-level cross-modal association pairs; (3) Strategy c: the proposed SCC strategy.

Table 6. Ablation study results of different strategies on the three datasets.

Strategy	MUUFL			Houston2013			Augsburg
	OA	AA	Kappa	OA	AA	Kappa	OA	AA	Kappa
a	88.27	88.47	84.76	96.78	97.19	96.51	96.36	82.67	94.78
b	88.50	88.82	85.01	96.12	96.72	95.81	96.31	79.50	94.70
c	89.58	88.63	86.28	97.28	97.61	97.06	96.54	82.70	95.03

Bold indicates the best or tied-best result in each column.

presents the experimental results of different semantic consistency constraint strategies. Strategy a achieves relatively stable performance on all three datasets, but is overall inferior to Strategy c. Strategy b yields slightly better AA than Strategy c on MUUFL, but its AA drops significantly on Houston2013 and Augsburg, indicating that while cross-level cross-modal matching is important, relying solely on cross-level associations is insufficient to handle semantic shifts in complex scenes. In contrast, the proposed SCC strategy constructs a complete set of multi-level semantic correspondences, which can more fully enhance the semantic consistency and classification reliability of the fused features.

5.3. Parameter Analysis

5.3.1. Impact of Learning Rate on Classification Performance

Considering the intrinsic differences in spectral details and geometric structures between HSI and LiDAR data, an appropriate learning rate can effectively coordinate the feature-learning process. Experiments were conducted on the candidate set {$1\times {10}^{-5}$, $5\times {10}^{-5}$, $1\times {10}^{-4}$, $5\times {10}^{-4}$, $1\times {10}^{-3}$, $5\times {10}^{-3}$}, and the variation in OA with learning rate on the three datasets is shown in Figure 10. The experimental results indicate that when the learning rate is too low, the model converges slowly, resulting in low OA values across all datasets. Conversely, an excessively high learning rate tends to cause training oscillation and leads to a significant performance decline. For the MUUFL dataset, the optimal learning rate is $1\times {10}^{-3}$, while the Houston2013 and Augsburg datasets achieve their best performance at $5\times {10}^{-4}$.

5.3.2. Impact of Patch Size on Classification Performance

The input patch size determines the range of local context that the model can perceive and is crucial for feature extraction from HSI and LiDAR data. To investigate its impact, we tested six patch sizes of 7, 9, 11, 13, 15 and 17 on the three datasets, with the results shown in Figure 11. The experimental results indicate that the selection of patch size requires a trade-off between local details and global context. A patch size that is too small fails to provide sufficient neighborhood information, preventing the model from adequately capturing the spatial structure and spectral context of land cover objects. Conversely, an excessively large patch size tends to introduce redundant information, interfering with the discrimination of local details and causing performance degradation. The optimal patch size varies across the three datasets. The MUUFL dataset achieves its best performance at $\mathrm{p} = 9$, while the Houston2013 and Augsburg datasets perform best at $\mathrm{p} = 11$ and $\mathrm{p} = 13$, respectively. This is because the Houston2013 dataset features a more complex scene that requires a slightly larger window to incorporate more contextual information, thereby enabling effective differentiation among different land cover classes.

Figure 10. Effect of different learning rates on classification performance. (a) MUUFL. (b) Houston2013. (c) Augsburg.

Figure 10. Effect of different learning rates on classification performance. (a) MUUFL. (b) Houston2013. (c) Augsburg.

Figure 11. Effect of different patch sizes on classification performance. (a) MUUFL. (b) Houston2013. (c) Augsburg.

Figure 11. Effect of different patch sizes on classification performance. (a) MUUFL. (b) Houston2013. (c) Augsburg.

5.3.3. Impact of Balance Coefficient $\mathsf{\mu }$ on Classification Performance

The balance coefficient $\mathsf{\mu }$ controls the contribution of the semantic consistency constraint loss ${\mathrm{L}}_{\mathrm{SCC}}$ to the total loss. When $\mathsf{\mu }$ is too small, the semantic consistency constraint contributes insufficiently, making it difficult to effectively suppress cross-modal semantic drift. When $\mathsf{\mu }$ is too large, the dominant role of the classification loss is weakened, and the model may focus excessively on feature alignment at the expense of discriminative ability. The value range of $\mathsf{\mu }$ was set to {0.1, 0.2, 0.3, 0.4, 0.5}. For each dataset, the model was trained with each candidate value and the resulting OA values were recorded, as shown in Figure 12. The MUUFL dataset achieves the highest OA at $\mathsf{\mu } = 0.2$, while the optimal μ for the Houston2013 and Augsburg datasets are 0.3 and 0.1, respectively. This can be attributed to the greater diversity of classes in the Houston2013 dataset, which requires a stronger constraint to guide cross-modal consistency.

5.4. Feature Visualization

To further validate the effectiveness of the HSCC framework, we employed the t-distributed stochastic neighbor embedding (t-SNE) algorithm to visualize the learned features and compared them with the best-performing baseline model, MICFNet. Figure 13 shows the feature distributions of the two models on the three datasets, where the first row presents the results of MICFNet and the second row those of HSCC. Overall, HSCC produces clearer class boundaries and higher intra-class compactness. For example, for class 7 (Buildings shadow) of the MUUFL dataset and class 12 (Park lot 1) of the Houston2013 dataset, the feature clusters of HSCC are more compact and well separated from other classes. This is mainly attributed to the SCC strategy, which effectively suppresses cross-modal semantic shift by explicitly pulling closer the semantic similarity of cross-modal and cross-level features. The above visualization results confirm the effectiveness of HSCC in cross-modal feature learning, facilitating more accurate land cover classification.

5.5. Computational Complexity Analysis

To comprehensively evaluate the trade-off between performance and computational cost for each method, we report the OA, number of parameters, and floating point operations (FLOPs) on the Houston2013 dataset. The results are shown in

Table 7. Comparison of performance and complexity of different methods on the Houston2013 dataset.

Methods	IDNet	DSymFuser	mPMCL	FDNet	MICFNet	AFDSE	MCFNet	PICNet	Our
OA	95.33	94.47	90.69	89.24	95.74	95.80	95.54	94.15	97.28
Parameters	111.79 K	848.66 K	172.67 K	345.86 K	167.68 K	441.84 K	252.31 K	27.12 M	425.44 K
FLOPs	21.35 M	112.01 M	7.59 M	11.68 M	20.09 M	48.86 M	39.06 M	1.23 G	35.88 M

Bold indicates the best or tied-best result in each column.

. HSCC achieves the highest OA, significantly outperforming all compared methods. Although mPMCL has the lowest FLOPs and a relatively small number of parameters, its OA is only 90.69%, indicating that excessive computational compression impairs the feature representation capability of the model. In contrast, PICNet, despite its higher computational cost, still yields a lower OA than HSCC, suggesting that its utilization efficiency of computational resources can be further improved. By comparison, HSCC achieves the best accuracy with moderate parameters and FLOPs, validating its feasibility as an efficient fusion framework.

6. Conclusions

Aiming at the problems of cross-modal semantic inconsistency and imbalanced modality contribution in HSI and LiDAR joint classification, this paper proposes a hierarchical semantic consistency constraint framework, HSCC. The framework is co-constructed by PIFNet and the SCC strategy. PIFNet progressively calibrates the semantic representations of multimodal features at different abstraction levels through CMSA and SCAT module to mitigate the information imbalance in deep interactions; the SCC strategy establishes both same-level and cross-level semantic associations to guide the network to autonomously maintain consistency between heterogeneous feature representations. A series of experimental results demonstrate that HSCC achieves superior performance in multi-source remote sensing data fusion classification tasks, validating its effectiveness.

Nevertheless, HSCC still has certain limitations. First, the training of the framework relies on a certain amount of high-quality labeled data; its performance may be constrained in scenarios where annotations are scarce or expensive to obtain. Second, although the introduced multi-attention mechanism helps improve classification accuracy, it also increases the training complexity of the model, requiring further optimization in resource-constrained scenarios. In future work, we will leverage self-supervised or semi-supervised strategies to mine effective supervisory signals from unlabeled data, thereby enhancing the model’s generalization ability under few-shot conditions. Meanwhile, we will further explore lightweight network architectures to effectively reduce the computational cost of the model while maintaining performance. In addition, we will consider introducing an uncertainty estimation mechanism to characterize the inherent ambiguity of land cover classes in complex scenes, providing interpretable confidence information for classification decisions.