Enhancing High-Resolution Land Cover Classification Using Multi-Level Cross-Modal Attention Fusion

Abstract

High-precision land cover classification is fundamental to environmental monitoring, urban planning, and sustainable land-use management. With the growing availability of multimodal remote sensing data, combining spectral and structural information has become an effective strategy for improving classification performance in complex high-resolution scenes. However, most existing methods predominantly rely on shallow feature concatenation, which fails to capture long-range dependencies and cross-modal interactions that are critical for distinguishing fine-grained land cover categories. This study proposes a multi-level cross-modal attention fusion network, Cross-Modal Cross-Attention UNet (CMCAUNet), which integrates a Cross-Modal Cross-Attention Fusion (CMCA) module and a Skip-Connection Attention Gate (SCAG) module. The CMCA module progressively enhances multimodal feature representations throughout the encoder, while the SCAG module leverages high-level semantics to refine spatial details during decoding and improve boundary delineation. Together, these modules enable more effective integration of spectral–textural and structural information. Experiments conducted on the ISPRS Vaihingen and Potsdam datasets demonstrate the effectiveness of the proposed approach. CMCAUNet achieves an mean Intersection over Union (mIoU) ratio of 81.49% and 84.76%, with Overall Accuracy (OA) of 90.74% and 90.28%, respectively. The model also shows superior performance in small object classification, with targets like “Car,” achieving 90.85% and 96.98% OA for the “Car” category. Ablation studies further confirm that the combination of CMCA and SCAG modules significantly improves feature discriminability and leads to more accurate and detailed land cover maps.

1. Introduction

In recent years, breakthroughs in Earth observation technology have significantly enhanced the availability of multimodal high-resolution remote sensing data, encompassing diverse sources such as optical, multispectral, and hyperspectral imagery; LiDAR; and Synthetic Aperture Radar (SAR) [1,2,3,4,5,6,7,8,9]. These heterogeneous data sources provide complementary information from different physical perspectives, enabling comprehensive characterization of Earth’s spectral features, texture properties, and structural characteristics. Consequently, the effective fusion of these multi-source heterogeneous data has become a key research focus in Earth and environmental sciences. As multimodal data fusion technology can substantially improve the accuracy and robustness of critical applications like fine terrain classification and land cover mapping [3,4,5,6], it provides more reliable decision-making foundations for change detection [1,2] and remote sensing analysis tasks [7,8,9].

Semantic segmentation, as a core task in intelligent remote sensing interpretation, aims to assign precise land cover category labels to each pixel in images. Traditional machine learning methods such as random forests [10], support vector machines [11,12], and conditional random fields [13] perform poorly when handling the complex spectral and spatial variations in high-resolution images. These methods struggle to extract high-level abstract semantic representations and thus fail to fully capture the nonlinear and intricate features of real-world terrain environments. With the rise of deep learning (DL) technologies, particularly convolutional neural networks (CNNs) that leverage their powerful capabilities for automatic learning of multi-scale spatial and semantic features, significant progress has been made in remote sensing image semantic segmentation [14,15,16,17,18,19,20,21,22,23,24,25]. However, a fundamental challenge persists: most CNN architectures are inherently designed for single-modal data; when processing multimodal inputs, they only employ basic fusion operations like early fusion for simple stitching or element-wise addition. These simplified strategies fail to fully exploit deep complementary information between modalities and complex cross-modal correlations, ultimately limiting the models’ discriminative ability and generalization performance in complex terrain classification tasks.

While single-modal semantic segmentation has achieved significant progress in computer vision [26,27,28,29], research on multimodal data fusion in remote sensing still faces numerous bottlenecks. Most existing methods integrate features based on the inherent attributes of individual modalities [14,30,31]. However, a fundamental challenge remains unresolved: How can we fully explore and capture spatial dependencies between land cover categories and cross-modal contextual information? This remains an open challenge in both computer vision and remote sensing. Multimodal data in remote sensing, by integrating spectral, textural, and structural features, can provide richer information than single modalities. However, inherent heterogeneity, spatial misalignment, and noise variations pose significant challenges to feature fusion. This challenge becomes particularly prominent in high-resolution scenarios where land cover categories exhibit complex spatial structures and significant differences in scale and morphology [3,32]. Traditional convolution-based fusion methods often struggle to capture the global contextual associations required for accurate differentiation of similar land cover categories (e.g., different types of roofs and vegetation) due to their limited receptive fields [33]. Although architectures like Vision Transformer (ViT) have introduced new paradigms for capturing long-range dependencies [34], designing efficient Multi-modal fusion mechanisms for convolutional architectures to establish cross-modal associations remains a challenging yet highly valuable research area.

To address these challenges, researchers have explored three mainstream multimodal fusion strategies: early fusion, late fusion, and intermediate fusion [17,33,35,36], as shown in

Table 1. Summary of the mainstream multimodal fusion strategies.

Fusion Strategy	Description
Early Fusion [17]	Directly merges raw data at the input layer; ensures spatial alignment accuracy, but is prone to introducing redundancy or task-irrelevant noise.
Late Fusion [35]	Integrates predictions from independently trained unimodal networks; ignores potential semantic associations between modalities at the feature level.
Intermediate Fusion [36]	Provides a compromise through feature-level interactions (e.g., summation or concatenation); however, single-layer operations may fail to capture hierarchical dependencies.
Attention-based Fusion [37]	Employs cross-modal attention mechanisms for coarse-grained tasks; often lacks the spatial accuracy and multi-scale refinement needed for fine-grained mapping.

. Early fusion directly merges raw data at the input layer, ensuring spatial alignment accuracy, but it is prone to introducing redundancy or task-irrelevant noise. Late fusion integrates predictions from independent networks while completely ignoring potential semantic associations between modalities at the feature level [17,35]. Intermediate fusion provides a more refined compromise through feature-level interactions [36], yet most methods still rely on single-layer operations (e.g., element-wise summation or concatenation) that fail to capture hierarchical dependencies and long-range correlations between cross-modal features [33]. TMFNet [37] proposes a Transformer-based multimodal fusion network specifically for semantic segmentation, which integrates a multimodal former module to capture hierarchical dependencies and a Boundary Region Attention module to refine small targets. CCFormer [38], which employs a transformer-based architecture for multimodal cross-attention fusion and is optimized for coarse-grained scene classification, lacks the spatial accuracy and multi-scale feature refinement required for fine-grained land cover mapping. Therefore, the task of land cover classification urgently requires an advanced feature-level fusion mechanism based on CNN that can be used for more detailed classification and is capable of deeply mining and integrating complementary cross-modal information across different scales.

This study proposes a Cross-Modal Cross-Attention UNet (CMCAUNet)—a novel multi-level cross-modal fusion framework designed to overcome the limitations of traditional CNN-based approaches in multimodal land object classification. CMCAUNet employs a dual-branch encoder to process visible light data and Digital Surface Model (DSM) data, respectively, while utilizing a designed Cross-Modal Cross-Attention Fusion (CMCA) module to model bidirectional information exchange between different modalities, effectively establishing long-distance dependencies and complementary relationships among multimodal data. In the decoder section, a Skip-Connection Attention Gate (SCAG) module is introduced, which not only refines spatial details such as land cover boundaries but also effectively bridges the gap between low-order visual features and high-order semantic features. This hierarchical progressive fusion architecture achieves refined integration of features from superficial to deep levels, significantly improving the network’s segmentation accuracy and robustness for complex land objects. The main contributions of this study can be summarized as follows:

• Novel Cross-Modal Cross-Attention Fusion Mechanism: The proposed CMCA module enables deep bidirectional information exchange across modalities, effectively capturing cross-modal long-distance dependencies critical for land cover classification, thereby significantly enhancing the discriminative power of feature representation.
• Hierarchical Feature Fusion Architecture: CMCAUNet integrates CMCA and SCAG modules through a dual-branch encoder and a carefully designed decoder structure, creating a multi-level feature fusion framework that progresses from shallow to deep layers, significantly enhancing the model’s ability to focus on semantically important regions.
• Comprehensive Experimental Validation and Performance Comparison: Extensive experiments on the public ISPRS Vaihingen and Potsdam datasets demonstrate that CMCAUNet outperforms existing state-of-the-art models in multimodal land cover classification, validating the effectiveness and superiority of the proposed method.

The structure of this paper is as follows: Section 2 provides an overview of relevant CNN segmentation methods; Section 3 details the CMCAUNet architecture; Section 4 validates the model’s performance through extensive experiments; Section 5 presents the discussion; and Section 6 summarizes the research.

2. Related Works

2.1. Single-Modal Semantic Segmentation

In the field of remote sensing object classification, deep learning-based semantic segmentation methods have become the mainstream paradigm. The pioneering work originated from the Fully Convolutional Network (FCN) [26], which achieved end-to-end pixel-level prediction for the first time, laying the foundation for subsequent research. However, the simplified upsampling operation in FCN’s decoder resulted in insufficient spatial detail restoration, often leading to issues such as blurred object boundaries and imprecise contours in segmentation results. To address this limitation, researchers proposed the UNet architecture based on the classic encoder–decoder network structure [27]. This architecture extracts hierarchical multi-scale features through progressive downsampling in the encoder, followed by symmetrical upsampling by the corresponding decoder to restore spatial resolution. Its core innovation lies in the introduced skip-connection mechanism, which fuses low-level high-resolution features (such as texture and edges) from the encoder with semantically rich high-level features in the decoder. This mechanism has been proven effective in preserving spatial details of objects, sharpening category boundaries, and thereby improving classification accuracy.

While the aforementioned architecture demonstrates remarkable performance in single-modal land classification, the inherent limitation of local receptive fields in convolutional neural networks [39] fundamentally constrains its performance ceiling. This architectural flaw makes it challenging for the model to effectively capture global contextual information and long-range spatial dependencies in input images [40]. Particularly in complex real-world terrain scenarios, this limitation becomes especially pronounced, as the model often fails to comprehend spatial layout relationships between land objects, thereby affecting accurate interpretation of the overall scene’s semantic structure. To address this issue, researchers have begun exploring network architectures capable of modeling global context, such as the pyramid scene parsing network [41], which enhances scene understanding by aggregating contextual information from different regions, providing a promising direction for future research.

2.2. Multimodal Semantic Segmentation Based on CNN

With the rapid advancement of Earth observation technologies, the acquisition of multimodal remote sensing data—such as multispectral imagery, optical photographs, and DSM—has become increasingly accessible. Among these, DSM stands out as an indispensable tool due to its ability to accurately encode elevation information of ground features. In tasks requiring precise ground feature classification, elevation data is crucial for distinguishing objects with similar spectral signatures but varying heights (e.g., low shrubs versus tall trees). It also provides clear structural boundaries, enabling accurate delineation of various targets. Consequently, multimodal semantic segmentation methods that integrate DSM with visible light data have emerged as a research hotspot [19,42].

Early deep learning approaches for multimodal remote sensing data fusion were relatively straightforward. For instance, the ResUNet-A model [43] achieved the most direct modal fusion by concatenating RGB images with DSM data across channels to form a four-channel composite input. While this method was simple and efficient, the crude channel-level fusion struggled to effectively address feature differences and noise inconsistencies across modalities [44,45], thereby limiting the model’s performance ceiling.

To achieve deeper feature interaction and enhance fusion performance, researchers have proposed various improved fusion architectures. FuseNet [14], an early representative, adopts a dual-branch parallel architecture to process RGB and DSM data in parallel, fusing features through element-wise addition after each convolutional block. Inspired by FuseNet, vFuseNet [15] introduces a multi-scale fusion strategy to further enhance the expressive power of multimodal features. However, these fusion strategies based on simple addition or concatenation remain fundamentally shallow feature interactions, failing to fully establish complex cross-modal relationships. To break through this bottleneck, CMGFNet [42] proposes a gated fusion module that integrates two modalities for feature extraction, achieving adaptive learning of discriminative features and removal of irrelevant information. Recent studies such as IIHN [46], CIMFNet [47], and ABHNet [48] explore feature fusion through more complex network architectures, including constructing hypergraph models, designing cross-layer gating mechanisms, or utilizing attention mechanisms. Although these methods have achieved performance improvements in specific tasks, they mostly focus on local or short-distance feature interactions, still lacking the global and long-distance cross-modal semantic dependencies required for modeling complex terrain categories [33]. Therefore, designing an advanced fusion mechanism capable of effectively capturing global contextual associations remains a critical challenge in the field of multimodal terrain classification.

3. Proposed Method

To address the limitations of conventional CNN-based multimodal semantic segmentation, we propose a Cross-Modal Cross-Attention UNet Network for mmultimodalhigh-resolution remote sensing imagery. In the backbone network, CMCAUNet employs the Cross-Modal Cross-Attention Fusion module to achieve cross-modal feature fusion. Specifically, the network extracts features from visible imagery (VIS) and DSM data using two separate CNN branches [49]. After each CNN module, the CMCA module performs cross-modal fusion and feature enhancement at multiple scales. The combined and reshaped CMCA fusion output is input to the next-level encoder. At the same time, it is input to the cascade decoder through skip connections, thereby achieving high-precision restoration of spatial information.

To illustrate, VIS and DSM data are used to demonstrate the proposed CMCAUNet, with VIS serving as the primary modality due to its richer spectral information and DSM acting as an auxiliary modality, providing structural and elevation information. Importantly, the proposed fusion scheme is generalizable and can be applied to other multimodal fusion tasks. The core components of CMCAUNet are described in detail in the following sections.

3.1. Architecture of CMCAUNet

The overall architecture of CMCAUNet is illustrated in Figure 1. The network architecture follows a typical encoder–decoder paradigm. It is distinguished by the strategic integration of two novel modules, the Cross-Modal Cross-Attention Fusion module and the Skip-Connection Attention Gate, which synergistically enhance multimodal feature fusion and the restoration of fine-grained spatial details.

The encoder is constructed with a symmetric dual-branch architecture, with each branch dedicated to processing a distinct modality independently. The DSM branch consists of four convolutional layers that progressively extract elevation and structural features from the DSM data. The VIS branch also consists of four convolutional layers designed to capture rich spectral and textural features from optical images. Each convolutional layer consists of two convolution operations, followed by batch normalization and activation functions. It should be noted that the VIS data contains three channels, while the DSM data has only one channel. At each encoding level (from Layer 1 to Layer 4), feature maps of the same scale from both modalities are fed into the designed Cross-Modal Cross-Attention Fusion module. Through its bidirectional cross-attention mechanism, the shallow features from both the DSM and VIS modalities are deeply fused, generating a discriminative feature representation that jointly encodes spectral, textural, and elevation information. These fused features are then used as the VIS-branch input for the next deeper encoding level. This hierarchical and progressive fusion strategy ensures effective integration of multimodal information from shallow to deep levels. After four levels of CMCA fusion and progressive downsampling, the resulting features are passed to a bottleneck layer (Bridge), which further extracts higher-level semantic representations through a single convolutional block.

The decoder aims to progressively restore the spatial resolution and generate the final segmentation map. It consists of four upsampling stages, each comprising an upsampling operation (UpConv), attention-gated fusion, and convolution operations. The upsampling step enlarges the feature map size using transposed convolution or bilinear interpolation. For attention-gated fusion, the upsampled features (as the guiding signal G) and the CMCA-fused skip connection features from the corresponding encoder level are fed together into the Skip-Connection Attention Gate module. The SCAG module adaptively selects and enhances the spatial details in the skip connections using the semantic context from the decoder, thus alleviating the semantic gap between shallow and deep features. The convolution then integrates the features processed by the SCAG module through a convolutional block, preparing them for the next upsampling stage. Finally, after four decoding stages, the output feature map is projected through a 1 × 1 convolution to produce per-pixel class probability maps.

Overall, CMCAUNet integrates hierarchical cross-modal fusion (via CMCA modules) and adaptive skip-connection refinement (via SCAG modules) within an efficient encoder–decoder structure, enabling fine-grained semantic segmentation of multimodal remote sensing data.

3.2. Cross-Modal Cross-Attention Fusion Module

To effectively capture the deep complementary relationships between VIS and DSM data, we designed a novel Cross-Modal Cross-Attention Fusion module. The core idea of this module is to draw inspiration from the cross-attention mechanism to achieve bidirectional information interaction and enhancement between the feature maps of the two modalities, thereby generating fused features that are more discriminative for object classification. As shown in Figure 2, the CMCA module receives two parallel feature maps as input, originating from the VIS branch (${\mathrm{F}}_{\mathrm{vis}}\in {\mathrm{R}}^{\mathrm{C} \times \mathrm{H} \times \mathrm{W}}$) and the DSM branch (${\mathrm{F}}_{\mathrm{dsm}}\in {\mathrm{R}}^{\mathrm{C} \times \mathrm{H} \times \mathrm{W}}$), where H and W denote the height and width of the feature map. To unify the feature dimensions for subsequent processing and extract high-level semantics, we perform separate 1 × 1 convolution operations on the two input feature maps to generate corresponding Query, Key, and Value for each modality. For the optical modality, we generate ${\mathrm{Q}}_{\mathrm{vis}}$, ${\mathrm{K}}_{\mathrm{vis}}$, ${\mathrm{V}}_{\mathrm{vis}}$, and for the DSM modality, we generate ${\mathrm{Q}}_{\mathrm{dsm}}$,${\mathrm{K}}_{\mathrm{dsm}}$, ${\mathrm{V}}_{\mathrm{dsm}}$, where $\mathrm{Q}, \mathrm{K}, \mathrm{V}\in {\mathrm{R}}^{\mathrm{C}\prime \times \mathrm{H} \times \mathrm{W}}$ and $\mathrm{C}\prime$ is the dimension of the hidden layer.

Directly computing attention on the original feature map dimensions (H × W) incurs significant computational overhead. To mitigate this issue while preserving performance, we adopt an efficient computational strategy. We apply adaptive average pooling to the core computational tensors (the Q, K, and V matrices) to downsample their spatial dimensions to a predefined, smaller scale. This scale is dynamically adjusted based on the size of the feature map at different network stages, such as 4 × 4 in deeper layers and 32 × 32 in shallower layers. To maintain the fine-grained spatial details of the original input, the resulting attention-enhanced feature map is upsampled back to its original H × W resolution via bilinear interpolation. Finally, this upsampled feature is residually connected with the original, unpooled Value tensor. This approach achieves computational efficiency while effectively fusing cross-modal contextual information with high-resolution detail.

The cross-attention computation is bidirectional. Attention from VIS to DSM ($\mathrm{Att}{\mathrm{n}}_{\mathrm{vis}-\mathrm{dsm}}$): This attention weight is used to measure the correlation between each $\mathrm{Q}$ of the VIS feature map and each $\mathrm{K}$ of the DSM feature map. Its calculation formula is as follows:

$$\mathrm{Att}{\mathrm{n}}_{\mathrm{vis}-\mathrm{dsm}} = \mathrm{Softmax}(\mathrm{Pool}{\left({\mathrm{Q}}_{\mathrm{vis}}\right)}^{\mathrm{T}}\cdot \mathrm{Pool}({\mathrm{K}}_{\mathrm{dsm}}\left)\right)$$

(1)

Attention from DSM to VIS ($\mathrm{Att}{\mathrm{n}}_{\mathrm{dsm}\to \mathrm{vis}}$): Similarly, this weight measures the degree of attention the DSM feature map pays to the VIS feature map:

$$\mathrm{Att}{\mathrm{n}}_{\mathrm{dsm}-\mathrm{vis}} = \mathrm{Softmax}(\mathrm{Pool}{\left({\mathrm{Q}}_{\mathrm{dsm}}\right)}^{\mathrm{T}}\cdot \mathrm{Pool}({\mathrm{K}}_{\mathrm{vis}}\left)\right)$$

(2)

After obtaining the attention weights, we apply them to the pooled Value tensors to aggregate contextual information from the other modality. The enhanced feature for the VIS modality, ${\mathrm{O}}_{\mathrm{vis}}$, is obtained by weighting the Values ${\mathrm{V}}_{\mathrm{dsm}}$ from the DSM modality:

$${\mathrm{O}}_{\mathrm{vis}} = \mathrm{Interpolate}\left(\mathrm{Pool}({\mathrm{V}}_{\mathrm{dsm}})·{\mathrm{Attn}}_{\mathrm{vis}-\mathrm{dsm}}^{\mathrm{T}}\right)$$

(3)

The enhanced feature for the DSM modality, ${\mathrm{O}}_{\mathrm{dsm}}$. is obtained by weighting the Values ${\mathrm{V}}_{\mathrm{v}\mathrm{i}\mathrm{s}}$ from the VIS modality:

$${\mathrm{O}}_{\mathrm{dsm}} = \mathrm{Interpolate}\left(\mathrm{Pool}({\mathrm{V}}_{\mathrm{vis}})·{\mathrm{Attn}}_{\mathrm{dsm}\to \mathrm{vis}}^{\mathrm{T}}\right)$$

(4)

where $\mathrm{Interpolate}\left(·\right)$ denotes a bilinear interpolation upsampling operation, which restores the aggregated features to the original H × W size. To preserve the original information of each modality and stabilize the training process, we perform a residual connection between the cross-modality aggregated features and the original Value tensor of that modality:

$${\tilde{\mathrm{O}}}_{\mathrm{v}\mathrm{i}\mathrm{s}}={\mathrm{O}}_{\mathrm{v}\mathrm{i}\mathrm{s}}+{\mathrm{V}}_{\mathrm{v}\mathrm{i}\mathrm{s}},{ \tilde{\mathrm{O}}}_{\mathrm{d}\mathrm{s}\mathrm{m}}={\mathrm{O}}_{\mathrm{d}\mathrm{s}\mathrm{m}}+{\mathrm{V}}_{\mathrm{d}\mathrm{s}\mathrm{m}}$$

(5)

Subsequently, the two enhanced feature maps are each passed through a 1 × 1 convolutional layer for feature integration. Finally, they are concatenated along the channel dimension to form the final fused feature map ${\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}\in {\mathbb{R}}^{2\mathrm{C}\prime \times \mathrm{H}\times \mathrm{W}}$:

$${\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}=\mathrm{C}\mathrm{a}\mathrm{t}\left[\begin{array}{c}Con{v}_{1\times 1}\left({\mathrm{V}}_{\mathrm{v}\mathrm{i}\mathrm{s}}+\mathrm{I}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{p}\mathrm{o}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{e}\left(\mathrm{Pool}({\mathrm{V}}_{\mathrm{dsm}})·{\mathrm{Softmax}}^{\mathrm{T}}(\mathrm{Pool}{\left({\mathrm{Q}}_{\mathrm{vis}}\right)}^{\mathrm{T}}·\mathrm{Pool}({\mathrm{K}}_{\mathrm{dsm}}\left)\right)\right)\right)\\ Con{v}_{1\times 1}\left({\mathrm{V}}_{\mathrm{d}\mathrm{s}\mathrm{m}}+\mathrm{I}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{p}\mathrm{o}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{e}\left(\mathrm{Pool}({\mathrm{V}}_{\mathrm{vis}})·{\mathrm{Softmax}}^{\mathrm{T}}(\mathrm{Pool}{\left({\mathrm{Q}}_{\mathrm{dsm}}\right)}^{\mathrm{T}}·\mathrm{Pool}({\mathrm{K}}_{\mathrm{vis}}\left)\right)\right)\right)\end{array}\right]$$

(6)

Through this series of meticulously designed operations, the CMCA module not only achieves deep, bidirectional information interaction between the two modalities but also effectively controls computational complexity through its pooling strategy. This enables it to be flexibly embedded into the shallow layers of the network, thereby establishing a powerful cross-modal feature representation early in the segmentation task.

3.3. Skip-Connection Attention Gate Module

In the encoder–decoder architecture, skip connections are key to restoring spatial details. They work by transferring high-resolution shallow features from the encoder path to the decoder, effectively compensating for the inevitable feature information lost due to continuous downsampling and pooling operations. However, while the shallow features from the encoder retain rich low-level spatial information such as edges and textures, their semantic representation capability is weak. In contrast, the deep features obtained by the decoder through successive layers of abstraction contain highly generalized contextual semantic information. Directly concatenating or adding these two feature maps with such a large disparity in information hierarchy would lead to a “semantic gap,” reducing segmentation accuracy. To address this issue, we have designed an innovative Skip-Connection Attention Gate module and integrated it into each skip connection path of the decoder. This module, also known as SCAG, functions to dynamically select and enhance the information most relevant to the current decoder’s semantics before fusing the encoder features.

As shown in Figure 3, the SCAG module receives two inputs: the upsampling feature map generated by the previous decoder layer (as the guidance signal, $\mathrm{G}\in {\mathrm{R}}^{{\mathrm{F}}_{\mathrm{g}} \times \mathrm{H} \times \mathrm{W}}$) and the skip-connection feature map corresponding to the encoder layer ($\mathrm{S}\in {\mathrm{R}}^{{\mathrm{F}}_{\mathrm{l}} \times \mathrm{H} \times \mathrm{W}}$). Unlike the bidirectional interaction of the CMCA module, the SCAG module performs an asymmetric, guidance-driven attention mechanism. Its objective is to leverage the semantic information $\mathrm{G}$ from the decoder to “query” and “filter” the useful information S from the encoder. To control the computational load, all tensors are first projected into an intermediate dimension ${\mathrm{F}}_{\mathrm{int}}$ (typically set to ${\mathrm{F}}_{\mathrm{l}}//4$). Similarly to CMCA, and to enhance efficiency, we first perform adaptive average pooling on the Query ($\mathrm{Q}$) and Key ($\mathrm{K}$) tensors, reducing their spatial dimensions to a smaller, dynamically adjusted size of $\mathrm{p}\times \mathrm{p}$ (where $\mathrm{p}$ does not exceed the actual dimensions of the feature map). The Query and Key tensors are generated by passing the pooled guidance feature $\mathrm{G}$ and the pooled skip-connection feature $\mathrm{S}$ through corresponding 1 × 1 convolutions, respectively. The Value ($\mathrm{V}$) tensor is generated by passing the original skip-connection feature $\mathrm{S}$ through a 1 × 1 convolution. The attention weights are obtained by calculating the similarity between the Query and the Key:

$$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{{\mathrm{Q}}^{\mathrm{l}}\cdot \mathrm{K}}{\sqrt{{\mathrm{F}}_{\mathrm{i}\mathrm{n}\mathrm{t}}}}}\right)$$

(7)

The scaling factor $\sqrt{{\mathrm{F}}_{\mathrm{i}\mathrm{n}\mathrm{t}}}$, a standard practice in attention mechanisms, is used to stabilize gradients when the dot products become excessively large. The resulting attention weight map is then applied to the Value ($\mathrm{V}$) tensor. This step signifies that the semantic information $\mathrm{G}$ from the decoder dictates which spatial regions and features from the encoder’s skip connection $\mathrm{S}$ should be attended to. The aggregated feature $\mathrm{O}$ is upsampled back to the original resolution H × W. To preserve the original information from the skip connection, the attention-filtered feature $\mathrm{O}$ is then residually added to the projected original skip connection feature:

$$\dot{\mathrm{O}}=\mathrm{O}+\mathrm{V}$$

(8)

Finally, a block composed of a convolution layer, a batch normalization layer, and an activation function restores the fused features from the intermediate dimension ${\mathrm{F}}_{\mathrm{int}}$ resize to the original skip connection channel count ${\mathrm{F}}_{\mathrm{l}}$. The output feature map $\tilde{\mathrm{S}}\in {\mathbb{R}}^{{\mathrm{F}}_{\mathrm{l}}\times \mathrm{H}\times \mathrm{W}}$ can then be directly concatenated with the decoder’s upsampled feature $\mathrm{G}$ to complete the information fusion. Through the SCAG module, our network can intelligently “focus” on the most useful parts of the encoder features for the current segmentation task while suppressing irrelevant or noisy information, thereby effectively mitigating the semantic gap problem and achieving more refined feature fusion.

4. Experiment

4.1. Datasets

4.1.1. Vaihingen

The Vaihingen dataset contains 16 ultra-high resolution true orthoimages with an average resolution of 2500 × 2000 pixels. Each orthoimage consisted of three channels, the near-infrared, red, and green bands (NIRRG), along with a normalized DSM with a 9 cm ground sampling interval. The dataset comprises five foreground classes: Building (Bui.), Tree (Tre.), Low Vegetation (Low.), Car, and Impervious Surface (Imp.), as well as one background class (Clutter). In addition, the 16 orthoimages were divided into a training set with 12 sample blocks and a test set with 4 sample blocks: the training set contains sample blocks with indices 1, 3, 7, 11, 13, 17, 23, 26, 28, 32, 34, and 37; the test set contains sample blocks 5, 21, 15, and 30.

4.1.2. Potsdam

The Potsdam dataset contains 24 high-resolution orthophotography images, each of resolution 6000 × 6000 pixels. The dataset provides four multispectral channels—infrared, red, green, and blue (IRRGB)—along with a normalized DSM with 5 cm ground sampling spacing. Its foreground categories are the same as the Vaihingen dataset, but the distribution differs due to geographical differences. In this dataset, RGB synthetic images are used for processing. The 24 orthophoto images were divided into 18 training samples and 6 test samples, with the specific allocation as follows: the training set contains samples numbered 2_10, 2_12, 3_11, 3_12, 4_11, 4_12, 5_10, 5_12, 6_7, 6_8, 6_9, 6_10, 6_12, 7_7, 7_8, 7_9, 7_10, and 7_11; the test set contains samples numbered 2_11, 3_10, 4_10, 5_11, 6_11, and 7_12.

Figure 4 shows sample images from the two datasets. To process large-scale images without pre-cropping, our methodology employs a sliding window strategy to dynamically generate training batches. During the training phase, the sliding window size is set to 256 × 256 with a stride of 256. In the testing phase, this is adjusted to a stride of 32. Using a smaller stride during testing allows for the averaging of prediction results in overlapping regions, which effectively suppresses boundary effects.

4.2. Evaluation Metrics

To evaluate the segmentation performance of the multimodal remote sensing data, we used Overall Accuracy (OA), mean F1-score (mF1), and mean Intersection over Union (mIoU) as evaluation metrics. These standardized statistical metrics effectively compare the performance of the proposed CMCAUNet method with other frontier networks. Specifically, we calculate the mF1 and mIoU Values for the five foreground classes, respectively. Furthermore, in the Overall Accuracy evaluation, we also include the clutter or background regions in the consideration, which is defined as follows:

$$\mathrm{O}\mathrm{A}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N}}}$$

(9)

where the TP, FP, TN, and FN represent True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN), respectively. Additionally, the F1-score and Intersection over Union (IoU) for each foreground class are calculated according to the following formulas:

$$\mathrm{F}\mathrm{I}=2\times {\displaystyle \frac{{\mathrm{Q}}_{\mathrm{c}}\times {\mathrm{R}}_{\mathrm{c}}}{{\mathrm{Q}}_{\mathrm{c}}+{\mathrm{R}}_{\mathrm{c}}}}$$

(10)

$$\mathrm{I}\mathrm{o}\mathrm{U}={\displaystyle \frac{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}}{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{N}}_{\mathrm{c}}}}$$

(11)

where ${\mathrm{T}\mathrm{P}}_{\mathrm{c}}$, ${\mathrm{F}\mathrm{P}}_{\mathrm{c}}$ and ${\mathrm{F}\mathrm{N}}_{\mathrm{c}}$ represent the True Positive, False Positive, and False Negative for class c, respectively. Finally, the calculation formulas for ${\mathrm{Q}}_{\mathrm{c}}$ and ${\mathrm{R}}_{\mathrm{c}}$ are

$${\mathrm{Q}}_{\mathrm{c}}={\displaystyle \frac{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}}{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{P}}_{\mathrm{c}}}}$$

(12)

$${\mathrm{R}}_{\mathrm{c}}={\displaystyle \frac{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}}{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{N}}_{\mathrm{c}}}}$$

(13)

We obtain the F1 and IoU scores of the five valid foreground classes based on the above definitions and then derive the mean Values, mF1 and mIoU.

4.3. Implementation Details

All experiments were performed using the v2.4.1 PyTorch framework on an NVIDIA GeForce RTX 4070 graphics card with 12 GB of VRAM. The model was trained using the Stochastic Gradient Descent (SGD) algorithm with the following parameter settings: learning rate of 0.001, momentum of 0.9, weight decay of 0.0005, and batch size of 10. After the samples are collected through the sliding window, the data is subjected to basic augmentation processing, such as rotation and flipping. It takes about 3 min to train an epoch on the Vaihingen dataset.

4.4. Performance Comparison

We compare the performance of the proposed CMCAUNet with eight representative frontier methods. These include ABCNet [50], PSPNet [41], MAResUNet [51], vFuseNet [15], FuseNet [14], ESANet [30], SA-GATE [52], and CMGFNet [42]. The experiments indicate that ABCNet, PSPNet, and MAResUNet utilize only the primary modality of visible light images. These single-modal methods intuitively demonstrate the influence of multimodal data on DSM analysis, highlighting the advantage of mmultimodalover single-modal approaches. The other methods, however, comprehensively consider the cross-data between different data sources. See Table 1 and

Table 2. Performance of the network on the Vaihingen dataset.

Method	OA (%)						mF1 (%)	mIoU (%)
	Bui.	Tre.	Low.	Car	Imp.	Total
ABCNet [50]	94.10	90.81	78.53	64.12	89.70	89.25	85.34	75.20
PSPNet [41]	94.52	90.17	78.84	79.22	92.03	89.94	86.55	76.96
MAResU-Net [51]	94.84	89.99	79.09	85.89	92.19	90.17	88.54	79.89
vFuseNet [15]	95.92	91.36	77.64	76.06	91.85	90.49	87.89	78.92
FuseNet [14]	96.28	90.28	78.98	81.37	91.66	90.51	87.71	78.71
ESANet [30]	95.69	90.50	77.16	85.46	91.39	90.61	88.18	79.42
SA-GATE [52]	94.84	92.56	81.29	87.79	91.69	91.10	89.81	81.27
CMGFNet [42]	97.75	91.60	80.03	87.28	92.35	91.72	90.00	82.26
CMCAUNet (our)	97.76	91.48	77.41	90.85	90.31	90.74	89.47	81.49

for details on the quantitative results.

4.4.1. Performance Comparison on the Vaihingen Dataset

The quantitative results presented in Table 2, compared to the baseline model FuseNet, show that our proposed CMCAUNet achieves highly competitive performance. The model achieves an impressive OA of 90.74%. For the more challenging metrics, it obtains scores of 89.47% in mF1 and 81.49% in mIoU, ranking among the top contenders. This validates the capability of the CMCAUNet to extract complementary information from auxiliary modalities, successfully perform cross-modal fusion, and effectively generate robust representations.

Notably, CMCAUNet demonstrates significant advantages in the segmentation of specific classes. Particularly in segmenting the small object Car, our model achieves an accuracy of 90.85%, an improvement of 3.06% over the existing method SA-GATE. This serves as strong evidence for the exceptional capability of our designed cross-modal attention mechanism in preserving and identifying fine-grained spatial details. Furthermore, for the Building class, CMCAUNet attains an accuracy of 97.76%, the best performance among all other compared models. This indicates its effectiveness in leveraging multimodal information to distinguish between objects with similar spectral signatures but different heights. However, the model’s performance is slightly inferior on certain classes, such as Low Vegetation and Impervious Surface, with accuracies of 77.41% and 90.31%, respectively, falling short of some advanced methods. This suggests that while pursuing superior small object segmentation performance, the model may make certain trade-offs in performance on some larger classes. Nevertheless, its outstanding advantage in small object segmentation makes it invaluable for application scenarios requiring fine-grained segmentation, such as vehicle counting and small object recognition.

To more intuitively demonstrate our model’s segmentation performance in terms of generalization, we provide a visual comparison of eight methods in Figure 5. Remote sensing images are more complex than natural images for two main reasons: first, buildings vary significantly in size, and while they have clear boundaries, their shapes are highly diverse; second, trees and low vegetation are intricately interwoven. The experimental results show that the proposed method can identify complex edges more accurately, generate complete and coherent objects, and effectively reduce isolated pixels. As shown in Figure 5a–l, the segmentation map generated by our CMCAUNet (Figure 5l) has clearer and more accurate boundaries. Specifically, for the cars in the image (the small yellow dots in Figure 5), the predictions of the compared models exhibit some blurring and adhesion to the road, whereas our model can precisely segment each Car individually with well-defined contours. This is completely consistent with the quantitative results in Table 2, among which the Car class achieved the highest accuracy rate. Meanwhile, at the boundaries between buildings and roads, the segmentation lines from CMCAUNet also adhere more closely to the ground truth, effectively reducing boundary confusion. In the two purple box regions in Figure 5, the upper-left box not only completely identifies the incomplete building at the edge of the image but also recognizes a white mobile cabin that was not annotated in the ground truth; the lower-right box accurately identifies the narrow low-vegetation road between two buildings while greatly preserving the original shape of the buildings. This indicates that the feature fusion through the CMCA module enables the model to focus more on the integrity of relevant target areas and some small object regions.

4.4.2. Performance Comparison on the Potsdam Dataset

The quantitative results presented in

Table 3. Performance of the network on the Potsdam dataset.

Method	OA (%)						mF1 (%)	mIoU (%)
	Bui.	Tre.	Low.	Car	Imp.	Total
ABCNet [50]	96.23	78.92	86.40	92.92	88.90	87.52	88.14	79.26
PSPNet [41]	97.03	83.13	85.67	88.81	90.91	88.67	88.92	80.36
MAResU-Net [51]	96.82	83.97	87.70	95.88	92.19	89.82	90.86	83.61
vFuseNet [15]	97.23	84.29	89.03	95.49	91.62	90.22	91.26	84.26
FuseNet [14]	97.48	85.14	87.31	96.10	92.64	90.58	91.60	84.86
ESANet [30]	97.10	85.31	87.81	94.08	92.76	89.74	91.22	84.15
SA-GATE [52]	96.54	81.18	85.35	96.63	90.77	87.91	90.26	82.53
CMGFNet [42]	97.41	86.80	86.68	95.68	92.60	90.21	91.40	84.53
CMCAUNet (our)	97.51	84.83	87.98	96.98	92.95	90.28	91.52	84.76

show that the proposed CMCAUNet achieves performance comparable to or even superior to various frontier networks in the multimodal semantic segmentation task. Specifically, CMCAUNet attains an overall F1-score of 91.52% and an mIoU of 84.76%, ranking it among the top-performing networks. It exhibits particularly outstanding results on the Building (97.51%), Car (96.98%), and Impervious Surface (92.95%) classes, which indicates that the CMCA module can effectively capture fine-grained spatial features and enhance the feature complementarity between VIS and DSM data.

Notably, CMCAUNet’s segmentation performance on the Car class surpasses mainstream models such as FuseNet and ESANet, effectively reducing adhesion and misclassification artifacts at vehicle edges. This result demonstrates that the CMCA module, integrated into the network, can better focus on target regions during the fusion of VIS and DSM data, thereby improving the recognition accuracy for small objects. Simultaneously, for the large-area land-cover classes Building and Impervious Surface, CMCAUNet achieved accuracies of 97.51% and 92.95%, respectively, both ranking first among all compared methods. This reflects the network’s exceptional capability in maintaining boundary consistency and structural integrity. Furthermore, while its performance on the Tree class (84.83%) is slightly inferior to the top-performing CMGFNet (86.80%), it outperforms FuseNet on the Low Vegetation class (87.98%) and remains competitive with other advanced methods like vFuseNet. This highlights the network’s stability in multi-scale feature representation and cross-modal feature fusion, enabling it to maintain high fidelity for both small objects and large-area land-cover types.

Figure 6 shows the visualized prediction results of each method on a 2000 × 2000 pixel Potsdam test sample. Figure 6a,b are the input visible image and DSM data, respectively. Figure 6c is the ground truth, and Figure 6d–l sequentially display the segmentation results of the eight methods, with key comparison areas marked by purple boxes in the figures. Overall, traditional multimodal fusion networks (such as ABCNet and PSPNet) exhibit some blurring and misclassification at building edges and vegetation boundary areas. Although MAResU-Net and ESANet are relatively accurate in segmenting large land-cover classes, they still suffer from discontinuities or adhesion issues in small object recognition and shadowed areas. In contrast, the segmentation result generated by CMCAUNet (Figure 6l) is visually clearer and more coherent, with smoother object boundaries and significantly fewer misclassified regions.

Two purple annotation boxes have been added in Figure 6a–l for comparison. In the upper purple box area, trees obscure buildings, and it can be seen that CMCAUNet provides more accurate segmentation of Tree classes in complex scenes. The lower purple box demonstrates CMCAUNet’s fine segmentation effect on detailed objects. As shown in Figure 6, CMCAUNet is the only network among all that precisely identifies the long, narrow wall fence and nearly completely segments its contour despite the influence of trees. This fully demonstrates the synergistic effect of the CMCA and SCAG modules: the CMCA module enhances feature discriminability through deep cross-modal fusion, while the SCAG module precisely preserves spatial details and boundary integrity. Together, they achieve high-fidelity and fine-grained segmentation of complex scenes.

4.5. Ablation Study

To ascertain the individual contribution of each proposed component, we conducted a series of ablation studies within the CMCAUNet framework, with the results detailed in

Table 4. Comparison of ablation experiments.

Structure		OA (%)	mF1 (%)	mIoU (%)
CMCA	SCAG
√		87.07	84.34	73.70
	√	87.34	84.19	73.55
√	√	90.74	89.47	81.49

√: Indicates that the corresponding module is included in the network structure.

. We designed two ablation studies based on this network structure. The first experiment involves removing the proposed CMCA module, which was originally used to fuse features from the two modalities, while retaining the SCAG module in the decoding process. The second experiment removes the SCAG module while retaining the CMCA module in the shallow CNN features extracted from the two branches.

The quantitative results reveal that when the CMCA or SCAG module is used individually, the model’s performance is comparable, with OA of 87.07% and 87.34% and mIoU of 73.70% and 73.55%, respectively. However, when the two modules work in conjunction, the model’s performance is significantly enhanced, with OA reaching 90.74% and mIoU increasing to 81.49%, representing improvements of approximately 3.5% and 8.1%, respectively. This result strongly demonstrates the complementary and synergistic effects of the CMCA and SCAG modules.

To more intuitively illustrate the role of each module, we present the visualization results of the ablation study on the 1000 × 1000 Vaihingen test set in Figure 7. A comparative analysis reveals that with the CMCA module removed (Figure 7d), the model exhibits a pronounced deficiency in modal fusion when processing complex scenes. Compared with the ground truth (Figure 7b), severe boundary blurring and internal voids are observed in building areas, with some buildings being misclassified as impervious surfaces or low vegetation. Concurrently, the continuity of linear features like roads is disrupted, showing signs of fragmentation. This indicates that the CMCA module effectively fuses the spectral information from visible images with the height information from the DSM via its cross-modal attention mechanism, providing a more comprehensive feature representation for the model and playing a crucial role, particularly in distinguishing between spectrally similar objects with different heights.

In the case where the SCAG module is removed (Figure 7e), the model performs relatively well in overall object recognition but shows significant deficiencies in detail processing and small object segmentation. It can be observed that the edges of buildings are not refined, exhibiting jagged boundaries; the boundaries between low vegetation and trees are confused, with some low vegetation being misidentified as trees. More critically, some small objects (such as small Building and Car) are missed. These results confirm the critical role of the SCAG module in bridging the semantic gap. By applying cross-attention within the skip connections, it dynamically refines the shallow features from the encoder, aligning them with the deep semantic context of the decoder. This selective fusion preserves high-frequency spatial details, which in turn leads to a marked improvement in the recognition of small objects and the precision of boundary delineation.

The qualitative results from the complete CMCAUNet network (Figure 7f) demonstrate a remarkable balance between semantic accuracy and spatial precision. The model produces sharp, intact boundaries for large objects like buildings, though minor internal voids are still observable when compared to the ground truth. Furthermore, linear features such as roads are rendered continuously and smoothly, and the transitions between different land-cover types are sharply delineated. Notably, the model also exhibits a strong capability for capturing and segmenting smaller targets. This superior outcome is attributable to the synergistic interplay between the CMCA and SCAG modules. The CMCA module establishes a robust, semantically rich foundation by deeply fusing multimodal features, ensuring accurate class-level discrimination. The SCAG module then operates on this foundation, meticulously refining the feature maps to preserve and integrate high-resolution spatial details, which is essential for achieving such fine-grained boundary definition.

5. Discussion

5.1. Computational Complexity and Efficiency

To comprehensively evaluate the computational efficiency of the proposed CMCAUNet, we employed a comprehensive set of evaluation metrics. These metrics include Floating Point Operations (FLOPs), the number of parameters, memory footprint, and Frames Per Second (FPS). The metrics quantitatively assess the model across four dimensions: theoretical computational complexity, storage requirements, actual resource consumption, and real-time processing capabilities, aiming to find an optimal balance between computational cost and segmentation performance.

As shown in

Table 5. Comparison of complexity and efficiency.

Method	Multimodal	FLOPs (G)	Parameter (M)	Memory (MB)	Speed (FPS)	MIoU (%)
ABCNet [50]	N	3.9	13.39	1598	15.87	75.20
PSPNet [41]	N	49.03	46.72	3124	66.01	76.96
MAResU-Net [51]	N	8.79	26.27	1908	10.62	79.89
vFuseNet [15]	Y	60.36	44.17	2618	16.93	78.92
FuseNet [14]	Y	58.37	42.08	2284	18.92	78.71
ESANet [30]	Y	7.73	34.03	1914	10.42	79.42
SA-GATE [52]	Y	41.28	110.85	3174	10.00	81.27
CMGFNet [42]	Y	19.51	64.20	2463	11.61	82.26
CMCAUNet (our)	Y	73.92	71.53	1299	106.87	81.49

, the comparative analysis of model complexity and efficiency reveals a notable characteristic of CMCAUNet: despite its high theoretical computational complexity, it demonstrates exceptional efficiency in practical inference. Specifically, CMCAUNet achieves an inference speed of 106.87 FPS, significantly outperforming all comparison methods. This outstanding performance is primarily attributed to the architectural design of the CMCA and SCAG modules, which feature highly parallelized computational processes that fully utilize modern GPU computing resources. Notably, while achieving high-speed inference, CMCAUNet consumes only 1299 MB of memory, far lower than memory-intensive multimodal models like CMGFNet and SA-GATE. This dual advantage of speed and memory efficiency enables CMCAUNet to maintain high precision (81.49% mIoU) while remaining highly practical. These data collectively demonstrate CMCAUNet’s successful balance between high segmentation performance and low resource consumption.

However, the limitations of CMCAUNet cannot be overlooked. Its theoretical computational complexity (FLOPs) is relatively high, reaching 73.92 G. This indicates that the model requires substantial computing resources. To fully leverage its high-speed inference capabilities, it is recommended to deploy it on high-end GPUs or cloud platforms with robust parallel computing power.

Overall, CMCAUNet achieves high inference speed and low memory consumption in practical deployment by increasing theoretical computational load. This unique trade-off makes it particularly suitable for real-time terrain classification applications such as disaster emergency monitoring and urban dynamic analysis, where it can operate efficiently with sufficient computing resources.

5.2. Generalization and Semantic Understanding

The generalization capability and semantic comprehension of CMCAUNet were evaluated through visual analysis on the Vaihingen dataset. As shown in Figure 8, the model demonstrates superior semantic understanding beyond the given annotations within the 380 × 380 test area. In the purple-highlighted region of Figure 8a, the NIRRG image clearly displays a white mobile cabin, while the ground truth incorrectly classifies it as a non-porous surface (Figure 8b). The prediction results in Figure 8d reveal the model’s exceptional performance in spectral confusion scenarios, successfully distinguishing spectral environmental features from surrounding similar objects. This indicates that the model establishes a sophisticated spectral ambiguity resolution mechanism by leveraging structural cues from DSM data. The reasoning results in the low-illumination region of Figure 8a, further validating the model’s precise identification of low-vegetation areas, with its predicted boundaries closely matching the subtle elevation variations in DSM data (Figure 8c). These cases collectively demonstrate that CMCAUNet not only learns surface pattern annotations but also captures the latent relationship between spectral and structural features, achieving a leap from pattern matching to semantic reasoning. This capability enables effective handling of unseen or misannotated scenarios.

The model’s exceptional generalization capability stems from the complementary design of its CMCA and SCAG modules. The CMCA module achieves deep integration of NIRRG spectral data and digital elevation model (DSM) elevation information through a cross-modal attention fusion mechanism. This process enables the model to acquire a global multi-perspective understanding of ground features, which is crucial for addressing spectral ambiguity in land cover classification—such as distinguishing white mobile cabins with similar colors but distinct structural outlines from impermeable surfaces. Meanwhile, the SCAG module, functioning as a detail-preserving module, selectively optimizes shallow-level features by integrating deep semantic context, thereby safeguarding critical spatial information throughout the decoding phase. This mechanism not only provides robust support for precise segmentation of small-sized or irregular land-covers but also significantly enhances the model’s adaptability to incomplete annotations.

At its core, the attention-based fusion strategy transforms CMCAUNet from a label-distribution-focused pattern matcher into a semantic reasoning engine that comprehends the intrinsic spectral-structural correlations of land cover types. This model effectively identifies inherent properties of land covers, avoiding overfitting to noisy labels or incomplete annotations. Such inherent robustness not only holds academic value but also addresses practical demands in large-scale remote sensing applications—particularly crucial in scenarios where annotation errors and data omissions are commonplace.

5.3. Limitations and Future Work

While CMCAUNet demonstrates outstanding performance, its high computational complexity poses challenges for deployment and scalability in environments with limited computational resources. Future research should prioritize designing lightweight or adaptive network architectures that significantly reduce floating-point operations and parameter counts without compromising segmentation fidelity. Regarding data dependency, the methodological approach is currently constrained by the availability of high-resolution images. Specifically, the presence of environmental factors such as shadows cast by buildings and trees, as well as cloud cover, can significantly degrade feature extraction accuracy. Consequently, the current framework primarily focuses on images obtained with Unmanned Aerial Vehicles (UAVs), which offer high resolution but limited coverage compared to satellite platforms. Additionally, enhancing the framework’s versatility to support diverse data combinations (e.g., SAR–optical or LiDAR–high spectral) will drive the development of more universal multimodal terrain classification paradigms. Crucially, improving cross-domain generalization capabilities remains vital to ensure robust performance across new geographic regions and sensor configurations. Recent advancements in unsupervised domain adaptation and prototype learning [23,24,25] offer promising solutions, warranting further exploration to achieve “plug-and-play” applicability in broader scenarios.

6. Conclusions

To address core challenges in fine-grained land object classification using multimodal remote sensing data—including ambiguous object boundaries, small-scale target omissions, and spectral confusion-induced misclassification—this study proposes the CMCAUNet. By integrating the CMCA and SCAG modules, the framework establishes a deep bidirectional interaction mechanism for feature fusion. This architecture achieves effective integration of deep features between VIS and DSM data, enabling precise feature extraction. Consequently, the proposed method significantly enhances both classification accuracy and model generalization capabilities.

Comprehensive experiments on the ISPRS Vaihingen and Potsdam open datasets demonstrate the robustness and effectiveness of CMCAUNet across diverse urban scenes. Quantitative analysis reveals that, while maintaining high OA and mIoU, the proposed method excels specifically in segmenting challenging small targets; notably, the accuracy for the Car category is significantly improved compared to state-of-the-art methods. Qualitative evaluations further confirm that CMCAUNet generates classification maps with superior boundary adherence and structural coherence, effectively mitigating the impact of noisy or erroneous annotations. The consistent performance across both distinct datasets validates the stability of the framework. Furthermore, the model demonstrates exceptional semantic generalization capabilities, accurately inferring missing or erroneous targets in real-world annotations. This advantage stems from the synergistic design of the CMCA and SCAG modules: CMCA captures deep complementary dependencies, while SCAG ensures precise reconstruction of local spatial details during decoding.

Future research will focus on several promising directions. First, improving the model’s accuracy in distinguishing spectrally similar but structurally distinct land cover categories (such as low vegetation and impermeable surfaces) remains a core challenge. Second, expanding the generalizability of fusion frameworks by integrating new data modalities like LiDAR point clouds or synthetic aperture radar (SAR) imagery into existing systems will significantly enhance model robustness and adaptability. Finally, combining CMCAUNet’s fusion mechanism with large-scale vision models like Vision Transformers (ViT) [16,20,28] will leverage ViT’s powerful global modeling capabilities to open new pathways for identifying novel or rare surface categories in complex remote sensing scenarios.