MMA-Net: A Semantic Segmentation Network for High-Resolution Remote Sensing Images Based on Multimodal Fusion and Multi-Scale Multi-Attention Mechanisms

Highlights

What are the main findings?

• We propose MMA-Net, a novel dual-branch network that integrates CLMF and MSMA modules to achieve state-of-the-art performance, with mIoU scores of 88.74% and 84.92% on the Potsdam and Vaihingen datasets, respectively.
• The CLMF module uses a two-stage fusion strategy to effectively preserve spatial details and suppress DSM noise. The MSMA module innovatively incorporates multi-scale depthwise separable convolutions into the attention block, which significantly boosts the model’s ability to perceive and segment ground objects of vastly different sizes.

What is the implication of the main finding?

• MMA-Net provides a robust solution for accurate semantic segmentation of high-resolution remote sensing images, particularly improving boundary clarity and small-object recognition, which is critical for applications such as urban planning and disaster monitoring.
• The proposed modular design offers a generalizable framework for multimodal data fusion, demonstrating significant potential for extending to other remote sensing tasks involving multi-source data.

Abstract

Semantic segmentation of high-resolution remote sensing images is of great application value in fields like natural disaster monitoring. Current multimodal semantic segmentation methods have improved the model’s ability to recognize different ground objects and complex scenes by integrating multi-source remote sensing data. However, these methods still face challenges such as blurred boundary segmentation and insufficient perception of multi-scale ground objects when achieving high-precision classification. To address these issues, this paper proposes MMA-Net, a semantic segmentation network enhanced by two key modules: cross-layer multimodal fusion module and multi-scale multi-attention module. These modules effectively improve the model’s ability to capture detailed features and model multi-scale ground objects, thereby enhancing boundary segmentation accuracy, detail feature preservation, and consistency in multi-scale object segmentation. Specifically, the cross-layer multimodal fusion module adopts a staged fusion strategy to integrate detailed information and multimodal features, realizing detail preservation and modal synergy enhancement. The multi-scale multi-attention module combines cross-attention and self-attention to leverage long-range dependencies and inter-modal complementary relationships, strengthening the model’s feature representation for multi-scale ground objects. Experimental results show that MMA-Net outperforms state-of-the-art methods on the Potsdam and Vaihingen datasets. Its mIoU reaches 88.74% and 84.92% on the two datasets, respectively. Ablation experiments further verify that each proposed module contributes to the final performance.

1. Introduction

As a core task in the intelligent interpretation of Earth observation, the semantic segmentation of high-resolution remote sensing images (HRRSI) enables pixel-level classification of ground objects. This technique provides critical data support for a wide range of applications, including natural disaster monitoring [1,2], object extraction [3,4], land cover mapping [5,6,7], and urban planning [8,9,10]. With the rapid development of remote sensing sensor technology, HRRSI has been widely applied. While its abundant spatial details and complex spectral information enhance the potential for ground object identification, they also pose numerous challenges: HRRSI contains ground objects with rich details and complex spatial structures, along with significant spectral similarity and intra-class variability. These issues cause semantic segmentation methods to frequently suffer from problems such as ambiguous edge segmentation, insufficient ability to perceive and model multi-scale ground objects, and category confusion, which restrict the further improvement of segmentation accuracy.

Traditional semantic segmentation methods for remote sensing images mainly rely on single-modal data, such as optical images, infrared images, and multispectral images. However, these approaches [11,12] often exhibit limitations in complex scenarios, as single-modal data may lack sufficient information to accurately discriminate between different categories of ground objects. Consequently, multimodal remote sensing image semantic segmentation technology has attracted the attention of researchers.

Multimodal semantic segmentation techniques leverage the complementarity of multi-source data to enhance feature representation capabilities. Although existing multimodal approaches demonstrate improved model performance compared to unimodal methods, they face several challenges. Methods such as SwinFusion [13] and FTransUNet [14] experience a gradual loss of shallow spatial details during encoder downsampling, resulting in incomplete segmentation of small-scale objects and blurred boundary localization. Most multimodal fusion strategies, such as SA-Gate [15], rely on simple operations like concatenation or summation. These approaches fail to fully exploit cross-modal long-range dependencies and complementarity, resulting in superficial fusion effects that struggle to effectively resolve category confusion issues.

To address the challenges mentioned above, this paper proposes a high-resolution remote sensing image segmentation network (MMA-Net) based on multimodal fusion and a multi-scale multi-attention mechanism. Unlike existing methods such as FTransUNet and LMFNet, MMA-Net introduces a more holistic and synergistic fusion strategy that explicitly addresses both feature-level hierarchy and cross-modal dependencies through dedicated module designs. The main contributions of this study are as follows:

(1)

We propose MMA-Net, a novel dual-branch encoder–decoder architecture that systematically integrates the Cross-Layer Multimodal Fusion (CLMF) module and the Multi-Scale Multi-Attention (MSMA) module. This design enables complementary feature enhancement across both spatial and semantic dimensions, leading to significant improvements in semantic segmentation accuracy. In experiments on the ISPRS Vaihingen and Potsdam datasets, the model achieved mIoU scores of 84.92% and 88.74%, respectively, outperforming state-of-the-art methods by 0.76% to 1.43%.

(2)

This paper proposes a novel Cross-Layer Multimodal Fusion (CLMF) module, which adopts a two-stage fusion strategy of “cross-layer fusion followed by multi-modal fusion” to effectively address the loss of spatial detail (shallow features) during the encoder’s downsampling process. In the Cross-Layer Fusion (CLF) stage, adjacent hierarchical features are integrated via concatenation and channel attention weighting, enabling a dynamic combination of shallow details and high-level semantics. The Multi-Modal Fusion (MMF) stage incorporates a redundancy filtering mechanism to suppress noise and interference caused by directly integrating raw DSM features.

(3)

This paper proposes an MSMA, which integrates Self-Attention (SA) and Cross-Attention (CA) to enable simultaneous capture of intra-modal local context and inter-modal complementary features. Adopting a three-layer structural design, this block effectively models the long-range correlations between RGB and DSM modalities. Moreover, we innovatively incorporate multi-scale depthwise separable convolutions into the attention block, which enhances the network’s ability to model ground objects of varying scales without significantly increasing computational cost.

This paper is organized into six sections. Section 1 presents the research background and significance of multimodal remote sensing image semantic segmentation, analyzes the limitations of existing methods, and outlines the scope and contributions of this study. Section 2 reviews related work on both single-modal and semantic segmentation of multimodal remote sensing imagery. Section 3 elaborates on the overall architecture of MMA-Net and details the design of each module. Section 4 evaluates the efficacy of the proposed method through comprehensive experiments. Section 5 provides a discussion of the findings, and Section 6 concludes with a synthesis of the study.

2. Related Work

Semantic segmentation of remote sensing imagery focuses on enabling machines to automatically identify the semantic categories of surface features within images, constituting a fundamental task in Earth observation data interpretation. With the continuous advancement of deep learning technologies, a rich research system has been formed in this field, ranging from single-modal to multimodal approaches, and from Convolutional Neural Networks (CNNs) to Transformers. This section systematically reviews the related research progress and analyzes the limitations of existing methods from two dimensions: single-modal and multimodal semantic segmentation.

2.1. Single-Modal Remote Sensing Image Semantic Segmentation Methods

Early semantic segmentation approaches depended on handcrafted features—such as texture and shape descriptors—yet their generalizability remained constrained in the intricate scenes of HRRSI. The rise of deep learning provided a powerful tool for automatic feature extraction. Among these, the FCN proposed by Long et al. [16] was the first end-to-end semantic segmentation model. Through the substitution of fully connected layers with convolutional layers, this approach enables pixel-wise classification of images of arbitrary dimensions. However, the decoder of FCN adopts a relatively simple upsampling method, which leads to blurring at the edges of segmentation results and loss of detailed information, thus failing to meet the demands of HRRSI for identifying fine structures.

To solve the limitations of FCN, researchers have proposed a series of improved architectures. UNet [17] adopts a symmetric encoder–decoder structure, where shallow low-level features from the encoder are fused with high-level semantic features from the decoder via skip connections. This substantially enhances the segmentation precision for small objects and intricate boundaries, making it a classic framework for subsequent remote sensing semantic segmentation methods. DeepLabV3+ [18] introduces Atrous Convolution to expand the receptive field and combines Atrous Spatial Pyramid Pooling (ASPP) to capture multi-scale features, further enhancing adaptability to ground objects of different sizes. MFATrack [19] captures shallow context and deep details in RGB and thermal infrared images through a multi-scale full-feature fusion strategy. By integrating an auto-attention interaction module to enhance target features, it improves accuracy while maintaining real-time performance. Targeting the characteristics of variable scales and complex boundaries of ground objects in high-resolution remote sensing images, BSNet [20] proposes dynamic hybrid gradient convolution technology, which can dynamically adjust gradient fusion strategies according to local features to accurately capture subtle edges and contours of small-scale targets. HMANet [21] strengthens the modeling of global context through a regional attention mechanism, improving the overall segmentation consistency of large-scale ground objects (e.g., urban blocks). However, single-modal data struggles to address the challenges of “large intra-class variance and small inter-class variance” in high-resolution remote sensing images. In complex scenarios such as illumination variations, shadowed areas, and spectral confusion, their information representation capacity is limited, which can easily lead to misclassification and omission.

2.2. Multimodal Remote Sensing Image Semantic Segmentation Methods

Multimodal semantic segmentation leverages the complementary nature of different modalities, compensating for the limitations of single data sources in feature representation and thereby enhancing overall feature characterization capabilities. Among these, DSM data, which encompasses elevation information of ground objects, is used to assist in distinguishing between spectrally similar but height-different objects [11,22], making it one of the core auxiliary modalities in remote sensing multimodal segmentation. When classified according to the fusion stage, multimodal fusion approaches fall into three categories:

①

Data-level fusion: This method involves directly concatenating or enhancing raw data before feeding it into the network [23]. However, this approach easily introduces multi-modal redundant information and fails to account for differences in physical properties between modalities, resulting in inefficient feature learning.

②

Decision-level fusion: This approach first generates classification results from single-modal models separately, then fuses decisions via voting, weighting, or other methods [24]. This approach offers high flexibility but neglects early feature interactions between modalities, making it difficult to capture deep semantic correlations.

③

Feature-level fusion: Fuses features across diverse modalities within the network’s intermediate layers, which is currently the mainstream strategy. Early methods such as FuseNet [25] and vFuseNet [26] adopted simple concatenation or summation for feature fusion, failing to fully explore inter-modal dependencies. In recent years, attention mechanisms have found extensive application in feature-level fusion. For example, ASFFuse [27] dynamically adjusts modal weights through spatial and channel attention, while SA-Gate [15] and CFNet [28] use gating mechanisms to filter effective features, significantly improving fusion efficiency.

As the advantages of Transformer in global modeling become increasingly prominent, Transformer-based multimodal fusion methods have emerged as a new trend. TransFuser [29] introduces a Transformer layer after the convolutional block and designs a BiFusion module to fuse the fine-grained local details from CNNs with the global dependencies captured by Transformers. SwinFusion [13] adopts Swin Transformer as its backbone and realizes deep interaction of multimodal features through self-attention mechanisms. In the remote sensing field, HMANet [21] enhances the modeling of global context through a regional attention mechanism, improving the overall segmentation consistency of large-scale ground objects. FTransUNet [14] combines CNNs and Transformers, utilizing self-attention for capturing long-range dependencies and increasing the recognition accuracy of complex ground object boundaries. LMFNet [30] designs multimodal feature fusion reconstruction layers and self-attention fusion layers to enhance complementarity between different modalities, while improving the model’s ability to recognize small-scale features through a max feature merging strategy. EDFT [31] and MFTransNet [32] further strengthen inter-modal complementarity through depth-aware self-attention modules. FTransDeepLab [33] builds upon the DeepLabv3+ encoder architecture by stacking multi-scale Segformer modules to enhance long-range dependency modeling and multi-scale feature extraction. However, existing multimodal methods still suffer from two major limitations. Firstly, the gradual loss of shallow details during the downsampling process leads to poor segmentation accuracy, compromising the model’s adaptability to multi-scale targets and resulting in inadequate feature representation. Secondly, they overlook the oversimplification of existing multimodal fusion methods, which fail to establish a cross-modal global dependency model and thus cannot fully leverage the deep complementarity between RGB and DSM data.

3. Methods

This paper proposes MMA-Net, a semantic segmentation network based on a dual-branch encoder–decoder architecture. The proposed model facilitates effective interaction between high-level and low-level features, as well as across multimodal data, through the CLMF module. Meanwhile, the MSMA enhances the network’s ability to capture objects at different scales. This section first introduces the overall network architecture, followed by a detailed explanation of the design principles and functions of the CLMF and MSMA modules.

3.1. Overview

MMA-Net builds on a dual-branch encoder–decoder architecture, using a CNN backbone (ResNet-50) to process both RGB images and DSM data modalities. Through a phased, multi-level feature fusion and enhancement mechanism, it achieves efficient integration of multimodal and multi-scale information. During the encoding phase, the preprocessed RGB and DSM images with a resolution of 256 × 256 are fed into a weight-shared backbone network. The network extracts multi-scale features (${\mathrm{R}}_1$ − ${\mathrm{R}}_5$, ${\mathrm{D}}_1$ − ${\mathrm{D}}_5$) through five convolutional blocks, with the resolution decreasing from 128 × 128 to 8 × 8 and the number of channels increasing from 64 to 1024. The first four layers (${\mathrm{R}}_1$ − ${\mathrm{R}}_4$, ${\mathrm{D}}_1$ − ${\mathrm{D}}_4$) of features are integrated with adjacent hierarchical and multi-modal information via the CLMF. The fifth layer (${\mathrm{R}}_5$, ${\mathrm{D}}_5$) is flattened and input into the MSMA, where semantic features are enhanced through multi-attention mechanisms and multi-scale convolutions. The decoder progressively restores the spatial resolution to 256 × 256 using skip connections that fuse multi-level encoder features, followed by cascaded upsampling, ultimately generating pixel-wise segmentation results. The overall framework of MMA-Net is illustrated in Figure 1.

3.2. Cross-Layer Multimodal Fusion Module (CLMF)

Semantic segmentation of HRRSI necessitates not only the integration of high-level and low-level features in RGB images (encompassing shallow details and deep semantic information) but also the effective utilization of elevation data contained in DSM to differentiate ground objects with similar spectral characteristics. However, existing multimodal fusion methods (such as FTransUNet and LMFNet) generally adopt a “single unified spatial fusion” approach, which directly places features from different modalities into a unified feature space for processing. This approach overlooks the hierarchical differences between the shallow details of RAG and the elevation semantics of DSM, and also fails to address the issue of redundant interference in the raw features of DSM. During this process, the feature maps of the two modalities always maintain the same spatial size and channel dimension, providing a scale-aligned basis for subsequent multi-modal feature fusion. The structure of the proposed CLMF module is shown in Figure 2, and its specific procedure is described below.

(1)

Cross-Layer Feature Fusion (CLF)

The purpose of this sub-module is to achieve collaborative optimization of “details—semantics” by dynamically fusing adjacent levels of RGB features, thereby addressing the segmentation gap between high-level and low-level representations. Shallow-level features preserve rich detailed information yet are deficient in high-level semantic content, making them difficult to directly use for category judgment. In contrast, deep features, after multiple convolutions and downsampling, have been abstracted into semantically discriminative features but lose fine spatial position information, which hinders the localization of small targets. Through the cross-layer feature fusion mechanism, high-level semantic information and low-level detailed features are effectively integrated. This not only preserves the accuracy of spatial position information but also enhances the expressive ability of multi-scale contextual features, thus realizing the collaborative optimization of semantic information and structural details in target localization tasks. The particular architecture of this module is depicted in Figure 3.

Two features from different layers are extracted from the CNN encoder, denoted as ${\mathrm{R}}_{\mathrm{i}+1}$ and ${\mathrm{R}}_{\mathrm{i}}$ respectively. Owing to the discrepancy in the channel count between the shallow feature ${\mathrm{R}}_{\mathrm{i}}$ and the deep feature ${\mathrm{R}}_{\mathrm{i}+1}$, a 1 × 1 convolution operation is first applied to ${\mathrm{R}}_{\mathrm{i}}$ to adjust its channel number to match that of ${\mathrm{R}}_{\mathrm{i}+1}$. The deep feature ${\mathrm{R}}_{\mathrm{i}+1}$ is upsampled using bilinear interpolation to restore its spatial size to be consistent with ${\mathrm{R}}_{\mathrm{i}}$. Feature mapping is performed through a 3 × 3 convolution layer to obtain ${\mathrm{R}}_{\mathrm{i}+1}^{\mathrm{u}\mathrm{p}}$, which adapts to the features of the lower level. Then, the adjusted ${\mathrm{R}}_{\mathrm{i}}$ and ${\mathrm{R}}_{\mathrm{i}+1}^{\mathrm{u}\mathrm{p}}$ are concatenated along the channel dimension, resulting in a feature${\mathrm{R}}_{\mathrm{c}\mathrm{a}\mathrm{t}}$ that integrates shallow details and high-level semantics. This step directly associates adjacent hierarchical features, enabling high-level semantic information to “guide” the weight allocation of shallow details and enhancing the semantic relevance of detailed features. To further highlight the semantically relevant key channels in the concatenated features (e.g., edge channels of buildings, texture channels of vegetation), dynamic attention weights are generated using the deep feature ${\mathrm{R}}_{\mathrm{i}+1}$: First, global average pooling (GAP) is applied to compress the spatial dimension of ${\mathrm{R}}_{\mathrm{i}+1}$ to 1 × 1, obtaining a 1 × 1 × C channel description vector that reflects the global importance of each channel. Applying a 1 × 1 convolution (to compress the number of channels for reducing computational complexity) and a Softmax activation function to the vector output by GAP, a normalized channel weight of size C × 1 × 1 is generated. The optimized RGB feature ${\mathrm{R}}_{\mathrm{i}+1}^{\mathrm{h}}$ is obtained by multiplying ${\mathrm{R}}_{\mathrm{c}\mathrm{a}\mathrm{t}}$ with these weights channel-wise to enhance the responses of important features and suppress irrelevant information, and applying max pooling to highlight salient features. The specific process is shown in Equation (1):

$${\mathrm{R}}_{\mathrm{i}+1}^{\mathrm{h}}=\mathrm{M}\mathrm{P}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}({\mathrm{R}}_{\mathrm{i}},\mathrm{U}\mathrm{p}({\mathrm{R}}_{\mathrm{i}+1})))\times \mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}(\mathrm{G}\mathrm{A}\mathrm{P}({\mathrm{R}}_{\mathrm{i}+1}))))$$

(1)

Among them, the GAP operation realizes the aggregation of spatial information into channel information, $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}$ represents the pixel-level channel concatenation operation, and $\mathrm{U}\mathrm{p}$ denotes bilinear interpolation upsampling.

The CLF module enables RGB features to preserve the spatial precision of shallow features while enhancing scene comprehension through high-level semantic guidance. This process establishes a robust feature foundation for subsequent multi-modal fusion.

(2)

Multi-Modal Feature Fusion (MMF)

DSM data documents the elevation details of surface features and assumes an irreplaceable role in distinguishing ground objects with similar spectra but different heights. However, DSM data is often contains noise and redundant information due to limitations in sensor accuracy, terrain undulations, and environmental interference. Directly fusing such data with RGB features could introduce invalid information or mislead the model. To address this, the multi-modal feature fusion module achieves efficient complementarity between RGB and DSM features through a two-step operation of “redundancy filtering—multi-scale fusion”. The specific process is as follows:

To mitigate noise and redundancy in DSM data, we design a parallel convolutional branch for feature filtering. One branch applies a 3 × 3 convolution to the DSM feature ${\mathrm{D}}_{\mathrm{i}+1}$ to extract basic elevation features and retain valid structural information. The other branch first extracts details from ${\mathrm{D}}_{\mathrm{i}+1}$ via a 3 × 3 convolution, then compresses the number of channels through a 1 × 1 convolution, and finally generates a filtering mask between 0 and 1 using the sigmoid activation function. The outputs of the two branches are multiplied element-wise to retain elevation features in high-importance regions and suppress noise and redundancy in low-importance regions, resulting in the filtered DSM feature ${\mathrm{D}}_{\mathrm{i}+1}^{\mathrm{f}}$. The specific process is shown in Equation (2):

$${\mathrm{D}}_{\mathrm{i}+1}^{\mathrm{f}}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}({\mathrm{D}}_{\mathrm{i}+1})\times \mathsf{\delta }({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}({\mathrm{D}}_{\mathrm{i}+1})))$$

(2)

wherein: $\mathsf{\delta }$ denotes the sigmoid function, and ${\mathrm{D}}_{\mathrm{i}+1}^{\mathrm{f}}$ represents the filtered feature of the ($\mathrm{i}+1$)-th layer.

In high-resolution remote sensing images, ground objects exhibit substantial scale variations, which cannot be effectively captured using convolution operations with a fixed receptive field. To address this, multi-modal feature fusion adopts multi-scale dilated convolutions to expand the receptive field. Specifically, the filtered DSM feature ${\mathrm{D}}_{\mathrm{i}+1}^{\mathrm{f}}$ and the cross-layer fused RGB feature ${\mathrm{R}}_{\mathrm{i}+1}^{\mathrm{h}}$ are concatenated along the channel dimension to obtain the fused joint feature. Subsequently, dilated convolutions with dilation rates of 1, 2, 4, and 8 are applied in parallel to this joint feature to capture feature information of small-scale, medium-scale, large-scale, and extra-large-scale objects respectively. Finally, the multi-scale features are processed through a 1 × 1 convolution to integrate channel information, resulting in the final multi-modal fused feature ${\mathrm{E}}_{\mathrm{i}+1}$. The detailed procedure is illustrated in Equation (3):

$${\mathrm{E}}_{\mathrm{i}+1}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}({∆}_{\mathrm{r}}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}({\mathrm{D}}_{\mathrm{i}+1}^{\mathrm{f}},{\mathrm{R}}_{\mathrm{i}+1}^{\mathrm{h}})))$$

(3)

wherein: ${∆}_{\mathrm{r}}\left(\mathrm{x}\right)$ denotes the dilated convolution operation with a dilation rate of $\mathrm{r}$ applied to feature $\mathrm{x}$, and $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}$ represents the concatenation operation.

The high-level features of the two modalities extracted from the 5th block of the CNN backbone network undergo spatial alignment and channel unification. Subsequently, two embedding layers and a reshaping operation are used to flatten the features into a sequence form. Position encoding is added to the flattened patch tokens to preserve positional information, resulting in vectorized patches ${\mathrm{L}}_{\mathrm{R}}^0$ and ${\mathrm{L}}_{\mathrm{D}}^0$. These enriched feature sequences are fed into the MSMA to achieve deeper multimodal fusion and multiscale enhancement, strengthening the interactive representation capability between multimodal inputs and improving the model’s perception ability for targets of different scales.

3.3. Multi-Scale Multi-Attention (MSMA)

The MSMA module is designed to decouple modal fusion from multi-scale modeling, thereby enhancing feature representation for multimodal targets and improving perception in scale-sensitive regions such as roads and riverbanks. After processing by the CLMF, the features already have good modal complementarity and scale adaptability. However, the segmentation of ground objects with similar spectra and variable scales still requires refined modeling of deep features. Existing methods are limited to “single-modal self-attention” or “unidirectional cross-attention,” making it difficult to fully explore bidirectional complementary relationships across modalities. In multi-scale modeling, reliance on “fixed receptive field convolutions” hinders adaptation to scenes with varying scales. To address this, the proposed MSMA module overcomes existing limitations through a “multi-modal attention-multi-scale augmentation” cascade architecture. On one hand, it employs multi-branch depth-separable convolutions to enable dynamic receptive field adjustment. On the other hand, it achieves dynamic feature optimization through the synergistic interaction between attention mechanisms and multi-scale convolutions. Its structure is shown in Figure 4.

The MSMA module consists of three layers. Let $L_R^n$ and $L_D^n$ denote the features of the $\mathrm{n}$-th layer in the optical image branch and DSM branch, respectively, where $\mathrm{n}\in \left[\mathrm{1,3}\right]$. Notably, the dimensions of the feature maps remain consistent throughout all layers. The input features of the two modalities are first processed by Layer Norm to reduce distribution shifts and improve feature stability. The processed features then enter the multimodal attention module, which simultaneously performs self-attention and cross-attention to capture intra-modal dependencies and establish inter-modal associations respectively. The output of the attention module is connected with the original input via a residual connection and undergoes LayerNorm again, providing normalized input for the subsequent nonlinear module. Then, the features are input into the multi-scale module, which adopts a multi-branch dilated convolution structure to capture spatial structural features under different receptive fields. Finally, a standard feed-forward neural network is used to further perform nonlinear mapping on the fused features and enhance their expressive ability.

To effectively capture intramodal local dependencies and inter-modal complementary relationships, MSMA adopts a parallel design of “self-attention + cross-attention” to enhance interactions between modalities while preserving the information within each modality. Its specific structure is shown in Figure 5.

First, $L_R^n$ and $L_D^n$ denote the input features from the two modalities respectively. In each layer, the input features are mapped through linear projections (using $Z_R^{qkv}$, $Z_D^{qkv}$ respectively) to obtain the matrices of $\mathrm{Q}$, $\mathrm{K}$, and $\mathrm{V}$. The specific operations are as follows (Equations (4) and (5)):

$$[{\mathrm{Q}}_{\mathrm{R}},{\mathrm{K}}_{\mathrm{R}},{\mathrm{V}}_{\mathrm{R}}]=L_R^n\times Z_R^{qkv}$$

(4)

$$[{\mathrm{Q}}_{\mathrm{D}},{\mathrm{K}}_{\mathrm{D}},{\mathrm{V}}_{\mathrm{D}}]=L_D^n\times Z_D^{qkv}$$

(5)

Next, multi-head attention is used to compute SA and CA. Each modality calculates its internal relationships through self-attention respectively, and the process can be expressed as Equations (6) and (7):

$${\mathrm{s}\mathrm{a}}_{\mathrm{R}}=\mathrm{\varnothing }({\displaystyle \frac{{\mathrm{Q}}_{\mathrm{R}}\times {\mathrm{K}}_{\mathrm{R}}^{⊺}}{\sqrt{\mathrm{d}}}}){\mathrm{V}}_{\mathrm{R}}$$

(6)

$${\mathrm{s}\mathrm{a}}_{\mathrm{D}}=\mathrm{\varnothing }({\displaystyle \frac{{\mathrm{Q}}_{\mathrm{D}}\times {\mathrm{K}}_{\mathrm{D}}^{⊺}}{\sqrt{\mathrm{d}}}}){\mathrm{V}}_{\mathrm{D}}$$

(7)

where: $\mathrm{\varnothing }$ denotes the softmax function, $\mathrm{d}$ is a normalization constant, and $\sqrt{\mathrm{d}}$ is the scaling factor for the feature dimension.

Self-attention calculates the internal dependency relationships within each modality.

In cross-attention (CA), cross-information between the two modalities is captured by exchanging queries and keys between modalities, enhancing mutual correlations between modalities. This process may be formulated as Equations (8) and (9):

$${\mathrm{c}\mathrm{a}}_{\mathrm{R}}=\mathrm{\varnothing }({\displaystyle \frac{{\mathrm{Q}}_{\mathrm{R}}\times {\mathrm{K}}_{\mathrm{D}}^{⊺}}{\sqrt{\mathrm{d}}}}){\mathrm{V}}_{\mathrm{D}}$$

(8)

$${\mathrm{c}\mathrm{a}}_{\mathrm{D}}=\mathrm{\varnothing }({\displaystyle \frac{{\mathrm{Q}}_{\mathrm{D}}\times {\mathrm{K}}_{\mathrm{R}}^{⊺}}{\sqrt{\mathrm{d}}}}){\mathrm{V}}_{\mathrm{R}}$$

(9)

Subsequently, to balance the contributions of SA and CA, an adaptive mechanism is employed to fuse SA and CA. The specific process can be expressed as Equations (10) and (11):

$${\mathrm{g}}_{\mathrm{n}}^{\mathrm{R}}={\mathsf{\lambda }}_{\mathrm{s}\mathrm{a}}^{\mathrm{R}}{\mathrm{s}\mathrm{a}}_{\mathrm{R}}+{\mathsf{\lambda }}_{\mathrm{c}\mathrm{a}}^{\mathrm{R}}{\mathrm{c}\mathrm{a}}_{\mathrm{R}}$$

(10)

$${\mathrm{g}}_{\mathrm{n}}^{\mathrm{D}}={\mathsf{\lambda }}_{\mathrm{s}\mathrm{a}}^{\mathrm{D}}{\mathrm{s}\mathrm{a}}_{\mathrm{D}}+{\mathsf{\lambda }}_{\mathrm{c}\mathrm{a}}^{\mathrm{D}}{\mathrm{c}\mathrm{a}}_{\mathrm{D}}$$

(11)

where: ${\mathsf{\lambda }}_{\mathrm{s}\mathrm{a}}^{\mathrm{R}}$, ${\mathsf{\lambda }}_{\mathrm{c}\mathrm{a}}^{\mathrm{R}}$, ${\mathsf{\lambda }}_{\mathrm{s}\mathrm{a}}^{\mathrm{D}}$, ${\mathsf{\lambda }}_{\mathrm{c}\mathrm{a}}^{\mathrm{D}}$ are learnable weighting coefficients.

Using multi-head attention, the input is projected into multiple distinct subspaces, allowing each attention head to learn different feature relationships in parallel. The outputs from each attention head are concatenated and then fused via a linear transformation, yielding the final multimodal feature representation. The outputs of all self-attention and cross-attention are fused via element-wise addition and connected with the input features through a residual connection. The entire process can be expressed as Equations (12) and (13):

$$Z_R^n=\mathrm{M}\mathrm{A}(\mathrm{L}\mathrm{N}\left(L_R^{n-1}\right),\mathrm{L}\mathrm{N}(L_D^{n-1}\left)\right)+L_R^{n-1}$$

(12)

$$Z_{D }^n=\mathrm{M}\mathrm{A}(\mathrm{L}\mathrm{N}\left(L_R^{n-1}\right),\mathrm{L}\mathrm{N}(L_D^{n-1}\left)\right)+L_D^{n-1}$$

(13)

To better capture information about ground objects with significant scale variations in remote sensing data, the multi-scale module further enhances the model’s ability to model local and global information in complex scenes by fusing features of different scales. It enables more accurate segmentation and recognition, especially for regions with blurred object boundaries or large morphological variations. Its structure is shown in Figure 6. Before being input to the multi-scale module, the features first pass through a LayerNorm layer. Then, the input features undergo a 1 × 1 convolution to adjust the number of channels, reducing the channel dimension C to 1/4 of its original size. Subsequently, three parallel depth-wise separable convolutions are used to capture spatial information at different scales, where $\mathrm{d}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v},{\mathrm{T}}_1$, $\mathrm{d}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v},{\mathrm{T}}_2$, and $\mathrm{d}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v},{\mathrm{T}}_3$ represent three depth-wise separable convolution operations with different receptive fields. The dilation rates ${\mathrm{T}}_{\mathrm{i}}(\mathrm{i}=\mathrm{1,2},3)$ correspond to {(1,3,5)}, {(3,5,7)}, and {(5,7,9)} for each Multiscale module in the three multi-scale MSMA blocks, respectively. Then, a 1 × 1 convolution is applied to integrate information across channels. The features from the three branches are fused via element-wise addition. Finally, adaptive pooling is used to adjust the spatial dimensions, and a 1 × 1 convolution restores the channel dimension to the original feature size. The specific process can be expressed as Equations (14)–(17):

$$\widetilde{X_R^n}=\mathrm{L}\mathrm{N}(Z_R^n)$$

(14)

$$X_R^i={\mathrm{d}\mathrm{w}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}^{{\mathrm{T}}_{\mathrm{i}}}({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}(\widetilde{X_R^n}))$$

(15)

$$\widetilde{X_R^i}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}(X_R^i)$$

(16)

$$X_R^{out}=\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}(\widetilde{X_R^1}+\widetilde{X_R^2}+\widetilde{X_R^3})\times \widetilde{X_R^n})$$

(17)

3.4. Loss Function

This paper employs the cross-entropy loss function as the optimization objective for model training. This loss function effectively measures the distributional discrepancy between predicted probabilities and true labels, precisely penalizing the model’s prediction bias across various categories. Loss function is expressed as (18):

$$\mathrm{L}=-{\sum }_{\mathrm{i}}^6{\mathrm{Y}}_{\mathrm{i}}\times \mathrm{l}\mathrm{o}\mathrm{g}({\mathrm{P}}_{\mathrm{i}})$$

(18)

where i represents the index of the six target categories; ${\mathrm{Y}}_{\mathrm{i}}$ is the binary true label indicator variable, such that ${\mathrm{Y}}_{\mathrm{i}}$ = 1 if the true category of the pixel is the i class, and ${\mathrm{Y}}_{\mathrm{i}}$ = 0 otherwise. ${\mathrm{P}}_{\mathrm{i}}$ is the probability that the predicted map belongs to the labeled class i.

4. Results

4.1. Dataset

We utilize two public remote sensing datasets from ISPRS: Potsdam and Vaihingen. Descriptions of these two datasets are provided below.

The Potsdam dataset comprises 38 high-resolution images, each with a size of 6000 × 6000 and a ground sampling distance (GSD) of 5 cm. Included in this dataset are channels for near-infrared, red, green, and blue, along with both DSMs and nDSMs. We divide the dataset into 24 images for training and 16 for testing. The test set consists of images with the following IDs: 2_14, 2_15, 3_14, 3_15, 4_14, 4_15, 4_16, 5_12, 5_13, 5_14, 6_12, 6_13, 6_14, and 7_13. In our experiments, we use only the red, green, and blue channels together with their corresponding nDSM images.

The Vaihingen dataset comprises 33 orthorectified image files, containing three spectral bands (red, green, near-infrared) and nDSM data with identical resolution. It exhibits a spatial resolution of 9 cm, an average pixel dimension of 2494 × 2494, and a GSD of 5 cm. We allocate 16 images for training and 17 for testing. The test images are identified by the following IDs: 3, 6, 9, 12, 15, 18, 21, 22, 24, 26, 29, 30, 32, 33, 35, 37, and 38. Corresponding nDSM data are also used during both training and testing phases.

Figure 7 presents some sample data from the two datasets, depicting six categories: impervious surfaces (white), low-lying vegetation (cyan), buildings (blue), cars (yellow), trees (green), and clutter background (red).

To better understand the dataset characteristics and guide the model design and evaluation, we analyze the class distribution in both datasets, as shown in Figure 8 below.

4.2. Evaluation Metrics

To comprehensively quantify the performance of the model in the task of semantic segmentation for multimodal HRRSI, this paper adopts three commonly used evaluation metrics: OA, mIoU, and mF1. The specific definitions of each metric are as follows:

OA measures the global classification correctness over all pixels. The specific calculation formula is as follows in Equation (19):

$$\mathrm{O}\mathrm{A}={\displaystyle \frac{{\sum }_{\mathrm{c}=1}^{\mathrm{C}}{\mathrm{T}\mathrm{P}}_{\mathrm{c}}}{{\sum }_{\mathrm{c}=1}^{\mathrm{C}}({\mathrm{T}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{P}}_{\mathrm{c}}+{\mathrm{T}\mathrm{N}}_{\mathrm{c}}+{\mathrm{F}\mathrm{N}}_{\mathrm{c}})}}$$

(19)

where C denotes the total number of classes. ${\mathrm{T}\mathrm{P}}_{\mathrm{c}}$, ${\mathrm{F}\mathrm{P}}_{\mathrm{c}}$, ${\mathrm{T}\mathrm{N}}_{\mathrm{c}}$, and ${\mathrm{F}\mathrm{N}}_{\mathrm{c}}$ represent the number of true positives, false positives, true negatives, and false negatives for class c, respectively. This formula is also used to evaluate the OA values of each category.

mIoU evaluates segmentation accuracy. The formula is presented as follows in Equations (20) and (21):

$${\mathrm{I}\mathrm{o}\mathrm{U}}_{\mathrm{c}}={\displaystyle \frac{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}}{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{N}}_{\mathrm{c}}}}$$

(20)

$$\mathrm{m}\mathrm{I}\mathrm{o}\mathrm{U}={\displaystyle \frac{1}{\mathrm{C}}}{\sum }_{\mathrm{c}=1}^{\mathrm{C}}{\mathrm{I}\mathrm{o}\mathrm{U}}_{\mathrm{c}}$$

(21)

The mean F1 score represents the harmonic average of precision and recall across each category. The formula is presented as follows in Equations (22)–(25):

$${\mathrm{F}1}_{\mathrm{c}}=2\times {\displaystyle \frac{{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{c}}\times {\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}_{\mathrm{c}}}{{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{c}}+{\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}_{\mathrm{c}}}}$$

(22)

$$\mathrm{m}\mathrm{F}1={\displaystyle \frac{1}{\mathrm{C}}}{\sum }_{\mathrm{c}=1}^{\mathrm{C}}{\mathrm{F}1}_{\mathrm{c}}$$

(23)

$${\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{c}}={\displaystyle \frac{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}}{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{P}}_{\mathrm{c}}}}$$

(24)

$${\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}_{\mathrm{c}}={\displaystyle \frac{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}}{{\mathrm{T}\mathrm{P}}_{\mathrm{c}}+{\mathrm{F}\mathrm{N}}_{\mathrm{c}}}}$$

(25)

where ${\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{c}}$ represents the proportion of pixels that are actually of category c among those predicted as category c, and ${\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}_{\mathrm{c}}$ denotes the proportion of pixels actually belonging to class c that are correctly predicted.

4.3. Experimental Details

We employ ResNet-50, pre-trained on the ImageNet dataset [34], as the backbone network. For comparative methods, single-modal experiments are trained using only RGB data, while multi-modal methods utilize both RGB and corresponding DSM data. To ensure fairness, all methods are conducted under identical experimental conditions, with training and testing performed on an NVIDIA GeForce RTX 3090 24G GPU. During network training, depth values in DSM images are normalized to the range [0, 255]. All models are optimized using the Stochastic Gradient Descent (SGD) algorithm with a learning rate of 0.0007 and a momentum of 0.9, trained for 150 epochs and the batch size is set to 10 during training. Data augmentation is achieved by performing random horizontal and vertical flipping operations.

4.4. Performance Comparison

In this paper, the performance of the proposed MMA-Net is benchmarked against 8 state-of-the-art methods: HRCNet [35], MANet [36], SSNet [37], CMANet [38], CIMFNet [12], VSGNet [39], FTransUNet [14], and LMFNet [30]. Among these, HRCNet, MANet, and SSNet are single-modal methods trained exclusively on RGB data, serving to illustrate the contribution of multimodal DSM information. The remaining methods all use both RGB and nDSM images as input. Quantitative comparison results are presented in

Table 1. Shows the experimental results on the Vaihingen dataset.

Method	Muti-Modal	mIoU (%)	mF1 (%)	OA (%)
				Imp.Surf	Building	Low Veg.	Tree	Car	Total
HRCNet (2020)	×	67.69	79.75	79.16	85.90	64.36	74.94	60.93	73.06
MANet (2022)	×	79.56	85.94	91.57	93.77	76.29	89.91	86.03	87.51
SSNet (2024)	×	76.68	85.91	80.10	87.53	69.17	71.76	76.34	76.98
CMANet (2022)	√	77.94	85.94	92.02	95.29	77.35	90.48	73.56	85.94
CIMFNet (2022)	√	81.44	89.31	92.19	95.33	77.96	90.32	88.76	88.91
VSGNet (2025)	√	84.16	91.58	93.61	98.16	83.02	88.78	91.07	91.09
FTransUNet (2024)	√	83.69	90.84	92.48	97.73	80.68	90.49	91.27	90.59
LMFNet (2025)	√	84.09	91.27	92.61	96.75	85.61	86.98	90.34	90.46
Ours (ResNet18)	√	84.25	91.75	92.34	97.59	84.86	91.30	91.63	91.34
Ours (ResNet50)	√	84.92	92.53	93.44	98.92	85.32	91.79	92.73	92.77
Ours (ResNet101)	√	84.42	91.92	93.19	98.01	84.80	90.93	92.70	91.62
Ours (VGG-16)	√	84.74	92.09	92.78	98.32	84.95	91.68	91.59	91.86

√ indicates the use of RGB and nDSM data, while × indicates the use of only RGB data. Bold and light gray background indicate the best and second-best performance, respectively.

and

Table 2. Presents the experimental results on the Potsdam dataset.

Method	Muti-Modal	mIoU (%)	mF1 (%)	OA (%)
				Imp.Surf	Building	Low Veg.	Tree	Car	Total
HRCNet (2020)	×	85.05	89.62	91.27	92.17	86.34	86.94	91.40	89.62
MANet (2022)	×	72.63	81.11	90.48	87.58	88.78	82.69	90.96	88.09
SSNet (2024)	×	77.67	85.92	80.35	88.09	69.58	72.81	77.54	77.67
CMANet (2022)	√	74.99	84.20	91.75	93.64	86.42	84.70	92.74	89.85
CIMFNet (2022)	√	75.67	84.80	91.72	93.48	87.56	85.02	85.42	88.64
VSGNet (2025)	√	87.31	92.29	85.40	98.14	91.63	90.47	93.42	91.81
FTransUNet (2024)	√	86.14	91.83	93.67	97.78	87.92	87.84	94.31	92.30
LMFNet (2025)	√	86.09	91.36	89.38	95.31	82.11	79.07	86.08	86.39
Ours (ResNet18)	√	87.97	92.73	91.75	97.44	92.23	91.74	94.74	93.57
Ours (ResNet50)	√	88.74	93.67	93.25	98.95	93.22	91.92	95.68	94.65
Ours (ResNet101)	√	87.68	92.65	93.23	98.56	91.68	90.19	94.40	92.89
Ours (VGG-16)	√	88.63	93.29	92.84	98.63	92.51	90.85	95.27	94.27

√ indicates the use of RGB and nDSM data, while × indicates the use of only RGB data. Bold and light gray background indicate the best and second-best performance, respectively.

.

4.4.1. Metrics Comparison

(1)

Vaihingen Dataset

As shown in Table 1, the proposed method achieves significant improvements in OA, mF1, and mIoU, reaching 92.77%, 92.53%, and 84.92% respectively. These results represent gains of 1.68%, 0.95%, and 0.76% over the best-performing baseline methods. This result confirms that the proposed MMA-Net can effectively extract information from nDSM images and successfully fuse it with features from RGB images, owing to the designed CLMF. Specifically, compared to single-modal semantic segmentation methods, multi-modal ones perform better in segmenting trees and low-lying vegetation, which indicates that introducing nDSM data during semantic segmentation is beneficial. When compared with other multi-modal methods, the proposed multi-modal approach achieves 1.30% and 0.76% improvements in the segmentation accuracy of trees and buildings, respectively. This enhanced performance stems from MMA-Net’s ability to effectively leverage the distinct elevation characteristics of these categories through DSM data. Many classification methods fail to adequately consider local and overall information; in contrast, our method achieves a 1.46% improvement in car category segmentation. This is attributed to the designed cross-layer fusion and multi-scale modules, which enhance the model’s ability to model local and global information in complex scenes. These experimental results indicate that the proposed MMA-Net method possesses superior generalization capability.

(2)

Potsdam Dataset

As indicated in Table 2, the proposed method exhibits a consistent trend on the Potsdam dataset, achieving mIoU, mF1, and OA values of 88.74%, 93.67%, and 94.65%, respectively. These results represent improvements of 1.43%, 1.38%, and 2.35% over existing methods, demonstrating enhanced segmentation accuracy across all categories. Notably, the model shows particular effectiveness in classifying small-scale objects such as cars. These performance gains can be attributed to the synergistic interaction between the CLMF module and the MSMA module within MMA-Net.

4.4.2. Visualization Results Comparison

Figure 9 below shows the visualization results of experiments on the two datasets. The proposed method achieves more accurate classification results for small-scale targets such as buildings and cars, achieved primarily through the coordinated action of the CLMF and MSMA modules. Specifically, the CLMF enhances both local and global information representation by first fusing adjacent-layer RGB features and then integrating them with DSM data. Meanwhile, the MSMA module enables the model to adaptively focus on salient information across feature spaces at different scales, thereby improving comprehension and processing of complex data. Purple boxes highlight key differences between our method and comparative approaches. In the first row, our method accurately identifies low vegetation, consistent with quantitative results. The second row reveals a clear contrast: while existing methods often produce scattered artifacts at object boundaries, our approach achieves precise building edge segmentation. The third and fourth rows demonstrate our method’s capability to correctly identify relevant categories in complex environments. The fifth row shows that our method provides more accurate car detection, benefiting from the multi-scale attention module’s ability to capture small-target features. Finally, the sixth row illustrates complete building recognition even in challenging scenarios.

4.4.3. Few-Shot Class Segmentation Performance Validation

The class imbalance problem in HRRSI poses a significant challenge for semantic segmentation. As shown in Figure 8, both datasets exhibit strongly skewed category distributions. In the Vaihingen dataset, the “car” class constitutes only 1.21% of pixels—approximately 23 times fewer than the “impervious surface” class (27.94%). Similarly, in the Potsdam dataset, “car” accounts for merely 1.74%, about 16 times lower than “impervious surface” (29.02%). Such extreme imbalance causes categories with limited samples, such as cars and low vegetation, to be easily overwhelmed by dominant classes during training, thereby limiting segmentation accuracy. To address this issue, MMA-Net demonstrates strong performance in segmenting under-represented categories. As reported in Table 2 and

Table 3. Ablation experiments of the proposed method, where CLMF (no CLF) denotes the fusion module without cross-layer interaction. √ indicates that the corresponding module is included, while × indicates that it is not.

Baseline	CLMF (No CLF)	CLMF	MSMA (No Multiscale)	MSMA (No Multimodal Attention)	MSMA	Potsdam			Vaihingen
						mF1 (%)	mIoU (%)	OA (%)	mF1 (%)	mIoU (%)	OA (%)
√	×	×	×	×	×	84.67	78.18	87.58	83.53	77.08	86.84
√	√	×	×	×	×	88.77	82.67	89.94	88.32	81.60	89.26
√	×	√	×	×	×	89.75	84.30	90.86	89.51	82.06	90.53
√	×	×	√	×	×	91.95	86.25	92.52	90.05	83.52	91.39
√	×	×	×	√	×	90.42	84.83	91.28	89.23	82.41	90.17
√	×	×	×	×	√	92.38	87.16	93.24	91.28	84.35	91.89
√	×	√	×	×	√	93.67	88.74	94.65	92.53	84.92	92.77

, on the Vaihingen dataset, our method improves the IoU of the “car” class by 1.46% over FTransUNet, reaching 92.73%. On the Potsdam dataset, the “car” class IoU exceeds that of FTransUNet by 1.37%, and “low vegetation” IoU increases by 1.59%. These results confirm the model’s robustness under imbalanced data conditions.

These improvements stem from several deliberate design choices: The CLMF module incorporates elevation cues from DSM, providing discriminative information for rare classes such as height differences between cars and roads, or between low vegetation and trees, thus reducing confusion caused by spectral similarity. In the MSMA module, multi-scale depthwise separable convolutions enhance local feature extraction for small objects. When coupled with attention mechanisms, the model dynamically emphasizes sparse pixels from minor classes, effectively alleviating the dominance of majority-class features.

4.5. Ablation Experiments

To validate the effectiveness of the proposed CLMF and MSMA modules and their contributions to overall performance, this section conducts ablation studies from two perspectives. First, by incrementally integrating the CLMF and MSMA modules, we systematically analyze the individual and combined impacts of each component on segmentation performance through both quantitative and qualitative assessments. Second, we conduct comparative experiments with different MSMA depths (1 to 4 layers) to investigate how the depth of attention modeling affects multi-scale feature fusion capability and computational efficiency. All experiments are constructed using ResNet50 as the backbone network for the baseline, with all other hyperparameters held consistent.

(1)

Impact of Individual Modules

To validate the efficacy and significance of the proposed modules, ablation experiments were performed by eliminating specific components under the premise that all other conditions remain unaltered. As depicted in Table 3, the baseline network uses ResNet50, with the CLMF module replaced by convolution and simple addition operations; other settings remain consistent with the complete MMA-Net. It can be clearly seen from Table 3 that removing either CLMF or MSMA will lead to a decline in experimental performance, thereby affirming the efficacy of each individual module.

The following ablation experiments are all based on the Potsdam dataset. First, we introduce the CLMF module without CLF, which improves mIoU to 82.67%, demonstrating the effectiveness of incorporating DSM elevation information. However, the complete CLMF module further boosts mIoU to 84.30%. This improvement primarily stems from the CLF mechanism’s successful dynamic integration of shallow spatial details with deep semantic information, effectively mitigating detail loss during the encoder’s downsampling process. The MSMA module, incorporating a multimodal attention submodule, achieves a significant mIoU boost. This clearly demonstrates its crucial role in establishing global dependencies between RGB and DSM features at high-level semantic scales, effectively resolving category confusion issues. The multi-scale convolution submodule expands the receptive field through multi-branch depth-separated convolutions, significantly enhancing the model’s feature extraction capability for objects with pronounced scale variations. This particularly improves recognition of small-to-medium-scale targets. The complete MSMA module elevates mIoU to 87.1%, demonstrating synergistic effects between multimodal attention and multi-scale convolutions.

As depicted in Figure 10, the visual results of the ablation studies clearly demonstrate that removing any proposed module leads to noticeable performance degradation. Specifically, in the first row, compared with the complete CLMF module, the method lacking the CLF has deviations in classifying cars and cannot accurately identify cars in hidden positions. This is because the absence of cross-layer fusion leads to the neglect of local features and the loss of detailed information. In the second row, if the MSMA multi-scale attention module is missing, the segmentation effect of buildings will be poor, and the segmentation boundaries will be relatively blurred. Due to the lack of complementarity between different modalities, the segmentation accuracy of some low vegetation and trees decreases. The results in the third row are similar; the absence of the MSMA module will lead to incorrect segmentation of buildings, which also verifies the data of the above ablation experiments.

(2)

Number of Multi-scale Multi-attention Layers

The depth of the MSMA module represents a crucial design choice, directly affecting both the model’s ability to capture multi-scale multimodal dependencies and its computational requirements. Too few layers hinder effective modeling of long-range dependencies across scales and modalities, limiting segmentation accuracy. Too many layers, on the other hand, introduce redundant computations, increase the model’s parameter size and inference time, and reduce its practical application value.

Table 4. Ablation experiments on the number of MSMA layers on the Potsdam dataset.

MSMA Layers	mIoU (%)	mF1 (%)	OA (%)	Params (M)	FLOPs (G)
1	85.21	90.17	91.35	42.63	38.72
2	87.35	92.04	93.12	48.95	45.26
3	88.74	93.67	94.65	53.28	51.83
4	88.62	93.51	94.52	61.57	63.41

presents the comparison results for different MSMA layer settings (1, 2, 3, and 4 layers). In the experiments, other modules (such as the CLMF module and decoder structure) and hyperparameters (12 attention heads, 512 channels) remain unchanged, with only the number of MSMA layers being adjusted.

The experimental results indicate a clear correlation between MSMA depth and model performance. As the number of layers increases from 1 to 3, all evaluation metrics (mIoU, mF1, OA) on the Potsdam dataset show consistent improvement, with mIoU rising from 85.21% to 88.74%. This suggests that augmenting the layer count enhances the model’s capability to capture multi-scale features and multi-modal dependencies. Nevertheless, when the layer number is increased to 4, performance improvements tend to plateau or even decline marginally, whereas complexity metrics rise substantially: the parameter count increases from 53.28 M to 61.57 M, and FLOPs grow from 51.83 G to 63.41 G. Therefore, a 3-layer configuration achieves the optimal balance between accuracy and efficiency, maximizing the multi-scale attention mechanism’s ability to explore intra-modal and inter-modal correlations while avoiding the significant computational overhead of excessive layer stacking.

4.6. Complexity Analysis

We evaluate the computational complexity of the proposed MMA-Net through the following metrics: Floating-Point Operations (FLOPs), model parameter count, memory consumption, and frames per second (FPS).

Table 5. Comparative analysis of model complexity based on the Potsdam dataset. √ indicates a multimodal method, × indicates a unimodal method.

Method	Multimodal	FLOPs (G)	Params (M)	Memory (MB)	Speed (FPS)	mIoU (%)
HRCNet	×	49.03	46.72	3124	66.01	85.05
MANet	×	42.58	38.96	2890	58.37	72.63
SSNet	×	39.21	35.42	2650	52.14	77.67
CMANet	√	68.35	59.87	3420	22.63	74.99
CIMFNet	√	72.16	63.54	3610	20.89	75.67
VSGNet	√	58.72	72.31	3850	18.42	87.31
FTransUNet	√	65.38	160.88	4210	10.25	86.14
LMFNet	√	55.62	68.93	3520	16.73	86.09
Ours (ResNet50)	√	56.82	68.45	3280	19.63	88.74

provides the complexity analysis results for all comparative methods included in this study. As shown in Table 5, MMA-Net (as a multimodal model)naturally exhibits higher FLOPs and parameter counts compared to single-modal methods like HRCNet, MANet, and SSNet, due to its dual-branch architecture processing both RGB and DSM data along with dedicated fusion modules. However, this increase is offset by a significant improvement in segmentation performance: on the Potsdam dataset, the mIoU rises from the previous highest of 87.31% to 88.74%.

Notably, compared to FTransUNet, our ResNet50-based model reduces the parameter size by 57.5% (68.45 M vs. 160.88 M) and nearly doubles the inference speed (19.63 FPS vs. 10.25 FPS), while achieving a 2.6% higher mIoU. This can be ascribed to the MSMA, which adopts depth-wise separable convolutions and a parallel branch architecture to mitigate complexity while boosting the capability of feature modeling. In comparison to LMFNet, which has a similar complexity level, Ours (ResNet50) achieves a 2.65% higher mIoU with 6.8% less memory usage, validating the efficiency of the collaborative design between CLMF and MSMA. Overall, MMA-Net (ResNet50) achieves an optimal balance between performance and complexity on the Potsdam dataset. With moderate computational costs (56.82 G FLOPs, 68.45 M parameters), it attains the highest mIoU (88.74%), exhibiting particularly strong performance on small objects (e.g., cars) and ambiguous boundary regions (e.g., between low vegetation and trees). This underscores the model’s practical value in semantic segmentation tasks for HRRSI.

The computational characteristics of MMA-Net on the Potsdam dataset are based on results from NVIDIA RTX 3090 servers. For deployment on resource-constrained edge devices (e.g., drones) or real-time systems, the model can be further optimized using lightweight backbone networks (e.g., MobileNet) or inference acceleration frameworks like TensorRT. Another practical consideration involves reliance on DSM data. When high-quality DSMs are unavailable, approximations can be generated from stereo imagery, or robustness can be enhanced through modal loss strategies during training.

5. Discussion

The excellent performance of MMA-Net on the ISPRS Potsdam and Vaihingen datasets stems primarily from the collaborative design of the CLMF and the MSMA module. In the CLF stage, this module dynamically associates shallow spatial details with deep semantic information by concatenating adjacent hierarchical RGB features and weighting them via channel attention, effectively avoiding the loss of shallow features during encoder downsampling. In the multimodal fusion stage, the parallel convolutional filtering branch efficiently suppresses noise and redundancy in DSM data. The MSMA module enhances the model’s ability to model multi-scale ground objects and capture multi-modal dependencies. SA and CA enable the model to not only explore intra-modal local context but also capture inter-modal complementary relationships. Additionally, the multi-scale depthwise separable convolutions employed in the MSMA module expand the receptive field to cover ground objects of different sizes, effectively resolving the limitation of “fixed receptive field” in traditional convolutions.

Despite its competitive performance, MMA-Net still has limitations. It struggles to accurately segment extremely small targets (e.g., individual street lamps) and extremely large targets (e.g., continuous urban blocks). This is because the receptive field of the multi-scale convolutions in the MSMA module relies on a fixed dilation rate and cannot be dynamically adjusted according to the scale characteristics of the input content. Moreover, in areas with complex terrain or severe noise, the model fails to eliminate high-frequency interference, which leads to distortion of elevation features and subsequently degrades the multimodal fusion effect.

6. Conclusions

This study centers on the semantic segmentation task for HRRSI.To address the issues of blurred boundary segmentation and insufficient multi-scale feature perception in existing multimodal semantic segmentation methods, we propose the MMA-Net architecture. This network aims to improve semantic segmentation accuracy by effectively fusing RGB and DSM multi-modal data. The introduced CLMF strategy reduces intra-class variations to a minimum while maximizing inter-class discrepancies, facilitating better segmentation of ground objects with similar spectra while effectively preventing the loss of local details. The introduced multi-scale multi-attention module, which incorporates both SA and CA mechanisms, not only enhances interactions between modalities while preserving intra-modal information but also combines multi-scale depth-wise separable convolutions to effectively model features of ground objects at different scales, improving adaptability to complex scenes. Experimental validation on the ISPRS Potsdam and Vaihingen datasets demonstrates that MMA-Net outperforms mainstream methods on key metrics including OA, mIoU, and mF1, fully validating the rationality and effectiveness of the proposed modules and overall network design.