MHBA-TransUNet: Shallow–Deep Collaborative RGB–DSM Fusion with Hybrid Bidirectional Attention for High-Resolution Remote Sensing Semantic Segmentation

Highlights

What are the main findings?

• MHBA-TransUNet coordinates shallow RGB–DSM fusion and deep global–local feature interaction through SCFFM and GLAM, achieving mIoU scores of 85.15%, 88.01%, and 85.49% on Vaihingen, Potsdam, and US3D, respectively.
• A lightweight SA–CA–SA dual-stream Transformer enables intra-modal context modeling and bidirectional cross-modal interaction with substantially reduced Transformer parameter overhead.

What are the implications of the main findings?

• Coordinating shallow multimodal fusion with deep semantic–spatial interaction provides an effective strategy for RGB–DSM semantic segmentation in complex urban scenes.
• The compact multimodal Transformer provides a practical approach to balancing cross-modal representation and computational efficiency in high-resolution remote sensing applications.

Abstract

High-resolution remote sensing semantic segmentation using RGB imagery and digital surface models (DSM) remains challenging because of insufficient shallow cross-modal fusion and inadequate coordination between global semantics and local spatial information. To address these issues, we propose MHBA-TransUNet, a hybrid encoder–decoder network for RGB–DSM semantic segmentation. In the encoder, a Shallow Cross-modal Feature Fusion Module (SCFFM) progressively integrates complementary RGB and DSM features through dynamic spatial recalibration, while a lightweight dual-stream Transformer with a self-attention–cross-attention–self-attention (SA–CA–SA) configuration models intra-modal context and bidirectional cross-modal interactions. During decoding, a Global–Local Alignment Module (GLAM) combines Block-Distance Linear Attention (BDLA), feature-wise modulation, and channel reweighting to coordinate deep semantic features with shallow spatial details. Experiments on the ISPRS Vaihingen, ISPRS Potsdam, and US3D datasets achieve mIoU scores of 85.15%, 88.01%, and 85.49%, respectively, outperforming representative state-of-the-art methods. Ablation studies further verify the effectiveness and complementarity of SCFFM and GLAM, while the US3D results demonstrate stable performance across diverse urban scenes.

1. Introduction

In recent years, the rapid development of Earth observation technologies has greatly improved both the acquisition capability and the usability of high-resolution remote sensing data. Remote sensing data are inherently multimodal in nature, including, for example, visible-light optical imagery, synthetic aperture radar (SAR) imagery, and digital surface models (DSM) generated from light detection and ranging (LiDAR) measurements [1]. Data from different modalities can characterize the spectral–textural properties, three-dimensional structures, and scattering behaviors of ground objects from distinct physical mechanisms and observational perspectives, thereby providing abundant complementary information for comprehensive and accurate representation of surface features. Consequently, multimodal data fusion, by integrating such complementary information, has become a key technical support for many core tasks in geographic research, with applications covering land-use and land-cover change detection [1], high-precision land-cover mapping [2], extraction of specific targets such as buildings and roads [3], image fusion [4,5,6], change detection [7], super-resolution [8], and object detection [9,10], as well as ecological monitoring and disaster assessment [11,12].

As a fundamental pixel-level classification task, semantic segmentation aims to assign each pixel in remote sensing imagery to a predefined land-cover category and serves as the basis for the aforementioned applications. Early studies mainly relied on traditional machine learning methods, such as random forests [13], support vector machines [14], and conditional random fields [15]. However, these methods are limited in their ability to automatically extract deep and abstract discriminative semantic features, making it difficult to meet the demands of fine-grained recognition for complex objects in high-resolution remote sensing imagery. In recent years, breakthroughs in deep learning have brought revolutionary advances to remote sensing semantic segmentation. Representative methods based on convolutional neural networks (CNNs) and their extensions have significantly improved segmentation accuracy and efficiency. Early CNN-based fusion approaches [16] established end-to-end learning frameworks for multimodal data, while subsequent encoder–decoder architectures [17,18] and hybrid attention-based designs [19,20,21] further advanced automatic multi-level feature representation from large-scale remote sensing data. Nevertheless, owing to the inherently local receptive fields of CNNs [22,23], natural limitations remain in modeling long-range dependencies among pixels and capturing global contextual information, and these limitations become particularly evident when complex remote sensing scenes with strong spectral heterogeneity and intricate structures are processed.

This limitation has driven the successful adoption of Transformer architectures in computer vision. In 2020,the Vision Transformer (ViT) [24] was the first to demonstrate the great potential of a pure Transformer architecture for image recognition tasks. Compared with CNNs, Transformers, by virtue of their global attention mechanism, are able to effectively model long-range dependencies among serialized image patches and exhibit superior capability in global context modeling. In particular, early Transformer-based segmentation backbones such as TransUNet [25] and Segmenter [26] demonstrated the potential of self-attention for dense prediction, while later efficient architectures including SegFormer [27] and UNetFormer [28] further adapted the paradigm to natural and remote sensing imagery. This advantage was soon introduced into the field of semantic segmentation, leading to efficient segmentation models such as SegFormer [27] and Segmenter [26], which achieved performance surpassing that of CNN-based methods on natural images.

While remarkable success has been achieved in single-modality semantic segmentation, the limitations of single-sensor data have also become increasingly evident. Constrained by imaging mechanisms and sensor characteristics, unimodal data often suffer from inherent information gaps. For example, optical imagery is easily affected by cloud and haze occlusion and is often inadequate for distinguishing highly similar spectral features; SAR imagery is subject to geometric distortions and speckle noise; and DSM lacks spectral–textural information [2,29]. By contrast, multimodal data can compensate for these deficiencies and provide more comprehensive information support for segmentation tasks.

Therefore, how to effectively integrate the complementary information contained in multimodal remote sensing data has become a central research direction for improving semantic segmentation performance. According to the stage at which fusion is performed, existing methods can generally be categorized into early fusion, intermediate fusion, and late fusion. Early fusion usually concatenates or stacks multimodal data directly at the input level. Although this strategy is straightforward, it is often inadequate for accommodating the significant differences in statistical characteristics and noise levels across modalities [30,31]. In late fusion, each modality is processed independently and the results are combined at the decision or output level. This scheme is easy to implement, but the exploitation of deep complementary correlations across modalities remains insufficient [31]. By contrast, intermediate fusion performs interaction and integration in the feature space, thereby enabling cross-modal dependencies to be captured more flexibly; for this reason, it is widely regarded as one of the most effective fusion strategies for semantic segmentation [32,33].

However, most existing multimodal fusion models built upon Transformer backbones, such as TransFuser [34] and SwinFusion [35], perform modal interaction mainly through a unidirectional cross-attention mechanism. Under such a design, the contextual features of the target modality are likely to be suppressed or overwhelmed when information from the source modality is absorbed, while the one-way information flow further restricts the efficient transmission and adaptive fusion of bidirectional complementary information across modalities [36]. In addition, when the global semantic features captured by Transformers are fused with the local detail features extracted by CNN encoders, existing methods often rely on simple addition or concatenation, without carefully handling or adaptively aligning their inherent differences in receptive field and abstraction level [36,37]. Such coarse fusion strategies are prone to causing semantic misalignment in the feature space, so that global contextual information and local fine-grained features interfere with each other, which tends to degrade the representation of fine-grained structures and the recognition of small targets.

To address the above issues, TransUNet [25] is adopted in this study as the baseline architecture, upon which a multilayer fusion network, termed Multimodal Hybrid Bidirectional Attention TransUNet (MHBA-TransUNet), is developed for RGB–DSM remote sensing image semantic segmentation. It should be noted that, throughout this paper, the term “RGB–DSM” is used in a general sense to denote three-channel optical imagery paired with elevation data (DSM/NDSM); the specific channel configurations of each dataset are described in Section 3.1. First, to mitigate the local response misalignment that arises in the shallow feature space of multimodal data, a Shallow Cross-modal Feature Fusion Module (SCFFM) is designed. Through a dynamic spatial recalibration mechanism, the shallow fine-grained features extracted from RGB imagery and DSM by the dual-branch CNN encoder are progressively interacted and fused, thereby effectively alleviating the shallow fusion difficulties caused by differences in imaging mechanisms, occlusions, and local inconsistencies, and providing high-quality feature representations for subsequent processing. Second, to overcome the insufficient collaboration between deep global semantics and shallow local details during decoding, a Global–Local Alignment Module (GLAM) is proposed. In this module, a Block-Distance Linear Attention (BDLA) mechanism and a feature-wise linear modulation strategy are introduced. While computational complexity is kept under control, long-range dependencies and local correspondences between deep semantic features and shallow spatial details are established, enabling adaptive alignment and fusion of cross-level features and improving the representation of fine-grained structures and the recognition of small targets in complex scenes. The main contributions of this study are summarized as follows:

1.

A Shallow Cross-modal Feature Fusion Module (SCFFM) is proposed, in which the progressive fusion of shallow RGB and DSM features is achieved through spatial interaction enhancement and attention-based recalibration. With this design, shallow mismatches between multimodal data can be effectively alleviated, and the representation of details in shadowed and low-texture regions can be enhanced.

2.

A Global–Local Alignment Module (GLAM) is proposed, within which a Block-Distance Linear Attention (BDLA) mechanism is introduced. Through the joint use of block-wise linear mixing, local positional enhancement, and feature modulation, efficient alignment and fusion of deep global semantics and shallow local details are achieved, thereby enhancing the fine-grained detail representation and small-target recognition performance of the model.

3.

Comprehensive comparative experiments and ablation studies are conducted on three remote sensing datasets, namely ISPRS Vaihingen, ISPRS Potsdam, and US3D (grss_dfc_2019) [38,39]. The results show that the proposed method outperforms a variety of representative state-of-the-art methods in terms of overall accuracy, class-wise F1 scores, and mIoU, thereby validating the effectiveness of the proposed framework.

The remainder of this paper is organized as follows. Section 2 reviews related work on unimodal and multimodal semantic segmentation. Section 3 introduces the datasets and presents the architecture of the proposed MHBA-TransUNet in detail. Section 4 describes the experimental setup, including implementation details, evaluation metrics, and compared methods. Section 5 reports the experimental results, including comparative experiments, generalization experiments, complexity analysis, ablation studies, and feature visualization. Section 6 discusses the findings, mechanism interpretation, and limitations of the proposed method. Finally, Section 7 concludes the paper.

2. Related Work

2.1. Unimodal Semantic Segmentation

In the early exploration of semantic segmentation, the Fully Convolutional Network (FCN) [40], as the first CNN-based end-to-end model, laid an important foundation for this task. However, the simplified upsampling strategy adopted in its decoder resulted in blurred boundaries and insufficient segmentation accuracy. To overcome this limitation, a key improvement was achieved by UNet [41] through its innovative encoder–decoder architecture: multiscale features are captured by the encoder through progressive downsampling, while contextual information is fused by the decoder via skip connections, thereby enabling effective recovery of spatial details. Although CNN-based methods have demonstrated robust performance in segmentation tasks, their inherent local receptive field property [22,23] restricts the ability to model global semantics and makes the handling of long-range dependencies particularly challenging.

This limitation has promoted the cross-domain adoption of Transformer architectures. In 2020, ViT [24] introduced Transformers into vision tasks for the first time, breaking the locality constraint of CNNs through sequence-based modeling. Compared with conventional CNNs, ViT has shown significant advantages in global context perception [25,26,27,28]. It has not only advanced the development of cross-modal tasks such as vision–language understanding, but has also become an important paradigm for subsequent research. Overall, unimodal methods have established mature encoder–decoder paradigms, yet the information bottleneck of single-sensor data fundamentally constrains their upper performance in complex scenes. Detail–semantic decoupling has also been explored to improve feature quality in related tasks [9].

2.2. CNN-Based Multimodal Semantic Segmentation

With the continuous advancement of Earth observation technologies, multimodal remote sensing data, such as optical imagery, multispectral data, and digital surface models (DSM), have become increasingly accessible. Prior-guided fusion strategies have proven effective for heterogeneous modality integration [4,5]. Among them, DSM provides precise height information of ground objects, which not only offers critical cues for identifying structures with height consistency, such as roads and buildings, but also significantly improves semantic segmentation accuracy owing to its naturally clear structural representation [42,43]. This advantage has stimulated extensive research on deep learning-driven multimodal fusion. Early approaches, such as ResUNet-a [44], attempted to directly stack RGB images and DSM into a four-channel input. However, such a strategy is inadequate for handling the inherent statistical heterogeneity and noise discrepancies across modalities [30,31]. To address this issue, FuseNet [16] innovatively adopted a dual-branch architecture, in which RGB and DSM data are processed in parallel during encoding, while feature fusion is performed through element-wise addition. Building upon this idea, vFuseNet [45] further introduced a multi-scale fusion mechanism, thereby significantly enhancing feature representation capability. Nevertheless, fusion schemes based on simple addition still fail to fully exploit the complementarity of multimodal features.

Subsequent studies have achieved further progress through more refined fusion mechanisms. ABCNet [33] improves segmentation efficiency by combining dual-branch context modeling with an attention mechanism, thereby balancing spatial detail preservation and semantic information aggregation. DDCM-Net [46] employs dense dilated convolutions to fuse features with different receptive fields, so that contextual representation of multiscale objects can be enhanced. MANet [47], from the perspective of multiple attention mechanisms, jointly models dependency relationships in both spatial and channel dimensions, thus improving the capture of discriminative features in complex scenes. MACU-Net [48], built upon the U-Net architecture, introduces multiscale skip connections and asymmetric convolutions to strengthen cross-level feature interaction and local structure representation. Although these methods have achieved favorable results in local detail recovery, multiscale information integration, and feature enhancement, they still rely primarily on convolutional operations at their core. As a result, their capability for modeling long-range spatial dependencies remains relatively limited, which to some extent constrains semantic segmentation performance in complex remote sensing scenes.

2.3. CNN- and Transformer-Based Multimodal Semantic Segmentation

Owing to its outstanding capability in context modeling and multimodal fusion, the Transformer architecture has gradually become a core paradigm for multimodal fusion in semantic segmentation. Under this trend, TransFuser [34] innovatively embeds multiple Transformer layers into the convolutional feature fusion stage, thereby enhancing representation capacity through dynamic modeling of cross-feature-map correlations. SwinFusion [35] goes a step further by using convolutional layers as shallow feature extractors while employing Swin Transformer [49] to build the backbone for deep feature generation, demonstrating that Transformers can replace conventional CNNs as the principal backbone in fusion tasks.

Studies in remote sensing semantic segmentation have shown that combining the local modeling capability of CNNs with the global context perception ability of Transformers has become an important direction for improving model performance. CMFGNet [43] enhances semantic associations across different modalities through cross-modal multiscale interaction and a Transformer-based fusion mechanism. MSFNet [37] progressively integrates multilevel multimodal features based on a multistage Transformer architecture, thereby improving segmentation performance on high-resolution remote sensing data. MFTransNet [36] achieves complementary representation of multimodal features by combining CNNs and Transformers. TMFNet [50] adopts a multimodal branch encoding scheme together with a Transformer-based fusion strategy, thereby strengthening the interaction among information from different sources. FTransUNet [51] further unifies the local detail modeling capability of CNNs and the global context perception ability of ViT within a multilevel multimodal fusion framework.MCFTNet [52] further corroborates the effectiveness of cross-layer multimodal Transformer fusion. Nevertheless, most existing methods have not fully addressed the problems of insufficient spatial alignment in multimodal feature fusion and inadequate coordination between global and local semantics. As a result, issues such as omission of fine-grained targets and insufficient overall semantic consistency still tend to occur in complex scenes.

Recent advances in related fields have also highlighted the importance of frequency-aware representation learning and local–global feature collaboration. Specifically, FSSFFormer [53] and FreDNet [54] leverage wavelet-based frequency decomposition to enhance boundary preservation in remote sensing image segmentation, and TSF-DETR [10] further applies frequency-domain enhancement to small-object detection, while ELGC-Transcoding [55] and DST [56] employ dual-scale local–global attention to improve feature compactness in image and video coding. Although these studies confirm the effectiveness of frequency-domain priors and global–local interaction in their respective tasks, they are not designed for multimodal RGB–DSM segmentation, where shallow cross-modal misalignment and deep cross-level semantic–spatial discrepancy remain unresolved. The proposed SCFFM and GLAM are therefore developed to bridge these specific gaps.

The proposed method differs from previous studies in the following aspects.

1.

In terms of shallow fusion strategy, previous CNN-based multimodal methods have largely been limited to feature concatenation, element-wise addition, or fixed channel-attention mechanisms. Such static fusion strategies are unable to handle the spatial misalignment between RGB textures and DSM geometric structures. In contrast, a Shallow Cross-modal Feature Fusion Module (SCFFM) is designed in this study. Instead of relying on static fusion, a dynamic spatial recalibration mechanism with resolution-reduced cross-branch interaction is adopted. Through lightweight learnable modulation between complementary contextual statistics, local response deviations across modalities are adaptively recalibrated, thereby enabling progressive feature fusion.

2.

In terms of deep decoding and feature fusion, existing CNN–Transformer hybrid architectures usually rely on fixed skip connections or simple feature concatenation during decoding to combine global semantics with local details, while lacking effective treatment of cross-level feature discrepancies. In contrast, a Global–Local Alignment Module (GLAM) is proposed in this study, in which Block-Distance Linear Attention (BDLA) and a feature-wise linear modulation mechanism are introduced. By establishing explicit correspondences between deep semantic features and shallow detail features, the direct stacking or simple concatenation adopted in existing methods is replaced with an adaptive alignment mechanism.

3.

In terms of the overall fusion framework, conventional methods tend to focus on feature fusion at a single stage or on simple stacking of network architectures, making it difficult to simultaneously account for shallow-detail complementarity and deep semantic coordination. In contrast, the multilayer multimodal fusion framework constructed in this study is not a mere combination of modules. Instead, targeted interaction and alignment mechanisms are separately designed for the early encoding stage and the decoding stage, thereby forming a cross-level collaborative fusion pathway.

3. Materials and Methods

3.1. Datasets

To evaluate the proposed method, experiments were conducted on three representative semantic segmentation datasets, namely the ISPRS Vaihingen dataset (https://www.isprs.org/resources/datasets/benchmarks/UrbanSemLab/semantic-labeling.aspx, accessed on 8 August 2026), the ISPRS Potsdam dataset (https://www.isprs.org/resources/datasets/benchmarks/UrbanSemLab/semantic-labeling.aspx, accessed on 8 August 2026), and the US3D dataset (https://ieee-dataport.org/open-access/urban-semantic-3d-dataset, accessed on 8 August 2026). The detailed descriptions of these datasets are provided in the following subsections.

3.1.1. Vaihingen Dataset

The Vaihingen dataset consists of 16 ultra-high-resolution true orthophotos with an average size of $2500\times 2000$ pixels. Each orthophoto contains three channels, namely near-infrared, red, and green (NIRRG), and is accompanied by a normalized digital surface model (NDSM) with a ground sampling distance (GSD) of 9 cm. In this study, the NIRRG three-channel imagery is directly used as the optical input of this dataset, and the NDSM serves as the elevation input. The dataset includes five foreground classes, namely building (Bui.), tree (Tre.), low vegetation (Low.), car, and impervious surface (Imp.), as well as one background class, i.e., clutter. The 16 orthophotos are divided into a training set containing 12 images and a test set containing 4 images. Specifically, the training set includes tiles with indices 1, 3, 23, 26, 7, 11, 13, 28, 17, 32, 34, and 37, whereas the test set consists of tiles 5, 21, 15, and 30.

3.1.2. Potsdam Dataset

The Potsdam dataset contains 24 ultra-high-resolution true orthophotos, each with a tile size of $6000\times 6000$ pixels. This dataset provides four multispectral channels, namely near-infrared, red, green, and blue (IRRGB), together with a normalized digital surface model (NDSM) with a ground sampling distance (GSD) of 5 cm. The foreground classes are the same as those in the Vaihingen dataset, although the class distribution differs because of the different geographic location. In this study, the RGB composite imagery is used as the optical input of this dataset, and the NDSM serves as the elevation input. The 24 orthophotos are divided into 18 training images and 6 test images. Specifically, the training set includes tiles 6_10, 7_10, 2_12, 3_11, 2_10, 7_8, 5_10, 3_12, 5_12, 7_11, 7_9, 6_9, 7_7, 4_12, 6_8, 6_12, 6_7, and 4_11, whereas the test set consists of tiles 2_11, 3_10, 4_10, 5_11, 6_11, and 7_12.

3.1.3. US3D Dataset

The US3D dataset contains 108 original tiles in total. To avoid patch-level data leakage, dataset partitioning was performed at the original-tile level rather than by randomly splitting the cropped patches. Specifically, each $512\times 512$ patch was first associated with its parent tile according to the tile identifier in its filename, and all patches derived from the same original tile were treated as an indivisible group. The 108 tile groups were then randomly shuffled using a fixed seed of 42 and assigned to the training, validation, and test sets with 80, 16, and 12 tiles, respectively. Consequently, patches originating from the same tile never appear in different subsets.

The validation set contains 16 tiles: JAX_022, JAX_079, JAX_167, JAX_175, JAX_179, OMA_026, OMA_042, OMA_134, OMA_172, OMA_181, OMA_212, OMA_225, OMA_292, OMA_342, OMA_353, and OMA_383. The test set contains 12 tiles: JAX_020, JAX_113, JAX_117, JAX_144, JAX_178, JAX_204, JAX_236, OMA_248, OMA_278, OMA_332, OMA_364, and OMA_382. The remaining 80 tiles are used for training. The three subsets all contain samples from both Jacksonville (JAX) and Omaha (OMA), enabling evaluation under different urban-scene distributions. For network input, the samples associated with each tile are represented as non-overlapping $512\times 512$ image patches, while the parent-tile assignment is preserved throughout dataset partitioning.

3.2. Overall Network Architecture

To fully exploit the textural and spectral information contained in RGB imagery as well as the elevation and structural information provided by DSM, a hybrid encoder–decoder framework for RGB–DSM semantic segmentation is developed in this study, as illustrated in Figure 1. The framework mainly consists of four components: (1) a shallow dual-branch convolutional encoder; (2) a Shallow Cross-modal Feature Fusion Module (SCFFM);(3) a lightweight dual-stream Transformer semantic encoder with a self-attention–cross-attention–self-attention (SA–CA–SA) configuration; and (4) a progressive decoder built upon the Global–Local Alignment Module (GLAM). Specifically, the input RGB image and the three-channel DSM representation are first fed into two parallel convolutional branches, where progressive cross-modal interaction and fusion are performed through SCFFM at the shallow stages. The deep features from the two branches are then mapped into token sequences and passed to a lightweight dual-stream Transformer encoder, where intra-modal semantic dependencies and bidirectional cross-modal interactions are modeled through successive self-attention and cross-attention operations.Finally, the fused deep semantic features are delivered to the decoder and are progressively aligned and fused, via GLAM, with multilevel features from the shallow encoder, thereby producing the final pixel-wise classification result.

Figure 1. Overall architecture of the proposed MHBA-TransUNet. The encoder follows a U-shaped dual-branch CNN with progressive SCFFM fusion (left), feeding into a lightweight SA–CA–SA dual-stream Transformer bottleneck (center; SA = self-attention, CA = cross-attention). The decoder progressively aligns and fuses multilevel features through cascaded GLAM blocks (right), recovering the full-resolution prediction. The detailed structure of the SA–CA–SA Transformer is provided in Section 3.4.

Let the input RGB image and DSM representation be denoted by $I_r\in {\mathbb{R}}^{3\times H\times W}$ and $I_d\in {\mathbb{R}}^{3\times H\times W}$, respectively. In the current implementation, the DSM branch adopts a three-channel elevation representation to encode height, slope, and aspect cues while maintaining an input dimensionality compatible with the pretrained convolutional stem. The detailed preprocessing protocol is provided in Section 4.1. The shallow dual-branch convolutional encoder outputs four levels of cross-modal features, $\{F_3,F_2,F_1,F_0\}$, where $F_3$ to $F_0$ denote the four levels of skip-connection features from deep to shallow. Meanwhile, the deepest features from the two branches are mapped into token sequences and fed into the Transformer encoder, yielding the RGB semantic feature $T_r$ and the DSM semantic feature $T_d$. These two features are subsequently fused through weighted aggregation to form a bottleneck semantic representation, which is then sent to the decoder for progressive recovery.

Different from conventional architectures that rely only on one-shot early fusion or fixed skip connections, the proposed method explicitly alleviates local response discrepancies across modalities at the shallow stages and explicitly models the cross-level alignment relationship between deep global semantics and shallow local details during decoding. Therefore, the objective of the overall framework is not a simple stacking of CNNs and Transformers, but the resolution of the key bottlenecks in RGB–DSM joint segmentation from two complementary perspectives, namely shallow-detail fusion and deep–shallow semantic coordination.

Backbone selection. The dual-branch encoder adopts ResNet-50 as the shallow feature extractor, following the established practice in RGB–DSM semantic segmentation [51,57]. ResNet-50 strikes a favorable balance between representational capacity and computational cost: compared with shallower variants (ResNet-18/34), its deeper architecture captures richer hierarchical features essential for discriminating complex urban objects; compared with heavier counterparts (ResNet-101/152), it incurs substantially lower memory and computational overhead, which is critical given the additional parameter budget introduced by the dual-stream Transformer and cross-modal fusion modules. Moreover, adopting ResNet-50 ensures strictly fair comparison with representative CNN–Transformer hybrid methods such as FTransUNet [51] and CSFAFormer [57], all of which employ ResNet-50 as the convolutional backbone under an end-to-end training paradigm. It is worth noting that recent foundation-model-based approaches (e.g., MFNet [58]) adopt a fundamentally different technical route based on frozen SAM encoders with parameter-efficient fine-tuning; these are therefore not directly comparable to our training-from-scratch framework. While lighter backbones may be preferred for resource-constrained deployment, they often compromise low-level detail preservation, which is vital for the subsequent SCFFM shallow fusion. We therefore select ResNet-50 as a robust, widely validated intermediate solution, and leave the exploration of more lightweight backbones to future work.

3.3. Shallow Cross-Modal Feature Fusion Module (SCFFM)

3.3.1. Shallow Fusion Motivation

In RGB–DSM multimodal segmentation, the local responses of different modalities exhibit significant discrepancies at the shallow stages. RGB is more sensitive to texture, color, and illumination variation, whereas DSM focuses more on geometric height and structural contours. If the two modalities are directly fused through addition, concatenation, or channel compression, spatial misalignment and local inconsistency across modalities are likely to be overlooked, leading to insufficient fusion of shallow fine-grained information. To address this issue, SCFFM is introduced at each shallow stage of the dual-branch convolutional encoder, so that the locally salient responses of the two modalities can be dynamically spatially recalibrated and cross-modal information can be progressively injected into the RGB main branch.

3.3.2. Dual-Branch Shallow Encoding

A dual-branch ResNet50 is adopted as the shallow convolutional encoder, in which RGB and DSM are equipped with independent root convolution layers and subsequent residual blocks. The RGB branch and the DSM branch are used to extract local texture features and geometric structural features from their respective modalities, and feature interaction is performed through SCFFM after each convolutional stage. It is worth noting that, in the current implementation, the RGB branch takes the fused feature as the input to the next stage, whereas the DSM branch preserves its own enhanced output and propagates it independently. In this way, the complementary geometric information from DSM can be continuously injected into the RGB branch, while the structural discriminability of DSM is prevented from being prematurely degraded at the shallow stages.

Let the outputs of the RGB and DSM branches at the l-th shallow stage be denoted by ${\mathbf{R}}_l$ and ${\mathbf{D}}_l$, respectively. The objective of SCFFM is to generate enhanced features ${\tilde{\mathbf{R}}}_l$ and ${\tilde{\mathbf{D}}}_l$ according to their local responses, and to produce the fused feature ${\mathbf{F}}_l$. Formally, this process is expressed as

$${\tilde{\mathbf{R}}}_l=\mathrm{DSRM}\left({\mathbf{R}}_l\right),{\tilde{\mathbf{D}}}_l=\mathrm{DSRM}\left({\mathbf{D}}_l\right),$$

(1)

$${\mathbf{F}}_l={\tilde{\mathbf{R}}}_l+{\tilde{\mathbf{D}}}_l,$$

(2)

where $\mathrm{DSRM}(·)$ denotes the Dynamic Spatial Re-calibration Module, and the enhanced feature $\tilde{\mathbf{X}}$ is obtained via Equation (8). In the current implementation, the fused feature ${\mathbf{F}}_l$ is fed into the subsequent layers of the RGB branch, while ${\tilde{\mathbf{D}}}_l$ continues to serve as the input to the DSM branch, thereby forming a progressive interaction pattern of “fusion in the main branch and preservation in the auxiliary branch”.

3.3.3. Dynamic Spatial Recalibration

As illustrated in Figure 2, the core of SCFFM is the Dynamic Spatial Re-calibration Module (DSRM). In shallow multimodal features, the high-frequency textures of RGB and the geometric contours of DSM are often subject to slight spatial misalignment. If they are fused directly, conflicting noise can easily be introduced. To address this issue, DSRM is designed to perform resolution-reduced cross-branch interaction to suppress background interference with minimal computational overhead.

Figure 2. Structure of the Dynamic Spatial Re-calibration Module (DSRM).

Given an input feature $\mathbf{X}\in {\mathbb{R}}^{C\times H\times W}$, we first compress it along the channel dimension by average pooling and max pooling to obtain complementary contextual statistics:

$${\mathbf{X}}_{\mathrm{avg}}={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}{\mathbf{X}}_c,{\mathbf{X}}_{\max }=\underset{c}{max}{\mathbf{X}}_c,$$

(3)

where ${\mathbf{X}}_{\mathrm{avg}}$ captures global background distribution and ${\mathbf{X}}_{\max }$ preserves salient local activations. Both are single-channel maps of size $H\times W$.

Downsampling for efficient interaction. To reduce the computational cost of processing high-resolution shallow features and to suppress local high-frequency noise commonly found in remote sensing imagery, we explicitly downsample the two response maps via strided convolutions:

$${\mathbf{X}}_{\mathrm{avg}}^{\downarrow }={\mathrm{Conv}}_{3\times 3,s=2}\left({\mathbf{X}}_{\mathrm{avg}}\right),{\mathbf{X}}_{\max }^{\downarrow }={\mathrm{Conv}}_{3\times 3,s=2}\left({\mathbf{X}}_{\max }\right).$$

(4)

This produces half-resolution maps ($H/2\times W/2$), enlarging the effective receptive field and lowering the sampling difficulty for subsequent spatial interaction.

Cross-branch adaptive interaction. Unlike conventional spatial attention that processes average and maximum responses independently, DSRM introduces lightweight cross-branch modulation between the two statistics. Specifically, two learnable $3\times 3$ convolutions (initialized to zero for stable training) generate residual modulation gates:

$${\mathbf{G}}_{\mathrm{avg}}=\sigma \left({\mathrm{Conv}}_{3\times 3}\left({\mathbf{X}}_{\mathrm{avg}}^{\downarrow }\right)\right),{\mathbf{G}}_{\max }=\sigma \left({\mathrm{Conv}}_{3\times 3}\left({\mathbf{X}}_{\max }^{\downarrow }\right)\right),$$

(5)

where $\sigma (·)$ denotes the Sigmoid function. These gates are then used to modulate the counterpart branch:

$${\mathbf{M}}_{\mathrm{avg}}={\mathbf{G}}_{\mathrm{avg}}\odot {\mathbf{X}}_{\max }^{\downarrow },{\mathbf{M}}_{\max }={\mathbf{G}}_{\max }\odot {\mathbf{X}}_{\mathrm{avg}}^{\downarrow }.$$

(6)

This cross-multiplication allows the average context to be sharpened by maximum saliency, and vice versa, producing more discriminative interaction features than simple concatenation.

Mask generation and upsampling. The modulated features are concatenated and convolved to produce a single-channel mask, which is then bilinearly upsampled back to the original resolution:

$$\mathbf{M}\left(\mathbf{X}\right)={\mathrm{BilinearUp}}_{\times 2}\left({\mathrm{Conv}}_{3\times 3}\left([{\mathbf{M}}_{\mathrm{avg}};{\mathbf{M}}_{\max }]\right)\right).$$

(7)

The final recalibrated feature is obtained by gating the original input:

$$\tilde{\mathbf{X}}=\mathrm{ReLU}\left(\mathbf{X}\odot \sigma \left(\mathbf{M}\right(\mathbf{X}\left)\right)\right).$$

(8)

Gradient scaling for training stability. Because shallow cross-modal interaction can suffer from optimization instability, we apply a customized gradient scaling strategy to the two interaction convolutions during backpropagation. Let $\mathcal{L}$ be the overall loss and ${\mathbf{W}}_{\mathrm{avg}}$, ${\mathbf{W}}_{\max }$ be the parameters of the two interaction branches. Their gradients are scaled by fixed coefficients:

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}_{\mathrm{avg}}}}={\gamma }_{\mathrm{avg}}·{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}_{\mathrm{avg}}^{\mathrm{raw}}}},{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}_{\max }}}={\gamma }_{\max }·{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}_{\max }^{\mathrm{raw}}}},$$

(9)

where ${\gamma }_{\mathrm{avg}}=0.4$ and ${\gamma }_{\max }=0.1$. This suppresses overly aggressive updates in the shallow interaction layers, preventing gradient explosion when RGB and DSM features are initially misaligned.

After the above spatial recalibration, the purified and enhanced features ${\tilde{\mathbf{R}}}_l$ and ${\tilde{\mathbf{D}}}_l$ are obtained. SCFFM then performs element-wise addition on the two features to generate the fused output at the current stage, as formulated in Equation (2). Here, ${\mathbf{F}}_l$ is taken as the input to the next stage of the RGB main branch, while ${\tilde{\mathbf{D}}}_l$ continues to serve as the input to the DSM branch. This design provides two advantages for shallow multimodal fusion. On the one hand, the purified structural information from DSM can provide complementary geometric cues for RGB features, thereby reducing the modal noise conflict introduced by direct addition. On the other hand, preserving an independent DSM propagation path is beneficial for maintaining the integrity of geometric structural features during subsequent deep encoding.

3.3.4. Multilevel Progressive Fusion

In the specific implementation, SCFFM is inserted into four shallow stages, corresponding to channel dimensions of 64, 256, 512, and 1024, respectively. Through such multilevel progressive fusion, the model can capture complementary local structural and textural information at lower levels and gradually form multimodal local representations with both geometric and textural expressiveness at deeper convolutional stages. Finally, the encoder outputs four levels of skip features, $F_3$, $F_2$, $F_1$, and $F_0$, which are used for cross-level fusion during decoding.

The progressive fusion process is illustrated in Figure 3. At each stage, the RGB and DSM branches interact via SCFFM, producing fused features with increasing channel capacity.

Figure 3. Progressive multilevel fusion via SCFFM. The channel dimensions increase from 64 to 1024 across four shallow stages, yielding skip features $F_0$ to $F_3$ for subsequent decoding.

3.4. Lightweight Dual-Stream Transformer Encoding

After shallow convolutional encoding, a lightweight dual-stream Transformer encoder is introduced to further model long-range semantic dependencies and cross-modal interactions between RGB and DSM features. Instead of adopting a deep ViT architecture with a large number of stacked Transformer blocks, the proposed encoder employs a compact three-layer configuration following an SA–CA–SA pattern. The embedding dimension is set to 256, while the hidden dimension of the feed-forward network is set to 1024. This compact configuration substantially reduces the parameter overhead of the Transformer component while retaining both intra-modal contextual modeling and explicit bidirectional interaction between the two modalities.

Let the deepest RGB and DSM features produced by the dual-branch convolutional encoder be denoted by $F_r$ and $F_d$, respectively. The two features are first projected into a common embedding space and rearranged into token sequences:

$$Z_r^0={\mathcal{E}}_r\left(F_r\right)+P,Z_d^0={\mathcal{E}}_d\left(F_d\right)+P,$$

(10)

where ${\mathcal{E}}_r(·)$ and ${\mathcal{E}}_d(·)$ denote the corresponding embedding operations, and P represents the positional embedding shared by the two streams. In this way, the RGB and DSM features are transformed into representations with the same token dimension while their modality-specific information is preserved.

The first Transformer layer adopts two independent standard self-attention (SA) blocks for the RGB and DSM streams. Specifically, the RGB tokens interact only with RGB tokens, while the DSM tokens interact only with DSM tokens. Therefore, the two streams independently model their own long-range contextual dependencies without exchanging information at this stage. The corresponding operations can be expressed as

$$Z_r^1={\mathrm{SA}}_r\left(Z_r^0\right),Z_d^1={\mathrm{SA}}_d\left(Z_d^0\right),$$

(11)

where ${\mathrm{SA}}_r(·)$ and ${\mathrm{SA}}_d(·)$ denote standard multi-head self-attention Transformer blocks with independent parameters. This modality-specific processing allows the optical stream to preserve its spectral–textural representation and the DSM stream to retain its geometric and elevation-related representation before cross-modal interaction.

The intermediate layer introduces a bidirectional cross-attention (CA) block to explicitly exchange semantic information between the two modalities. Its structure is illustrated in Figure 4. Unlike a conventional one-way cross-attention mechanism, the proposed CA block performs interaction in both RGB-to-DSM and DSM-to-RGB directions while simultaneously retaining the self-attention responses of each individual modality.

Figure 4. Structure of the bidirectional cross-attention (CA) block. Each modality retains an independent self-attention pathway while receiving complementary information from the other modality through cross-attention. The two responses are adaptively combined through learnable weighting coefficients.

For clarity, let the two inputs of the CA block be denoted by $Z_x$ and $Z_y$, corresponding to the RGB and DSM token streams, respectively. For the $Z_x$ stream, an intra-modal self-attention response is first obtained via multi-head attention (MHA) as

$$A_x^{\mathrm{self}}=\mathrm{MHA}(Z_x,Z_x,Z_x),$$

(12)

where the three inputs of $\mathrm{MHA}(·)$ correspond to the query, key, and value, respectively. Meanwhile, cross-modal information from $Z_y$ is introduced by taking $Z_x$ as the query and $Z_y$ as the key and value:

$$A_{x\leftarrow y}^{\mathrm{cross}}=\mathrm{MHA}(Z_x,Z_y,Z_y).$$

(13)

The updated representation of the x stream is then obtained by adaptively combining the intra-modal and cross-modal responses:

$${\widehat{Z}}_x={\lambda }_{11}{A}_x^{\mathrm{self}}+{\lambda }_{12}{A}_{x\leftarrow y}^{\mathrm{cross}},$$

(14)

where ${\lambda }_{11}$ and ${\lambda }_{12}$ are learnable weighting coefficients controlling the relative contributions of intra-modal contextual information and complementary information from the other modality.

The same operation is symmetrically performed for the $Z_y$ stream. Its self-attention and cross-attention responses are given by

$$A_y^{\mathrm{self}}=\mathrm{MHA}(Z_y,Z_y,Z_y),$$

(15)

and

$$A_{y\leftarrow x}^{\mathrm{cross}}=\mathrm{MHA}(Z_y,Z_x,Z_x),$$

(16)

respectively. The resulting representation is expressed as

$${\hat{Z}}_y={\lambda }_{21}{A}_y^{\mathrm{self}}+{\lambda }_{22}{A}_{y\leftarrow x}^{\mathrm{cross}}.$$

(17)

Through this symmetric design, each modality maintains its own semantic representation while selectively incorporating complementary contextual information from the other modality. Consequently, bidirectional information exchange can be achieved within a single CA layer without repeatedly stacking a large number of cross-modal Transformer blocks.

To further encode spatial relationships among remote sensing tokens, a two-dimensional relative positional bias is incorporated into the attention computation. The bias is constructed according to the relative coordinates of tokens on the two-dimensional feature grid and is added to the attention correlation matrix before normalization. For a generic attention operation, the attention weights can be written as

$$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^{\mathrm{T}}}{\sqrt{d_k}}}+B_{\mathrm{rel}}\right)V,$$

(18)

where $d_k$ denotes the dimension of the key vectors and $B_{\mathrm{rel}}$ represents the learnable two-dimensional relative positional bias. The same positional modeling strategy is used in both self-attention and cross-attention operations, providing explicit relative spatial information during contextual aggregation.

After bidirectional cross-modal interaction, the resulting token sequences ${\hat{Z}}_x$ and ${\hat{Z}}_y$ are respectively passed through the final modality-specific SA blocks:

$$T_r={\mathrm{SA}}_r^{\mathrm{out}}\left({\hat{Z}}_x\right),T_d={\mathrm{SA}}_d^{\mathrm{out}}\left({\hat{Z}}_y\right),$$

(19)

where $T_r$ and $T_d$ denote the final deep semantic representations of the RGB and DSM streams, respectively. The final SA layer further refines the contextual representation within each modality after cross-modal information exchange.

Before entering the decoder, the two semantic representations are adaptively aggregated through learnable scalar weights:

$$T=\alpha T_r+\beta T_d,$$

(20)

where $\alpha$ and $\beta$ are learnable parameters that control the contributions of the RGB and DSM semantic streams. The resulting fused bottleneck representation T is subsequently fed into the progressive decoder and interacts with multilevel shallow features through GLAM for semantic reconstruction.

3.5. Global–Local Alignment Module (GLAM)

3.5.1. Motivation

Although the Transformer encoder is capable of providing powerful global semantic representations, its output features often exhibit limited sensitivity to local structures during decoding. In contrast, the skip features from the shallow convolutional encoder are more advantageous in preserving fine-grained texture details, yet they lack sufficient global semantic discriminability. Therefore, if only simple concatenation, addition, or fixed skip connections are adopted, inconsistency between deep semantics and shallow local details is likely to arise in the representation space, which in turn may lead to omission of fine-grained targets and loss of local structural detail. To address this issue, GLAM is introduced in the decoding stage, where deep semantic features and shallow detail features are explicitly aligned through a process of global relation modeling, reference-guided modulation, and adaptive fusion enhancement.

Let the shallow local feature at the s-th decoding stage be denoted by $L_s$, and let the deep global feature from the upper decoder stage or the bottleneck layer be denoted by $G_s$. As illustrated in Figure 5, GLAM first projects the two features into a unified dimension through $1\times 1$ convolutions and aligns their spatial resolutions. BDLA is then employed to establish cross-level global–local correspondences, after which a Feature-wise Linear Modulation (FiLM)-based modulation mechanism is introduced to adaptively recalibrate the deep semantic responses by taking the shallow feature as guidance. Finally, the aligned output feature is generated through convolutional fusion, channel attention, and residual connection.

Figure 5. Structure of the Global–Local Alignment Module (GLAM).

3.5.2. Global–Local Relation Modeling via BDLA

As illustrated in Figure 6, Block-Distance Linear Attention (BDLA) is introduced into GLAM to model the cross-level long-range dependencies between deep semantic features and shallow details while keeping the computational cost under control. Given the reference feature $L_s$ and the feature to be aligned $G_s$, linear projections are first applied to obtain $Q^{\prime }$, $K^{\prime }$, and V:

$$Q^{\prime }=\mathrm{RMSNorm}\left(W_q{L}_s\right),K^{\prime }=\mathrm{RMSNorm}\left(W_k{G}_s\right),V=W_v{G}_s.$$

(21)

where $W_q$, $W_k$, and $W_v$ are learnable projection matrices.

Figure 6. Structure of Block-Distance Linear Attention (BDLA).

The features are then partitioned from the two-dimensional spatial domain into M non-overlapping blocks, and the tokens within each block are reorganized into local sequences. Different from standard attention with quadratic complexity, BDLA adopts a linear attention formulation, in which local statistics are computed within each block, thereby substantially reducing the computational burden during high-resolution decoding.

For the intra-block computation, a nonnegative feature mapping function, $\varphi (·)=\mathrm{ReLU}(·)+\epsilon$, is employed, giving

$$Q^{+}=\varphi \left(Q^{\prime }\right),K^{+}=\varphi \left(K^{\prime }\right).$$

(22)

Local key-value statistics are then computed within each block as

$$K{V}_{\mathrm{local}}={\left(K^{+}\right)}^{\top }V,Z_{\mathrm{local}}=\sum K^{+},$$

(23)

where $K{V}_{\mathrm{local}}$ represents the semantic aggregation within the block, and $Z_{\mathrm{local}}$ is used for normalization.

3.5.3. Block-Wise Mixing with Distance Prior

To prevent local blocks from being treated independently, a dynamic mixing mechanism with a spatial locality prior is introduced in BDLA at the block level, as illustrated in Figure 7. Unlike conventional block-wise attention, where spatial regions are hard-partitioned, a learnable inter-block mixing matrix $\mathbf{W}\in {\mathbb{R}}^{M\times M}$ is constructed to enable adaptive contextual aggregation across different spatial blocks. The key idea is that, while preserving the local block structure, the spatial correlations among blocks are explicitly modeled, so that each block can aggregate information not only from its own local statistics but also selectively from neighboring or even distant blocks.

Figure 7. Illustration of the locality-biased block mixing in BDLA.

Specifically, suppose that the feature map is divided into M spatial blocks. To endow the model with an initial awareness of geometric structure, the pairwise Euclidean distance matrix $\mathbf{D}\in {\mathbb{R}}^{M\times M}$ is first computed according to the two-dimensional coordinates of the block centers. Based on this matrix, a local bias prior $\mathbf{B}$ is generated through a cosine transform, defined as

$$B_{i,j}=cos\left({\displaystyle \frac{D_{i,j}}{max\left(\mathbf{D}\right)}}·{\displaystyle \frac{\pi }{4}}\right),$$

(24)

where $i,j\in \{1,2,\dots ,M\}$ denote the spatial block indices. With this design, blocks that are spatially closer are naturally assigned larger correlation weights at initialization, whereas more distant blocks are given weaker but still nonzero connection strengths. In this way, a prior consistent with spatial neighborhood regularity is provided for subsequent training.

The bias matrix $\mathbf{B}$ is then row-normalized, and its logarithm is taken to initialize an unconstrained learnable parameter matrix $\mathbf{L}\in {\mathbb{R}}^{M\times M}$:

$$L_{i,j}^{\left(0\right)}=ln\left({\displaystyle \frac{B_{i,j}}{{\displaystyle \sum _{k=1}^M}{B}_{i,k}}}+\epsilon \right),$$

(25)

where $ϵ$ is a stabilizing term introduced to avoid numerical overflow. During training, $\mathbf{L}$ is continuously updated through backpropagation, enabling the model to learn more suitable cross-block mixing relationships from data rather than being restricted to predefined local connection patterns.

At each forward pass, the parameter matrix $\mathbf{L}$ is mapped by the Softmax function into a strictly nonnegative mixing weight matrix $\mathbf{W}$, whose rows sum to 1:

$$W_{i,j}={\displaystyle \frac{exp\left(L_{i,j}\right)}{{\displaystyle \sum _{k=1}^M}exp\left(L_{i,k}\right)}}.$$

(26)

The resulting $\mathbf{W}$ can be interpreted as a block-level probability distribution: the i-th output block aggregates information from all input blocks j according to the weights $W_{i,j}$. It should be noted that this spatial mixing matrix $\mathbf{W}$ is shared across all attention heads. As a result, the number of parameters does not increase linearly with the number of heads, which helps improve cross-block modeling capability while effectively controlling model complexity.

Different from conventional block-wise methods that aggregate only local key-value terms, the same mixing operation is synchronously applied here to the local key-value statistics ${\mathbf{KV}}_{\mathrm{local}}\in {\mathbb{R}}^{M\times d_k\times d_v}$ and the normalization term ${\mathbf{Z}}_{\mathrm{local}}\in {\mathbb{R}}^{M\times d_k}$. Specifically, for the i-th block, the mixed results are given by

$$K{V}_{\mathrm{mix}}^{\left(i\right)}={\displaystyle \sum _{j=1}^M}{W}_{i,j}K{V}_{\mathrm{local}}^{\left(j\right)},Z_{\mathrm{mix}}^{\left(i\right)}={\displaystyle \sum _{j=1}^M}{W}_{i,j}{Z}_{\mathrm{local}}^{\left(j\right)}.$$

(27)

This consistent inter-block weighting strategy ensures that the numerator and denominator are updated under the same spatial mixing rule, thereby avoiding the numerical mismatch that may arise when only the key-value statistics are mixed while the normalization term is ignored.

After the mixed block-level statistics are obtained, the output response of BDLA can be written as

$${\mathbf{O}}^{\left(i\right)}={\displaystyle \frac{Q^{+\left(i\right)}K{V}_{\mathrm{mix}}^{\left(i\right)}}{Q^{+\left(i\right)}{Z}_{\mathrm{mix}}^{\left(i\right)}+\epsilon }},i=1,\dots ,M,$$

(28)

where $Q^{+\left(i\right)}\in {\mathbb{R}}^{N_b\times d_k}$ denotes the mapped query tokens of the i-th block ($N_b$ tokens per block, $d_k$ dimensions each); $K{V}_{\mathrm{mix}}^{\left(i\right)}\in {\mathbb{R}}^{d_k\times d_v}$ and $Z_{\mathrm{mix}}^{\left(i\right)}\in {\mathbb{R}}^{d_k\times 1}$ are the mixed key–value aggregation and the mixed normalization term of that block, respectively. The matrix multiplications are performed along the shared $d_k$ dimension, yielding the block-wise output ${\mathbf{O}}^{\left(i\right)}\in {\mathbb{R}}^{N_b\times d_v}$. In our implementation, $d_v=d_k=$ head_dim. The final output $\mathbf{O}$ is obtained by concatenating all block outputs ${\left\{{\mathbf{O}}^{\left(i\right)}\right\}}_{i=1}^M$ and reshaping them back to the spatial feature map. By applying exactly the same spatially weighted mixing to both $\mathbf{KV}$ and $\mathbf{Z}$, the proposed method extends the receptive field from a single local block to cross-block context while strictly preserving the stability of the normalization structure in linear attention, thereby enhancing the model’s ability to represent complex spatial structures and long-range dependencies.

3.5.4. Local Positional Enhancement and FiLM-Guided Modulation

Although linear attention is effective for efficient contextual aggregation, fine-grained positional information may still be lost during local detail recovery. To address this issue, a locally enhanced positional encoding (LEPE) is introduced at the output of BDLA, where a local positional compensation term is generated through a depthwise separable convolution applied to V. In the current implementation, a $5\times 5$ convolution kernel is adopted, and the resulting positional compensation is added to the block-wise reconstructed attention output so that local neighborhood structural information can be restored:

$${\mathbf{O}}^{\prime }=\mathbf{O}+\mathrm{LEPE}\left(\mathbf{V}\right).$$

(29)

This design facilitates the preservation of local spatial information and the representation of small-scale objects.

After the BDLA output is obtained, GLAM does not directly perform feature concatenation. Instead, a Feature-wise Linear Modulation (FiLM)-based modulation mechanism is further introduced, in which the shallow local feature $L_s$ is taken as guidance to adaptively recalibrate the deep semantic response. Specifically, $L_s$ is first serialized and normalized by Layer Normalization, after which a one-dimensional semantic descriptor is generated through global average pooling. Two independent multilayer perceptrons with bottleneck structures are then used to regress the channel-wise modulation parameters $\gamma$ and $\beta$. In the current implementation, the channel reduction ratio is set to 4.

To ensure numerical stability during cross-level feature modulation, Sigmoid and Tanh activation functions are applied to constrain the ranges of $\gamma$ and $\beta$ to $(0,1)$ and $(-1,1)$, respectively. The aligned feature representation is therefore given by

$$G_s^{\mathrm{align}}=(\mathbf{1}+\gamma )\odot {\mathbf{O}}^{\prime }+\beta ,$$

(30)

where $\gamma$ and $\beta$ are broadcast to match the spatial dimensions of ${\mathbf{O}}^{\prime }$. In this way, shallow local details serve not only as the features to be fused during reconstruction, but also as reference signals that actively guide the redistribution of deep semantic features, thereby effectively bridging the cross-level representational gap between CNNs and Transformers.

3.5.5. Adaptive Fusion and Residual Enhancement

After the aligned deep feature $G_s^{\mathrm{align}}$ is obtained, GLAM concatenates it with the shallow local feature $L_s$ along the channel dimension. To fully fuse the heterogeneous information carried by the two features and to alleviate the instability of batch normalization under small-batch training conditions, a $3\times 3$ convolution followed by Group Normalization and ReLU activation is adopted for local feature extraction:

$${\mathbf{F}}_s=\mathrm{ReLU}\left(\mathrm{GN}\left({\mathrm{Conv}}_{3\times 3}\left(\left[L_s,G_s^{\mathrm{align}}\right]\right)\right)\right),$$

(31)

where $[·,·]$ denotes channel-wise concatenation.

Subsequently, to suppress redundant features introduced during multimodal fusion and to emphasize discriminative responses, a channel-attention mechanism is further incorporated to reweight the fused features. In the current implementation, this channel-attention branch adopts a channel reduction ratio of $1/4$, so that effective responses can be enhanced while the additional parameter overhead remains controlled. Finally, the reweighted fused feature is added residually to the original deep feature $G_s$, after unified dimensional projection, to form the final output:

$${\mathbf{F}}_{\mathrm{out}}=\mathrm{CA}\left({\mathbf{F}}_s\right)\odot {\mathbf{F}}_s+{\mathbf{G}}_s.$$

(32)

With this cascaded structure of alignment, fusion, reweighting, and residual enhancement, GLAM functions not merely as an attention module, but as an integrated fusion unit specifically designed for cross-level semantic coordination and fine-grained feature representation.

3.6. Progressive Decoding and Prediction Head

In the decoding stage, the fused Transformer bottleneck feature is first projected into the convolutional feature space. A four-stage progressive recovery strategy is then adopted, in which GLAM-based alignment and fusion are sequentially performed with the four skip features, $F_3$, $F_2$, $F_1$, and $F_0$, from the encoder. After each fusion stage, a PixelShuffle refinement block is employed for $2\times$ upsampling. Compared with conventional transposed convolution, PixelShuffle first maps the channels to the dimension required for high-resolution reconstruction through a $1\times 1$ convolution and then performs channel rearrangement, thereby enabling more efficient upsampling. Subsequently, a residual convolution branch is used to smooth and refine the reconstructed features, so that checkerboard artifacts can be alleviated and local continuity can be enhanced. After the four-stage progressive recovery process, a high-resolution feature map with 64 channels is obtained and is further mapped into the category space through a $3\times 3$ segmentation head, yielding the final semantic segmentation result.

3.7. Loss Function

In multimodal high-resolution remote sensing image semantic segmentation, the distribution of ground objects is often highly imbalanced. For example, buildings usually occupy relatively large regions, whereas targets such as vehicles correspond to only a small number of fine-grained pixels. Meanwhile, spectral confusion is also likely to occur between similar categories, such as trees and low vegetation, which exhibit strong visual similarity. As a result, it is often difficult to sufficiently optimize the network when only a single loss function is employed. To address this issue, a hybrid loss function incorporating pixel-level contrastive learning is constructed and embedded into a multilevel deep supervision framework. For an arbitrary prediction head, the basic joint loss is defined as a weighted combination of cross-entropy loss, Dice loss, and pixel-level contrastive loss:

$$\mathcal{L}={\lambda }_{\mathrm{CE}}{\mathcal{L}}_{\mathrm{CE}}+{\lambda }_{\mathrm{Dice}}{\mathcal{L}}_{\mathrm{Dice}}+{\lambda }_{\mathrm{Con}}{\mathcal{L}}_{\mathrm{Con}}.$$

(33)

Here, ${\mathcal{L}}_{\mathrm{CE}}$ provides the basic pixel-wise classification supervision signal and ensures stable category discrimination ability of the network. The Dice loss ${\mathcal{L}}_{\mathrm{Dice}}$ directly optimizes the overlap between predicted regions and ground-truth annotations, thereby effectively alleviating the class imbalance caused by large variations in foreground object sizes. To further enhance discriminability in the feature space, an InfoNCE-style pixel-level contrastive loss ${\mathcal{L}}_{\mathrm{Con}}$ is additionally introduced. By pulling valid pixels of the same category closer together while pushing pixels from different categories farther apart in the feature space, this loss improves the model’s ability to distinguish hard samples and effectively mitigates the confusion between spectrally similar categories. In the current experimental setting, the weighting coefficients of the three loss terms are empirically set to ${\lambda }_{\mathrm{CE}}=1.0$, ${\lambda }_{\mathrm{Dice}}=1.0$, and ${\lambda }_{\mathrm{Con}}=0.5$.

4. Experimental Setup

4.1. Implementation Details

All experiments were implemented using the PyTorch (v2.0.1) framework and conducted on a single NVIDIA A40 GPU (NVIDIA Corporation, Santa Clara, CA, USA) with 48 GB memory. A two-stage optimization strategy was adopted for network training. In the first stage, the parameters of the ImageNet-pretrained ResNet-50 convolutional backbone were frozen for 10 epochs, while all newly initialized components, including the lightweight dual-stream Transformer encoder, SCFFM, GLAM, relative positional parameters, and decoder, were optimized. In the second stage, the ResNet-50 backbone was unfrozen and the entire network was jointly fine-tuned in an end-to-end manner for 100 epochs. AdamW was employed as the optimizer. The initial learning rate in the first stage was set to $7\times {10}^{-5}$. During the second stage, learning rates of $1.05\times {10}^{-4}$, $7\times {10}^{-5}$, and $3.5\times {10}^{-5}$ were assigned to the newly initialized Transformer and fusion modules, the decoder, and the pretrained convolutional backbone, respectively. The weight decay coefficient was set to ${10}^{-3}$, and the learning rate was scheduled using cosine annealing with linear warm-up. Owing to GPU memory limitations, the physical batch size was set to 4, while an effective batch size of 16 was achieved through four-step gradient accumulation. Automatic mixed-precision training was also enabled to reduce memory consumption and improve training efficiency.

For the ISPRS Vaihingen and Potsdam datasets, the original tile images were cropped into patches as network inputs following the data preprocessing and augmentation protocol described above. For the US3D dataset, all tiles were cropped into $512\times 512$ pixel patches during training, and the training, validation, and test splits followed the protocol described in Section 3.1.

The raw single-channel NDSM is first normalized to $[0,1]$ via min–max scaling. Horizontal and vertical gradients are then extracted using Sobel operators, and the absolute values of both gradient maps are independently re-normalized to $[0,1]$. The resulting three-channel DSM representation is obtained by stacking the horizontal gradient, vertical gradient, and normalized elevation along the channel dimension. During training, input patches of size $512\times 512$ are randomly cropped from the full-resolution tiles. Synchronized online data augmentation, including random horizontal flipping, random vertical flipping, and random rotation by $90°$, $180°$, or $270°$, is applied consistently to the RGB image, the three-channel DSM, and the corresponding label map. During inference, each test tile is processed using a sliding window of size $512\times 512$ with a stride of 32 pixels. The softmax probability maps predicted from overlapping windows are accumulated by pixel-wise addition, and the final segmentation result is obtained via an argmax operation over the aggregated probability volume.

In terms of model architecture, the proposed network employs dual ResNet-50 convolutional branches for shallow feature extraction, corresponding to the RGB and DSM modalities, respectively. The convolutional branches are initialized using ImageNet-pretrained ResNet-50 weights, whereas the subsequent Transformer encoder and the remaining task-specific modules are initialized from scratch.Although the DSM branch receives a three-channel elevation representation (NDSM and spatial gradients) whose statistics differ from natural images, bootstrapping its encoder from pretrained visual backbones follows the established practice in RGB–DSM semantic segmentation [57,59]: the low-level structural priors (edge detectors, texture filters) learned from ImageNet transfer effectively to elevation-derived features, and the subsequent end-to-end fine-tuning described below allows the DSM branch to adapt its early layers to the specific input distribution. After shallow convolutional encoding, the deepest features from the two modalities are projected into a common embedding space and processed by the lightweight dual-stream Transformer encoder introduced in Section 3.4. The Transformer adopts a compact three-layer SA–CA–SA configuration, consisting of two modality-specific self-attention layers and one intermediate bidirectional cross-attention layer. The embedding dimension is set to $C_{\mathrm{emb}}=256$, the feed-forward hidden dimension is set to $C_{\mathrm{ffn}}=1024$, and each attention layer employs $H=8$ attention heads. A learnable two-dimensional relative positional bias is incorporated into both self-attention and cross-attention operations to encode relative spatial relationships between tokens. With this compact configuration, the Transformer component contains approximately 5.1 M parameters, substantially reducing the parameter overhead associated with deep Transformer encoding.

The decoder adopts a four-stage progressive upsampling structure with channel dimensions of $512\to 256\to 128\to 64$. At each decoding stage, the current deep feature is aligned and fused with the corresponding shallow encoder feature through GLAM, followed by PixelShuffle-based $2\times$ upsampling. After the final decoding stage, the reconstructed high-resolution representation is mapped to the semantic category space by the segmentation head to generate the final pixel-wise prediction.

To ensure strictly fair comparison, all baseline methods listed in Section 4.3 were re-implemented by the authors in PyTorch and retrained from scratch under the identical data splits, data augmentation, optimizer setting, and hardware environment described above.

4.2. Evaluation Metrics

To quantitatively evaluate the performance of the proposed method on multimodal semantic segmentation, three standard metrics widely used in remote sensing image segmentation are adopted, namely overall accuracy (OA), mean intersection over union (mIoU), and F1 score. Overall accuracy measures the proportion of correctly classified pixels among all pixels in the image and reflects the overall prediction accuracy of the model. It is defined as

$$\mathrm{OA}={\displaystyle \frac{{\sum }_{i=1}^k{p}_{ii}}{N}},\mathrm{where}N={\displaystyle \sum _{i=1}^k}{\displaystyle \sum _{j=1}^k}{p}_{ij}.$$

(34)

The mean intersection over union is obtained by first computing the intersection over union between the prediction and the ground truth for each class and then averaging over all classes. This metric effectively reflects the overall segmentation capability of the model across different categories and is one of the most important evaluation criteria in semantic segmentation. It is defined as

$$\mathrm{mIoU}={\displaystyle \frac{1}{k}}{\displaystyle \sum _{i=1}^k}{\displaystyle \frac{p_{ii}}{{\sum }_{j=1}^k{p}_{ij}+{\sum }_{j=1}^k{p}_{ji}-p_{ii}}}.$$

(35)

The F1 score is the harmonic mean of precision and recall, and it provides a comprehensive measure of recognition performance under class imbalance. For a single class, precision, recall, and F1 score are defined as

$$P_i={\displaystyle \frac{p_{ii}}{{\sum }_{j=1}^k{p}_{ji}}},R_i={\displaystyle \frac{p_{ii}}{{\sum }_{j=1}^k{p}_{ij}}},\mathrm{F}{1}_i={\displaystyle \frac{2P_i{R}_i}{P_i+R_i}}.$$

(36)

In the above formulations, k denotes the number of classes over which the metric is averaged. Following the common evaluation protocol of the ISPRS benchmarks, the reported mF1 and mIoU for the Vaihingen and Potsdam datasets are averaged over the five foreground classes, explicitly excluding the clutter/background class (i.e., $k=5$), whereas OA is computed over all annotated pixels of all classes. The term $p_{ii}$ denotes the number of pixels whose ground-truth class is i and that are correctly predicted as class i, i.e., the true positives. The term $p_{ij}$ denotes the number of pixels whose ground-truth class is i but are incorrectly predicted as class j, whereas $p_{ji}$ denotes the number of pixels whose ground-truth class is j but are incorrectly predicted as class i. Accordingly, for class i, $p_{ij}$ corresponds to false negatives and $p_{ji}$ corresponds to false positives. In addition, the mean F1 score (mF1) is also used in the experimental analysis to assess the average recognition performance of the model over all foreground classes.

4.3. Compared Methods

To ensure transparent and reproducible evaluation, the compared methods are explicitly categorized according to their input modality configuration, as summarized in Table 1. Single-modal methods—TransUNet [25], UNetFormer [28], SFFNet [60], EIGNet [61], D2SL [62], CIGFormer [63], and CGGLNet [64]—were trained and evaluated using only the optical imagery (RGB for Potsdam and US3D; NIRRG for Vaihingen) as input. Their original single-stream architectures and input channel configurations were preserved without modification; specifically, no early channel concatenation or extra DSM branch was appended, so that they serve as references for the single-modal performance ceiling. Multimodal methods—MIEFNet [65], CSFAFormer [57], MFNet [58], and FTransUNet [51]—were trained and evaluated using the full RGB–DSM input pair following their respective multimodal fusion protocols. The proposed MHBA-TransUNet also falls into the multimodal group. All methods in both groups follow the unified training protocol detailed in Section 4.1.

Table 1. Input modality configuration of the compared methods.

Method	Type	Input Modality
TransUNet [25]	Single-modal	Optical only
UNetFormer [28]	Single-modal	Optical only
SFFNet [60]	Single-modal	Optical only
EIGNet [61]	Single-modal	Optical only
D2SL [62]	Single-modal	Optical only
CIGFormer [63]	Single-modal	Optical only
CGGLNet [64]	Single-modal	Optical only
MIEFNet [65]	Multimodal	RGB–DSM
CSFAFormer [57]	Multimodal	RGB–DSM
MFNet [58]	Multimodal	RGB–DSM
FTransUNet [51]	Multimodal	RGB–DSM
Ours	Multimodal	RGB–DSM

To ensure the comprehensiveness and fairness of the comparison, two categories of representative methods were selected as baselines. None of the quantitative results reported in Table 2, Table 3, Table 4 and Table 5 were taken directly from the original publications.

5. Results

5.1. Comparison on the Vaihingen Dataset

Table 2 reports the quantitative comparison on the ISPRS Vaihingen dataset. The proposed method achieves the best overall performance, with OA, mF1, and mIoU reaching 92.58%, 91.74%, and 85.15%, respectively. Compared with the baseline TransUNet, the proposed method improves OA, mF1, and mIoU by 3.45, 3.99, and 6.61 percentage points, respectively.

In terms of class-wise F1 scores, the proposed method attains the best results in four categories, namely Building (97.74%), Low vegetation (82.02%), Car (93.33%), and Impervious surface (93.66%). Notably, for the Car class, the proposed method surpasses FTransUNet and MFNet by more than 2 percentage points, indicating stronger capability in recognizing small and spatially discrete targets. Figure 8 presents the qualitative comparison, where the proposed method shows more coherent prediction regions and fewer visually apparent misclassifications across different land-cover categories.

Figure 8. Visual comparison of prediction results on the Vaihingen dataset. The purple boxes mark the regions where the differences among the compared methods are most evident. (a) NIRRG images; (b) NDSM; (c) ground truth; (d) TransUNet; (e) UNetFormer; (f) SFFNet; (g) MIEFNet; (h) EIGNet; (i) D2SL; (j) CIGFormer; (k) CGGLNet; (l) CSFAFormer; (m) MFNet; (n) FTransUNet; (o) Ours.

Table 2. Comparison results on the Vaihingen dataset (%). The best results are highlighted in bold, and the second-best results are underlined.

Method	Type	Bui.	Tre.	Low.	Car	Imp.	OA	mF1	mIoU
TransUNet [25]	Single	93.53	90.57	78.35	86.80	89.51	89.13	87.75	78.54
UNetFormer [28]	Single	95.89	90.80	80.17	92.22	91.95	90.98	90.21	82.57
SFFNet [60]	Single	96.73	91.43	81.25	92.32	92.88	91.82	90.92	83.75
MIEFNet [65]	Multi	96.39	90.95	81.41	93.22	92.59	91.49	90.91	83.72
EIGNet [61]	Single	96.92	91.18	80.96	91.39	92.72	91.69	90.64	83.28
D2SL [62]	Single	96.01	91.40	81.64	91.79	92.15	91.43	90.60	83.15
CIGFormer [63]	Single	96.96	90.86	80.82	92.80	93.03	91.71	90.89	83.74
CGGLNet [64]	Single	96.99	90.89	81.16	92.88	92.97	91.74	90.98	83.87
CSFAFormer [57]	Multi	97.39	92.57	80.58	92.66	92.36	92.01	91.11	84.10
MFNet [58]	Multi	97.55	92.11	81.92	91.21	93.34	92.46	91.23	84.32
FTransUNet [51]	Multi	97.36	91.94	81.49	91.27	93.01	92.40	91.01	84.23
Ours	Multi	97.74	91.94	82.02	93.33	93.66	92.58	91.74	85.15

Note: The results of Ours correspond to the single run with the best overall performance among 3 independent runs. The mean ± std over these 3 runs with different random seeds are OA = 92.56 ± 0.03, mF1 = 91.78 ± 0.11, and mIoU = 85.13 ± 0.15.

Table 2. Comparison results on the Vaihingen dataset (%). The best results are highlighted in bold, and the second-best results are underlined.

Method	Type	Bui.	Tre.	Low.	Car	Imp.	OA	mF1	mIoU
TransUNet [25]	Single	93.53	90.57	78.35	86.80	89.51	89.13	87.75	78.54
UNetFormer [28]	Single	95.89	90.80	80.17	92.22	91.95	90.98	90.21	82.57
SFFNet [60]	Single	96.73	91.43	81.25	92.32	92.88	91.82	90.92	83.75
MIEFNet [65]	Multi	96.39	90.95	81.41	93.22	92.59	91.49	90.91	83.72
EIGNet [61]	Single	96.92	91.18	80.96	91.39	92.72	91.69	90.64	83.28
D2SL [62]	Single	96.01	91.40	81.64	91.79	92.15	91.43	90.60	83.15
CIGFormer [63]	Single	96.96	90.86	80.82	92.80	93.03	91.71	90.89	83.74
CGGLNet [64]	Single	96.99	90.89	81.16	92.88	92.97	91.74	90.98	83.87
CSFAFormer [57]	Multi	97.39	92.57	80.58	92.66	92.36	92.01	91.11	84.10
MFNet [58]	Multi	97.55	92.11	81.92	91.21	93.34	92.46	91.23	84.32
FTransUNet [51]	Multi	97.36	91.94	81.49	91.27	93.01	92.40	91.01	84.23
Ours	Multi	97.74	91.94	82.02	93.33	93.66	92.58	91.74	85.15

Note: The results of Ours correspond to the single run with the best overall performance among 3 independent runs. The mean ± std over these 3 runs with different random seeds are OA = 92.56 ± 0.03, mF1 = 91.78 ± 0.11, and mIoU = 85.13 ± 0.15.

5.2. Comparison on the Potsdam Dataset

Table 3 reports the quantitative comparison on the ISPRS Potsdam dataset. The proposed method achieves the best overall performance, with OA, mF1, and mIoU of 92.29%, 93.47%, and 88.01%, respectively. Compared with the baseline TransUNet, it improves OA, mF1, and mIoU by 3.09, 3.35, and 5.77 percentage points, respectively.

In terms of class-wise F1 scores, the proposed method attains the best results in three categories, namely Building (98.04%), Tree (89.32%), and Impervious surface (94.55%). For Low vegetation and Car, the proposed method achieves competitive F1 scores of 88.39% and 97.07%, respectively. Overall, the proposed method still maintains the best performance across the comprehensive evaluation metrics. Figure 9 provides a qualitative comparison of the prediction results, showing generally coherent predictions across buildings, vegetation, and other land-cover categories.

Figure 9. Visual comparison of prediction results on the Potsdam dataset. The purple boxes mark the regions where the differences among the compared methods are most evident. (a) IRRGB images; (b) NDSM; (c) ground truth; (d) TransUNet; (e) UNetFormer; (f) SFFNet; (g) MIEFNet; (h) EIGNet; (i) D2SL; (j) CIGFormer; (k) CGGLNet; (l) CSFAFormer; (m) MFNet; (n) FTransUNet; (o) Ours.

Table 3. Comparison results on the Potsdam dataset (%). The best results are highlighted in bold, and the second-best results are underlined.

Method	Type	Bui.	Tre.	Low.	Car	Imp.	OA	mF1	mIoU
TransUNet [25]	Single	95.51	86.65	85.31	95.17	91.49	89.20	90.12	82.24
UNetFormer [28]	Single	96.40	87.13	86.75	95.17	93.10	90.58	91.71	84.94
SFFNet [60]	Single	97.36	88.94	87.98	96.56	94.11	91.90	92.99	87.14
MIEFNet [65]	Multi	96.90	89.14	87.96	96.46	93.39	91.28	92.77	86.73
EIGNet [61]	Single	96.43	87.84	86.75	96.23	92.80	90.58	92.01	85.47
D2SL [62]	Single	97.42	88.88	87.90	96.47	94.06	91.81	92.95	87.07
CIGFormer [63]	Single	97.09	88.45	87.02	96.64	93.96	91.36	92.63	86.55
CGGLNet [64]	Single	97.05	88.73	88.02	96.43	93.93	91.52	92.83	86.86
CSFAFormer [57]	Multi	97.95	86.75	86.55	97.31	93.51	91.30	92.41	86.80
MFNet [58]	Multi	97.93	87.81	87.89	97.11	93.61	91.60	92.87	86.99
FTransUNet [51]	Multi	97.78	88.27	88.48	96.31	93.17	91.34	92.41	86.20
Ours	Multi	98.04	89.32	88.39	97.07	94.55	92.29	93.47	88.01

Note: The results of Ours correspond to the single run with the best overall performance among 3 independent runs. The mean ± std over these 3 runs with different random seeds are OA = 92.30 ± 0.17, mF1 = 93.49 ± 0.14, and mIoU = 88.09 ± 0.08.

Table 3. Comparison results on the Potsdam dataset (%). The best results are highlighted in bold, and the second-best results are underlined.

Method	Type	Bui.	Tre.	Low.	Car	Imp.	OA	mF1	mIoU
TransUNet [25]	Single	95.51	86.65	85.31	95.17	91.49	89.20	90.12	82.24
UNetFormer [28]	Single	96.40	87.13	86.75	95.17	93.10	90.58	91.71	84.94
SFFNet [60]	Single	97.36	88.94	87.98	96.56	94.11	91.90	92.99	87.14
MIEFNet [65]	Multi	96.90	89.14	87.96	96.46	93.39	91.28	92.77	86.73
EIGNet [61]	Single	96.43	87.84	86.75	96.23	92.80	90.58	92.01	85.47
D2SL [62]	Single	97.42	88.88	87.90	96.47	94.06	91.81	92.95	87.07
CIGFormer [63]	Single	97.09	88.45	87.02	96.64	93.96	91.36	92.63	86.55
CGGLNet [64]	Single	97.05	88.73	88.02	96.43	93.93	91.52	92.83	86.86
CSFAFormer [57]	Multi	97.95	86.75	86.55	97.31	93.51	91.30	92.41	86.80
MFNet [58]	Multi	97.93	87.81	87.89	97.11	93.61	91.60	92.87	86.99
FTransUNet [51]	Multi	97.78	88.27	88.48	96.31	93.17	91.34	92.41	86.20
Ours	Multi	98.04	89.32	88.39	97.07	94.55	92.29	93.47	88.01

Note: The results of Ours correspond to the single run with the best overall performance among 3 independent runs. The mean ± std over these 3 runs with different random seeds are OA = 92.30 ± 0.17, mF1 = 93.49 ± 0.14, and mIoU = 88.09 ± 0.08.

5.3. Comparison on the US3D Dataset

Table 4 reports the quantitative comparison on the US3D dataset. The proposed method achieves the best overall performance, with OA, mF1, and mIoU reaching 94.37%, 92.05%, and 85.49%, respectively. Compared with FTransUNet, the strongest competing method, the proposed method improves OA, mF1, and mIoU by 0.42, 0.41, and 0.70 percentage points, respectively.

In terms of class-wise F1 scores, the proposed method attains the best results in four categories, namely Ground (96.42%), High vegetation (85.89%), Building (93.71%), and Elevated road (90.25%). For the Water class, FTransUNet achieves the highest F1 score (94.19%), while the proposed method remains competitive (94.02%). Figure 10 presents the qualitative comparison, where the proposed method produces more coherent regions and fewer misclassified pixels across diverse urban scenes.

Figure 10. Visual comparison of prediction results on the US3D dataset. (a) RGB images; (b) DSM; (c) ground truth; (d) D2SL; (e) CIGFormer; (f) CGGLNet; (g) MIEFNet; (h) CSFAFormer; (i) MFNet; (j) FTransUNet; (k) Ours.

Table 4. Comparison results on the US3D dataset (%). The best results are highlighted in bold, and the second-best results are underlined.

Method	Type	Ground	High-Veg	Bui.	Water	Elv.-Rd	OA	mF1	mIoU
D2SL [62]	Single	91.63	73.81	87.74	92.67	67.88	88.21	82.75	71.79
CIGFormer [63]	Single	91.05	72.44	86.64	89.91	63.17	87.43	80.64	68.93
CGGLNet [64]	Single	92.10	73.86	88.03	92.55	68.66	88.71	83.04	72.19
MIEFNet [65]	Multi	95.23	81.77	92.12	94.13	88.70	92.78	90.39	82.81
CSFAFormer [57]	Multi	95.86	85.15	93.61	90.38	89.72	93.73	90.94	83.60
MFNet [58]	Multi	96.04	84.47	93.12	93.94	89.41	93.77	91.40	84.41
FTransUNet [51]	Multi	96.14	85.24	93.02	94.19	89.59	93.95	91.64	84.79
Ours	Multi	96.42	85.89	93.71	94.02	90.25	94.37	92.05	85.49

Note: The results of Ours correspond to the single run with the best overall performance among 3 independent runs. The mean ± std over these 3 runs with different random seeds are OA = 94.34 ± 0.05, mF1 = 92.08 ± 0.09, and mIoU = 85.55 ± 0.10.

Table 4. Comparison results on the US3D dataset (%). The best results are highlighted in bold, and the second-best results are underlined.

Method	Type	Ground	High-Veg	Bui.	Water	Elv.-Rd	OA	mF1	mIoU
D2SL [62]	Single	91.63	73.81	87.74	92.67	67.88	88.21	82.75	71.79
CIGFormer [63]	Single	91.05	72.44	86.64	89.91	63.17	87.43	80.64	68.93
CGGLNet [64]	Single	92.10	73.86	88.03	92.55	68.66	88.71	83.04	72.19
MIEFNet [65]	Multi	95.23	81.77	92.12	94.13	88.70	92.78	90.39	82.81
CSFAFormer [57]	Multi	95.86	85.15	93.61	90.38	89.72	93.73	90.94	83.60
MFNet [58]	Multi	96.04	84.47	93.12	93.94	89.41	93.77	91.40	84.41
FTransUNet [51]	Multi	96.14	85.24	93.02	94.19	89.59	93.95	91.64	84.79
Ours	Multi	96.42	85.89	93.71	94.02	90.25	94.37	92.05	85.49

Note: The results of Ours correspond to the single run with the best overall performance among 3 independent runs. The mean ± std over these 3 runs with different random seeds are OA = 94.34 ± 0.05, mF1 = 92.08 ± 0.09, and mIoU = 85.55 ± 0.10.

5.4. Complexity Analysis

To evaluate the accuracy–efficiency trade-off, the parameter counts, floating point operations (FLOPs), and inference speed of representative methods are compared under identical settings. All measurements are conducted on an NVIDIA A100 40G GPU with an input size of $512\times 512$ and a batch size of 1. The inference speed is reported as the average frames per second (FPS) over 100 formal runs after 20 warm-up iterations. FLOPs are measured with the THOP toolkit (https://github.com/Lyken17/pytorch-OpCounter, accessed on 8 August 2026). The test script records total parameters, trainable parameters, FLOPs, and average inference time to ensure a fair comparison.

As shown in Table 5, the proposed method achieves a favorable accuracy–efficiency balance. With 72.12 M parameters and 63.12G FLOPs, it runs at 69.68 FPS. Compared with FTransUNet (208.13 M parameters) and MFNet (317.25 M parameters), the proposed model significantly reduces the parameter overhead. Meanwhile, its FLOPs are much lower than those of CSFAFormer (141.76 G) and CIGFormer (135.50 G). In terms of inference speed, the proposed method is close to SFFNet (71.99 FPS) and markedly faster than MFNet (5.89 FPS) and FTransUNet (36.64 FPS), demonstrating competitive computational efficiency.

Table 5. Complexity comparison of different methods. Parameters and FLOPs are measured in millions (M) and giga (G), respectively. FPS is reported as mean ± std.

Method	Type	Parameters (M)	FLOPs (G)	FPS
D2SL [62]	Single	92.45	91.09	51.61 ± 0.73
CIGFormer [63]	Single	34.38	135.50	69.02 ± 2.36
CGGLNet [64]	Single	34.83	152.08	53.58 ± 1.78
MIEFNet [65]	Multi	66.75	44.72	39.57 ± 1.28
CSFAFormer [57]	Multi	54.34	141.76	12.03 ± 0.15
MFNet [58]	Multi	317.25 (8.98) a	360.68	5.89 ± 0.06
SFFNet [60]	Single	34.18	51.67	71.99 ± 0.15
FTransUNet [51]	Multi	208.13	57.33	36.64 ± 0.55
Ours	Multi	72.12	63.12	69.68 ± 1.38

^a 8.98 denotes the trainable parameters of MFNet in millions.

Table 5. Complexity comparison of different methods. Parameters and FLOPs are measured in millions (M) and giga (G), respectively. FPS is reported as mean ± std.

Method	Type	Parameters (M)	FLOPs (G)	FPS
D2SL [62]	Single	92.45	91.09	51.61 ± 0.73
CIGFormer [63]	Single	34.38	135.50	69.02 ± 2.36
CGGLNet [64]	Single	34.83	152.08	53.58 ± 1.78
MIEFNet [65]	Multi	66.75	44.72	39.57 ± 1.28
CSFAFormer [57]	Multi	54.34	141.76	12.03 ± 0.15
MFNet [58]	Multi	317.25 (8.98) a	360.68	5.89 ± 0.06
SFFNet [60]	Single	34.18	51.67	71.99 ± 0.15
FTransUNet [51]	Multi	208.13	57.33	36.64 ± 0.55
Ours	Multi	72.12	63.12	69.68 ± 1.38

^a 8.98 denotes the trainable parameters of MFNet in millions.

5.5. Ablation Studies

5.5.1. Module-Level Ablation

To verify the effectiveness of SCFFM and GLAM, ablation experiments were conducted on the Vaihingen and Potsdam datasets while keeping the dual-branch backbone unchanged. As shown in Table 6, on the Vaihingen dataset, introducing SCFFM alone improves mIoU by 0.77 percentage points, while introducing GLAM alone improves mIoU by 1.68 percentage points. The complete model achieves the best performance (OA 92.58%, mIoU 85.15%). A similar trend is observed on Potsdam. These results demonstrate that both modules make positive contributions and exhibit clear complementarity, with GLAM bringing the more notable individual gain, as also visualized in Figure 11.

Table 6. Module-level ablation on the Vaihingen, Potsdam, and US3D datasets (%). The best results for each dataset are highlighted in bold.

Dataset	Method	OA	mF1	mIoU
Vaihingen	Baseline	90.41	90.22	82.44
	Baseline + SA-CA-SA	90.99	90.56	82.86
	Baseline + SA-CA-SA + SCFFM	91.32	90.99	83.63
	Baseline + SA-CA-SA + GLAM	91.82	91.57	84.54
	Baseline + SA-CA-SA + SCFFM + GLAM	92.58	91.74	85.15
Potsdam	Baseline	90.13	91.98	85.57
	Baseline + SA-CA-SA	90.92	92.47	86.21
	Baseline + SA-CA-SA + SCFFM	91.36	92.78	86.74
	Baseline + SA-CA-SA + GLAM	91.84	93.12	87.37
	Baseline + SA-CA-SA + SCFFM + GLAM	92.29	93.47	88.01
US3D	Baseline	92.26	89.82	82.31
	Baseline + SA-CA-SA	92.76	90.41	82.79
	Baseline + SA-CA-SA + SCFFM	93.52	90.88	83.57
	Baseline + SA-CA-SA + GLAM	94.03	91.44	84.87
	Baseline + SA-CA-SA + SCFFM + GLAM	94.37	92.05	85.49

Figure 11. Module-level ablation results on the Vaihingen dataset. Both SCFFM and GLAM contribute positively to mIoU, with their combination yielding the largest gain.

5.5.2. Ablation on SCFFM Fusion Strategies

To further verify the internal design of SCFFM, comparative experiments were conducted on the Vaihingen dataset against three common fusion strategies. As shown in Table 7, direct addition yields the lowest mIoU (83.88%), as it ignores spatial misalignment. Concatenation followed by $1\times 1$ convolution slightly improves performance (mIoU 84.03%). SE-Attention fusion further raises mIoU to 84.55% through channel-wise recalibration. The proposed SCFFM achieves the best results (OA 92.58%, mIoU 85.15%), outperforming SE-Attention by 0.60 percentage points in mIoU. This gain is attributed to the DSRM, which adaptively captures and corrects local deviations between RGB and DSM.

Table 7. Comparison of fusion strategies in SCFFM on the Vaihingen dataset (%).

Method	OA	mF1	mIoU
Direct Addition	92.01	90.77	83.88
Concat + $1\times 1$ Conv	92.15	90.92	84.03
SE-Attention Fusion	92.40	91.28	84.55
SCFFM	92.58	91.74	85.15

5.5.3. Component Ablation of GLAM

To investigate the key components within GLAM, ablation experiments were conducted on the Vaihingen dataset. As shown in Table 8, based on the baseline with SCFFM (mIoU 83.63%), adding FiLM alone improves mIoU to 84.25%, demonstrating its ability to bridge the cross-level representational gap. Adding BDLA alone further improves mIoU to 84.87%, verifying its effectiveness in establishing long-range dependencies. The complete GLAM module achieves the best performance (mIoU 85.15%), indicating that BDLA and FiLM are highly complementary: BDLA handles global relation modeling, while FiLM performs fine-grained feature distribution modulation. Figure 12 further visualizes these component-level results.

Table 8. Component ablation of GLAM on the Vaihingen dataset (%).

Method	OA	mF1	mIoU
Baseline + SCFFM	91.32	90.99	83.63
Baseline + SCFFM + FiLM	92.04	91.16	84.25
Baseline + SCFFM + BDLA	92.44	91.53	84.87
Baseline + SCFFM + GLAM	92.58	91.74	85.15

Figure 12. Component ablation within GLAM on the Vaihingen dataset. BDLA and FiLM exhibit complementary effects, jointly achieving the best performance.

5.5.4. Ablation on Transformer Layer Configuration

To investigate the sensitivity of the Transformer encoder to layer depth and operation ordering, we conduct experiments on the Vaihingen dataset while fixing $C_{\mathrm{emb}}=256$, $C_{\mathrm{ffn}}=1024$, and $H=8$. As shown in Table 9, the two-layer configurations (SA–CA and CA–SA) achieve comparable mIoU scores around $84.8\%$, indicating that a single SA and a single CA are insufficient regardless of ordering. The proposed three-layer SA–CA–SA significantly improves the mIoU to $85.15\%$ with $5.12$ M parameters, benefiting from the “intra-modal aggregation → cross-modal fusion → intra-modal refinement” pattern. However, further deepening does not help: the four-layer variant achieves $85.02\%$ with $50\%$ more parameters, and the six-layer stack degrades to $84.66\%$, even underperforming the two-layer baselines. This suggests that, under the compact embedding setting, excessive depth introduces optimization difficulties and feature redundancy. Consequently, the SA–CA–SA configuration represents the optimal trade-off between computational efficiency and representation capability.

Table 9. Ablation on Transformer layer configurations on the Vaihingen dataset. All settings use $C_{\mathrm{emb}}=256$, $C_{\mathrm{ffn}}=1024$, and $H=8$.

Configuration	Layers	Trans. Params (M)	mIoU (%)
SA–CA	2	3.82	84.89
CA–SA	2	3.82	84.76
SA–CA–SA (Ours)	3	5.12	85.15
SA–CA–CA–SA	4	7.68	85.02
$2\times$SA–$2\times$CA–$2\times$SA	6	10.24	84.66

5.5.5. Ablation on Loss Function

To clarify the individual contribution of the hybrid loss function, we conduct experiments on the Vaihingen dataset using the complete proposed network while varying only the loss terms. As reported in Table 10, using CE loss alone achieves an mIoU of $84.32\%$. Incorporating Dice loss improves the mIoU to $84.89\%$, demonstrating its effectiveness in handling class imbalance caused by large object-size variations. The full hybrid loss with the additional pixel-level contrastive term further raises the mIoU to $85.15\%$. This progressive improvement indicates that the contrastive supervision effectively pulls same-class pixels closer and pushes different-class pixels apart in the feature space, thereby mitigating confusion between spectrally similar categories such as tree and low vegetation.

Table 10. Ablation on loss function configurations on the Vaihingen dataset (%).

Loss Configuration	OA	mF1	mIoU
CE only	91.87	91.95	84.32
CE + Dice	92.32	91.42	84.89
CE + Dice + Contrastive (Full)	92.58	91.74	85.15

5.6. Feature Visualization of GLAM

To intuitively examine how GLAM reshapes feature representations during decoding, feature maps before and after GLAM at the stage closest to the segmentation head are visualized in Figure 13. The first two rows show samples from the Vaihingen dataset, and the last two rows show samples from the Potsdam dataset. For each sample, the four columns show, from left to right: the optical image; the ground truth; the feature map before GLAM (upsampled input to the last GLAM block); and the feature map after GLAM.

Figure 13. Visualization of decoder features before and after GLAM at the stage closest to the segmentation head. (Rows 1–2): Vaihingen dataset; (Rows 3–4): Potsdam dataset. For each sample: (1st col.) optical image; (2nd col.) ground truth; (3rd col.) feature map before GLAM; (4th col.) feature map after GLAM.

As qualitatively evidenced, after passing through GLAM, the activation responses become more spatially coherent within target regions, while small and spatially discrete objects exhibit stronger foreground–background contrast. Background responses are meanwhile suppressed, indicating that the FiLM-guided modulation helps deep semantic features better exploit shallow spatial details rather than being overwhelmed by them. These observations are consistent with the quantitative improvements in the Car class F1 score reported in Table 2 and Table 3. It should be emphasized that this visualization serves as a qualitative interpretation of the module’s behavior; the quantitative evidence of performance gains is provided by the overall metrics, class-wise results, and ablation studies reported above.

6. Discussion

Systematic experiments on the ISPRS Vaihingen, ISPRS Potsdam, and US3D datasets demonstrate the effectiveness of MHBA-TransUNet for RGB–DSM semantic segmentation in complex urban scenes. The proposed method achieves consistent improvements in the overall evaluation metrics, including OA, mF1, and mIoU, while also showing favorable recognition performance for several challenging object categories. On the Vaihingen and Potsdam datasets, MHBA-TransUNet achieves competitive or superior performance compared with the evaluated state-of-the-art methods. The results on US3D further indicate that the proposed framework can maintain stable segmentation performance under different geographic environments, imaging conditions, and urban layouts.

The results on the US3D dataset provide additional evidence regarding the robustness of the proposed multimodal modeling strategy. As shown in Table 4, MHBA-TransUNet achieves an mIoU of 85.49% on the US3D test set, exceeding FTransUNet by 0.70 percentage points. The proposed method also achieves strong class-wise performance, including an F1 score of 90.25% for the Elevated road category. Considering the differences in scene composition and urban morphology represented in the US3D data, these results suggest that the proposed framework can effectively exploit complementary information from optical imagery and elevation data and maintain stable performance across diverse urban scenes. This observation is consistent with the intended roles of SCFFM and GLAM, which perform cross-modal interaction at shallow stages and global–local feature coordination during decoding, respectively.

Compared with conventional CNN-based multimodal fusion approaches, the proposed framework combines local feature extraction with long-range semantic modeling at different stages of the network. SCFFM progressively integrates complementary RGB and DSM information in the shallow encoder through dynamic spatial recalibration, while GLAM establishes interactions between deep semantic features and shallow spatial information during decoding through block-distance linear attention and feature-wise modulation. The ablation results further show that both modules contribute positively to segmentation performance and that their combination provides a larger overall improvement than either module alone. These findings indicate that coordinating shallow cross-modal fusion with deep global–local feature interaction is beneficial for multimodal semantic segmentation, particularly in scenes containing heterogeneous textures, shadow interference, small objects, and complex spatial arrangements.

Despite the overall improvements, the category-level results also reveal several remaining challenges. On the Potsdam dataset, the Low vegetation class does not achieve the highest F1 score among the compared methods, indicating that confusion between visually similar vegetation categories remains difficult to fully resolve. A similar issue can occur between Tree and Low vegetation in the Vaihingen dataset because of their similar spectral and textural characteristics. For the Car category, although MHBA-TransUNet achieves strong performance compared with most competing methods, very small, densely distributed, or partially occluded vehicles may still be missed. In highly complex urban scenes, cluttered backgrounds, fragmented object regions, and strong intra-class appearance variations may also introduce local prediction errors. These observations suggest that further improvements are still needed for extremely small-object recognition, discrimination between spectrally similar categories, and semantic segmentation under highly heterogeneous urban conditions.

To better understand the cases where the proposed method does not achieve the highest F1 score, the class-wise precision and recall are further analyzed in Table 11 and Table 12. On the Vaihingen dataset, the Tree F1 (91.94%) trails CSFAFormer (92.57%) mainly because of its lower precision (91.66% vs. recall 92.21%), indicating that some low-vegetation or shadowed regions are misclassified as Tree. The Car class, despite attaining the best F1 (93.33%), exhibits a large precision–recall gap (91.12% vs. 95.64%), suggesting a tendency toward over-detection. On the Potsdam dataset, the Low vegetation F1 (88.39%) is marginally below FTransUNet (88.48%); its recall (90.13%) is satisfactory, yet the precision (86.72%) reveals false positives along shadow boundaries. These observations suggest that future refinement should target class-specific error modes—suppressing spectral confusion for vegetation and introducing shape constraints for small objects.

Table 11. Class-wise precision and recall on the Vaihingen dataset (%).

Class	Precision	Recall	F1	Best F1 Among Compared Methods
Impervious surface	93.80	93.52	93.66	93.66 (Ours)
Building	97.22	98.27	97.74	97.74 (Ours)
Low vegetation	83.06	81.01	82.02	82.02 (Ours)
Tree	91.66	92.21	91.94	92.57 (CSFAFormer)
Car	91.12	95.64	93.33	93.33 (Ours)
Clutter	0.00	0.00	0.00	${—}^{ \mathrm{a}}$

^a Clutter is excluded from the mF1/mIoU computation following the ISPRS benchmark protocol.

Table 12. Class-wise precision and recall on the Potsdam dataset (%).

Class	Precision	Recall	F1	Best F1 Among Compared Methods
Impervious surface	93.81	95.30	94.55	94.55 (Ours)
Building	97.55	98.54	98.04	98.04 (Ours)
Low vegetation	86.72	90.13	88.39	88.48 (FTransUNet)
Tree	90.12	88.53	89.32	89.32 (Ours)
Car	96.89	97.25	97.07	97.31 (CSFAFormer)
Clutter	77.27	53.97	63.55	${—}^{ \mathrm{a}}$

^a Clutter is excluded from the mF1/mIoU computation following the ISPRS benchmark protocol.

In addition to these category-level challenges, the hybrid CNN–Transformer architecture introduces additional feature transformation and multimodal interaction operations, resulting in higher computational complexity than lightweight single-branch networks. As shown in Table 5, MHBA-TransUNet contains 72.12 M parameters and achieves an inference speed of 69.68 FPS, providing a favorable balance between segmentation accuracy and computational efficiency among the evaluated methods. Nevertheless, the dual-branch encoder and multilayer feature interaction mechanisms still require considerable computational resources. Therefore, improving the balance among cross-modal information utilization, segmentation performance, and computational efficiency remains an important direction for practical deployment, especially in large-scale remote sensing image processing and real-time applications.

Future work will focus on developing more lightweight and efficient multimodal collaborative modeling strategies while retaining the complementary advantages of optical imagery and DSM elevation data. Possible directions include lightweight frequency-aware modules, reduced-channel feature transformations, and more efficient dynamic interaction mechanisms to reduce parameter counts and computational cost. Additional data modalities, such as SAR imagery and multispectral data, may also be incorporated to investigate the robustness of multimodal segmentation under more diverse imaging and environmental conditions. Furthermore, lightweight collaborative architectures and multi-scale adaptive inference strategies will be explored to improve the trade-off among segmentation accuracy, computational efficiency, robustness, and practical deployment. For large-scale remote sensing applications, overlap-based tiling and parallel inference strategies may also be investigated to reduce prediction inconsistencies introduced during tiled processing while improving inference efficiency.

7. Conclusions

This paper proposes a multimodal hybrid bidirectional attention framework for RGB–DSM semantic segmentation of high-resolution remote sensing images. The SCFFM and GLAM modules in MHBA-TransUNet achieve collaborative modeling of shallow cross-modal feature fusion and deep global–local semantic alignment through dynamic spatial recalibration, block-distance linear attention, and feature-wise modulation, thereby improving the segmentation accuracy and robustness of the model in complex scenes. Experimental results demonstrate that MHBA-TransUNet achieves robust and competitive segmentation performance on the ISPRS Vaihingen, ISPRS Potsdam, and US3D datasets, with mIoU values of 85.15%, 88.01%, and 85.49%, respectively, indicating its broad applicability across diverse urban benchmarks and application potential. Overall, the proposed method provides an effective modeling paradigm for multimodal semantic segmentation of high-resolution remote sensing images. Future work will explore lightweight architectures and additional modal information fusion to further enhance the generalization across diverse urban scenes and segmentation performance.