GIMMNet: Geometry-Aware Interactive Multi-Modal Network for Semantic Segmentation of High-Resolution Remote Sensing Imagery

Highlights

What are the main findings?

• A novel architecture named GIMMNet is proposed to explicitly model the geometric structure embedded in nDSM and use it to guide the spatial distribution of semantic categories. The network comprises three core modules: the Geometric Position Prior Module, the Geometry-Guided Disentangled Fusion Module, and the Geometry-Attentive Context Module.

What are the implications of the main findings?

• The proposed method explicitly models the 3D geometric priors embedded in the nDSM and integrates them into multimodal fusion, thereby significantly improving semantic segmentation accuracy in complex urban scenes—a capability that is essential for high-precision land-cover recognition.
• The experiments validated the critical role of geometric priors in remote sensing semantic understanding, providing important inspiration for future research to explore the potential of other geometric or physical priors.

Abstract

Remote sensing semantic segmentation holds significant application value in urban planning, environmental monitoring, and related fields. In recent years, multimodal approaches that fuse optical imagery with normalized Digital Surface Models (nDSM) have attracted widespread attention due to their superior performance. However, existing methods typically treat nDSM merely as an additional input channel, failing to effectively exploit its inherent 3D geometric priors, which limits segmentation accuracy in complex urban scenes. To address this issue, we propose a Geometry-aware Interactive Multi-Modal Network (GIMMNet), which explicitly models the geometric structure embedded in nDSM to guide the spatial distribution of semantic categories. Specifically, we first design a Geometric Position Prior Module (GPPM) to construct 3D coordinates for each pixel based on nDSM and extract intrinsic geometric priors. Next, a Geometry-Guided Disentangled Fusion Module (GDFM) dynamically adjusts fusion weights according to the differential responses of each modality to the geometric priors, enabling adaptive multimodal feature integration. Finally, during decoding, a Geometry-Attentive Context Module (GACM) explicitly captures the dependencies between land-cover categories and geometric structures, enhancing the model’s spatial awareness and semantic recovery capability. Experimental results on two public remote sensing datasets—Vaihingen and Potsdam—show that the proposed GIMMNet outperforms existing mainstream methods in segmentation performance, demonstrating that enhancing the model’s geometric perception capability effectively improves semantic segmentation accuracy. Notably, our method achieves an mIoU of 85.2% on the Potsdam dataset, surpassing the second-best multimodal approach, PACSCNet, by 2.3%.

1. Introduction

Semantic segmentation of remote sensing imagery is a fundamental technique [1] for analyzing data acquired from aerial platforms such as satellites and unmanned aerial vehicles. Its primary objective is to accurately classify each pixel in an image into predefined semantic categories—such as buildings, roads, forests, and water bodies—thereby enabling intelligent extraction and analysis of geospatial information to support applications in environmental monitoring, urban planning, and disaster emergency response [2].

Early deep learning–based approaches primarily relied on optical remote sensing images and achieved promising results across various scenarios. However, single-modality optical data are highly susceptible to illumination variations, cloud occlusion, and mutual occlusion among ground objects, leading to unstable performance in complex scenes [3]. With the advancement of multi-source remote sensing sensors, fusing complementary information from heterogeneous modalities has emerged as an effective strategy to enhance both accuracy and robustness. Among these, the fusion of optical images (OPT) and normalized Digital Surface Models (nDSM) has attracted significant attention: as shown in Figure 1, OPT provides rich spectral and textural cues that characterize surface appearance, while nDSM encodes elevation information [4]. The two are highly complementary across information dimensions, and their effective fusion enables superior feature representation.

Despite extensive research on dual-modality semantic segmentation using OPT and nDSM, existing methods share a critical limitation: they treat nDSM merely as an additional input channel, failing to fully exploit its explicit 3D geometric priors. For instance, Fan et al.’s PACSCNet [5] employs symmetric ResNet-50 [6] backbones for both OPT and nDSM but overlooks their substantial disparity in information density—nDSM contains sparse semantic details, causing its deep branch to suffer from over-parameterization, feature redundancy, and degradation. EDGNet [7] focuses on boundary cues from OPT yet ignores the geometric boundaries induced by abrupt elevation changes in nDSM, thus underutilizing its structural prior. Meanwhile, G2GNet [8], SERNet [9], and CIMFNet [10] either neglect low-level spatial details or rely on computationally expensive repetitive fusion modules, failing to establish an efficient mechanism to integrate the spatial layout encoded in nDSM into semantic reasoning.

Notably, these approaches commonly overlook a key insight: nDSM is not merely a height map but a rich source of 3D geometric priors that complement spectral information. These geometric priors capture structural cues such as height continuity, relative elevation differences between neighboring objects, and characteristic spatial layouts of different land-cover types. For example, high-rise buildings typically form compact, block-like clusters with uniform elevation; roads appear as low, continuous planar surfaces; and trees exhibit irregular, undulating canopy shapes with variable heights. By integrating these geometric cues with RGB-derived semantic features, a model can more effectively distinguish spectrally similar but geometrically distinct regions, such as low-rise buildings versus tree crowns, thereby reducing misclassification in complex urban scenes.

To address the insufficient exploitation of spatial positional information in nDSM by existing methods, we propose GIMMNet (Geometry-aware Interactive Multi-Modal Network), a novel dual-modality semantic segmentation framework. GIMMNet deeply harnesses the geometric value of nDSM through differentiated feature extraction and position-aware cross-modal interaction, enabling synergistic optimization between geometric priors and semantic understanding. The main contributions of this paper are as follows:

• Geometric Position Prior Module (GPPM): This module generates pixel-level 3D coordinates from the nDSM and extracts intrinsic 3D geometric priors.
• Geometry-Guided Disentangled Fusion Module (GDFM): This module quantifies the response discrepancy of each modality to 3D geometric priors, separately extracts shared and modality-specific features, and dynamically generates adaptive fusion weights, thereby achieving a more comprehensive multimodal fusion.
• Geometry-Attentive Context Module (GACM): This module explicitly models the contextual dependencies among semantic information and those among geometric positions, enhancing the model’s joint perception of geometric structure and semantic layout, thereby significantly improving segmentation accuracy and spatial consistency.
• Experimental results on the two public remote sensing datasets, Vaihingen and Potsdam, demonstrate that the proposed GIMMNet outperforms existing models.

2. Related Works

2.1. Semantic Segmentation

Early semantic segmentation primarily relied on Convolutional Neural Network (CNN) architectures. Since Long et al. introduced FCN [11], end-to-end CNN-based semantic segmentation methods have rapidly become the dominant paradigm in this field. Subsequent works have continuously optimized segmentation performance through encoder–decoder structures (U-Net [12]), dilated convolutions (DeepLab series [13]), multi-scale context aggregation (PSPNet [14]), and attention mechanisms (like CCNet [15], EncNet [16]). These methods typically use CNN backbone networks such as ResNet [17] and HRNet [18] for feature extraction, combined with skip connections or refined upsampling strategies to recover spatial details. Additionally, to address the high cost of annotation, weakly supervised learning approaches have also been widely explored; for practical deployment, lightweight models like ICNet [19], BiSeNet [20], Fast-SCNN [21] have significantly improved inference efficiency while maintaining accuracy. Overall, CNNs have established a solid foundation in single-modal RGB image semantic segmentation due to their local perception, translation invariance, and computational efficiency.

In recent years, with the success of ViT [22], semantic segmentation methods based on Transformers [23] have gradually emerged. These methods effectively overcome the limitation of CNNs’ limited receptive fields by leveraging self-attention mechanisms. Representative works such as SETR [24] were the first to apply pure Transformers to segmentation tasks, achieving highly consistent semantic predictions through global context modeling; SegFormer [25] introduces hierarchical Transformer encoders and lightweight decoders; others like UPerNet [26] and Mask2Former [27] further integrate pyramid features and dynamic mask mechanisms, continuously pushing performance improvements. Although Transformers excel at modeling global relationships, their computational overhead and reliance on large-scale data remain challenges. Current trends lean towards designing more efficient attention variants (such as window attention in Swin Transformer [28]) or combining CNNs with Transformers to fully exploit their complementary strengths.

2.2. Remote Sensing Multimodal Semantic Segmentation

Unlike natural scene images, remote sensing data typically encompasses rich multimodal information, such as visible (RGB), near-infrared (NIR), thermal infrared, synthetic aperture radar (SAR), LiDAR point clouds, and digital surface models (DSM/DTM). Among these, the combination of optical imagery and normalized digital surface models (nDSM) has gradually drawn increasing attention due to their strong complementarity at both semantic and geometric levels: optical images provide rich spectral details from a nadir perspective, while nDSM effectively captures the 3D height structure of ground objects.

To fully exploit this complementarity, numerous multimodal fusion methods have emerged in recent years. CIMFNet [10] captures multiscale contextual features through hierarchical feature interaction. However, it relies heavily on addition and multiplication operations and repeatedly stacks identical modules in the latter part of the network, resulting in structural redundancy and poor interpretability. EDGNet [7] leverages boundary priors from optical images to alleviate modality distribution discrepancies but fails to exploit the geometric boundary cues inherently present in nDSM. Moreover, it employs identical feature extraction backbones for both modalities, lacking differentiated modeling for auxiliary modalities like nDSM. Similarly, G2GNet [8] and SERNet [9] fuse only the deepest semantic features output by the backbone networks, overlooking the critical role of mid- and low-level spatial details in boundary refinement.

To enhance multi-scale collaboration, Fan et al. proposed PACSCNet [5], which adopts a symmetric architecture in the fusion stage to facilitate cross-scale information flow, yet still employs the same ResNet-50 backbone for both optical and nDSM modalities during feature extraction. Given the relatively low information density of nDSM, an overly deep network branch tends to over-smooth or even lose its structural details. Recently, FtransUNet [29] attempted to combine CNNs with Transformers by using a Transformer to fuse high-level semantic features and applying channel-wise weights—derived from globally averaged pooled vectors—to reweight features from the other modality. However, this approach inadequately models the dynamic interplay and inherent differences between modalities and suffers from excessive model parameters, hindering practical deployment.

Although the aforementioned methods achieve effective fusion of optical and nDSM modalities, they generally overlook the geometric prior knowledge embedded in nDSM—particularly the intrinsic relationship between object height structures and semantic layouts. This lack of explicit modeling of the geometry–semantics coupling limits the model’s segmentation performance on fine-grained objects (e.g., low-rise buildings, densely packed rooftops, bridges).

3. Materials and Methods

3.1. Overall Framework

The GIMMNet architecture is illustrated in Figure 2, adopting an encoder–decoder structure. Considering that the nDSM modality contains only elevation values and exhibits a significantly lower information density compared with optical images, we employ a lightweight convolutional branch to extract features from the nDSM modality, while using EfficientNet-B2 [30] as the primary feature extractor for optical images. The two feature extraction branches are each divided into four stages, corresponding to layers 1 to 4 in the figure. The features extracted at each stage represent progressively deeper spatial details and semantic abstraction. We denote the optical modality features at stage i as $z_i^{\left(opt\right)}$ and the nDSM modality features at stage i as $z_i^{\left(nDSM\right)}$, where $i=1,2,3,4$.

3.2. Geometric Position Prior Module (GPPM)

Geometric position information plays a critical role in both major deep learning architectures: CNNs and Transformers. CNNs implicitly model 2D spatial relationships through local receptive fields, whereas Transformers require explicit positional encodings to preserve spatial order [22]. To fully exploit 3D geometric priors, we propose constructing pixel-wise 3D coordinates from the nDSM image and extracting multi-scale geometric position features via a hierarchical convolutional network.

For a pixel located at row i and column j in the nDSM image, denoted as ${\mathbf{p}}_{i,j}$, its 3D coordinate is defined as

$${\mathbf{p}}_{i,j}=(i,j,h_{i,j})$$

(1)

where $h_{i,j}$ denotes the normalized elevation value at that pixel. By stacking the 3D coordinates of all pixels in spatial order, we construct a 3D coordinate tensor ${\mathbf{I}}_{\mathrm{pos}}\in {\mathbb{R}}^{H\times W\times 3}$, with H and W representing the height and width of the input image, respectively. Since the 2D spatial coordinates and elevation values differ significantly in numerical scale, we normalize them separately to map both into a unified numerical range, ensuring balanced treatment of all dimensions in subsequent network layers.

Since the 3D coordinates encode both spatial location and elevation information, they carry richer semantic content than conventional 2D coordinates. Therefore, we treat ${\mathbf{I}}_{\mathrm{pos}}$ as a unified input and extract multi-scale geometric priors through a four-stage convolutional encoder. To maintain spatial alignment with the optical (OPT) branch, each stage employs a $3\times 3$ convolution with stride $s=2$ and padding $=1$, followed by a ReLU activation function, to achieve downsampling and capture hierarchical geometric features.

The resulting multi-scale geometric position features are denoted as ${\mathbf{z}}_1^{\mathrm{pos}},{\mathbf{z}}_2^{\mathrm{pos}},{\mathbf{z}}_3^{\mathrm{pos}},{\mathbf{z}}_4^{\mathrm{pos}}$, whose spatial resolutions match those of the corresponding stages in the OPT branch:

$${\displaystyle \frac{H}{2}}\times {\displaystyle \frac{W}{2}}\times C_1, {\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}\times C_2, {\displaystyle \frac{H}{8}}\times {\displaystyle \frac{W}{8}}\times C_3, {\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}\times C_4,$$

where $C_1,C_2,C_3,C_4$ are the output channel dimensions at each stage, respectively.

By processing the 3D coordinates through this multi-scale convolutional pipeline, GPPM explicitly models the dependencies among pixels in both spatial and elevation dimensions, laying the foundation for the two subsequent modules.

3.3. Geometry-Guided Disentangled Fusion Module (GDFM)

In multimodal learning, feature representations can be categorized into two complementary types: shared semantic features, which reflect the common perception of the same land cover across different modalities, and modality-specific features, which capture the unique discriminative capabilities of individual sensors. Effectively disentangling and fusing these two types is essential for improving segmentation accuracy [31]. To this end, we propose the Geometry-Guided Disentangled Fusion Module (GDFM), as illustrated in Figure 3. GDFM leverages geometric position priors as a “probe” to quantify the response discrepancy between optical (OPT) and nDSM modalities to 3D spatial structures, enabling adaptive disentanglement and fusion of shared and specific features. This mechanism simultaneously enhances modality-specific representation and promotes cross-modal semantic alignment.

Specifically, GDFM consists of three key steps:

• Shared Feature Quantification: Feature Similarity Vector

Let ${\mathbf{z}}_i^{\left(\mathrm{opt}\right)},{\mathbf{z}}_i^{\left(\mathrm{nDSM}\right)}\in {\mathbb{R}}^{H\times W\times C}$ denote the OPT and nDSM features at stage i, respectively. We first apply Global Average Pooling (GAP) to both:

$${\mathbf{v}}_i^{\left(\mathrm{opt}\right)}=\mathrm{GAP}\left({\mathbf{z}}_i^{\left(\mathrm{opt}\right)}\right), {\mathbf{v}}_i^{\left(\mathrm{nDSM}\right)}=\mathrm{GAP}\left({\mathbf{z}}_i^{\left(\mathrm{nDSM}\right)}\right)$$

(2)

This yields global feature vectors of shape $1\times 1\times C$. The feature similarity vector is then computed as their element-wise product followed by a Sigmoid activation:

$${\mathbf{r}}_i=\sigma \left({\mathbf{v}}_i^{\left(\mathrm{opt}\right)}\odot {\mathbf{v}}_i^{\left(\mathrm{nDSM}\right)}\right)$$

(3)

where ⊙ denotes element-wise multiplication and $\sigma (·)$ is the Sigmoid function. The resulting vector ${\mathbf{r}}_i\in {\mathbb{R}}^{1\times 1\times C}$ captures the strength of shared semantic responses across channels.

• Modality-Specific Feature Quantification: Positional Response Discrepancy Vector

Using the geometric position embedding ${\mathbf{z}}_i^{\left(\mathrm{pos}\right)}\in {\mathbb{R}}^{H\times W\times C}$ at the same scale as a geometric prior, we compute the modality-specific responses to 3D structure via element-wise multiplication:

$${\mathbf{R}}_i^{\left(\mathrm{opt}\right)}={\mathbf{z}}_i^{\left(\mathrm{opt}\right)}\odot {\mathbf{z}}_i^{\left(\mathrm{pos}\right)}, {\mathbf{R}}_i^{\left(\mathrm{nDSM}\right)}={\mathbf{z}}_i^{\left(\mathrm{nDSM}\right)}\odot {\mathbf{z}}_i^{\left(\mathrm{pos}\right)}$$

(4)

The positional response discrepancy vector is obtained by summing the absolute differences over spatial dimensions and applying Sigmoid:

$${\mathbf{d}}_i=\sigma \left(\sum _{h=1}^H\sum _{w=1}^W\left|{\mathbf{R}}_i^{\left(\mathrm{opt}\right)}(h,w,:)-{\mathbf{R}}_i^{\left(\mathrm{nDSM}\right)}(h,w,:)\right|\right)$$

(5)

This vector ${\mathbf{d}}_i\in {\mathbb{R}}^{1\times 1\times C}$ measures the confidence that each channel corresponds to modality-specific features.

• Fusion Weight Generation and Feature Fusion

We concatenate ${\mathbf{r}}_i$ and ${\mathbf{d}}_i$ along the channel dimension and feed them into a Channel Attention Module (CA), consisting of two fully connected layers with ReLU activation and Sigmoid normalization, to generate modality-specific fusion weights:

$${\mathbf{w}}_{\mathrm{opt}}=\mathrm{CA}\left([{\mathbf{r}}_i;{\mathbf{d}}_i]\right), {\mathbf{w}}_{\mathrm{nDSM}}=\mathbf{1}-{\mathbf{w}}_{\mathrm{opt}}$$

(6)

where $[·;·]$ denotes channel concatenation. The final fused feature is computed as

$${\mathbf{M}}_i={\mathbf{w}}_{\mathrm{opt}}\odot {\mathbf{z}}_i^{\left(\mathrm{opt}\right)}+{\mathbf{w}}_{\mathrm{nDSM}}\odot {\mathbf{z}}_i^{\left(\mathrm{nDSM}\right)}$$

(7)

The multi-scale fused features $\{{\mathbf{M}}_2,{\mathbf{M}}_3,{\mathbf{M}}_4\}$ are subsequently passed to the decoder to produce the final semantic segmentation map of geospatial objects.

3.4. Geometry-Attentive Context Module (GACM)

The spatial distribution of objects in remote sensing imagery follows specific geographical and physical laws: roads typically connect buildings or intersections, farmlands are often located at urban peripheries or along rivers, and water bodies tend to be situated in low-lying areas adjacent to wetlands and vegetation zones [32]. Such spatial prior knowledge is crucial for semantic reasoning. While CNNs struggle with long-range dependencies due to local receptive fields, Transformers’ self-attention excels at capturing global context. Inspired by this, we design the Geometry-Attentive Context Module (GACM), which fuses semantic context and geometric context to explicitly model the dependencies between object categories and three-dimensional spatial structures, thereby improving both spatial reasoning and semantic segmentation accuracy in complex scenes.

The core idea of GACM is to introduce geometric position embeddings derived from nDSM as an auxiliary context signal, based on the standard self-attention mechanism, and adaptively fuse the two types of attention responses through learnable weights. Its structure is shown in Figure 4, and it consists of the following four steps:

• Q, K, V Generation

For the fused feature ${\mathbf{M}}_i\in {\mathbb{R}}^{H\times W\times C}$ at stage i, $1\times 1$ convolutions (stride 1, padding 0) are used to generate Query, Key, and Value features:

$${\mathbf{Q}}_M,{\mathbf{K}}_M,{\mathbf{V}}_M=f_{\mathrm{conv}}\left({\mathbf{M}}_i\right)$$

(8)

where $f_{\mathrm{conv}}$ denotes the $1\times 1$ convolution operation, and the output channel number is set to d (e.g., $d=256$) to ensure dimensional consistency for subsequent attention computations.

Meanwhile, for the geometric position embedding ${\mathbf{z}}_i^{\left(\mathrm{pos}\right)}\in {\mathbb{R}}^{H\times W\times 3}$ at the same scale, only $1\times 1$ convolutions are applied to generate its corresponding Query and Key:

$${\mathbf{Q}}_{\mathrm{pos}},{\mathbf{K}}_{\mathrm{pos}}=f_{\mathrm{conv}}^{\mathrm{geo}}\left({\mathbf{z}}_i^{\left(\mathrm{pos}\right)}\right)$$

(9)

where $f_{\mathrm{conv}}^{\mathrm{geo}}$ is an independent lightweight convolution layer, with the output dimension also set to d.

• Dual Visual Attention Computation

Semantic Context Attention ${\mathbf{G}}_M$: Measures the spatial-semantic associations within the fused feature ${\mathbf{M}}_i$, computed as

$${\mathbf{G}}_M=\mathrm{Softmax}\left({\displaystyle \frac{{\mathbf{Q}}_M{\mathbf{K}}_M^{\top }}{\sqrt{d}}}\right)$$

(10)

where $\sqrt{d}$ is a scaling factor used to improve the numerical stability of attention weights, and ${\mathbf{G}}_M\in {\mathbb{R}}^{HW\times HW}$ is the attention matrix in the spatial dimension.

Geometric Context Attention ${\mathbf{G}}_{\mathrm{pos}}$: Measures the spatial-height associations encoded by the geometric position embedding. The computation is given by

$${\mathbf{G}}_{\mathrm{pos}}=\mathrm{Softmax}\left({\displaystyle \frac{{\mathbf{Q}}_{\mathrm{pos}}{\mathbf{K}}_{\mathrm{pos}}^{\top }}{\sqrt{d}}}\right)$$

(11)

where ${\mathbf{G}}_{\mathrm{pos}}\in {\mathbb{R}}^{HW\times HW}$, consistent with the dimensions of ${\mathbf{G}}_M$ for subsequent fusion.

• Adaptive Weighted Fusion and Feature Enhancement

To balance the contributions of semantic and geometric contexts, a learnable parameter $\alpha \in [0,1]$, initialized to 0.5, is introduced to dynamically adjust through backpropagation. The two types of attention weights are linearly fused to generate an adaptive fused attention matrix:

$${\mathbf{G}}_{\mathrm{fuse}}=\alpha ·{\mathbf{G}}_M+(1-\alpha )·{\mathbf{G}}_{\mathrm{pos}}$$

(12)

Subsequently, ${\mathbf{G}}_{\mathrm{fuse}}$ is applied to the value feature ${\mathbf{V}}_M$, followed by a linear transformation $\mathcal{L}$ to match the channel number with ${\mathbf{M}}_i$. The result is added to the original feature via a residual connection to achieve feature enhancement:

$${\mathbf{M}}_i^{\prime }={\mathbf{M}}_i+\mathcal{L}({\mathbf{G}}_{\mathrm{fuse}}·{\mathbf{V}}_M)$$

(13)

• Multi-Scale Decoding and Output

In the decoding phase, enhanced features from different stages $\{{\mathbf{M}}_2^{\prime },{\mathbf{M}}_3^{\prime },{\mathbf{M}}_4^{\prime }\}$ are progressively fused through upsampling and skip connections. Finally, the output feature map is upsampled to the original input image resolution via bilinear interpolation, yielding the semantic segmentation results for geospatial objects.

4. Results

4.1. Dataset

To validate the proposed method, experiments are conducted on two open-access ISPRS benchmark datasets: Vaihingen and Potsdam. Both contain six semantic classes—impervious surfaces, buildings, low vegetation, trees, cars, and clutter (the number of samples in each class is shown in Figure 5)—and provide high-resolution true orthophotos (TOP) with three spectral bands (near-infrared, red, and green), corresponding nDSM data, and pixel-level labels. The label maps include an eroded-boundary version to mitigate edge ambiguity; following standard practice, this version is used for training, while eroded boundary pixels are excluded during evaluation.

In our experiments, the TOP images (serving as optical remote sensing inputs, denoted as OPT) and their corresponding nDSM images are used as the two input modalities:

• Vaihingen Dataset: Comprises 33 orthophotos with an average size of $2494\times 2064$ pixels and a ground sampling distance (GSD) of 9 cm. Following the official split and standard protocols in prior work, 16 images are used for training and 17 for testing.
• Potsdam Dataset: Contains 38 orthophotos, each of fixed size $6000\times 6000$ pixels and a higher spatial resolution of 5 cm GSD. As per prior studies, 24 images are allocated for training and 14 for testing.

4.2. Implementation Details

All experiments were performed under a consistent configuration using the PyTorch 2.4.1 on an NVIDIA GeForce RTX 3090 GPU. The model was optimized with SGD, using a batch size of 4, an initial learning rate of 0.007, a weight decay of 5 × ${10}^{-3}$, and a momentum of 0.9. Input images were cropped into 512 × 512 patches and augmented via random flipping, rotation, scaling, cropping, and concatenation to improve generalization. Training lasted 300 epochs on the Potsdam dataset and 200 epochs on the Vaihingen dataset. For the optical (OPT) branch, EfficientNet-B2 [30] served as the backbone network, initialized with ImageNet-pretrained weights.

4.3. Evaluation Metrics

We adopt five widely used evaluation metrics for semantic segmentation: Intersection over Union (IoU), F1 score, and overall accuracy (OA). Their definitions are as follows:

$${\mathrm{IoU}}_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c+F{N}_c}}$$

(14)

$$F{1}_c={\displaystyle \frac{2\times {\mathrm{Precision}}_c\times {\mathrm{Recall}}_c}{{\mathrm{Precision}}_c+{\mathrm{Recall}}_c}}$$

(15)

$$OA={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{{\sum }_{c=1}^C(T{P}_c+F{P}_c)}}$$

(16)

where $T{P}_c$, $F{P}_c$, and $F{N}_c$ denote the number of true positive, false positive, and false negative pixels for class c, respectively, and C is the total number of semantic classes.

4.4. Comparative Experiments

We evaluated the proposed GIMMNet against eight representative mainstream methods as baselines, namely DABNet [33], FANet [34], EANet [35], CMTFNet [10], DP-CTNet [36], SA-Gate [37], CMX [38], PACSCNet [5], and FTransUNet [29]. Among these, DABNet, FANet, EANet, CMTFNet, and DP-CTNet are unimodal approaches that utilize only optical imagery, while the remaining methods are multimodal approaches that fuse both optical and nDSM data. The inclusion of unimodal methods aims to reveal the impact of the nDSM modality on segmentation performance. Comprehensive comparative experiments were conducted on the Vaihingen and Potsdam datasets, with results reported in

Table 1. Comparison results on Vaihingen.

Method	Modality	F1					mF1	mIoU	OA
		Bul.	Imp.	Tre.	Low.	Car.
DABNet [33]	OPT	88.8	87.8	84.9	74.3	60.2	79.2	70.2	84.3
FANet [34]	OPT	93.9	90.0	87.3	82.2	44.5	79.6	69.4	88.2
EANet [35]	OPT	94.5	91.7	89.2	83.1	80.0	87.7	78.7	89.7
CMTFNet [10]	OPT	95.4	92.1	88.6	81.9	84.7	88.6	80.1	90.8
DP-CTNet [36]	OPT	92.4	89.9	78.9	84.7	74.2	84.0	73.0	87.0
SA-Gate [37]	OPT + nDSM	93.6	89.7	88.5	80.5	79.5	86.4	76.7	89.3
CMX [38]	OPT + nDSM	95.7	92.4	88.6	81.1	85.5	88.7	80.3	90.9
PACSCNet [5]	OPT + nDSM	94.8	91.5	88.9	82.5	86.5	88.8	80.5	90.6
FTransUnet [29]	OPT + nDSM	92.2	89.4	86.8	80.1	75.9	84.9	75.0	88.3
GIMMNet (ours)	OPT + nDSM	95.5	92.0	88.9	82.4	85.9	88.9	80.6	91.0

and

Table 2. Comparison results on Potsdam.

Method	Modality	F1					mF1	mIoU	OA
		Bul.	Imp.	Tre.	Low.	Car.
DABNet [33]	OPT	93.2	89.9	82.3	83.6	92.6	88.3	79.6	86.7
FANet [34]	OPT	96.1	92.0	87.8	86.0	94.5	91.3	84.2	89.8
EANet [35]	OPT	95.7	92.0	85.7	84.3	95.1	90.6	83.4	88.7
CMTFNet [10]	OPT	95.0	92.7	85.7	84.8	93.7	90.4	82.8	90.7
DP-CTNet [36]	OPT	93.1	88.6	81.4	81.0	87.9	86.4	76.4	86.3
SA-Gate [37]	OPT + nDSM	96.2	91.9	80.5	81.4	90.5	88.1	79.4	89.3
CMX [38]	OPT + nDSM	95.3	92.2	84.2	83.7	92.9	89.7	81.7	90.3
PACSCNet [5]	OPT + nDSM	94.9	91.5	87.2	86.2	92.7	90.5	82.9	90.8
FTransUnet [29]	OPT + nDSM	95.5	91.0	80.2	80.4	86.8	86.8	77.3	88.5
GIMMNet (ours)	OPT + nDSM	96.8	93.5	86.0	86.7	95.9	91.8	85.2	91.9

, respectively. Boldface values indicate the best performance for each metric.

4.4.1. Results on the Vaihingen

As shown in Table 1, in terms of overall metrics, the proposed GIMMNet achieves the best performance with OA of 91.0%, mF1 of 88.9%, and mIoU of 80.6%—all surpassing those of the current state-of-the-art model, PACSCNet (OA: 90.6%, mF1: 88.8%, mIoU: 80.5%).

At the category level, unlike unimodal methods (which typically show significant drops in performance for certain categories), GIMMNet demonstrates stable and balanced results across all land cover categories, showing the effectiveness of introducing the nDSM modality in improving overall segmentation accuracy. Notably, GIMMNet outperforms most existing multimodal methods in segmenting two pairs of spectrally similar and highly confusable classes: Building vs. Impervious Surface and Tree vs. Low Vegetation. This underscores its powerful capability to achieve fine-grained semantic discrimination through deep fusion of geometric and spectral information.

These outstanding results are attributed to GIMMNet’s effective fusion mechanism, which jointly leverages optical imagery and nDSM elevation data. Unlike methods such as CMTFNet that rely solely on unimodal inputs or shallow feature fusion, GIMMNet simultaneously exploits spatial texture cues and 3D structural information. This capability is particularly advantageous for distinguishing land cover classes with pronounced elevation differences—such as buildings versus impervious surfaces—where depth-guided feature enhancement significantly improves inter-class separability. Moreover, compared with SA-Gate and CMX, GIMMNet achieves more balanced per-class segmentation performance while maintaining high overall accuracy, as evidenced by its mF1 score of 88.9%, demonstrating strong stability and generalization across diverse semantic categories.

4.4.2. Results on the Potsdam

On the more complex and larger-scale Potsdam dataset, the proposed GIMMNet also demonstrates superior performance, further validating its strong generalization capability and robustness. As shown in Table 2, GIMMNet achieves outstanding segmentation results on four key classes—Building, Impervious Surface, Car, and Low Vegetation—with F1-scores of 96.8%, 93.5%, 95.9%, and 86.7%, respectively, significantly outperforming all competing methods. Notably, it attains the highest accuracy among all approaches for the building class, highlighting its exceptional ability to recognize buildings with complex structures. For the Car class, GIMMNet achieves the highest F1-score of 95.9%, demonstrating its strength in small-object detection and boundary preservation. The Impervious Surface class also achieves a high score of 93.5%, reflecting its strong adaptability to large, homogeneous regions.

In terms of overall metrics, GIMMNet achieves 91.9% OA, 91.8% mF1, and 85.2% mIoU on the Potsdam dataset, outperforming the second-best method by 1.1% OA, 0.5% mF1, and 1.0% mIoU, respectively. Compared with its performance on the Vaihingen dataset, these results further highlight GIMMNet’s superior performance and strong adaptability in more complex and challenging urban scenes.

Moreover, unlike methods such as SA-Gate and CMX that rely on simple feature concatenation or basic attention mechanisms, GIMMNet explicitly incorporates positional information, significantly enhancing the model’s awareness of global spatial structure. This design not only facilitates deeper fusion of multimodal features but also enables more refined semantic enhancement. As a result, GIMMNet maintains high segmentation accuracy while substantially improving model stability and prediction consistency. With an mF1 of 91.8%, it demonstrates well-balanced performance across all classes—particularly maintaining high accuracy when distinguishing between spectrally and structurally similar, easily confused categories such as Tree and Low Vegetation—showcasing its strong fine-grained discrimination capability.

4.5. Visualization Analysis

To evaluate and analyze model performance more intuitively, this paper conducts visual analysis on both the Vaihingen and Potsdam datasets. To evaluate and analyze model performance more intuitively, this paper conducts visual analysis on both the Vaihingen and Potsdam datasets. The selected competing models include CMTFNet [10], SA-Gate [37], CMX [38], PACSCNet [5], and FTransUNet [29].

4.5.1. Visualization on Vaihingen

Figure 6 shows the visual results of all compared methods on the Vaihingen dataset, covering four representative and challenging scenes. Based on the key regions highlighted by the red bounding boxes, we analyze the performance in each scene as follows:

• Scene (a): Irregular building segmentation

This scene challenges the precise delineation of edges for two irregularly shaped buildings. Our method achieves high-accuracy building segmentation with the fewest fragmented artifacts—particularly in the dark blue building region. In the small connecting structure marked by the red box (a typical small-scale target), our method and SA-Gate yield the best results, demonstrating superior capability in capturing fine architectural details at small scales.

• Scene (b): “T”-shaped impervious surface segmentation

The ground truth (GT) labels a “T”-shaped impervious surface within the red box, exhibiting complex geometry. Only FTransUNet and our method fully preserve the protruding segment in the upper-right corner; all other approaches suffer from partial or complete structural loss. This validates the robustness and geometric fidelity of our method when handling objects with intricate shapes.

• Scene (c): Multi-scale buildings and fine structural details

The red box encloses a small, isolated building adjacent to a large structure, serving as a test for multi-scale segmentation capability. Only our method and PACSCNet accurately segment this small target, with our approach producing cleaner and more complete boundaries. Moreover, only CMX, PACSCNet, and our method successfully and accurately reconstructed the vertically aligned parallel linear impervious surfaces beneath the building, further demonstrating the superiority of our approach in jointly handling objects at varying scales and fine geometric structures.

• Scene (d): Dense vehicles and right-angle building edges

In dense vehicle scenarios, all comparison methods suffer from either vehicle merging or missed detections, whereas our method clearly and independently segments each individual car. Moreover, at the right-angled building corners in the ground-truth (GT) annotations (highlighted within the red box), only our method accurately captures the straight, sharp edges of the building, fully demonstrating its superior performance in distinguishing small objects and preserving geometric boundaries.

4.5.2. Visualization on Potsdam

The visualization results on Potsdam are shown in Figure 7. Given that the Potsdam dataset exhibits greater land cover complexity and more intricate spatial configurations compared to Vaihingen, we further conducted an error visualization analysis on Potsdam, as shown in Figure 8, and performed a qualitative assessment in conjunction with the segmentation results in Figure 7.

• Scene (a): Multi-class mixed region

The left and right regions of this scene represent a typical multi-class mixed area, where buildings, trees, low vegetation, and impervious surfaces are highly intermingled. The spatial layout is cluttered, and class boundaries are irregular and ambiguous, forming an extremely challenging region for accurate prediction. All competing methods exhibit significant misclassifications in these two zones, and our method is no exception. However, it is evident that our approach produces the fewest red error pixels in this region, demonstrating superior discriminative capability.

• Scene (b): Mixed area of low vegetation and trees

The right portion of the image contains densely interwoven low vegetation and trees. Since these two land cover types exhibit similar spectral signatures in optical imagery, this region is primarily used to evaluate whether multimodal methods can effectively leverage elevation information from the nDSM modality to achieve accurate discrimination. Among all compared approaches, our method yields the fewest misclassified pixels in this area. Notably, the single-modality method CMTFNet achieves performance comparable to ours, whereas other multimodal methods produce significantly more errors due to insufficient fusion or ineffective utilization of elevation cues.

Furthermore, near the center-bottom of the image lies an approximately elliptical building. The nDSM data reveal a distinct cross-shaped elevation discontinuity on its roof. Influenced by this structural characteristic, most multimodal methods erroneously classify parts of the roof as impervious surfaces. In contrast, both our method and CMTFNet successfully segment the entire building, demonstrating superior segmentation accuracy.

4.6. Model Complexity Analysis

This paper employs three commonly used metrics to evaluate model complexity: the number of parameters, Multiply-Accumulate Operations (MACs), and Frames Per Second (FPS). The number of parameters reflects the model’s size and memory footprint—smaller values typically indicate a lighter architecture. MACs measure the computational cost of a single forward pass; lower values signify reduced computational load. FPS represents inference speed, with higher values indicating better real-time performance. The results are presented in

Table 3. The computational complexity analysis is based on measurements obtained by processing 512 × 512 images on a single NVIDIA GeForce RTX 3090 GPU. The mF1 scores are reported on the Potsdam dataset. Boldface values indicate the best performance for each metric.

Method	Modality	Parameter (M)	MACs (G)	FPS	mF1
CMTFNet	OPT	30.1	8.5	51.7	90.4
SA-Gate	OPT + nDSM	63.4	164.6	20.6	88.1
CMX	OPT + nDSM	66.6	56.9	24.4	89.7
PACSCNet	OPT + nDSM	90.0	150.0	18.5	90.5
FTransUnet	OPT + nDSM	160.9	56.0	19.4	86.8
GIMMNet (ours)	OPT + nDSM	25.2	40.7	31.7	91.8

.

As shown in Table 3, the proposed GIMMNet achieves the highest F1 score and the fewest number of parameters among all compared methods, clearly demonstrating its lightweight advantage. Although GIMMNet does not outperform the single-modality method CMTFNet in terms of MACs and FPS, it consistently surpasses other multimodal approaches—including SA-Gate, CMX, PACSCNet, and FTransUNet—on both metrics, highlighting its superior computational efficiency.

4.7. Ablation Study

To validate the effectiveness of the key components in the proposed GIMMNet, we conduct ablation experiments on the Vaihingen dataset for the three modules: GPPM, GDFM, and GACM. Specifically, for GPPM—whose core functionality lies in leveraging 3D geometric priors—we replace the input nDSM elevation data with a zero tensor during ablation to completely eliminate the 3D geometric prior. For GDFM, we replace its dual-modality feature fusion mechanism with a simple channel concatenation followed by a convolutional layer to assess the necessity of its designed fusion strategy. As for GACM, we remove this module entirely during the upsampling stage and instead employ a conventional upsampling approach to process multi-scale features, thereby evaluating its contribution to feature enhancement. The ablation results for each module are reported in

Table 4. Ablation study results of each module on the Vaihingen dataset. Boldface values indicate the best performance for each metric.

GPPM	GDFM	GACM	mIoU	mF1	OA
			77.3	86.8	89.7
	✓	✓	78.6	87.7	90.0
✓	✓		79.8	88.4	90.5
✓		✓	80.0	88.6	90.8
✓	✓	✓	80.6	89.0	91.0

.

As shown in Table 4, all three modules proposed in this paper—GPPM, GDFM, and GACM—demonstrate significant and positive contributions to model performance. In particular, the ablation study on GPPM reveals that when the elevation information provided by nDSM is removed and the model relies solely on 2D geometric features, performance drops noticeably across all evaluation metrics. This clearly demonstrates that effectively leveraging 3D geometric information substantially enhances the model’s ability to perceive spatial structures, thereby improving semantic segmentation accuracy.

Furthermore, comparing variants that employ only GDFM or only GACM shows that the full GIMMNet consistently achieves performance gains across all three core metrics—mIoU, mF1, and OA—albeit to varying degrees. This strongly validates the effectiveness of explicitly integrating the geometric priors offered by the nDSM modality into semantic segmentation.

Specifically, GDFM addresses the disparity in 3D perception capabilities between different modalities by adopting a “feature disentanglement–re-fusion” strategy, enabling more comprehensive and refined cross-modal fusion. This mechanism significantly strengthens the model’s capacity to model spatial structures, allowing it to more accurately recognize critical 3D characteristics such as building outlines and height variations. Moreover, GDFM goes beyond local geometric modeling; by incorporating a long-range contextual modeling mechanism, it effectively captures intrinsic relationships among distant geometric locations, thereby reinforcing the spatial consistency and structural integrity of ground objects at a global scale.

In summary, GPPM, GDFM, and GACM are complementary and work synergistically: GPPM establishes a solid foundation for geometric perception by incorporating reliable 3D geometric priors; building upon this, GDFM leverages these priors to achieve more comprehensive and refined multimodal feature fusion; meanwhile, GACM further exploits the internal relational cues within the 3D geometric priors to enhance the model’s sensitivity to fine-grained geometric structures. Together, these three modules form an integrated and cohesive framework that serves as the core driving force behind GIMMNet’s high-performance multimodal fusion capability.

5. Discussion

5.1. The Effectiveness of 3D Geometric Priors

This paper thoroughly explores the 3D geometric priors embedded in the nDSM modality and proposes three core modules: GPPM, GDFM, and GACM. Specifically, GPPM is dedicated to extracting pixel-wise 3D geometric priors from the nDSM, while GDFM and GACM effectively embed these geometric priors into the semantic segmentation process from complementary perspectives.

As shown in the ablation study in Table 4, leveraging height information from the nDSM for geometric prior extraction yields significantly better performance than the variant without height information: mF1 improves by 1.3%, mIoU by 2.0%, and OA by 1.0%. This result strongly validates the central premise of this work—that the 3D geometric priors inherent in nDSM can effectively guide semantic segmentation and enhance the model’s capacity to understand spatial structures.

Further analysis reveals that GDFM and GACM are complementary in embedding geometric priors. As illustrated in the figure, adding GACM to a model that uses only GDFM—or conversely, adding GDFM to a model that uses only GACM—consistently leads to performance gains across all evaluation metrics. This demonstrates that the two modules jointly enhance the model’s ability to learn geometry-aware semantic representations, with GDFM focusing on cross-modal feature fusion and GACM emphasizing contextual modeling of geometric relationships.

5.2. Future Directions

The generation of the nDSM modality is influenced by a variety of physical factors, including sensor type, imaging geometry, and surface reflectance properties, which can introduce significant uncertainty into the data quality. Future work should deepen the understanding of the physical mechanisms underlying nDSM formation and systematically analyze its error sources and uncertainty distributions. This would enable the construction of more reliable and robust representations of 3D geometric priors, thereby improving model adaptability in complex terrains or under low-quality nDSM conditions.

Moreover, the current GPPM employs a relatively simplistic approach to modeling 3D geometric priors—primarily relying on pixel-wise coordinate extraction. Future efforts could leverage remote sensing–specific foundation models to more comprehensively and deeply uncover the geometric structures and semantic relationships embedded within nDSM, thereby enriching the expressiveness of geometric priors. Moreover, we plan to systematically evaluate the proposed method across different backbone networks (e.g., ResNet-50, ViT-Tiny, etc.) to further validate its generalizability and robustness and to provide practical guidance for model selection in real-world applications.

Finally, it is essential to explore more efficient and adaptive mechanisms for embedding geometric priors. Such approaches should aim to maintain high segmentation performance while reducing computational overhead and enhancing generalization across heterogeneous scenarios—such as varying geographic regions or spatial resolutions—thus advancing geometry-guided multimodal semantic segmentation toward practical deployment and lightweight implementation.

6. Conclusions

This paper focuses on a key limitation in existing multimodal remote sensing segmentation methods: the underutilization of 3D geometric structures in nDSM, which are often treated as an additional spectral channel rather than explicit spatial priors. To this end, we propose GIMMNet, a geometry-aware interactive multimodal network that explicitly models and leverages geometric information throughout the segmentation pipeline.

Specifically, unlike conventional approaches, GIMMNet first reconstructs per-pixel 3D coordinates from nDSM via the Geometric Position Prior Module (GPPM), transforming elevation into a structured geometric context. It then resolves modality conflicts through the Geometric-guided Decoupled Fusion Module (GDFM), which dynamically balances optical and nDSM features based on their differential responses to geometric cues—enabling adaptive fusion rather than naive concatenation. Finally, the Geometric Attention Context Module (GACM) explicitly captures the dependencies between semantic categories and 3D structures during decoding, enhancing boundary coherence and spatial consistency.

Experiments on the Vaihingen and Potsdam benchmarks show that GIMMNet achieves strong segmentation performance, with mIoU scores of 80.6% and 85.2%, respectively—outperforming several recent multimodal methods. These results validate that explicit geometric modeling, rather than merely using multimodal inputs, is crucial for high-precision semantic segmentation in complex urban scenes.