Multimodal Image Segmentation with Dynamic Adaptive Window and Cross-Scale Fusion for Heterogeneous Data Environments

Abstract

Multi-modal image segmentation is a key task in various fields such as urban planning, infrastructure monitoring, and environmental analysis. However, it remains challenging due to complex scenes, varying object scales, and the integration of heterogeneous data sources (such as RGB, depth maps, and infrared). To address these challenges, we proposed a novel multi-modal segmentation framework, DyFuseNet, which features dynamic adaptive windows and cross-scale feature fusion capabilities. This framework consists of three key components: (1) Dynamic Window Module (DWM), which uses dynamic partitioning and continuous position bias to adaptively adjust window sizes, thereby improving the representation of irregular and fine-grained objects; (2) Scale Context Attention (SCA), a hierarchical mechanism that associates local details with global semantics in a coarse-to-fine manner, enhancing segmentation accuracy in low-texture or occluded regions; and (3) Hierarchical Adaptive Fusion Architecture (HAFA), which aligns and fuses features from multiple modalities through shallow synchronization and deep channel attention, effectively balancing complementarity and redundancy. Evaluated on benchmark datasets (such as ISPRS Vaihingen and Potsdam), DyFuseNet achieved state-of-the-art performance, with mean Intersection over Union (mIoU) scores of 80.40% and 80.85%, surpassing MFTransNet by 1.91% and 1.77%, respectively. The model also demonstrated strong robustness in challenging scenes (such as building edges and shadowed objects), achieving an average F1 score of 85% while maintaining high efficiency (26.19 GFLOPs, 30.09 FPS), making it suitable for real-time deployment. This work presents a practical, versatile, and computationally efficient solution for multi-modal image analysis, with potential applications beyond remote sensing, including smart monitoring, industrial inspection, and multi-source data fusion tasks.

1. Introduction

Multimodal image semantic segmentation is a fundamental technique for extracting pixel-level semantic information from complex image data [1], particularly when dealing with heterogeneous modalities (e.g., optical, radar, and elevation data [2]). By assigning semantic labels to objects and generating high-resolution segmentation maps [3], this technology serves as a key enabler for a wide range of applications, including urban development [4], natural resource surveying [5], environmental analysis and infrastructure monitoring [6], intelligent transportation systems [7], and public safety management [8]. Its significance extends not only to strategic initiatives such as smart city construction and integrated monitoring systems but also to practical tasks involving scene understanding and object recognition.

With the rapid development of remote sensing and aerial imaging platforms, modern datasets increasingly feature high-resolution images containing dense small objects, complex scene layouts, and significant variations in object size and scale. In this context, overlapping structures (such as buildings and vehicles in urban environments) pose significant challenges for accurate segmentation. Effectively addressing these challenges requires not only modeling the spatial relationships between targets but also leveraging the complementary information provided by various data modalities. These demands highlight the need for advanced, generalizable segmentation methods capable of handling complex, multi-modal, and high-resolution visual inputs. Traditional single-modal semantic segmentation methods, which depend on manual feature extraction, exhibit limited automation and inadequate accuracy when dealing with complex scenes [9]. Although deep learning-based methods, including convolutional neural networks (CNNs) (e.g., U-Net [10], PSPNet [11]) and transformers (e.g., ViT [12], Swin-Transformer [13], Swin-Unet [14]), have made substantial progress in feature extraction and global context modeling, they are still restricted by the limited information intrinsic to unimodal data. This limitation restricts their ability to effectively address the segmentation challenges in complex scenarios (for example, overlapping objects of different scales and heterogeneous data sources) [15], highlighting the necessity of multimodal fusion strategies that leverage the complementarity of multi-source data.

Multimodal data (such as RGB spectral data, DSM elevation data, infrared images, etc.) can more comprehensively describe surface features through complementary heterogeneous information, thereby improving segmentation robustness. At the same time, multiscale contextual information (for example, the dense connection mechanism in DFCN [16] and the local–global feature mining strategy in CMFNet [17]) has been proven to be crucial for achieving high segmentation accuracy. However, the existing multimodal fusion-based semantic segmentation methods exhibit significant limitations: the relevance of different modalities and semantic contexts is essentially unequal [18], and naively connecting multimodal data may lead to incompatible feature spaces due to heterogeneous statistical properties, thus reducing segmentation performance [19,20]. Although RDFNet [21] and HRNet [22] attempted to improve fusion strategies, and ACNet [23] introduced attention mechanisms to enhance contextual information association, challenges such as modal-specific feature neglect and feature redundancy still exist. Furthermore, Transformer-based approaches (e.g., MFTransNet [24] and STransFuse [25]) enhance performance by modeling long-range dependencies. Nevertheless, their single-scale window mechanism is inadequate for capturing the multiscale characteristics of remote sensing objects. Additionally, mismatches between window sizes and feature maps necessitate padding adjustments, which disrupt network structural consistency and compromise computational efficiency.

To address the aforementioned limitations, inspired by ASMFNet [26] and the dynamic window vision transformer [27,28], this study proposes a novel multimodal image segmentation network, DyFuseNet, by leveraging Swin-Transformer as the backbone and exploring multimodal fusion through an adjacent-scale perspective. The key contributions are outlined as follows:

(1)

Dynamic Window Module (DWM): this study presents a dynamic window module, which adjusts the size and position of the window dynamically by integrating continuous positional deviations. This addresses the key limitation of the fixed window strategy in Swin-Transformer that fails to capture the multi-scale features of irregular objects, significantly improving the recognition accuracy of complex terrain targets.

(2)

Cross-Scale Contextual Attention (SCA): this study designed a cross-scale contextual attention module, which employs a strategy from coarse to fine, extracting and integrating features from adjacent scales. It addresses the fundamental challenge of the semantic gap between local details and global context in conventional attention mechanisms, thereby improving the model’s ability to understand spatial relationships in heterogeneous scenarios.

(3)

Hierarchical Adaptive Fusion Architecture (HAFA): a hierarchical adaptive fusion architecture is proposed, which designs a heterogeneous modality synchronizer module (HMS) at the shallow network layer and employs an Efficient Channel Attention (ECA) mechanism at the deep network layer. This effectively fuses complementary multi-modal information while maintaining accuracy and significantly reducing the number of model parameters.

Experiments on the ISPRS Vaihingen and Potsdam datasets show that the method improves the segmentation accuracy of irregular features (such as building edges and road networks), enhances robustness in shadow and low-texture areas, maintains reliable classification in high-variance backgrounds, and achieves an optimal balance between accuracy and processing speed. These results collectively validate the effectiveness of the proposed DyFuseNet in complex remote sensing segmentation tasks.

2. Datasets and Data Preprocessing

This section introduces the characteristics of the Vaihingen and Potsdam datasets, discussing their spatial resolution, spectral composition, geographical coverage, and labeling systems. In addition, it details the processing of the raw datasets in this paper, including the sliding window cropping to address the mismatch between GPU memory limitations and the original high-resolution image ratio, ensuring compatibility with the input requirements of the proposed model.

2.1. Datasets

This section introduces the core features of the Vaihingen and Potsdam datasets, compares key attributes, and clarifies their complementary roles in data complexity and model validation, providing a basis for method adaptability analysis.

2.1.1. Vaihingen Dataset

The Vaihingen dataset comes from aerial images of the city of Vaihingen an der Enz in southwestern Germany, characterized by dense building clusters and a complex spatial configuration of scattered forest areas. This heterogeneity makes it particularly suitable for multi-class land cover classification problems in fragmented scenarios involving urban and vegetation elements.

The dataset contains 33 orthorectified aerial images with a spatial resolution of 9 cm per pixel, capable of capturing detailed features of small-scale objects such as individual buildings, roads, and patches of vegetation. Spectrally, the images are acquired in three bands: near-infrared (NIR), red (R), and green (G) (represented as an IR-R-G band combination), which are widely used to distinguish vegetation (through near-infrared reflectance) and built-up areas (through red and green bands).

Geographically, the area of the Vaihingen region is approximately 0.5 square kilometers, concentrated in residential areas, industrial facilities, and vegetation zones. Full labeled annotations are provided for all 33 images to ensure comprehensive ground truth for model training and evaluation. The dataset includes six semantic classes: roads, buildings, low vegetation, trees, cars, and background, with the first five categories defined as primary foreground objects of interest.

2.1.2. Potsdam Dataset

The Potsdam dataset comes from aerial survey images of the city center area of Potsdam, a historic city in Germany, known for its densely arranged large buildings and narrow streets, which presents unique challenges for segmentation tasks (such as distinguishing closely adjacent structures and fine urban features). This dataset consists of 38 orthorectified aerial images, with a higher spatial resolution of 5 cm per pixel, providing finer details compared to the Vaihingen dataset and assisting in the analysis of small-scale objects.

In the spectrum, the Potsdam images expand the Vaihingen band combination by including an additional blue (B) channel, thus forming a four-band configuration (NIR, R, G, and B). The addition of the blue band enhances contrast differentiation and improves the robustness of feature extraction in complex urban scenes. Furthermore, out of 38 images, 24 are fully labeled, serving as the primary source for model training and validation. The annotations follow the same six-class semantic system as the Vaihingen dataset, with consistent definitions for foreground and background categories. A key advantage of the Potsdam dataset lies in its high-resolution four-band imagery and dense urban structure, which provides a complementary contrast to the urban-forest mixed complexity and full label coverage of the Vaihingen dataset.

2.2. Data Preprocessing

In order to address the GPU memory limitations associated with handling high-resolution images and to align the large-scale input data with the input requirements of the proposed model, a sliding window clipping strategy is used for dataset preprocessing.

The original image is divided into 512 × 512 pixel sub-images with a stride of 512 pixels. This stride is chosen to balance computational efficiency and contextual continuity: a smaller stride reduces edge discontinuities within segmented objects, while a larger stride minimizes redundant computation. The sliding window method effectively suppresses boundary artifacts that may arise from sudden transitions at the edges of cropped sub-images, thereby improving the model’s generalization ability to different spatial distributions of ground objects. Additionally, to ensure the statistical independence and balanced representation of each semantic class, a stratified sampling strategy is used to randomly split the generated sub-images into training (70%), validation (20%), and testing (10%) sets. This method maintains the proportional distribution of foreground and background classes across all subsets, thus preventing bias in model evaluation and ensuring reliable assessment of the model’s multi-class segmentation performance.

By integrating the complementary attributes of the Vaihingen and Potsdam datasets with targeted preprocessing (sliding-window clipping and stratified splitting), this study establishes a robust and scalable dataset framework that enables rigorous validation of the proposed method’s effectiveness in multi-scale, multimodal remote sensing image analysis.

3. Methods

In response to the challenges of heterogeneity in DSM (Digital Surface Model) and IRRB (Infrared Radiation Brightness Image) images and the problem of multi-scale object segmentation, a multi-modal segmentation network called DyFuseNet based on dynamic windows and cross-scale fusion is proposed. As shown in Figure 1, the workflow of this algorithm consists of the following key steps:

(1)

Dual-branch Feature Extraction: the workflow starts with the input image and employs a parallel dual-branch architecture for multimodal feature encoding. Left branch: The IRRB image is processed using a Swin-T encoder integrated with DWM to extract the intermediate feature $R_i$, which is then further refined through the SCA module to obtain $R_h$. Right branch: The DSM image is processed using a unique Swin-T encoder integrated with the DWM to generate the intermediate feature $D_i$, which is then refined with SCA to produce $D_h$.

(2)

Cross-modal Feature Fusion: after obtaining the high-dimensional features $R_h$ and $D_h$ from the dual branches, the workflow proceeds to two core fusion stages: HMS and ECA. The HMS performs an initial fusion of $R_h$(from the optical/infrared branch) and $D_h$ (from the DSM branch), enabling the complementary integration of features from these two heterogeneous data sources. Subsequently, the ECA module applies channel—and/or spatial—wise attention weighting or feature reorganization to the features output by HMS, facilitating deeper interaction and integration of multimodal information, thereby providing more discriminative fused features for the subsequent decoding stage.

(3)

Decoding and Output: after feature extraction by the dual branches and cross-modal fusion, the final features are fed into the Swin-T Decoder for upsampling and refined classification operations, yielding the final output.

DyFuseNet is an end-to-end network, and its overall network framework is shown in Figure 2. Let the IRRB image and DSM image be used separately $R\in R^{C_R\times H\times W}$ and $D\in R^{C\times H\times W}$, where $H$ and $W$ represent the height and width of the image, respectively, and $C$ represents the channel size. $R_0$ is obtained by dividing $R$ into different patches with positional embeddings, and through several consecutive stages, its output form is $I_i\in R^{2^{i}C\times {\displaystyle \frac{H}{2^{i+2}}}\times {\displaystyle \frac{W}{2^{i+2}}}}$, where $i$ represents the number of layers, $C=96$. Similarly, the DSM encoder $R_i$ will export $D_i$ with the same size.

3.1. Dynamic Window Module

In the field of computer vision, the window attention mechanism has become a major design paradigm due to its inherent trade-off between computational efficiency and modeling capability. However, traditional fixed window strategies exhibit significant limitations when handling multi-scale visual analysis tasks, greatly restricting the generalization ability of the model.

To overcome this limitation, this study proposes a Dynamic Window Module. The specific key algorithms can be found in Appendix B.1. As illustrated in Figure 3, DWM integrates three key sub-modules: the Adaptive Window Partitioning (AWP) module, the Continuous Relative Position Bias Generator (CRPB) module, and the Multi-scale Dynamic Window Attention (MDWA) module.

(1)

Adaptive Window Partitioning (AWP): dynamically determine the optimal window size based on the input feature map dimensions to adapt to the characteristics of remote sensing images, ensuring that the window size evenly divides the feature map dimensions while eliminating redundant padding.

(2)

Continuous Relative Position Bias Generator (CRPB): use a lightweight multi-layer perceptron (MLP) to replace the pre-computed position bias table, generating position codes that are independent of window size through real-time normalization of spatial coordinates.

(3)

Multi-scale Dynamic Window Attention (MDWA): integrates outputs from the preceding two sub-modules to enable genuine multi-scale perception, effectively addressing the limitations of fixed windows when processing variable-sized feature maps.

3.1.1. Adaptive Window Partitioning Module

By dynamically adjusting the window size, the AWP module improves the calculation efficiency and reduces the semantic separation at the boundary, which is expressed by the mathematical formula:

$$M_d=\max \left\{m\left|m\le M_{base},m\right|H\right\}$$

(1)

$$N_d=\mathrm{m}\mathrm{a}\mathrm{x}\left\{n\left|n\le N_{base},n\right|W\right\}$$

(2)

The height and width of the input feature map are represented as $H$ and $W$, respectively. In addition, $M_{base}$ and $N_{base}$ are predefined basic window sizes, with a default value of (8,8), which has been experimentally verified to be optimal in terms of computational efficiency and semantic integrity. Here, $m$ is a variable related to the window size, and the condition $m|H$ (indicating that m can be divided by $H$) is used to ensure the rational division of windows.

The AWP module takes the size of the input image and the preset basic window information as input, and dynamically determines the optimal window size that can evenly divide the dimensions of the feature map based on the aspect ratio of the feature map and computational efficiency. After calculating the optimal window size, it outputs the actual size of the window for subsequent processing, while segmenting the feature map into corresponding windows based on this data. This module can eliminate the padding requirements of feature maps with different input scales, improve the calculation efficiency, and ensure a consistent number of windows for different input feature maps. By dynamically adjusting the window size, the AWP module alleviates the semantic segmentation issues at the boundaries while maintaining computational efficiency. For the configuration parameters involved in this module and the complete derivation process, please refer to Appendix A.1.

3.1.2. Continuous Relative Position Bias Generator

In this paper, a continuous relative position bias generator module is developed by combining normalized coordinates and employing a lightweight MLP instead of a pre—calculated position bias table. The specific calculation formula is presented as follows:

$$∆h_{ij}=h_i-h_j, ∆w_{ij}=w_i-w_j$$

(3)

$$\widehat{∆}{h}_{ij}={\displaystyle \frac{∆h_{ij}}{M_d-1}},\widehat{∆}{w}_{ij}={\displaystyle \frac{∆2_{ij}}{N_d-1}}$$

(4)

$$B_{ij}=W_2·ReLU(W_1·{\left[\widehat{∆}{h}_{ij},\widehat{∆}{w}_{ij}\right]}^T+b_1)$$

(5)

Let $∆h_{ij}$ and $∆w_{ij}$ be the generated relative coordinates, $\widehat{∆}{h}_{ij}$ and $\widehat{∆}{w}_{ij}$ be the results of coordinate normalization, and $B_{ij}$ be the position offset matrix generated by the Multi-Layer Perceptron (MLP) transformation. In this context, the coordinate normalization operation is performed on the generated relative coordinates. By mapping the physical coordinates (represented by $∆h_{ij}$ and $∆w_{ij}$) to a canonical space through this normalization process, the module enables adaptation to any window size. This adaptability eliminates the need for additional adjustments when dealing with different window sizes, thereby reducing the number of parameters required for the model.

The use of continuous coordinate representation achieved through coordinate normalization allows for more detailed modeling of the positional relationships between elements. This detailed modeling provides a more accurate description of the spatial structure within the feature map, thereby enhancing the model’s ability to extract image information. Furthermore, this module utilizes MLP to learn a generic function that describes positional relationships. This function is capable of generating positional offsets in real time, dynamically adapting to different window sizes. The MLP is a standard multilayer perceptron, its specific details and configuration are explained in Appendix A.1. By learning such functions, the model can better generalize to a variety of input data that require different window sizes, improving its generalization capability.

3.1.3. Multi-Scale Dynamic Window Attention Module

MDWA module is the computing core of DWM, which integrates the outputs of AWP and CRPB modules. Mathematically, these operations can be expressed as:

$$Attention=Softmax({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}+B)V$$

(6)

Among them, $Q$, $K$, and $V$ represent the matrices of queries, keys, and values derived from the linear transformation of $d_k$. This represents the dimension of the key vector, and B is the offset matrix generated by CRPB. Specifically, this module integrates the input window features with the learned position bias matrix $B$ and combines a dynamic window mechanism with continuous position encoding to achieve multi-scale adaptive attention.

Compared to traditional window attention mechanisms, the proposed multi-scale dynamic window attention achieves unified multi-scale processing, supports feature maps of different resolutions within a single network architecture, eliminates padding, and generates window-adaptive position offset codes in real-time. This submodule can reduce the computational complexity of the model while enhancing its ability to recognize objects in multi-scale images.

3.2. Cross-Scale Context Attention Module

To address the challenge of cross-scale semantic information interaction in remote sensing images, this paper proposes a Cross-Scale Context Attention module, as shown in Figure 4. Through the collaborative design of Multilevel Feature Interaction (MFI) and Channel Semantic Recalibration (CSR), the SCA module enables effective interaction of semantic information across scales, significantly enhances feature representation capability, and thereby improves the accuracy of semantic segmentation.

The core innovation of the SCA module lies in establishing a fusion mechanism between high-level semantic information and low-level detail features, effectively addressing the representation inconsistency caused by the scale differences in remote sensing images. Below, the architectural design and implementation principles of the MFI and CSR submodules are elaborated in detail. Taking the IRRB encoder as an example, the specific working mechanisms of each submodule and their contributions to the semantic segmentation task of remote sensing images are introduced. The key algorithm implementation of the SCA module can be found in Appendix B.2.

3.2.1. Multilevel Feature Interaction Module

MFI builds a foundation for cross-scale feature fusion to connect high-level and low-level features. As shown in Figure 4, in the IRRB encoder, this sub-module connects features at different levels through a bidirectional feature propagation mechanism. Mathematically, the above can be expressed as:

$$R_{low}={Conv}_{1\times 1}\left(R_i\right)$$

(7)

$$R_{high}={Conv}_{3\times 3}\left(Upsample\left(R_{i+1}\right)\right)$$

(8)

$$R_{cat}=Concat\left(R_{low},R_{high}\right)$$

(9)

where ${Conv}_{i\times i}$ denotes the convolution operation, $Upsample(·)$ represents bilinear upsampling, and $Concat$ indicates channel-wise splicing, the Multilevel Feature Interaction module operates as follows:

1.

The feature map $R_i$ is first refined through a $1\times 1$ convolution layer to adapt its channel dimensions for subsequent fusion;

2.

The adjusted $R_{i+1}$, which is upsampled (to match the resolution of $R_i$) and processed via a $3\times 3$ convolution layer to ensure feature compatibility;

3.

The concatenated features are integrated to generate the output feature map $R_{cat}$(with the same spatial and channel dimensions as $R_i$), which preserves both local details from $R_i$ and global context from the $R_{i+1}$.

By establishing cross-level contextual interaction pathways, the MFI module enables the model to simultaneously utilize local fine-grained features and global semantic information, significantly enhancing the model’s ability to extract image information in complex scenarios. The output $R_{cat}$ provides a feature representation for subsequent CSR sub-modules that allows for spatial semantic collaborative optimization, forming the core innovation of the SCA mechanism.

3.2.2. Channel Semantic Recalibration Module

The Channel Semantic Recalibration module adapts and optimizes feature channels through an intelligent weighting mechanism guided by high-level semantic features, significantly improving the accuracy of semantic segmentation. This module utilizes the semantic prior knowledge of high-level features to assess the importance of each channel (including channels in the feature map and $R_{cat}$ outputted by the MFI module), thereby enhancing the representation of key semantic information.

More specifically, CSR module first compresses the spatial dimension through a global average pooling layer. This operation extracts the global features at the channel level and integrates the adjacent higher-level features, as illustrated in Figure 4, which depicts the spatial dimension compression and channel-level feature extraction process.

Subsequently, a series of convolution layers are employed to learn the dependencies between channels, thereby generating attention weights. The Softmax activation function is then applied to these attention weights to produce a normalized attention weight distribution, achieving channel recalibration.

Next, the attention weights are multiplied by the output $R_{cat}$ of the MFI module. The final enhanced feature representation is derived using feature representations at different levels, thus achieving cross-scale contextual feature enhancement. This process significantly improves the model’s ability to extract features from remote sensing images. The above-mentioned operations can be formulated mathematically as follows:

$$R_i^h=Conv(Concat(R_i,Up(R_{i+1})))\times (Softmax(Conv\left(GAP(R_{i+1})\right))$$

(10)

where $GAP(·)$, $Concat(·)$, $Up(·)$ and $Softmax(·)$ respectively represent the global average pooling layer, cascade layer, up-sampling layer and Softmax function. In the same way, $D_i^h$ can be deduced in the same way.

Through the channel recalibration mechanism, the CSR module significantly enhances the discriminative ability of feature representations, thereby addressing key challenges in remote sensing images such as high inter-class similarity and the difficulty in identifying small targets, and provides an important guarantee for accurate semantic segmentation.

3.3. Hierarchical Adaptive Fusion Architecture

In heterogeneous data feature fusion, features at different levels contain different semantic information. Shallow features typically contain rich spatial details, while deep features carry higher-level semantic information. To fully leverage the complementary information in multimodal data, this paper proposes a Hierarchical Adaptive Fusion Architecture (HAFA), which consists of two sub-modules:

(1)

Heterogeneous Modality Synchronizer (HMS), which is used to fuse features from different modes, is mainly composed of two parts: Feature Enhancement in spatial direction and adaptive feature fusion in channel direction. The spatial correlation is calculated by using the multimodal features $R_i^h$ and $D_i^h$ derived from SCA module, which is the key to realize the adaptive fusion of multimodal features.

(2)

Efficient Channel Attention (ECA): in the deep network, due to the low resolution of feature map but rich semantic information of channels, this paper uses ECA module [29] to enhance the expression of significant channels in multimodal features, which can improve the response of important channels without increasing too much computational burden, thus strengthening the semantic consistency of cross-modal features. The hierarchical adaptive fusion architecture is illustrated in Figure 5.

Specifically, the output of the heterogeneous modal synchronizer, denoted as $A_i^s$, represents spatial correlations (its detailed operation is described in the next section). Meanwhile, the extracted multimodal features $R_i^h$ and $D_i^h$ are input into two ECA modules, and their outputs are summed to generate $A_i^c$ (i.e., channel correlations). Subsequently, $A_i^c$ and $A_i^s$ are processed through a Fully Connected (FC) layer, and the resulting outputs are combined to produce the final cross-modal feature $M_i^F$. The above operations are mathematically formulated as follows:

$$M_i^F=FC\left(AF\left(R_i^h,D_i^h,Conv(FE(R_i^h,D_i^h))\right)+FC\left(ECA(R_i^h,D_i^h)\right)\right), i=1,2,3$$

(11)

where $FC(·)$ consists of FC layer, ReLU and Sigmoid function. Among them, $FE(·)$ and $AF(·)$ represent feature enhancement operation and feature fusion, respectively. $ECA(·)$ is the operation of ECA module.

3.3.1. Heterogeneous Modality Synchronizer

Due to the incomplete spatial alignment between DSM data and IRRB data during the spatial correlation calculation process, this study proposes a Heterogeneous Modality Synchronizer (HMS) to fuse heterogeneous information. As shown in Figure 6, this module consists of two components: spatial feature enhancement (FE) and adaptive channel fusion.

(1)

Spatial Feature Enhancement (FE): firstly, process the multimodal data (including IRRB and DSM) through global average pooling (GAP) to extract spatial information. Then, input the aligned and fused features of IRRB and DSM into the adaptive fusion process, where the weight matrix is learned from the features of adjacent stages.

(2)

Adaptive Channel Fusion: to mitigate the impact of noisy or missing DSM signals on IRRB data, channel attention mechanisms are employed to adaptively adjust the weights of different modal features, enabling more effective cross-modal feature fusion.

The adaptive feature fusion process integrates spatially enhanced features $E_i^h$ with modality-specific inputs $R_i^h$ and $D_i^h$ as follows:

1.

The spatially enhanced features $E_i^h$ are first concatenated with $R_i^h$ and $D_i^h$, respectively, followed by a $1\times 1$ convolutional layer to adjust channel dimensions and a ReLU activation for non-linearity;

2.

The concatenated outputs are fed into a softmax layer to generate the weighting coefficients $\alpha$ and $\beta$, which adaptively balance the contributions of the two branches;

3.

Third item. The final spatial correlation map $A_i^s$ is computed by weighted fusion of the intermediate features using $\alpha$ and $\beta$. Mathematically, the operations are formulated as:

$$A_i^s=\alpha \times R_i^h+\beta \times D_i^h$$

(12)

$$\alpha ={\displaystyle \frac{Conv(Concat(R_i^h,E_i^h))}{Conv\left(Concat(R_i^h,E_i^h)\right)+Conv(Concat(D_i^h,E_i^h))}}$$

(13)

$$\beta ={\displaystyle \frac{Conv(Concat(D_i^h,E_i^h))}{Conv\left(Concat(R_i^h,E_i^h)\right)+Conv(Concat(D_i^h,E_i^h))}}$$

(14)

The ReLu function is omitted from the formula, which makes the formula more concise. In this module, the learnable potential weighting coefficients α and β provide strong adaptability for the process.

3.3.2. Efficient Channel Attention Module

As shown in Figure 7, in response to the problem of information redundancy caused by direct concatenation of multimodal remote sensing image features, this paper adopts Efficient Channel Attention (ECA) to improve the performance of the model through cross-channel interaction.

The input data are first processed by global average pooling (GAP) to obtain the spatially aggregated features $F_{rgb}\in R^{C\times 1\times 1}$, where $C$ denotes the number of channels. Subsequently, a convolutional layer is applied to these features.

Specifically, in the ECA module, the size of the convolution kernel is derived through non-linear mapping based on the channel dimension. The output of the convolutional layer is passed through a Sigmoid function to generate a vector of channel attention weights $F_{dsm}$. Finally, $F_{dsm}$ is element-wise multiplied with the input features to produce the final weighted features. Mathematically, the operations are formulated as:

$${\alpha }_c=\sigma \left(MLP\left(GAP(F_{rgb}\oplus F_{dsm})\right)\right)$$

(15)

$${\alpha }_s=\sigma \left(Conv([F_{rgb};F_{dsm}])\right)$$

(16)

$$F_{fused}={\alpha }_c·{\alpha }_s·(W_r{F}_{rgb}+{W_{d}F}_{dsm})$$

(17)

where $\oplus$ represents element wise addition, $W_r$ and $W_d$ are learnable weights.

3.4. Loss Function

Different objects in remote sensing imagery are adjacent to and overlapping each other, which makes the boundaries between neighboring objects blurry, leading to a higher risk of misclassification. In addition, there is a significant imbalance in the number of samples among different ground target categories, and most existing models tend to prioritize the correct classification of majority samples while neglecting minority samples, resulting in overfitting. To address these challenges, this paper adopts a loss function based on error point correction, which consists of two parts, defined as follows:

$${Loss}_{total}=\left(1-\alpha \right){Loss}_{EPC}+\alpha {Loss}_{LSR}$$

(18)

In this context, $\alpha$ is a hyperparameter, ${Loss}_{EPC}$ is a loss function for error-prone points and misclassification points, and ${Loss}_{LSR}$ is a loss function after smooth regularization for labels over-fitted by the network.

${Loss}_{EPC}$ is a specialized loss function designed to focus on points prone to errors and misclassifications. In remote sensing images, due to factors such as complex object boundaries and interference from objects with similar appearances, these regions or pixels are more likely to be misclassified. ${Loss}_{EPC}$ can assign higher weights to points that are prone to errors and misclassification during training, enabling the model to adjust its parameters accordingly to reduce the misclassification rate in these areas. ${Loss}_{LSR}$ is a loss function that performs label smoothing regularization to prevent network overfitting. When a model overfits the training data, it may memorize the noise and characteristics in the labels instead of learning the underlying patterns. ${Loss}_{LSR}$ helps prevent this overfitting by adding a regularization term to the loss function. It smooths the influence of overfitted labels, allowing the model to learn more generalizable features and improving its recognition performance on new data. Additionally, detailed mathematical expressions and derivations for ${Loss}_{EPC}$ and ${Loss}_{LSR}$ can be found in the Appendix A.2.

4. Results and Discussion

To validate the effectiveness of DyFuseNet, systematic experiments were conducted using the Vaihingen and Potsdam standard remote sensing datasets released by the International Society for Photogrammetry and Remote Sensing (ISPRS). This section first systematically introduces the evaluation metrics used in the experiments, followed by a presentation of the comparative experimental results and ablation study results. Subsequently, it focuses on discussing the advantages of the proposed DyFuseNet compared to existing models, and provides an in-depth analysis of the enhancement mechanisms of each module’s contribution to the model’s performance.

All experiments were conducted in an Ubuntu 20.04 environment, with hardware configured as a single NVIDIA RTX 3090 GPU (24 GB VRAM; NVIDIA Corporation, Santa Clara, CA, USA) paired with a 15-core Intel Xeon Platinum 8358 P 2.60 GHz processor (Intel Corporation, Santa Clara, CA, USA), 90 GB RAM, and 80 GB storage capacity, including a 30 GB system disk and a 50 GB data disk. The software environment is based on PyTorch 1.11.0, Python 3.8, and CUDA 11.3. The training parameters are set as follows: using the Stochastic Gradient Descent (SGD) optimizer, the decoder learning rate is 0.001, the encoder learning rate is 0.0005, and a fixed batch size of 10.

4.1. Evaluation Metrics

In this paper, the following indexes are used to evaluate the segmentation performance, including overall accuracy (OA), mean F1-score (mF1) and mean intersection-over-union (mIoU). The formulas are as follows:

$$OA={\displaystyle \frac{TP}{TP+TF+FP+FN}}$$

(19)

$${Precision}_c={\displaystyle \frac{{TP}_c}{{TP}_c+{FP}_c}},{Recall}_c={\displaystyle \frac{{TP}_c}{{TP}_c+{FN}_c}}$$

(20)

$${F1}_c={\displaystyle \frac{2\times {Precision}_c\times {Recall}_c}{{Precision}_c+{Recall}_c}}$$

(21)

$$mF1={\displaystyle \frac{1}{C}}\sum _{c=1}^C{F1}_c$$

(22)

$${IoU}_c={\displaystyle \frac{{TP}_c}{{TP}_c+{FP}_c+{FN}_c}}$$

(23)

$$mIoU={\displaystyle \frac{1}{C}}\sum _{c=1}^{C}Io{U}_c$$

(24)

In this context, TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively, and the subscript c indicates the metrics calculated for the c-th class of land cover objects (e.g., building, vegetation, water body, etc.)

OA reflects the proportion of correctly classified pixels to the total number of pixels, with higher values indicating better overall segmentation accuracy; similarly, higher values of mF1 and mIoU demonstrate the model’s superior semantic segmentation performance in preserving detailed features and accurately segmenting object boundaries.

In addition, to further analyze the computational efficiency of DyFuseNet, this study introduces a commonly used three-dimensional evaluation framework in semantic segmentation: Parameters (Params) reflect the model’s memory consumption, FLOP (Floating Point Operations) represents computational complexity, and FPS (Frames Per Second) measures inference speed. Ideally, an efficient model should have lower parameter and FLOP values to reduce computational overhead while maintaining a high FPS for real-time performance. These three metrics are quantitatively defined as follows:

$$Params=\sum _{l=1}^L(K_{l,width}\times K_{l,height}\times C_{l,in}\times C_{l,out}+C_{l,out})$$

(25)

$$FLOPs=\sum _{l=1}^L\left({2\times K}_{l,width}\times K_{l,height}\times C_{l,in}\times C_{l,out}\right)\times {\displaystyle \frac{H}{s_l}}\times {\displaystyle \frac{W}{s_l}}$$

(26)

$$FPS={\displaystyle \frac{N_{frames}}{{\sum }_{i=1}^{N_{frames}}{t}_{infer,i}}}$$

(27)

where $K_l$ represents the convolution kernel size of the $l$ layer, C represents the total number of classes, and $S_l$ represents the down-sampling rate of the $l$ layer. Ideally, an effective model should have lower values of the first two parameters and higher FPS.

All efficiency metric experiments were tested under the condition of input feature map resolution of 512 × 512 and batch size set to 10, consistent with the training configuration to ensure fairness. The number of channels is set according to the model input layer, with RGB images set to 3 channels and DSM images set to single channel. In addition, to avoid the impact of GPU initial state fluctuations, 10 warm-up forward inferences are performed before formal timing (only forward propagation time is counted, excluding backpropagation or data loading). The specific measurement methods and procedures for FLOPs and FPS metrics are as follows: FLOPs are calculated using PyTorch’s torch.profiler module by counting the multiply-accumulate operations of each layer during forward propagation and converting them; FPS is calculated based on the end-to-end time of 100 consecutive inferences (excluding data loading and warm-up iterations), taking the average value and keeping two decimal places. For the Canny edge detection algorithm, this paper uses boundary density(Bd) and the standard deviation of boundary density (stdBd) as experimental evaluation metrics. Boundary density refers to the proportion of boundary pixels between specific categories to the total number of pixels in a multi-class segmentation task. It reflects the clarity and separation degree of boundaries between different categories. The standard deviation of boundary density represents the dispersion of boundary densities across images in the dataset and can indicate the consistency or variability of boundary density within the dataset; a larger value indicates a higher degree of dispersion. The specific definition formula is:

$$Bd\left(\%\right)=\left({\displaystyle \frac{boundarypixelsnumbers}{totalpixelsnumbers}}\right)\times 100$$

(28)

$$stdBd\left(\%\right)=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N({Bd}_i\left(\%\right)-Mean{Bd}_i\left(\%\right))}$$

(29)

Here, $N$ is the number of images in the dataset, ${Bd}_i$ is the boundary density of the i-th image, and $Mean{Bd}_i\left(\mathrm{\%}\right)$ is the average boundary density of all images. When the boundaries of a dataset are relatively regular, that is, the boundaries between different categories are relatively clear and simple, with smoother and more orderly lines, the proportion of boundary pixels in the image is relatively low, meaning the boundary density is low. This is because in the case of regular boundaries, the transition areas between categories are relatively narrow, so only a small number of pixels are located on the boundaries.

4.2. Comparative Experiments

In this study, DyFuseNet uses a unified encoder to process IRRB and DSM data. Under the same experimental conditions, the performance of the proposed DyFuseNet is compared with the performance of six state-of-the-art segmentation models on the ISPRS Vaihingen and Potsdam datasets: PSPNet [11], Swin-Unet [14], ACNet [23], RDFNet [21], CMFNet [17], and MFTransNet [24]. Among these, Swin-Unet and PSPNet are unimodal methods that only utilize IRRB data, while the other models are multimodal methods. The inclusion of unimodal methods in the comparative experiments aims to validate the performance of classical methods, thereby providing a clearer benchmark for evaluating the proposed methods. For all comparison methods, at least 5 independent training and testing runs were conducted under the same dataset split, hyperparameters, and evaluation protocols, recording the mIoU results for each run. The detailed comparison results are presented in

Table 1. Performance comparison of DyFuseNet with SOTA single-modal and multimodal fusion methods for semantic segmentation on Vaihingen and Potsdam datasets.

Dataset	Method	F1					OA	mF1	mIoU
		Roa.	Bui.	Low.	Tre.	Car
Vaihingen	Swin-Unet	0.9021	0.9518	0.8463	0.6923	0.7591	0.8294	0.8303	0.7209
	PSPNet	0.9008	0.9467	0.8831	0.7383	0.7113	0.8374	0.8361	0.7292
	ACNet	0.8904	0.9314	0.8750	0.8135	0.7387	0.8419	0.8478	0.7590
	RDFNet	0.8981	0.9517	0.8921	0.8170	0.7392	0.8570	0.8596	0.7610
	CMFNet	0.9058	0.9545	0.9077	0.8196	0.7405	0.8640	0.8656	0.7708
	MFTransNet	0.9229	0.9629	0.9101	0.8254	0.7513	0.8715	0.8745	0.7849
	DyFuseNet	0.9201	0.9677	0.9102	0.8561	0.7900	0.8511	0.8869	0.8040
Potsdam	Swin-Unet	0.8983	0.9508	0.8870	0.7951	0.7136	0.8500	0.8490	0.7466
	PSPNet	0.9008	0.9536	0.8690	0.7948	0.7391	0.8455	0.8515	0.7490
	ACNet	0.9098	0.9526	0.8893	0.7883	0.7292	0.8536	0.8538	0.7538
	RDFNet	0.9037	0.9582	0.9015	0.8117	0.7470	0.8620	0.8644	0.7688
	CMFNet	0.9143	0.9601	0.9026	0.8247	0.7747	0.8677	0.8753	0.7843
	MFTransNet	0.9159	0.9657	0.9082	0.8449	0.7594	0.8720	0.8788	0.7908
	DyFuseNet	0.9194	0.9697	0.9160	0.8703	0.7970	0.8980	0.8987	0.8085

.

In addition, DyFuseNet achieved mIoUs of $80.40\%\pm 0.4\%$ and $80.85\%\pm 0.3\%$ on Vaihingen and Potsdam, respectively, demonstrating stable results. Moreover, a paired t-test was performed to analyze the multiple experimental results of DyFuseNet and the SOTA baselines. The paired t-test p-values on both datasets were less than 0.01. Multiple experiments and statistical tests indicate that the proposed method shows significant and stable improvements over other SOTA methods across various metrics.

Comparing the single-modal methods (Swin-Unet and PSPNet) with the multi-modal methods in the comparison table, it can be seen that the multi-modal data fusion methods outperform the single-modal methods across all evaluation metrics. This finding indicates that incorporating DSM elevation information can effectively enhance semantic segmentation performance, with the mIoU increasing by an average of 7.28%.

Further analysis of the class-level F1 scores for the two datasets indicates that the proposed DyFuseNet method achieves the best segmentation results for the building, tree, and vehicle categories. Notably, the improvements in the tree categories are particularly significant, exceeding the second-best method METransNet by 3.87% and 3.76%, respectively. Although DyFuseNet shows slightly lower performance than METransNet in the road and low vegetation categories, it still outperforms other comparative methods in these categories. Furthermore, among all multimodal methods, DyFuseNet achieved the highest mIoU metrics across both datasets, with particularly significant improvements in the tree and vehicle categories.

In urban scenarios, including the datasets used in this paper, Vaihingen and Potsdam, low vegetation (such as grass and shrubs) is often adjacent to buildings, roads, and other land cover types, forming numerous “vegetation-building” and “vegetation-road” boundary areas. These boundaries are challenging for semantic segmentation and place high demands on a model’s ability to distinguish local features. Therefore, an improvement in the F1 score for low vegetation (Low.) not only indicates more accurate classification of this category itself but also implicitly reflects the model’s precise segmentation ability at the “edges where vegetation meets other land cover types.” If a model can correctly classify boundary pixels of low vegetation, it demonstrates that it has captured local features at the edges. Combined with the overall improvement in mF1, it shows that the model not only classifies low vegetation more accurately but also, by optimizing edge local features, enhances the boundary segmentation accuracy of neighboring land cover types. In Table 1, DyFuseNet’s Low. F1 scores reach 0.9102 and 0.9160 on the Vaihingen and Potsdam datasets, respectively, demonstrating its significant advantage in classifying low vegetation/boundary pixels compared to other methods. These results validate the effectiveness of DyFuseNet in addressing the challenges of complex boundary segmentation.

From the perspective of comprehensive evaluation indicators, DyFuseNet ranks first in both OA and mF1 across two datasets, achieving mIoU values of 80.40% and 80.85%, which are 1.91% and 1.77% higher than the suboptimal method METransNet, respectively. These results indicate that, despite both the IRRB and DSM modalities using the same encoder, DyFuseNet demonstrates significantly superior overall segmentation performance compared to other multimodal methods, highlighting its balance and robustness in multi-class segmentation tasks. Taking the Potsdam dataset as an example, the comparative segmentation results, as shown in Figure 8, demonstrate that DyFuseNet achieves significant improvements in boundary segmentation accuracy and the recognition of objects at varying scales.

The results in the figure further support Low.’s finding that F1 can represent boundary segmentation accuracy: compared with the baseline, the segmentation maps generated by DyFuseNet show clearer and more precise boundaries, consistent with Low.’s quantitative advantage. This consistency between F1 visual and quantitative evidence reinforces the claim that DyFuseNet has excellent edge-handling capabilities.

To further assess the performance of the proposed DyFuseNet,

Table 2. Efficiency and Accuracy Comparison of DyFuseNet and SOTA Instance Segmentation Methods.

Method	Multimodal	FLOPs (G)	Parameter (MB)	FPS	mIoU (%)
Swin-Unet	N	16.54	34.68	38.64	74.66
PSPNet	N	51.23	46.72	68.24	74.90
ACNet	Y	12.96	62.37	18.64	75.38
RDFNet	Y	60.44	42.08	20.72	76.88
CMFNet	Y	80.67	112.44	9.82	78.43
MFTransNet	Y	9.52	41.36	16.68	79.08
DyFuseNet	Y	26.19	82.77	30.09	80.85

presents a comprehensive comparison among different methods under identical experimental conditions.

This evaluation provides a comprehensive assessment of method performance by simultaneously analyzing computational efficiency metrics and segmentation accuracy. Computational efficiency is quantified using three key indicators: FLOPs (measured in billions of floating-point operations), parameter count (reported in megabytes), and inference speed (expressed in frames per second, FPS). Segmentation accuracy is primarily evaluated through mIoU (mean Intersection over Union, reported as a percentage). Additionally, a multimodal indicator (marked as Y/N) is incorporated to specify whether each compared method employs multimodal fusion (Y) or relies on single-modal data (N) for semantic segmentation.

Compared to multi-modal fusion segmentation methods (CMFNet and MFTransNet), the proposed DyFuseNet achieves lower FLOPs, fewer parameters, and higher FPS, demonstrating the optimal balance between computational efficiency and segmentation performance. Specifically, The FPS model proposed in this paper is 1.8 times better than the suboptimal method. Although the parameters have slightly increased, the overall segmentation performance of DyFuseNet is still significantly superior to other methods.

In order to systematically evaluate the competitive advantages of DyFuseNet in multimodal remote sensing segmentation methods, we generated a scatter plot based on the quantitative data from Table 2 to intuitively illustrate the trade-off between FLOPs and mIoU. As shown in Figure 9, the horizontal axis represents floating-point operations (FLOPs, in billions), and the vertical axis represents mean Intersection over Union (mIoU, in %), clearly displaying the balance between the computational resource consumption and segmentation accuracy of different methods.

In the figure, DyFuseNet is marked with a black pentagram, occupying the best area, with a FLOPs of about 30 G FLOPs and an mIoU close to 81%; the multimodal methods are marked with orange circles, including MFTransNet, ACNet, CMFNet, and RDFNet, which are relatively dispersed in distribution, with generally lower mIoU values; the unimodal methods are marked with blue triangles, including Swin-Unet and PSPNet, achieving mIoUs of 74.5% and 75%, respectively.

Based on the comparative analysis in the figure, three core advantages of the method proposed in this paper can be drawn: First, compared to other typical multimodal methods, DyFuseNet achieves a higher mIoU at lower FLOPs, demonstrating its synergistic optimization in computational efficiency and segmentation accuracy. Secondly, as shown in the figure, unimodal methods cannot excel in a single metric; although Swin-Unet has low FLOPs, its mIoU is also low, indicating insufficient segmentation accuracy, while PSPNet has a high mIoU but also very high FLOPs, leading to both methods failing to simultaneously optimize computational load and segmentation performance. In contrast, DyFuseNet achieves a balanced performance of 26.19 G FLOPs and 80.65% mIoU, significantly surpassing the compared unimodal methods. Thirdly, Pareto optimality—from the overall distribution in the FLOPs-mIoU scatter plot, DyFuseNet is situated near the Pareto front, indicating that it maximizes segmentation accuracy under the given computation budget, reflecting the synergistic optimization of efficiency and accuracy.

In summary, DyFuseNet exhibits significant superiority in multimodal remote sensing segmentation: Its computational efficiency and segmentation performance cooperatively optimize resource consumption and accuracy, offering both practical utility and theoretical value as a representative method in the field.

4.3. Ablation Experiments

To better understand the influence of each improved module in DyFuseNet on the semantic segmentation performance of multimodal remote sensing images, this study conducts ablation experiments by progressively integrating modular components. The experimental design focused on three core objectives: (1) verifying the performance gains achieved by replacing the fixed window mechanism of the Swin Transformer with the DWM; (2) evaluating the effectiveness of the SCA Module in enhancing feature extraction cross-scales; and (3) analyzing the superiority of the HAFA in collaboratively integrating IRRB imagery and DSM information. All experiments are performed under consistent environmental conditions.

The baseline model is the Swin-Transformer, designed for semantic segmentation of multimodal remote sensing images. The ablation process is as follows: initially, the fixed-size window mechanism of Swin-Transformer is replaced with a dynamic window module (DWM) to enhance feature localization. Subsequently, a cross-scale contextual attention (SCA) module is incorporated to capture long-range dependencies. Finally, a hierarchical adaptive fusion architecture (HAFA) is constructed, where the shallow network employs a heterogeneous modal synchronizer to align multimodal features, and the deep network introduces efficient channel attention to reduce feature redundancy. The ablation experimental results are presented in

Table 3. DyFuseNet model ablation experiment results.

Dataset	Baseline	DWM	SCA	HAFA	OA	mF1
Vaihingen	√				0.8481	0.8532
	√	√			0.8537	0.8581
	√	√	√		0.8702	0.8748
	√	√	√	√	0.8764	0.8867
Potsdam	√				0.8536	0.8538
	√	√			0.8677	0.8753
	√	√	√		0.8764	0.8867
	√	√	√	√	0.8780	0.8897

√: Indicates that the corresponding module in the model; Blank: Indicates that the corresponding module is not included.

.

In addition, since the primary purpose of the ablation experiments in this paper is to independently verify the contribution of each improved module to the model’s accuracy, the focus is therefore on changes in performance metrics, including mIoU and mF1. As a result, computational efficiency metrics like FLOPs are not the main concern of this ablation study, for the following reasons: Firstly, the essence of ablation experiments is ‘controlled variables’: by gradually removing or adding specific modules, we observe fluctuations in performance metrics to clarify the direct impact of the module on accuracy. Comparing FLOPs simultaneously would introduce additional variables (such as the number of module parameters and the non-linear accumulation of computational complexity), which would interfere with the pure analysis of the module’s performance contribution. Then, computational efficiency metrics should be evaluated in the context of the complete model: metrics like FLOPs are more suitable for comparisons between complete models to assess the overall efficiency of a design. The focus of ablation experiments is on the local impact of modules, and the FLOPs of the complete model can be analyzed uniformly in comparative experiments. The impact of adding or removing a module on FLOPs and FPS is indirect and can be inferred: for example, a dynamic window module may replace global fusion with local window computation, theoretically reducing redundant calculations; hierarchical fusion may reuse multi-scale features without significantly increasing FLOPs. These logical inferences can be supplemented through theoretical analysis, eliminating the need to repeatedly calculate them in the ablation experiments.

The results of the ablation experiments indicate that the gradual integration of these three improved modules—the dynamic window module, the cross-scale contextual attention module, and the hierarchical adaptive fusion architecture—has produced significant and quantifiable performance improvements, validating the scientific design and effectiveness of each component.

The DWM replaced the fixed-size window mechanism with dynamic window scaling, significantly enhancing the perception of structures at different scales, including large buildings and small cars. It achieved a 2.86% increase in mIoU in Vaihingen, rising from 77.34% to 80.20%, and a 3.05% increase in Potsdam, from 77.60% to 80.65%, confirming that this module improved the segmentation accuracy of multi-scale target boundary details in complex scenarios.

The SCA module further improved the model’s segmentation accuracy, with the mIoU metric increasing by 1.93% on the Vaihingen dataset and by 1.77% on the Potsdam dataset. Although the gain of the model after adding this module is slightly lower than that of the dynamic multi-scale window module, it optimizes contextual dependencies through cross-scale feature interaction, enhancing the segmentation of fine-grained categories. It is evident that the segmentation accuracy of low vegetation and roads in the Vaihingen dataset has significantly improved. Additionally, the mF1 score for trees and vehicles in the Potsdam dataset has seen a substantial increase, indicating that this module compensates for the model’s limitations in global context modeling.

The addition of the HAFA module indicates the dependency of the model DyFuseNet proposed in this paper on the dataset. From the data in the table, it can be seen that this module increases the mIoU on the Vaihingen dataset by 1.25%, from 78.95% to 80.20%, by adjusting modal differences and suppressing redundancy. However, on the Potsdam dataset, the mIoU only improved by 0.45%, increasing from 79.20% to 79.65%. The reason for this result may be that the model is very sensitive to the modal fusion requirements of specific datasets, which needs further validation.

By comparing the annotation files of the original datasets, it can be observed that the Potsdam dataset has more regular boundaries, generally featuring organized block-like building areas and grid-like roads, whereas Vaihingen contains complex, fragmented edges with abundant scattered vegetation and exhibits gear-shaped boundary lines where irregular buildings and roads intersect. HAFA can significantly improve classification accuracy by integrating the complementary information from DSM elevation noise and IRRB data. However, in areas with clearly defined boundaries, the original information from DSM and IRRB is already sufficiently reliable, leaving limited room for optimization. Therefore, the effect of the HAFA module is more pronounced on Vaihingen.

Based on the Canny edge detection algorithm, we calculated the boundary pixel proportions for key categories such as ‘building-vegetation’ and ‘road-vegetation’ in the Vaihingen and Potsdam datasets (Figure 10), further quantifying this phenomenon. The implementation is performed via a Python script: the original annotation map is input, edges are detected using OpenCV’s Canny function (low threshold 0.1, high threshold 0.3), and then the total number of edge pixels related to each land cover category is counted.

The experimental results are shown in the figure. Experimental data show that the average boundary density in Potsdam is 0.6% ± 0.3% (standard deviation 0.3%), significantly lower than the value of 1.1% ± 0.2% (standard deviation 0.2%) in the Wageningen area. This difference directly reflects the contrast between the regularity of surface cover boundaries in Potsdam and the complexity in Wageningen, explaining why the HAFA module achieves only an average improvement of 0.45% in overlap ratio in Potsdam, while it can improve by 1.2–1.5% in Wageningen. This also confirms that the effectiveness of the HAFA algorithm is closely related to boundary complexity. Appendix B.3 provides the boundary density calculation script, and the revised paper adds the boundary calculation formula in Section 3.1 on evaluation metrics.

Overall, the three modules—namely the dynamic window module that optimizes local feature perception, the cross-scale context attention module that enhances global context modeling, and the hierarchical adaptive fusion architecture that refines multimodal fusion—work together to drive the overall performance improvement of DyFuseNet. The design logic of each module is closely related to the typical challenges in remote sensing images, such as complex scenes, multi-scale targets, modal differences, and boundary ambiguity, confirming the innovation and effectiveness of the DyFuseNet model proposed in this paper.

4.4. Multimodal Noise Robustness Experiments

In this experiment, we simulated two types of noise scenarios to evaluate the robustness of the DyFuseNet under multimodal noise conditions, specifically:

(1)

DSM noise: Gaussian noise was added to the elevation values of the digital surface model (DSM) with standard deviations of 0.1 m, 0.5 m, and 1.0 m.

(2)

Multispectral noise: Gaussian noise was added to each spectral channel of the multispectral images with standard deviations of 2, 5, and 10.

We applied DyFuseNet and Swin-T to the test sets of the Vaihingen and Potsdam datasets, respectively, and tested the overall accuracy (OA) for the following four input combinations: original annotated images (Origin), noise added only to DSM (OnlyDSM), noise added only to IRRB images (OnlyIRRB), and noise added to both DSM and IRRB images (Both). Keeping all experimental settings and parameters unchanged, we conducted a longitudinal evaluation of the impact of different noise conditions on the segmentation overall accuracy of DyFuseNet, while also performing a lateral comparison of the robustness of the proposed improved model DyFuseNet with the baseline model Swin-T under different noise conditions. The comparison results are shown in

Table 4. Multimodal Noise Robustness Experiments.

Datasets	Noisy	Swin-T	DyFuseNet	OA-Change
Vaihingen	Origin	0.8322	0.8789	0
	OnlyDSM	0.8215	0.8774	0.0015
	OnlyIRRB	0.8143	0.8721	0.0068
	Both	0.8112	0.8695	0.0094
Potsdam	Origin	0.8436	0.8789	0
	OnlyDSM	0.8325	0.8782	0.0007
	OnlyIRRB	0.8276	0.8743	0.0046
	Both	0.8105	0.8711	0.0078

. The comparison results are shown in Table 4. In addition, the OA-change listed in the table represents the change in OA of DyFuseNet compared to the original dataset without any added noise.

As can be seen from the table, under all noise conditions, the improved DyFuseNet proposed in this paper consistently outperforms the baseline methods in terms of OA. In particular, under multimodal noise conditions, the segmentation accuracy of the Swin-T model on Vaihingen decreases by 2% compared to the noiseless case. However, after incorporating the improvements proposed in this paper, the decline in OA of DyFuseNet is significantly smaller than before the modifications. A longitudinal comparison also shows that, compared with noise-free conditions, DyFuseNet’s OA values do not significantly decrease under different noise conditions, demonstrating the model’s stability and robustness.

5. Conclusions

In this study, we proposed DyFuseNet, a novel multimodal image segmentation framework that integrates dynamic adaptive windows and cross-scale fusion to address key challenges in segmenting complex images with significant scale variations and heterogeneous data sources.

The proposed network consists of three synergistically designed modules: (1) the Dynamic Window Module (DWM), which adaptively adjusts receptive fields to enhance the modeling of objects at varying scales and improve the extraction of fine-grained, edge-rich features; (2) the Cross-Scale Contextual Attention (SCA), which establishes hierarchical interactions between local and global semantics to refine boundary delineation in challenging regions such as shadows and low-texture areas; and (3) the Hierarchical Adaptive Fusion Architecture (HAFA), which aligns features across modalities in the shallow layers while suppressing redundancy in deeper layers through channel attention, achieving an effective balance between segmentation accuracy and computational efficiency.

Extensive experiments on the Vaihingen and Potsdam datasets demonstrate that DyFuseNet achieves state-of-the-art performance, with mIoUs reaching 80.40% and 80.85%, which are 2.83% and 2.99% higher than the baseline model Swin-Trans, respectively, and 1.71% and 1.57% better than the sub-optimal model MFTransNet. Moreover, the model shows high segmentation accuracy in complex scenes, especially for the edges of large buildings and the shadow areas of small objects, with an average F1 score of up to 85%. Finally, DyFuseNet maintains low model complexity and computation efficiency while improving segmentation accuracy, with a FLOP count of only 26.19 G and achieving 30.09 FPS, indicating that the model is suitable for real-time deployment in practical applications.

The method proposed in this article not only provides a robust and general solution for image segmentation under heterogeneous data scenarios, but also introduces innovative strategies of dynamic window modeling and cross-scale feature fusion, contributing to theoretical progress and practical utility. These contributions hold significant value for a wide range of applications, including urban planning, infrastructure monitoring, environmental analysis, and intelligent scene understanding, where accurately and efficiently segmenting complex multimodal visual data is crucial.