A Multi-Branch Attention Fusion Method for Semantic Segmentation of Remote Sensing Images

Abstract

In recent years, advancements in remote sensing image observation technology have significantly enriched the surface feature information captured in remote sensing images, posing greater challenges for semantic information extraction from remote sensing imagery. While convolutional neural networks (CNNs) excel at understanding relationships between adjacent image regions, processing multidimensional data requires reliance on attention mechanisms. However, due to the inherent complexity of remote sensing images, most attention mechanisms designed for natural images underperform when applied to remote sensing data. To address these challenges in remote sensing image semantic segmentation, we propose a highly generalizable multi-branch attention fusion method based on shallow and deep features. This approach applies pixel-wise, spatial, and channel attention mechanisms to feature maps fused with shallow and deep features, thereby enhancing the network’s semantic information extraction capability. Through evaluations on the Cityscapes, LoveDA, and WHDLD datasets, we validate the performance of our method in processing remote sensing data. The results demonstrate consistent improvements in segmentation accuracy across most categories, highlighting its strong generalization capability. Specifically, compared to baseline methods, our approach achieves average mIoU improvements of 0.42% and 0.54% on the WHDLD and LoveDA datasets, respectively, significantly enhancing network performance in complex remote sensing scenarios.

1. Introduction

In recent years, with the advancement of Earth observation technology, the acquisition cost of remote sensing images has been significantly reduced, leading to their widespread applications in transportation, agriculture, forestry, fisheries, and other fields [1,2,3,4,5,6,7,8,9]. Simultaneously, rapid developments in optical sensor technology and continuous innovations in sensor-carrying platforms have diversified the types of remote sensing data. From RGB images to multispectral and hyperspectral data, and from 2D data to 3D point cloud data, the resolution of acquired remote sensing images has become increasingly higher, with these high-resolution images containing richer object-level information [10]. Early researchers often employed mathematical morphology analysis methods for feature extraction [11], while Fourier transforms and maximum likelihood classification could meet the requirements of different tasks [12]. Some researchers integrated machine learning methods with remote sensing [13]; compared to the former approaches, these methods often achieved superior speed and accuracy when processing data with complex feature spaces [14]. However, the semantic information extraction capabilities of these methods remain insufficient to meet current task demands, often failing to deliver satisfactory results when handling high-resolution data. At present, researchers predominantly focus on integrating deep learning with remote sensing tasks. For instance, in 2022, attention mechanisms were proven to enhance performance in few-shot learning tasks [15], and in 2023, a YOLOv8-based instance segmentation method utilizing multispectral remote sensing data was applied to tobacco plant counting [16]. Yet, as this integration deepens, new challenges have emerged.

Compared to natural images, remote sensing images are typically captured by satellite or UAV-mounted sensors from high altitudes. Influenced by factors such as flight altitude, sampling area, and acquisition time, remote sensing images generally exhibit characteristics including more diverse target categories, larger object scale variations, and more complex terrain patterns. These characteristics endow remote sensing images with richer semantic information, more severe class imbalance, and increased interference compared to natural images. Additionally, uncontrollable factors such as solar flares and wind speed further degrade data quality. These challenges collectively render semantic information extraction from remote sensing data significantly more difficult than from natural images, indicating that attention mechanisms designed for natural images cannot effectively interpret the semantics of remote sensing data.

To address these issues, researchers have proposed various methods. For instance, HRCNet enhances spatial information extraction in remote sensing images through a boundary-aware loss (BA loss) function and a boundary-aware (BA) module [17]. The ResUNet-a architecture introduces a gradient-driven Tanimoto loss to improve performance in aerial image semantic segmentation tasks [18]. Furthermore, numerous methods have been developed to enhance model performance in remote sensing tasks [19,20,21,22,23]. Despite their successful applications in specific domains, these approaches are often tailored to address singular limitations or specialized datasets. To date, no universally applicable method has been proposed for general remote sensing tasks.

To address the aforementioned challenges, we select remote sensing semantic segmentation—a critical component of RSIR (Remote Sensing Image Retrieval)—as our research focus. Our objective is to design a highly generalizable, plug-and-play attention mechanism that enhances feature extraction capabilities for remote sensing image data. To this end, we propose an attention method that integrates deep and shallow features of the model. Within the module, we employ a three-branch architecture incorporating pixel-wise, spatial, and channel attention mechanisms to further strengthen the model’s ability to extract semantic information from remote sensing images. This module enhances the network’s semantic understanding by simultaneously learning both deep and shallow features of the input image.

During testing, we selected the Cityscapes dataset, LoveDA dataset, and WHDLD dataset to evaluate the performance of the attention module under different scenarios and resolutions. This approach ultimately led to a significant improvement in the model’s performance on remote sensing datasets.

The main contributions of this paper are as follows:

(1)

We designed and proposed a multi-branch attention fusion method based on shallow and deep semantic information, specifically tailored for remote sensing data and exhibiting high generalization capability. Compared to the attention mechanisms employed in baseline methods, our approach demonstrates superior effectiveness while maintaining competitive inference speed. As illustrated in Figure 1, our method produces more intuitive results, with building regions exhibiting more precise boundaries and road regions showing enhanced connectivity.

(2)

We conducted comprehensive tests across multiple datasets, comparing the effects of applying our proposed attention mechanism to both natural image datasets and remote sensing datasets. The results reveal that this multi-branch attention mechanism exhibits superior performance on remote sensing imagery, achieving enhanced overall performance.

2. Related Work

This section discusses technologies relevant to our research, including semantic segmentation methods, the PP-LiteSeg network, and multi-branch feature fusion techniques.

2.1. Fully Convolutional Network (FCN)-Based Semantic Segmentation

Unlike region-based semantic segmentation approaches, FCN-based semantic segmentation [24] operates at the pixel level, aiming to assign a category label to each pixel in an image. In deep learning, FCNs typically support more flexible input scales, enabling broader applicability across scenarios. For instance, using SDFCNv2 [25], researchers addressed the differences between remote sensing images and natural scene images by proposing a Hybrid Basic Convolution (HBC) block and a Spatial-Channel Fusion Squeeze-and-Excitation (SCFSE) module. These components enhance the model’s feature extraction capability by expanding the receptive field while reducing parameters [26]. Subsequently, following the design principles of FCNs, researchers have proposed various improved methods.

2.2. PP-LiteSeg

In remote sensing tasks, models must not only meet accuracy requirements but also maintain inference speed. Therefore, we select the lightweight semantic segmentation network PP-LiteSeg [27] as the baseline model for our study. PP-LiteSeg is a meticulously designed lightweight network that successfully balances model performance with significantly improved inference efficiency, achieving an optimal trade-off between lightweight design and high efficiency.

PP-LiteSeg primarily consists of three components: an encoder, a Flexible and Lightweight Decoder (FLD), and a computationally efficient Simple Pyramid Pooling Module (SPPM). The encoder employs the lightweight feature extraction network STDC proposed in STDCSeg [28], with PP-LiteSeg offering two variants—STDC1 and STDC2—whose detailed differences are summarized in

Table 1. Architectural details of PP-LiteSeg variants.

Model	Encoder	Channels in Decoder
PP-LiteSeg-T	STDC1	32, 64, 128
PP-LiteSeg-B	STDC2	64, 96, 128

. The input image first undergoes multi-layer convolution in the encoder to generate shallow feature maps, which are then fed into the SPPM module for contextual information aggregation. The output from SPPM is further processed by the FLD module. Within FLD, researchers introduced a Unified Attention Fusion Module (UAFM), which effectively fuses low-level and high-level features by applying channel attention or spatial attention to feature maps, thereby enhancing the network’s representational capacity and segmentation performance.

2.3. Multi-Branch Feature Fusion Techniques

In the field of artificial intelligence, multi-branch is a network design methodology distinct from single-branch architectures. In multi-branch structures, data can be processed in "parallel" streams without mutual interference, enabling specialized processing of specific data features. This design philosophy has been applied across diverse tasks including medical imaging, remote sensing, and deblurring. Using CMFNet [29], researchers proposed a multi-branch network architecture that mimics the parvocellular layers (P-cells), koniocellular layers (K-cells), and magnocellular layers (M-cells) in the visual neural system, thereby enhancing the model’s perception of image shape, color, and illumination variations.

3. Methods

Due to the large data volume and high complexity of remote sensing imagery, semantic segmentation models often struggle to extract sufficient effective information during training. Using PP-LiteSeg, researchers employ the Unified Attention Fusion Module (UAFM) (Figure 2a), which simultaneously inputs low-level and high-level features into the attention module to generate attention weights. This mechanism enables the attention module to integrate features across different receptive fields. Using CMFNet, the researchers enhance model performance by incorporating pixel attention, spatial attention, and channel attention mechanisms within a multi-branch architecture, simulating human visual cells (P-cells, K-cells, and M-cells), as illustrated in Figure 2b. However, the former approach utilizes only spatial attention in its module, while the latter significantly increases parameter count when integrated with U-Net architectures, resulting in suboptimal feature extraction capabilities. Therefore, inspired by CMFNet and PP-LiteSeg, we propose a multi-branch attention fusion method based on shallow and deep semantic information. As illustrated in Figure 2c, we adopt the three attention mechanisms (pixel, spatial, and channel attention) from CMFNet and hybridize them with the UAFM framework from PP-LiteSeg. This integration enables the module to concurrently apply multiple attention operations to both shallow and deep features. This design preserves the feature extraction capability of the module, and the additional parameters introduced by this design are constrained to approximately 2.6 M, ensuring compatibility with resource-sensitive remote sensing applications. Furthermore, we introduce a learnable weighting parameter that allows the network to dynamically adjust the contributions of different attention mechanisms.

The main workflow of this module is formalized in Equation (1), where $X_c,X_p,X_s$ represent the outputs of the channel attention, pixel attention, and spatial attention mechanisms, respectively, and w denotes a learnable weight. As shown in Figure 2c, shallow features are first upsampled and convolved before being fused with deep features through a learnable fusion parameter. The fused features are then fed into three parallel branches for pixel attention, spatial attention, and channel attention processing, generating feature maps f1, f2 and f3. These feature maps are subsequently aggregated, and the combined output is passed through a 3 × 3 convolutional layer to produce the final feature map. This paper provides two versions of the method: V01 and V02. As indicated by the dashed lines in Figure 2c, the V02 method incorporates residual connections to reduce overfitting by adding the fused feature maps to the outputs of the multi-branch attention.

To more intuitively demonstrate the differences between the V01 and V02 methods, we implemented the proposed approach in a concrete network. Taking PP-LiteSeg as an example (as shown in Figure 3), we replaced its original UAFM mechanism with our proposed multi-branch attention fusion mechanism (MAFM)—a novel module specifically designed based on the method presented in this work. Using the building areas as a case study, we selected two representative sets of remote sensing images for heatmap analysis. As shown in Figure 4, in the first two image rows, the V01 method demonstrates significantly superior responses to the building regions compared to both the baseline method and the V02 method. This occurs because the first two rows exhibit image blurring and irregular building shapes, the blurred images contain more noise, and complex regional boundaries require learning richer features, posing substantial challenges to the network’s feature extraction capabilities. Consequently, the V01 method, with stronger learning capacity, outperforms other methods in these scenarios. In the last two rows, where the images are clearer and the building edges are more regular, the demands placed on feature extraction are reduced. Under these conditions, the V01 method exhibits imbalanced responses to target regions, while the V02 method achieves more balanced responses across all target areas in the images.

$$Out=Conv({\displaystyle \frac{1-w}{2}}·X_c+w·X_p{\displaystyle \frac{1-w}{2}}·w{X}_s)$$

(1)

4. Experiments

4.1. Datasets

4.1.1. Cityscapes

The Cityscapes dataset [30] is an urban scene dataset designed for semantic segmentation, constructed by collecting data from vehicles across 50 cities in Germany and neighboring countries. The dataset comprises 5000 high-quality manually annotated images with the following specifications:

• Training set: 2975 images (59.5%)
• Validation set: 500 images (10%)
• Test set: 1525 images (30.5%)
• Evaluation categories: 19 urban scene classes
• Resolution: 2048 × 1024 pixels

As shown in Figure 5, this dataset is utilized to test the performance of the attention mechanism in processing non-remote sensing data.

4.1.2. LoveDA

The LoveDA dataset [31] is a remote sensing dataset designed for domain adaptive semantic segmentation. Collected from 18 Chinese administrative regions, it contains two distinct scenarios:

• Urban subset:

  −

  Training images: 1156;

  −

  Validation images: 677;

  −

  Total: 1833 samples;

  −

  Spatial resolution: 0.3 m;

  −

  Spectral channels: RGB.

• Rural subset:

  −

  Training images: 1366;

  −

  Validation images: 992;

  −

  Total: 2358 samples;

  −

  Spatial resolution: 0.3 m;

  −

  Spectral channels: RGB.

As shown in Figure 6, the left portion displays urban data and their masks, while the right portion shows rural data and their masks. A comparison with Figure 5 reveals that the LoveDA dataset exhibits three characteristics: nadir-view acquisition perspective, semantically rich single images, and imbalanced urban-rural class distribution. These characteristics impose higher demands on the model’s feature extraction capability.

4.1.3. WHDLD

The WHDLD dataset [32,33] is a densely annotated urban dataset with the following characteristics:

• Source: Wuhan metropolitan area, China;
• Total samples: 4,940 RGB images;
• Image dimensions: 256×256 pixels;
• Spatial resolution: 2 m;
• Land cover classes: 6 categories (buildings, roads, pavements, vegetation, bare soil, water).

• Compared with LoveDA, WHDLD features:

• Smaller spatial coverage per image;
• Lower spatial resolution;
• Higher class imbalance.

As shown in Figure 7, compared with LoveDA and Cityscapes datasets, this dataset features more detailed annotation masks. Specifically, the LoveDA annotations rely on simplified polygonal outlines composed of basic line segments, whereas the WHDLD dataset provides pixel-level annotations with finer granularity and superior data quality. Consequently, we adopted this dataset to rigorously evaluate model performance across varying annotation quality levels.

4.2. Data Augmentation

In our experiments, we introduced randomized adjustments to the brightness, contrast, and saturation of all training data to simulate real-world imaging variations under different temporal and meteorological conditions. Additionally, considering the diversity of acquisition perspectives in practical scenarios, we applied random flipping and scaling to the training data. The combination of these data augmentation methods enables our attention module to focus on more diverse information, thereby enhancing the model’s generalization capability.

4.3. Evaluation Metrics

In this experiment, we adopt the mean Intersection over Union (mIoU) as the evaluation metric for model performance. mIoU is a critical assessment metric in semantic segmentation tasks. As shown in Equation (2), K represents the total number of classes, typically including all target categories to be segmented (potentially encompassing the background class), and denotes the Intersection over Union (IoU) for class c. This metric comprehensively evaluates the model’s predictive performance across all categories. During testing, higher mIoU values indicate superior model performance.

$$\mathit{mIoU}={\displaystyle \frac{1}{K}}\sum _{c=1}^{K}Io{U}_c$$

(2)

4.4. Experiments on Non-Remote Sensing Datasets

Application Experiment on Cityscapes Dataset

This experiment aimed to test the effectiveness of our proposed attention mechanism on natural image datasets. Specifically, we applied the V01 version of the multi-branch attention fusion module to pp-LiteSeg with the stdc1 backbone, and the V02 version to pp-LiteSeg with the stdc2 backbone. Both backbone networks were pre-trained. As shown in Figure 8, the convergence speed of the V01 method was comparable to the baseline under pre-training conditions, while the V02 method exhibited slightly slower convergence.

The experimental results are presented in

Table 2. Performance comparison of attention on WHDLD dataset.

Model	InputSize	Val mIoU (%)	Test mIoU (%)
MobileNet V3-Large	-	-	72.6
STDC1-Seg50	512 × 1024	72.2	71.9
LEDNet	512 × 1024	-	70.6
pp_liteseg_stdc1_V01	512 × 1024	75.84	73.37
pp_liteseg_stdc1_Spa	512 × 1024	73.10	72.00
pp_liteseg_stdc2_V02	1024 × 1024	78.80	77.21
pp_liteseg_stdc2_Spa	1024 × 1024	78.20	77.50

Note: Bold values indicate the best result in each experiment.

. With the stdc1 backbone, our attention mechanism achieves 73. 37% on the mIoU test, outperforming the baseline result of 72.0%. However, with the stdc2 backbone, our method shows superior performance only in the validation set. These results indicate that, while the proposed attention mechanism provides modest improvements in general-purpose datasets, the performance gains are not substantial. Furthermore, the stdc2-based configuration is prone to overfitting.

4.5. Experiments on Remote Sensing Datasets

4.5.1. Performance Comparison on WHDLD Dataset

To validate the effectiveness of our method for remote sensing datasets, we conducted experiments using the WHDLD dataset. As shown in

Table 3. Performance comparison of attention on WHDLD dataset.

Model	Build	Road	Pavement	Vegetation	Bare Soil	Water	Parameter	Val mIoU (%)	Test mIoU (%)
U-net	55.32	43.14	28.16	84.68	34.85	83.19	-	-	54.89
FactSeg	55.31	48.35	29.46	85.27	34.21	84.43	-	-	56.17
HRNetV2 [34]	56.11	52.41	31.74	85.65	31.58	86.63	-	-	57.35
EAR-HRNet [35]	58.56	55.42	32.96	87.12	39.06	88.56	-	-	60.28
SSNet [36]	56.00	59.76	42.64	82.03	37.28	93.90	-	61.93	60.63
pp-liteseg_stdc2_V01	55.98	60.25	40.70	79.05	37.37	93.71	15.0	62.63	61.17
pp-liteseg_stdc2_V02	56.59	60.45	41.08	79.02	37.83	93.79	15.0	62.54	61.45
pp-liteseg_stdc2_Spa	55.09	60.30	40.75	78.92	37.62	93.55	12.38	62.51	61.03
pp-liteseg_stdc1_V01	55.76	60.48	39.97	79.04	37.66	93.74	10.41	62.57	61.10
pp-liteseg_stdc1_V02	55.52	59.87	40.31	78.69	36.58	93.79	10.41	62.34	60.77
pp-liteseg_stdc1_Spa	54.84	58.04	40.73	78.61	36.15	93.49	8.15	61.96	60.31

Note: Bold values indicate the best result in each experiment.

, under the STDC1 module, both the V01 attention mechanism and the V02 that use the residual connection attention mechanism outperformed the baseline in both validation and test datasets. Specifically, the V01 method achieved the best mIoU of 61.10%, surpassing the baseline by 0.79%. In the experiments using the STDC2 module, both V01 and V02 methods also outperformed the baseline on the validation and test sets. However, unlike the previous group of experiments, while the V01 method still achieved the best mIoU of 62.63% on the validation set, the V02 method outperformed it on the test set with an mIoU of 61.45%, which was 0.42% higher than the baseline. These improvements are primarily attributed to the application of an attention mechanism with stronger feature extraction capabilities in the network, while the use of residual connections helped maintain the network’s generalization ability as much as possible. Analyzing the results of the two groups of experiments, we found that at a 2 m resolution scale, our attention mechanism demonstrated a more pronounced improvement on lightweight networks. Meanwhile, within lightweight networks, the performance of the V02 method was inferior to that of V01. Furthermore, comparing our results with the baseline model, taking the experimental results of the STDC2 module as an example, the V02 method improved the recognition performance for the categories of buildings, roads, pavements, vegetation, bare soil, and water by 1.5%, 0.15%, 0.32%, 0.1%, 0.21%, and 0.24%, respectively. This indicates that our method can consistently enhance performance across all categories in this dataset, ultimately achieving better overall performance. In addition, the experimental data in Table 3 reveal that replacing the baseline’s backbone network from STDC1 to STDC2 increases the model parameters from 8.15 M to 12.38 M (a 4.23 M increase) while improving performance from 60.31% to 61.03% (a 0.73% increase). In contrast, our attention mechanism innovation achieves a higher performance of 61.10% (a 0.79% improvement) with only a 2.26 M increase in parameters. This demonstrates that a well-designed attention module can enhance network efficiency more effectively.

As shown in Figure 9, based on the comparison within the black dashed box, it can be observed that both the V01 and V02 methods outperform the original method in road recognition, with the detected road regions demonstrate improved spatial continuity. In the detection of building areas, the regions identified by the V02 method are more refined compared to the original attention mechanism. Additionally, as seen in the fifth image, the V02 method demonstrates superior performance in water body recognition compared to both the V01 method and the original method, avoiding large-scale misidentification of water body areas.

4.5.2. Performance Comparison on LoveDA Dataset

In this experiment, we trained the network using the LoveDA dataset with higher resolution and richer semantic information. The experimental results are shown in

Table 4. Performance comparison of attention on LoveDA dataset.

Model	Bg	Build	Road	Water	Barren	Forest	Agri	Param (M)	Val mIoU (%)	Test mIoU (%)
pp-liteseg_stdc1_V01	43.06	53.45	55.56	78.20	17.83	45.83	61.19	10.41	46.69	50.73
pp-liteseg_stdc1_V02	43.63	55.52	53.69	78.11	17.06	43.04	58.80	10.41	45.64	49.98
pp-liteseg_stdc1_Spa	44.19	54.75	54.57	78.37	12.10	45.17	62.29	8.15	45.53	50.21
pp-liteseg_stdc2_V01	41.72	54.05	51.52	77.62	16.91	44.62	58.16	15.0	51.63	49.23
pp-liteseg_stdc2_V02	44.14	55.30	55.54	79.01	14.64	45.38	59.02	15.0	51.18	50.43
pp-liteseg_stdc2_Spa	43.79	51.30	54.24	77.29	15.88	45.19	61.53	12.38	50.26	49.89

Note: Bold values indicate the best result in each experiment.

. In the experiments using the STDC1 backbone, the V01 method achieved the best performance for both the validation set (46.69%) and test set (50.73%), surpassing the baseline by 0.52%, while the V02 method underperformed on the test set. In the STDC2 backbone experiments, it can be observed that in the test set, the V02 method achieves segmentation accuracies of 44.14% (background), 55.30% (buildings), 55.54% (roads), 79.01% (water bodies), and 45.38% (forested areas), outperforming other methods. Compared to the baseline, the V02 method improves IoU for these categories by 0.35%, 4%, 1.3%, 1.72%, and 0.19%, respectively. Notably, the most significant improvements are observed for building segmentation, with roads and water bodies also showing substantial gains. Additionally, the V01 method attains a 16.91% IoU for the barren category, surpassing both V02 and the baseline by 2.26% and 1.03%, respectively. Overall, although the V01 method achieved the highest validation mIoU in both experimental groups, its test set superiority was limited to the lightweight backbone (STDC1). In contrast, the V02 method demonstrated more prominent performance (50.43% test mIoU) with the STDC2 backbone. This result aligns with earlier findings: the V01 method exhibited stronger feature extraction ability but suffered from severe overfitting, whereas the V02 method mitigated overfitting at the cost of slightly reduced feature extraction capacity. In addition, compared to backbone replacement, attention mechanism modifications yielded more efficient performance improvements.

Furthermore, observations on the recognition performance of farmland areas reveal that when identifying categories with blurred boundaries, the V01 method achieves only 58.16% mIoU. This indicates that ambiguous boundaries may cause the model to learn incorrect features. Critically, the V01 method, which lacks residual connections, is more prone to learning erroneous local features, thereby inducing severe overfitting. This issue demands particular attention.

As shown in Figure 10, both V01 and V02 methods achieve higher segmentation accuracy for categories with distinct boundaries (e.g., urban buildings and water bodies), but exhibit degraded performance for regions with ambiguous boundaries, such as farmland. Furthermore, the V02 method achieves superior road segmentation performance compared to other methods, with the segmented road regions demonstrating enhanced connectivity. Notably, these observations align precisely with the conclusions drawn in the preceding subsection.

4.5.3. Comparative Experiments with Other Networks

In this experiment, we further reduce the volume of training data utilizing only the urban subset of the LoveDA dataset. The results are summarized in

Table 5. Comparison results with other models.

Model	Bg	Build	Road	Water	Barren	Forest	Agri	Param (M)	Val mIoU (%)
RS3Mamba	39.72	58.75	57.92	61.00	37.24	39.67	33.98	43.32	46.90
TransUNet	35.13	58.90	57.57	67.61	25.93	37.01	20.85	105.32	43.29
ABCNet	35.15	46.37	48.61	44.31	28.43	37.58	11.10	13.67	35.94
UNetformer	37.61	52.78	51.89	63.47	39.84	34.23	11.44	11.69	41.61
pp-liteseg_stdc2_V01	40.03	63.04	61.90	67.62	26.35	43.26	41.53	15.0	49.12
pp-liteseg_stdc2_V02	41.13	62.32	59.29	67.81	28.41	45.45	44.50	15.0	49.84
pp-liteseg_stdc2_Spa	41.98	61.62	60.18	68.17	24.71	43.12	44.61	12.38	49.19

Note: Bold values indicate the best result in each experiment.

. The V02 method achieves IoU scores of 28.41% for barren land and 45.45% for forested areas, while the V01 method attains 63.04% and 61.90% IoU for buildings and roads, respectively. Notably, all models exhibit stronger performance on barren land segmentation in urban scenarios, with the V02 method delivering a 3.7% improvement over the original spatial attention mechanism. Remarkably, this enhancement is achieved without significant degradation in other categories, yielding a 0.65% overall mIoU gain.

5. Discussion

The proposed multi-branch attention fusion method effectively addresses challenges posed by multi-scale targets and complex scenes in remote sensing image segmentation. Experiments confirm that this approach improves mIoU while maintaining a manageable parameter scale.

While these findings are encouraging, our current implementation utilizes only lightweight pixel, spatial, and channel attention mechanisms, limiting its performance ceiling. Notably, while advanced attention mechanisms, such as Vision Transformer’s patch-based self-attention, lead to superior performance, they incur significantly higher computational costs. Future work will focus on distilling and integrating such high-performance attention modules while balancing model complexity.

Finally, the proposed method is not designed to achieve optimal performance across all tasks with a single configuration, but rather to adapt to diverse task requirements in a lighter-weight, more comprehensive, flexible, and efficient manner. As demonstrated by our V01 and V02 variants, where merely adding or removing residual connections, the model achieves notable improvements on both WHDLD and LoveDA datasets. Additionally, we plan to extend this framework to cross-modal data fusion scenarios, further enhancing its applicability in remote sensing analytics.

6. Conclusions

In this paper, we explore multi-attention fusion strategies and ultimately propose a multi-branch attention fusion mechanism tailored for remote sensing semantic segmentation. This mechanism comprehensively extracts features from remote sensing data by integrating channel, spatial, and pixel-wise attention operations. Experimental results demonstrate that while our attention mechanism offers limited improvements on natural image segmentation tasks, it significantly enhances recognition accuracy for most categories in remote sensing imagery and delivers robust overall performance gains. Unlike many models designed for single-category scenarios, our fused attention mechanism demonstrably excels in handling complex environmental data.