DAMFusion: Multi-Spectral Image Segmentation via Competitive Query and Boundary Region Attention

Highlights

What are the main findings?

• A novel multi-source image fusion branch named DAMFusion is proposed, which integrates a Competitive Query Module, a Multimodal Fusion Module, and a Boundary-Aware Attention Multi-level Fusion Module. This architecture effectively alleviates cross-modal interference and low small-target segmentation accuracy in multispectral farmland imagery. It achieves state-of-the-art performance on the self-built OUC-UAV-MSEG dataset, with an F1-score of 0.927 and a mean Intersection over Union (mIoU) of 0.897.
• Ablation experiments verify the independent effectiveness of each core module in DAMFusion: the modal competitive query selection strategy outperforms learnable queries in feature representation. The combination of cross-modal attention fusion and boundary-region attention enhancement significantly improves the model’s ability to recover fine boundary details and distinguish similar farmland objects.

What are the implications of the main findings?

• The dynamic modal feature selection and cross-layer detail fusion design of DAMFusion provide a new technical paradigm for multispectral remote sensing image segmentation, which can be extended to other complex scene segmentation tasks such as urban land cover classification and natural disaster damage assessment.
• The self-built OUC-UAV-MSEG multispectral farmland dataset fills the gap of dedicated benchmark datasets in the field of agricultural UAV image segmentation. The high-precision segmentation capability of DAMFusion lays a technical foundation for practical smart agriculture applications including precise farmland management, variable-rate operations, and crop growth monitoring.

Abstract

To address the challenges of modal differences in multimodal farmland images and insufficient segmentation accuracy for small targets, this paper proposes a multi-source image fusion branch (DAMFusion) based on modal competitive selection. The branch dynamically selects infrared and visible light features through the Competitive Query Module (CQM) using Top-K screening, combined with IOU-aware loss optimization to avoid cross-modal interference. The multimodal fusion module (MMFormer) employs cross-modal attention and symmetric mechanisms, enhancing single-modal features through a self-enhancement module and unifying multimodal distributions via linear projection. The Boundary Region Attention Multi-level Fusion Module (BRM) extracts boundary information through feature differencing, strengthens it with spatial attention, and fuses it with shallow features to achieve cross-layer detail recovery. Through the collaborative design of dynamic modal feature selection, cross-modal distribution unification, and boundary region enhancement, DAMFusion effectively solves the problems of multimodal differences and small target segmentation in multispectral images, providing precise feature representation for fine farmland segmentation. Experiments on the OUC-UAV-MSEG dataset show that DAMFusion achieves 93.25% OA, 91.71% F1, and 89.70% mIoU, demonstrating clear advantages over representative comparison methods. In addition, ablation results verify the effectiveness of the proposed modules, where CQM improves OA from 91.00% to 93.25%, confirming the importance of discriminative modality selection before fusion.

1. Introduction

Against the backdrop of the global food security crisis and intensifying resource and environmental constraints, traditional farmland management methods that rely heavily on experience urgently require technological innovation. In the context of smart agriculture, intelligent analysis of UAV aerial imagery [1] is emerging as an important breakthrough for transforming agricultural production paradigms. UAV remote sensing systems [2], equipped with multispectral sensors and empowered by deep learning models, enable intelligent perception of crop growth conditions and spatial feature extraction, thereby promoting the shift in agricultural production from experience-oriented management to data-driven decision-making. Such technology supports variable-rate operations based on micro-scale differences in farmland through crop feature recognition and geographic element segmentation.

Benefiting from high-precision aerial imagery acquired by UAV platforms and feature fusion network [3] models, multidimensional farmland monitoring systems have been progressively established. Through comparative analysis of time-series images, early biological stress characteristics can be effectively identified and disaster-affected areas quantitatively assessed, providing a reliable data foundation for pest and disease warning models. Meanwhile, by integrating historical growth records with environmental parameters, yield prediction models can be constructed to support agricultural scheduling and resource allocation.

From the perspective of technological evolution, advances in multimodal remote sensing [4] data analysis have substantially expanded the agricultural application scenarios [5] of UAV systems. By improving network architectures and optimizing feature extraction modules, the generalization capability [6] of algorithms in complex farmland environments has continued to improve, driving agricultural remote sensing analysis toward practical engineering deployment. The widespread adoption of such intelligent sensing technologies is reshaping traditional decision-making patterns in agriculture and injecting new momentum into digital transformation [7].

To cope with the challenges posed by crop morphological diversity and spatial heterogeneity in farmland ecosystems, recent image analysis algorithms have significantly improved canopy coverage estimation accuracy by introducing attention mechanisms [8] and multi-scale feature fusion strategies [9]. These advances provide reliable spatial information support for digital farmland mapping, sowing optimization, and variable-rate cultivation, thereby enhancing the controllability and repeatability of agricultural operations. Existing studies further indicate that the engineering application of UAV remote sensing [10] and intelligent analysis technologies is highly consistent with the development needs of modern intensive agriculture. By integrating advanced sensing devices and intelligent algorithms, a collaborative farmland information perception [11] network between aerial and ground platforms is gradually being established, providing technical support for the digital management of the entire agricultural production process.

Despite these advances, the core challenge of multispectral farmland segmentation remains insufficiently addressed. Existing studies have demonstrated the potential of multimodal fusion for agricultural scene understanding; however, most of them primarily emphasize how to fuse multimodal information, while paying less attention to what should be selected before fusion and how to suppress cross-modal interference during fusion. This issue is particularly critical in farmland scenarios. On the one hand, infrared and visible modalities differ substantially in imaging mechanisms and spectral responses, which often leads to inconsistent feature distributions and unstable cross-modal correspondence. Direct fusion under such conditions may introduce redundant or misleading information, thereby weakening the exploitation of multimodal complementarity. On the other hand, farmland scenes are inherently complex: crops of different categories may exhibit similar appearances at certain growth stages, weeds and crops are often spatially interwoven, and fragmented small targets are widely distributed. As a result, segmentation models are prone to boundary ambiguity, local omissions, and inaccurate predictions in mixed regions.

Therefore, the problem addressed in this paper is not merely multimodal feature fusion itself, but how to construct a task-oriented segmentation framework for farmland scenes that can: (1) select more discriminative modal information before fusion to reduce early-stage cross-modal interference; (2) achieve effective cross-modal interaction and distribution alignment during fusion; and (3) enhance boundary-sensitive and small-target regions after fusion to improve fine-grained segmentation performance. From this perspective, existing methods still suffer from three major limitations. First, many approaches rely on fixed learnable queries or direct concatenation of multimodal features, lacking an explicit mechanism for selecting discriminative modal cues prior to fusion. Second, most fusion modules focus on feature interaction itself, but fail to balance unimodal representation preservation and cross-modal alignment. Third, current strategies mainly emphasize global semantic modeling while lacking dedicated enhancement for ambiguous boundaries and fragmented small targets, which are especially important in farmland segmentation.

To address these issues, this paper proposes a unified framework termed DAMFusion for multispectral farmland image segmentation, which follows a progressive pipeline of modality selection, modality fusion, and boundary enhancement. Specifically, a Competitive Query Module (CQM) is first designed to select more discriminative modal features from infrared and visible encoded representations through a Top-K competition strategy, thereby reducing invalid modal interference before fusion. Next, a Multimodal Fusion Module (MMFormer) is introduced to preserve unimodal representation capability while performing cross-modal interaction and distribution alignment. Finally, a Boundary Region Attention-based Multilevel Fusion Module (BRM) is developed to explicitly enhance ambiguous boundary regions and fragmented small targets through shallow-deep feature differencing and spatial attention, thereby improving fine-grained segmentation performance in complex farmland scenes.

The main contributions of this paper are summarized as follows:

(1) A competitive query mechanism for multispectral segmentation is proposed. Unlike existing methods that rely on fixed learnable queries or direct feature concatenation, the proposed CQM performs Top-K competitive selection on infrared and visible features before fusion, thereby reducing early-stage cross-modal interference at the source.

(2) A fusion module that jointly preserves unimodal representations and enables cross-modal alignment is proposed. The proposed MMFormer preserves the independent representation capability of each modality while achieving cross-modal information interaction and distribution alignment, thereby improving the efficiency of complementary multimodal feature utilization.

(3) A multilevel fusion strategy for boundary detail recovery is proposed. To address blurred boundaries, confusing regions, and fragmented small targets in farmland scenes, the proposed BRM explicitly enhances boundary-region responses, thereby enabling cross-level detail recovery and improving fine-grained segmentation performance.

2. Related Works

2.1. Unsupervised Fusion

Unsupervised multimodal fusion semantic segmentation algorithms combine the advantages [12] of unsupervised learning and multimodal fusion. Without manual annotation labels, they use [13] multimodal data to understand scenes and improve the accuracy and robustness of semantic segmentation. For example, ref. [3] in autonomous driving scenarios, visual images from vehicle cameras and LiDAR point cloud data can be fused for pixel-level classification.

TransFusion [4] can directly fuse 2D color images and 3D point cloud data without lossy preprocessing of point clouds. Compared to the baseline FCN model using depth images, mIoU improved by 4% and 2% on the Vaihingen and Potsdam datasets, respectively.

MMADTD [14] proposes an unsupervised domain adaptation method based on multimodal inputs of RGB and depth images. By designing a Depth Fusion Block (DFB) to calibrate depth information and ref. [15] using Depth Adversarial Training (DAT) to bridge the depth difference between source and target domains, it enhances sensitivity to depth information and shows significant performance improvement in multiple [16] synthetic-to-real adaptation benchmarks.

MultiDAN [17] proposes an unsupervised, multi-stage, multi-source, and multi-target domain adaptation network (MultiDAN), involving Multi-Source and Multi-Target Domain Adaptation (MSMTDA), Entropy-based Clustering (EC), and Multi-stage Domain Adaptation (MDA). However, in ref. [7], unsupervised fusion algorithms struggle to fully explore potential relationships between different modal data without prior [18] knowledge, potentially missing important information and affecting fusion effectiveness and semantic segmentation accuracy.

2.2. Contrastive Fusion

Contrastive learning is a self-supervised learning method [19]. Its core idea is to learn feature representations by comparing similarities and [20] differences between samples. In the context of semantic segmentation, contrastive learning compares different regions within an image or between different images. In semantic segmentation, fusion can occur between data of different modalities (such as simultaneously using image and LiDAR data) or between data of the same modality but different features (such as fusing texture features and color features of images). Contrastive [21] fusion algorithms incorporate contrastive learning mechanisms during the fusion of these data. After extracting features of different modalities or [22] types, contrastive fusion semantic segmentation algorithms fuse these features. During fusion, instead of simple addition or concatenation, fusion weights or methods are determined based on contrastive learning.

FTransUNet [23] constructs a hierarchical feature collaboration framework for remote sensing image analysis, with innovation reflected in the design [24] of multi-scale feature interaction mechanisms. The network adopts a dual-stream heterogeneous architecture: the local perception stream captures high-resolution texture details through improved residual modules, while [25] the global modeling stream uses improved visual Transformers to extract cross-regional semantic correlations. Notably, contrastive fusion algorithms have high [26] computational complexity when processing large-scale datasets, with long training times, which is not conducive [1] to application scenarios with high real-time requirements.

2.3. Based on Generative Adversarial Networks

Generative Adversarial Network (GAN)-based multimodal image feature fusion methods use GANs to learn features from different modal images. DRMF proposes a degradation-resistant multimodal image fusion method that enhances the robustness of image fusion through composable diffusion priors. This method is based on composable diffusion priors. It fully utilizes the complementary potential of diffusion models under different modalities and degradation conditions to mitigate various degradation issues. M-GAN [27], introduces a generative adversarial network based on multi-attribute learning and multimodal feature fusion for text-to-image synthesis tasks. M-GAN alleviates data imbalance issues through multi-attribute learning, enhances heterogeneous vocabulary and category-related labels, promotes the propagation of attribute information to images, and generates images that better meet task requirements. Diff-IF [28] proposes a diffusion model-based multimodal image fusion method that utilizes fusion knowledge priors to improve fusion effects. GAN-based multimodal image feature fusion methods have certain advantages in learning [29] features from different modal images but also have some shortcomings. GAN training is usually complex, prone to mode collapse and gradient vanishing, making it difficult [30] for the model to converge to the optimal solution. In multimodal image fusion, the distribution differences of [31] different modal data are large, and GANs may encounter difficulties when learning these differences, causing fluctuations [32] in the quality and accuracy of the generated fused images.

2.4. Multi-Task Fusion Optimization

Multi-task fusion optimized multimodal image feature fusion is a technique that combines multiple modal image data (such as visible light, infrared, text, etc.), aiming to simultaneously optimize multiple related tasks through a deep learning framework to improve image fusion effects and downstream task performance. CDDFuse proposes a frequency-domain-aware multimodal feature collaboration framework, innovatively constructing a heterogeneous feature extractor and a statistically dependent constrained joint optimization mechanism. The architecture adopts a dual-stream heterogeneous design: the frequency domain decoupling [33] module captures high-frequency detail features through invertible neural networks, while the lightweight global [34] modeling module extracts low-frequency semantic representations using an improved attention mechanism. Its core technical breakthrough lies in proposing a mutual information-guided feature reorganization strategy—by establishing a cross-modal feature correlation [35] matrix, multi-source inputs are decomposed into shared primitives and modal-specific features, and dynamic weight allocation mechanisms are designed to achieve adaptive fusion in feature space. Its innovative design is reflected in two aspects: the invertible feature transformation module maintains the integrity of high-frequency information through fidelity constraints, and the frequency domain contrast loss function effectively enhances the interpretability of fused images by maximizing mutual information of shared features and minimizing the correlation of specific features.

Co-FAFusion [36] is a dual-task-driven fusion architecture that effectively manages feature differences in image fusion and object detection tasks through a feature adaptive interconnection module, achieves adaptive selection of features in a unified space, and improves fusion effects and detection accuracy through alternating training. SegMiF [37] is a semantic feature-guided multi-interactive feature learning architecture for image fusion and segmentation. SegMiF aims to address the difficulty of finding suitable features for visual perception and semantic understanding tasks and overcome the limitations of existing multimodal datasets. However, multi-task fusion optimization methods usually require designing complex network structures and loss functions to achieve collaboration between tasks, which increases model complexity and training difficulty [38]. In practical applications, for different datasets and task scenarios, the model requires extensive parameter tuning and optimization to achieve good results, lacking certain universality and scalability.

In addition, some classical methods in camouflaged object segmentation (COS) are introduced for reference, since the fragmented and occluded characteristics of farmland plots are similar to camouflaged targets. Meanwhile, recent SAM-based remote sensing segmentation methods are systematically reviewed. Different from the original SAM focusing on general prompt-based segmentation, this paper improves the feature extraction and fusion paradigm according to the spectral complexity and boundary blur characteristics of farmland scenes, and integrates multimodal information to better adapt to small-scale and fragmented farmland segmentation tasks.

2.5. Comparison with State-of-the-Art Remote Sensing Segmentation Methods

In addition, several state-of-the-art multimodal remote sensing segmentation methods are closely compared, including CMX, AMMFuseNet, NCGLF², HyperLi-Net, and PFGFE-Net. CMX focuses on cross-modal transformer fusion but lacks explicit modality selection before fusion. AMMFuseNet enhances feature aggregation but ignores discriminative information screening and boundary enhancement. NCGLF² integrates global and local features but does not handle modality competition and small-target details. HyperLi-Net and PFGFE-Net pursue efficient fusion yet lack unified cross-modal distribution alignment and boundary-aware refinement.

Different from these methods, DAMFusion follows a unified pipeline: modality competitive selection, cross-modal distribution unification, and boundary region enhancement, which explicitly solves modality discrepancy, cross-modal interference, and blurred small-target boundaries. This is the essential reason why DAMFusion achieves stronger segmentation performance than the above methods.

3. Difficulty Analysis

3.1. Difficulties in RGB and Multispectral Image Fusion

Current deep learning technologies face significant application bottlenecks in the analysis of low-altitude multispectral remote sensing data. The core constraints stem from the complexity of data modalities: multi-source spectral images not only contain high-dimensional features of more than four spectral channels but also contain rich information on ground object reflectance differences. Such data can effectively enhance vegetation identification and land cover classification discrimination through combination operations of visible and non-visible light bands. However, the existing technical system has dual constraints: first, there are spectral response differences between multispectral imaging payloads and visible light sensors at the hardware level, which can easily lead to reduced cross-modal image registration accuracy; second, the collaborative representation mechanism between high-dimensional spectral features and spatial features is not yet perfect, making it difficult for network models to establish stable mapping relationships between spectral dimensions and ground object categories. The spatial registration errors and feature fusion obstacles of such multi-source heterogeneous data severely restrict the engineering deployment of end-to-end deep learning frameworks.

3.2. Difficulties Caused by Farmland Application Scenarios

(1) High inter-class similarity. Farmland crops are diverse with different growth stages, resulting in large morphological differences within the same crop category and small differences between different crops or even between crops and weeds. As shown in Figure 1, the first row shows the segmentation model mistakenly dividing the same crop into two crop instances. The second row shows the segmentation network mistaking two different crops for instances of the same crop. At specific growth nodes, different crop varieties appear very similar, such as wheat and peanuts at the seedling stage, where leaf morphology and color are almost identical. When shot from high altitude by UAVs, plant height differences are not obvious, and leaf textures are difficult to distinguish, making precise division of different crop areas very difficult. Moreover, plants change significantly in appearance during growth, such as corn at maturity and seedling stages appearing completely different. Additionally, due to different planting times, the same type of vegetation at different growth stages is distributed alternately, seriously interfering with image recognition-based model detection and greatly increasing the difficulty of accurate identification and segmentation.

(2) Blurred vegetation growth boundaries. As shown in Figure 2, weeds and crops often grow together. Weeds are very similar to crops in growth morphology, leaf shape, and plant height, intertwining with each other and blurring visual boundaries, making it extremely difficult to accurately distinguish between crop and weed boundaries in images. Meanwhile, due to diverse farming patterns, plants of different varieties and growth stages grow mixed together, with complex spatial relationships between plants, further aggravating boundary blurring. Additionally, crops are susceptible to pests and diseases during growth. Once infected, the morphology and color of diseased parts change. For example, leaves infected with fungi develop disease spots with unclear boundaries between diseased and normal tissues, interfering with precise segmentation in images and hindering image-based analysis.

3.3. Difficulties in Model Design

When designing network models, it is necessary to balance model size (number of parameters and computational complexity), inference speed, and segmentation accuracy. Laboratory training resources are limited. Excessively large model parameter counts may cause memory overflow, preventing training from starting; excessive computational complexity can slow training speed, affecting experimental progress. Successful model design requires finding a balance between model size and computational efficiency while improving inference speed and segmentation accuracy.

When designing network models, key factors such as model size (including parameter count and computational complexity), model inference speed, and segmentation accuracy must be considered. Under the constraints of laboratory hardware resources, the engineering implementation of deep learning models faces dual challenges. First, limited by the memory capacity of the experimental environment, when the complexity of the network architecture exceeds hardware capacity thresholds, the system triggers storage resource exhaustion alerts, causing the training process to be interrupted during initialization. This technical barrier makes it difficult for algorithm designs with theoretical superiority to obtain effective support from empirical research. Second, for model structures that meet memory constraints, the spatiotemporal complexity of their computational graphs may still cause gradient update abnormalities: excessively high floating-point operation requirements not only significantly reduce iteration rates but may also cause computational unit scheduling disorders, extending model convergence periods to several weeks. This severe imbalance in computational efficiency not only restricts the controllability of experimental cycles but also directly affects the overall progress of research tasks. Therefore, successful model design requires finding a balance between model size and computational efficiency. Additionally, it is necessary to improve model inference speed and segmentation accuracy as much as possible while ensuring efficient model design. Fast inference speed can provide users with a good experience in practical applications, while good segmentation accuracy is a core metric for measuring model performance. Only by comprehensively considering and balancing these factors can practical and efficient models be designed [39].

4. Algorithm Design

To address the problem of insufficient data fusion in farmland segmentation scenarios and modal interference issues, a modal competitive query selection strategy is designed, combined with a multimodal fusion module and a boundary region attention multi-level fusion module to improve fusion effects; the segmentation scheme draws on camouflaged target segmentation ideas, using an information prompt enhancement module to optimize the SAM model, solving problems such as blurred crop edges and incomplete segmentation.

4.1. Overall Algorithm Framework

Given a pair of infrared and visible images, this paper uses two parallel encoders to extract and encode their features. As shown in Figure 3, subsequently, the encoded features are flattened, concatenated, and input into the Competitive Query Module (CQM). This module selects significant modal features as initial object queries.

Next, this paper designs a Transformer-based MMformer to unify the distribution of multimodal features, fusing features between visible and invisible light. Then, a Boundary Region Attention-based Multi-level Fusion Module (BRM) integrates cross-layer features and utilizes details around boundaries to enhance small target segmentation, recovering detailed information around boundary regions. Finally, these refined object queries obtain the final result output through the segmentation head. The following will elaborate on the proposed key components in detail.

4.2. Competitive Query Module

Since learnable object queries [40] have no clear physical meaning, their optimization is difficult. In infrared and visible images, the gap between the two modal features further increases the complexity of optimizing learnable object queries, making it difficult to adapt to task requirements.

In infrared-visible target detection tasks, complementary features have dynamically changing characteristics. Selecting object queries from encoded feature maps can better adapt to these dynamic changes compared to learnable object queries. The Top-K [41,42,43] method selects features from encoded feature maps and, compared to learnable object queries, can better handle dynamic feature changes.

To address the above modal difference challenges, we design the Competitive Query Module (CQM) as follows. Specifically, the encoded feature sequences from infrared and visible modalities are concatenated and fed into a linear projection layer to obtain feature point scores. From this combined feature representation, the Top-K scoring features are selected as initial object queries. These Top-K features originate from infrared or visible features, each representing an object instance specific to its respective modality. This approach can be defined as follows:

$$\begin{array}{c}\hfill z=\mathrm{Top}\text{-}K\left(\mathrm{Linear}\right(concat(I,V)\left)\right),\end{array}$$

(1)

where z represents the selected set of K features, and I and V represent the flattened encoded infrared feature sequence and visible feature sequence, respectively.

As mentioned, infrared or visible images may contain useless interference information that could confuse the network. Modality-specific features are competitively selected to construct a significant feature representation for each object. This approach helps prevent the introduction of interference from other modalities in the early stages and provides useful prior information for refining queries in subsequent decoders, emphasizing that the representation of the queried object should prioritize the modality from which it originated. Additionally, this article uses an optimization strategy with IOU-aware classification loss [40,44] to further improve the quality of selected features. The IoU-aware loss is jointly optimized with the overall network to promote the quality of selected features in CQM, rather than acting as an independent component. Therefore, independent ablation verification of this loss function is not separately conducted in this paper.

4.3. Multimodal Fusion Module

The Multimodal Fusion Module (MMformer) consists of cross-modal attention and a multilayer perceptron (MLP) [45]. Cross-modal attention is the key fusion component in the layer, unifying and fusing cross-modal features. This process can be described as follows:

$$\begin{array}{c}\hfill X_{i1}^{\mathrm{out}}=MLP\left(CrossAtten\left(LN\left(X_i\right)\right)+X_i\right)\end{array}$$

(2)

where $X_i$ represents the visible light characteristic $R_i$ or the normalized infrared characteristic $D_i$ of the i-th encoder layer, and LN represents the layer normalization operation.

Figure 4 shows the details of the multimodal fusion module. Two self-enhancement modules further emphasize the independent feature expression of each modality, while the cross-modal key–query correlation module reconstructs and fuses multimodal features and global attention. Then, the outputs of each module are fused through token addition and concatenation operations. The fused features are updated into RGB and NIR feature streams via linear projection and sent to the MLP.

As shown in Figure 4, this paper uses two symmetric attentions in the module, concatenating RGB and NIR features from the two streams to generate Q and completing the initial combination. K and V are generated by each modal feature stream in symmetric attention. The specific generation process formula is as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}Q=Linear\left(AvgPool\left(concate\left(R_{ij},D_{ij}\right)\right)\right)\hfill \\ K=Linear\left(X_{ij}\right)\hfill \\ V=Linear\left(X_{ij}\right),X_{ij}=LN\left(X_{i\dot{c}}^{\mathrm{out}}\right)\hfill \end{array}\right.\end{array}$$

(3)

where $X_{ij}$ represents the enhanced RGB feature $R_{ij}\in {\mathbb{R}}^{H\times W\times C^{RGB}}$ or the enhanced NIR feature $D_{ij}\in {\mathbb{R}}^{H\times W\times C^{NIR}}$. Effective Cross-modal key–query correlation can be described as: $X_{ij}^{\mathrm{fusion}}=Upsample(Softmax({\displaystyle \frac{Q{K}^{\mathrm{T}}}{\sqrt{d}}})·V)$.

4.4. Boundary Region Attention Multi-Level Fusion Module

To address the challenges of blurred boundaries, fragmented small targets, and confusing regions in farmland scenarios analyzed in Section 3, this paper designs BRM to recover spatial detail features and enhance fine-grained segmentation performance by fusing cross-layer features.

To recover spatial detail features of small targets, this paper designs BRM to fuse cross-layer features. As can be seen from Figure 5, the model consists of two components, marked in red and blue. In the red part, element-wise subtraction is used to calculate the feature difference between shallow and deep layers. Considering that shallow layers contain more detailed information than abstract features, the subtraction result mainly contains detailed information including information on the target boundary region. The boundary region refers to the features of several types of targets around the boundary that may be mistaken for other categories. Therefore, this paper uses spatial attention [46] to emphasize this part of the features to improve the prediction accuracy of boundaries that are important for small target prediction. Here, SA’ denotes the refined spatial attention features, and CBR denotes the convolution-batch normalization-ReLU activation module used for feature transformation. The red part content can be calculated as:

$$\begin{array}{c}\hfill F_{\mathrm{dif}}=CBR\left[\left({\mathrm{SA}}^{\prime }\left(F_{\mathrm{deep}}-F_{\mathrm{low}}\right)\right)\odot F_{\mathrm{deep}}+F_{\mathrm{low}}\right]\end{array}$$

(4)

where SA represents the spatial attention layer in BRM, composed of a 3 × 3 convolutional layer, 1 × 1 convolutional layer, BN, and Sigmoid function. CBR represents the combination of 3 × 3 convolutional layer, BN, and RELU.

4.5. Loss

To further enhance the reliability of prediction confidence, we introduce an IoU-aware classification loss, which explicitly aligns the predicted confidence with the localization quality. Specifically, the classification score is encouraged to reflect not only the probability of being foreground but also the overlap quality between prediction and ground truth. The loss is defined as:

$${\mathcal{L}}_{\mathrm{iou}\text{-}\mathrm{cls}}=-\left[y·\mathrm{IoU}·log\left(p\right)+(1-y)·log(1-p)\right],$$

(5)

where p denotes the predicted confidence, $y\in \{0,1\}$ is the ground-truth label, and $\mathrm{IoU}\in [0,1]$ measures the overlap between prediction and ground truth. By weighting positive samples with their IoU scores, this loss encourages high-confidence predictions to correspond to high-quality regions, thereby improving prediction ranking and robustness.

The overall loss of PopNet is defined as

$${\mathcal{L}}_{\mathrm{total}}^{\mathrm{new}}={\mathcal{L}}_{\mathrm{dep}}+{\lambda }_1{\mathcal{L}}_{\mathrm{loc}}+{\lambda }_2{\mathcal{L}}_{\mathrm{wtv}}+{\alpha }_1{\mathcal{L}}_{\mathrm{sep}}+{\alpha }_2{\mathcal{L}}_{\mathrm{sem}}+\beta {\mathcal{L}}_{\mathrm{iou}\text{-}\mathrm{cls}}.$$

(6)

Here, ${\mathcal{L}}_{\mathrm{dep}}=\mathrm{SSIM}(D_{po},D_{sf})$ preserves the structural consistency of the source-free depth, ${\mathcal{L}}_{\mathrm{loc}}$ enforces local smoothness within object regions to suppress noisy depth variations, and ${\mathcal{L}}_{\mathrm{wtv}}$ enhances depth discontinuities along semantic boundaries, improving foreground-background separability. The loss ${\mathcal{L}}_{\mathrm{sep}}=\mathrm{BCE}(S_s,G)$ supervises the pseudo semantics derived from the contact surface, while ${\mathcal{L}}_{\mathrm{sem}}$ provides direct supervision on the final semantic prediction.

The hyperparameters ${\lambda }_1$, ${\lambda }_2$, ${\alpha }_1$, ${\alpha }_2$, and $\beta$ control the balance among geometric regularization, depth-to-semantic transformation, direct semantic supervision, and quality-aware classification. In particular, ${\mathcal{L}}_{\mathrm{iou}\text{-}\mathrm{cls}}$ complements ${\mathcal{L}}_{\mathrm{sem}}$ by explicitly encoding prediction quality into confidence estimation, leading to more reliable and well-calibrated outputs.

5. Experiments and Evaluation

5.1. Dataset Introduction

Facing the current lack of specialized multispectral UAV segmentation benchmarks for agricultural scenarios, this study independently constructs a dedicated dataset. Data acquisition uses the DJI M3M multispectral aerial survey system, completing aerial collection by planning grid waypoints in Qingdao farmland areas. After radiometric correction and geometric registration, systematic annotation and quality verification are performed on the acquired images. To comprehensively evaluate model performance, the experimental plan includes a dual verification system: in addition to the self-built dataset, internationally public Vaihingen [47] and Potsdam benchmark datasets are simultaneously adopted. The Vaihingen dataset provides 20 cm resolution aerial images, and the Potsdam dataset contains 5 cm resolution urban land cover data, both providing important references for verifying the generalization ability of segmentation algorithms.

UAV-OUC-MSEG Dataset.

The main acquisition device for the farmland image semantic segmentation dataset captured in this paper is the DJI M3M(SZ DJI Technology Co., Ltd., Shenzhen, China). Figure 6 shows the actual situation of the UAV.

In this study, the collected multispectral farmland image semantic segmentation dataset is OUC-UAV-MSEG, which covers five categories: Peanut, Corn, Soil (soil, open space), Tree (trees, weeds), and Other (roads, buildings, ponds), aiming to fully reflect the complex ecological composition of farmland.

To obtain farmland image information from different perspectives, the UAV performed shooting tasks at multiple altitude segments. Specifically, the UAV descended stepwise from 1500 m high altitude in a nadir manner to about 50 m. During this, in the 1000–1500 m altitude range, data were collected at 1500 m and 1000 m altitude; in the 100–500 m altitude range, shooting was performed sequentially at every 100 m altitude difference. Such fine altitude stratified collection aims to find the most suitable shooting altitude that can ensure ideal farmland image segmentation results, accurately identifying various crops and environmental elements, while maximizing the coverage of farmland area under the limited field of view of the UAV. For this purpose, this study carefully compiled farmland datasets at different shooting altitudes for experimental analysis. Figure 6 shows partial sample images of farmland datasets at different shooting altitudes, with 500 farmland images shot at each altitude, over 2000 available images, with pixel specifications of 3840 × 2160, fully ensuring data richness and image clarity to meet high-precision academic research needs. To improve the credibility and reproducibility of the proposed dataset, systematic annotation and quality control procedures were further conducted after radiometric correction and geometric registration. The semantic annotation was performed using Labelme, and all images were labeled at the pixel level according to a unified farmland interpretation protocol. Specifically, the OUC-UAV-MSEG dataset was organized into five semantic categories, including Peanut, Corn, Soil, Tree, and Other. Annotation work was independently completed by 3 master’s students majoring in computer science with more than 2 years of experience in remote sensing image semantic annotation and proficient in using the Labelme tool. A senior doctoral student served as the quality inspector to conduct a secondary check on all samples, with a 100% correction rate for fuzzy boundaries and mislabeled regions.

To ensure annotation consistency, the labeling process followed a manual annotation and verification workflow. After the initial annotation, quality verification was carried out to correct ambiguous boundaries and mislabeled regions. In addition, the dataset construction process included radiometric correction, geometric registration, and systematic annotation quality checking, which helped improve the spatial consistency between visible and multispectral images and enhanced the reliability of downstream segmentation experiments.

5.2. Experimental Details

Before training, the OUC-UAV-MSEG dataset was further preprocessed to improve sample diversity and model generalization. Specifically, spatial-domain augmentation strategies, including axial flipping, random rotation, and regional cropping, were applied synchronously to the images and their corresponding labels. After augmentation, the dataset was expanded to 1438 sample pairs and was divided into the training, validation, and test sets at a ratio of 8:1:1.

This paper trains on 4 NVIDIA RTX 4090Ti GPUs with a batch size of 8. The AdamW optimizer is used with ${\beta }_1=0.9,{\beta }_2=0.999$, and weight decay of 0.01. During the warm-up phase, the learning rate is updated via ${\displaystyle \frac{\mathrm{iter}}{\mathrm{warm}\text{\_}\mathrm{iter}}}$, then set by multiplying by ${\left(1-{\displaystyle \frac{\mathrm{iter}}{\mathrm{warm}\text{\_}\mathrm{iter}}}\right)}^{0.9}$. This paper conducts 300 iterations of training on the Vaihingen and Potsdam datasets, and 500 iterations on the UAV-OUC-MSEG dataset. Validation is performed in both quantitative and qualitative aspects.

5.2.1. Qualitative Experiments

This section will conduct comparative analysis experiments with other semantic segmentation models on the OUC-UAV-MSEG dataset, selecting mainstream semantic segmentation architectures such as DeepLabV3+ and MMNet as baseline models. Because they are widely used in academia and industry, representative of different technical approaches; achieve good results on public datasets, serving as evaluation references to measure the performance of new methods; and have diverse technological innovation points, enabling comprehensive testing of the technical advantages and competitiveness of this work. Experiments follow the principle of controlling variables, uniformly configuring Res2Net-101 as the feature extraction backbone, and freezing parameters to ensure fairness.

As shown in Figure 7, by comparing the forward inference results of the UAV-OUC-MSEG validation set and test set, visual analysis shows that compared to the latest models such as CMX, $NCGL{F}^2$, and DFMRTFNet, this method has significant advantages in maintaining farmland boundary integrity and segmentation accuracy of small fragmented plots. Qualitative evaluation results and quantitative index calculations form complementary verification, jointly supporting the technical superiority of the proposed multispectral fusion mechanism in complex agricultural scenarios.

The visualization results on UAV-OUC-MSEG are shown in Figure 6. From left to right are the original image, invisible light image, Ground Truth, DeepLabV3+, MMNet, AMMFuseNet, CMX and NCGLF², DFMRTFNet, and the segmentation effect of the method proposed in this paper.

In the UAV-OUC-MSEG dataset, blue represents corn, green represents peanuts, brown represents trees and weeds uniformly labeled as (tree), gray-pink represents land, and purple represents buildings uniformly labeled as (other). It can be seen that compared to networks such as NCGLF² and DFMRTFNet, the method in this paper performs better when handling boundaries between different crops. In the Potsdam and Vaihingen datasets, green represents trees, blue represents buildings, yellow represents vehicles, cyan represents other low vegetation, and red represents the background. From the experimental results, it can be seen that the method proposed in this paper shows better category discrimination ability, and the segmentation effect is closest to Ground Truth.

5.2.2. Quantitative Experiments

From the visualization results of qualitative experiments, it can be seen that the method in this paper has achieved good results in farmland scenarios. However, to more accurately analyze the performance of the method, a series of quantitative experiments are needed.

Experimental data show that due to the significant spectral-geometric feature complexity of the UAV-OUC-MSEG dataset, the mean intersection over union of all comparison models on this dataset has decreased.

Meanwhile, we also evaluate the model complexity. Our model has 223.9 M parameters, 107.3 G MACs, 16.4 FPS, and 263.6 MB GPU memory usage, achieving a good balance between model scale, efficiency and segmentation accuracy.

Table 1. Comparison of Validation Results of Various Network Models in OUC-UAV-MSEG Dataset.

	Pub./Year	OA	MP	Recall	F1	mIoU	FWIoU
DeepLabV3+ [16]	ECCV/2018	79.84	73.66	75.07	73.80	69.03	70.03
MMNet [12]	Appl. Intell/2022	83.97	78.63	82.21	81.28	78.33	78.39
CMX [13]	T-ITS/2023	83.12	83.49	84.07	84.27	81.97	83.94
NCGLF2 [3]	Inform Fusion/2024	85.83	86.33	84.88	86.10	83.03	84.16
AMMFuseNet [4]	Appl Earth Obs/2024	87.49	88.33	86.69	87.51	85.73	86.34
Ours		93.25	92.70	91.87	91.71	89.70	88.83

Note: Bold values indicate the best performance (state-of-the-art) in each column.

shows that our method outperforms all competitors by a large margin. Such improvements are mainly attributed to the CQM for discriminative modality selection, MMFormer for effective cross-modal fusion, and BRM for boundary and small target enhancement, which together improve the segmentation performance comprehensively.

5.3. Ablation Experiments

(1) Effectiveness of Modal Competitive Query Selection.

To verify the effectiveness of the Competitive Query Module (CQM), competitive queries were first set as learnable queries. As shown in

Table 2. Ablation Experimental Results of Modal Competitive Query Selection on OUC-UAV-MSEG Dataset.

Method	Evaluation Metrics (%)
	OA	MP	Recall	F1	mIoU	FWIoU
Learnable Query	91.00	90.74	88.87	89.30	89.09	88.32
Modal Competitive Selection Query	93.25	92.70	91.87	91.71	89.70	90.73

Note: Bold values indicate the best performance (state-of-the-art) in each column.

, it can be seen that using CQM brings better results because it provides stronger prior information for the subsequent multimodal fusion module in the multispectral decoder.

(2) Effectiveness of Multimodal Fusion Module MMformer.

In the baseline network, this paper removed MMformer, used concatenation operation followed by a convolutional layer to basically fuse multimodal features, and replaced BRAFM with a simple addition operation. Then, MMformer and BRAFM were introduced sequentially to evaluate each module. Figure 8 shows the visualized prediction output when introducing the multimodal fusion module. Compared with simple concatenation, the multimodal fusion module can better fuse complementary information of visible and invisible light, accurately predict building areas obscured by shadows, and distinguish tree and low vegetation categories with similar textures. Experimental results are shown in

Table 3. Ablation Experimental Results of MMformer and BRM.

Method			OA	MP	Recall	F1	mIoU	FWIoU
MMformer	w/o Token	BRM
✓			85.43	86.30	88.16	83.04	80.24	79.97
	✓		84.21	85.00	86.45	82.37	79.64	78.02
✓		✓	87.24	89.06	90.11	87.45	82.90	83.27

. Compared with without token addition fusion, the average scores of OA, F1, and mIoU increased by 1.22%, 0.67%, and 0.6%, respectively. Experimental results prove the effective contribution of the token addition fusion operation.

(3) Effectiveness of Boundary Region Attention Multi-level Fusion Module BRM. Figure 9 shows the effect diagram of introducing the boundary region attention multi-level fusion module. BRM uses subtraction to focus on boundary regions that are easily mistaken and uses addition operations to focus on target regions. To further verify the effectiveness of the proposed method in small target segmentation, we provide visual comparison in Figure 9. It can be observed that DAMFusion achieves clearer boundary retention and more complete segmentation for small, fragmented farmland plots compared with other methods. The qualitative results in Table 3 fully demonstrate that our method can effectively enhance the representation of small targets and alleviate the problem of missing or incomplete segmentation for small and scattered instances in farmland scenarios.

6. Conclusions

This paper proposes DAMFusion, a unified framework for multispectral farmland image segmentation, which addresses cross-modal interference between infrared and visible features as well as challenges in boundary and small-target segmentation through a progressive pipeline of modality selection, fusion, and boundary enhancement. Specifically, CQM performs discriminative modality selection, MMFormer facilitates cross-modal interaction and distribution alignment, and BRM enhances boundary details. Experimental results on the OUC-UAV-MSEG dataset demonstrate that DAMFusion achieves 93.25% OA, 91.71% F1, and 89.70% mIoU, outperforming existing methods. Ablation studies further validate the effectiveness of each module, particularly the role of modality selection in suppressing cross-modal interference. Overall, the study highlights that pre-fusion modality selection, cross-modal alignment with unimodal representation preservation, and explicit boundary enhancement are key to improving fine-grained segmentation in complex farmland scenarios. Future work will focus on model lightweighting and dataset expansion to enhance practical applicability.