YOLO-MOD: An Instance Segmentation Algorithm for Pomelo Fruit and Fruit Stem Based on YOLOv11-Seg

Abstract

This study aims to develop an instance segmentation model for the joint segmentation of pomelo fruits and stems in complex natural orchard environments, with particular emphasis on slender, small-scale, and easily occluded stem targets. To this end, YOLO-MOD, an improved instance segmentation algorithm based on YOLOv11-seg, is proposed. Specifically, Omni-Dimensional Dynamic Convolution (ODConv) is introduced into the C3k2 module to enhance complex feature representation; a Multi-Scale Dilated Attention (MSDA) module is embedded to improve the multi-scale semantic perception of slender stem regions; and the original upsampling operator is replaced with DySample to strengthen fine-grained boundary recovery. Experimental results show that, compared with the original YOLOv11-seg, YOLO-MOD improves the Box mAP@50 and Mask mAP@50 by 2.9% and 3.9%, respectively. For the Stem class, the Box mAP@50 and Mask mAP@50 increase from 71.9% to 77.8% and from 68.4% to 76.2%, respectively. These results indicate that YOLO-MOD can achieve fine-grained segmentation of pomelo fruits and stems on the dataset used in this study. However, its generalization capability across different orchards, seasons, pomelo varieties, and fruit types still requires further evaluation, and its practical effectiveness in an integrated robotic harvesting system remains to be further validated.

1. Introduction

Pomelo is an important economic citrus fruit, rich in vitamins, dietary fiber and various bioactive components, with high edible and processing value [1,2]. In pomelo harvesting, fruit picking, grading and handling usually account for high labor costs. Against the backdrop of labor shortages and rising labor expenses, developing automated harvesting equipment for natural orchards has become a major focus of the industry [3,4]. The visual perception system is a key component of harvesting robots. Accurate recognition and segmentation of fruits and stems directly determine the reliability of subsequent structural parsing and the system’s robustness in complex scenarios [5,6]. However, pomelo orchards have typical unstructured characteristics: dense canopies lead to severe branch and leaf occlusion, frequent fruit surface reflections and shadows, and large fruits with obvious posture changes. Meanwhile, fruit stems are slender, small-scale, and easily confused with fine branch textures. Under dense fruiting and mutual occlusion, missed and fractured segmentation often occur, reducing segmentation quality and the stability of key structural information extraction. These factors increase the difficulty of pomelo detection and segmentation tasks [5,7].

In orchard object segmentation, early studies mainly adopted traditional computer vision techniques. Target regions were extracted through color space conversion, threshold segmentation, edge detection, and morphological processing [8,9]. Although these methods perform adequately under stable illumination or simple backgrounds, they lack robustness to reflections and shadows in natural orchard environments. They also exhibit poor cross-scene generalization. Moreover, for slender structures such as fruit stems, traditional methods are highly vulnerable to occlusion and similar textures. This often results in incomplete boundaries and weak connectivity, making it difficult to meet precise segmentation requirements in complex scenes.

As computer vision continues to advance, visual object detection and spatial localization in harvesting robots have progressively shifted from traditional image-processing methods toward deep learning approaches [10,11]. Models from the YOLO series are extensively utilized for object detection, and their subsequent versions, along with extended variants, demonstrate outstanding performance in image segmentation as well. Tao et al. [12] embedded the GAM attention mechanism and SPD-Conv convolution blocks into YOLOv8n, and improved the Wise-IoU loss function. Their method achieved 75.9% mAP for apple detection and 75.7% mAP for apple segmentation under complex unstructured orchard conditions. Zhang et al. [13] developed a lightweight YOLOv4-based apple detection model. They reconstructed the neck and YOLO head using depthwise separable convolutions, and added an attention module to the Feature Pyramid Network (FPN). On the apple dataset, the improved model reached a mAP of 95.72%, 3.45% higher than the original YOLOv4, and performed well under various lighting conditions. For tomato stem detection, Jing et al. [14] introduced the EDI-YOLO model aimed at segmenting the main stems and branches of tomatoes. This approach achieved a mean Average Precision (mAP) score of 79.3%, which contributes significantly to the efficiency of tomato harvesting robotics. Furthermore, Li et al. [15] explored citrus fruit and stem segmentation, alongside determining picking points, using an advanced version of the YOLOv8n-seg algorithm. Their method involved incorporating lightweight convolutional networks, attention mechanisms, and a specialized layer for detecting small objects, thereby improving the model’s capacity to identify and segment both citrus fruits and their stems. Additionally, Baek et al. [16] developed the AS-YOLO model, which incorporates Ghost bottlenecks and a global attention mechanism within the YOLO framework to enhance apple stem segmentation. The findings indicated that this approach preserved the model’s lightweight nature while significantly boosting performance in segmenting slender stem structures. For fruit picking-point localization, Li et al. [17] presented a mango picking-point localization method combining improved YOLOv8 object detection and instance segmentation. It realized synchronous recognition of mango fruits and bearing branches, and completed picking-point localization based on instance segmentation outputs, providing a reference for collaborative perception of fruits and stems and picking-point localization. Ghose et al. [18] proposed YOLO-SAM AgriScan for ripe strawberry detection and segmentation by combining YOLO-based detection with SAM-based mask generation. More recently, Rana et al. [19] conducted a multi-domain evaluation of YOLOv8–YOLOv12 models for grape cluster detection under complex vineyard conditions, considering challenges such as occlusion, heterogeneous illumination, spectral variability, RGB/NIR domain shifts, and image degradation. Their findings highlighted that the evaluation of agricultural vision models in real-field environments should extend beyond overall accuracy and also consider robustness to environmental interference and domain variation. This perspective is particularly relevant to the present study, since the joint segmentation of pomelo fruits and stems is similarly affected by branch and leaf occlusion, uneven illumination, low-contrast slender stems, and cluttered natural backgrounds. Accordingly, improving the robustness of stem segmentation under challenging orchard conditions is essential for establishing reliable visual perception in intelligent pomelo harvesting systems. Despite the progress made in fruit detection, stem segmentation, and picking-point localization, studies focusing on joint instance segmentation of pomelo fruit and stem remain relatively limited.

Especially in complex natural orchard scenarios, existing methods still face several common challenges: First, affected by branch and leaf occlusion, fruit surface reflection, and fruit overlapping, models lack sufficient ability to extract local distinctive features of target regions, often leading to missed detection of stems. Second, fruit stems are slender, small-scale, and low-contrast. Existing models show limited capability in multi-scale semantic representation for such targets under complex backgrounds. Third, traditional upsampling and feature recovery strategies are insufficient in reconstructing fine boundary details, which easily causes stem mask fragmentation, blurred boundaries, and structural discontinuities. Therefore, how to improve the fine segmentation performance of pomelo fruits and stems in complex natural environments while controlling model complexity is still a problem worthy of in-depth investigation.

To address the task-specific challenges described above, this study develops YOLO-MOD, an application-oriented instance segmentation framework based on YOLOv11-seg for joint segmentation of pomelo fruits and stems in complex orchard environments. Rather than introducing a new generic segmentation paradigm, the proposed framework is designed according to three major difficulties encountered in this task: local structural ambiguity between stems and background branches, insufficient multi-scale representation of small and partially occluded stems, and boundary discontinuity during mask reconstruction. Accordingly, the following targeted modifications are introduced:

(1)

ODConv is integrated into the C3k2 module to enhance the adaptive representation of local structural differences among pomelo fruits, stems, and background branches.

(2)

The MSDA module is incorporated to strengthen multi-scale contextual modeling of small-scale and partially occluded stem regions under complex background interference.

(3)

The original upsampling operator is replaced with DySample to improve feature reconstruction quality and preserve the boundary continuity of slender stem masks.

Through these task-oriented modifications, YOLO-MOD aims to improve the joint instance segmentation of pomelo fruits and stems under occlusion, uneven illumination, and complex background interference, while maintaining moderate model complexity. The effectiveness and complementary contributions of ODConv, MSDA, and DySample are evaluated through comparative and ablation experiments. It should be noted that the present study focuses on front-end visual perception based on instance segmentation and does not include downstream picking-point localization, three-dimensional pose estimation, or robotic harvesting experiments. Therefore, the proposed framework is positioned as a visual perception component for subsequent studies on picking-point localization and robotic harvesting, and its practical performance in a complete robotic harvesting system remains to be further validated.

2. Experimental Setup and Evaluation Metrics

This section mainly introduces the dataset construction, experimental environment configuration, and model evaluation metrics used in this study. First, the image collection scenarios, sample sources, and data distribution of pomelo fruits and stems are described. Second, the data annotation rules and dataset production process are elaborated. Finally, the experimental platform configuration and definition of evaluation metrics are presented. The above content lays an experimental foundation for subsequent model training, result comparison, and performance analysis.

2.1. Dataset Image Collection

Dataset images used in this work were captured in late autumn during the full ripening period of pomelos from a pomelo orchard nearby Chengdu Youleyuan, Xindu District, Chengdu, Sichuan, China (30.87° N, 104.08° E). The orchard is located within the Huaxiang Guoju pomelo production area, where standardized Xindu pomelo orchards cover more than 2000 mu. The varieties of pomelo involved were Xindu Pomelo No. 1 and No. 2. From 1 October to 28 October 2025, image acquisition was conducted on 10 selected days between 10:00 and 18:00, with approximately 100 images collected per acquisition day. For image acquisition, a Nikon Z30 camera (Nikon Corporation, Tokyo, Japan) was used, with a resolution of 5568 × 3712 pixels, and the images were saved in JPEG format.

The sampled orchard was managed under a fixed-spacing dense planting system, which facilitates pomelo harvesting operations. Routine management practices included pruning, fertilization, irrigation, weed control, and pest and disease management. However, under natural conditions, image collection still suffers from complex backgrounds and severe occlusion between fruits. To better represent real orchard conditions, images were not collected from a single fixed angle or standardized camera position. Instead, a controlled multi-view acquisition strategy was adopted. During image collection, the camera was positioned at different viewing angles and distances around the canopy, with shooting distances maintained between 30 cm and 80 cm. The main acquisition criterion was that pomelo fruits and visible stems should appear clearly in the image while retaining natural interference from branches, leaves, overlapping fruits, shadows, and illumination changes. The acquisition settings included different perspectives, varying shooting distances, different lighting situations, including front light and backlight, and different types of occlusion, such as branch occlusion, mutual fruit occlusion, and leaf occlusion. During image acquisition, no leaves, branches, or other occluding objects were artificially removed, and all images were collected under natural orchard conditions to preserve the complexity of real production scenes. These acquisition settings increased the diversity of samples within the current orchard environment and provided representative challenging cases for model training and evaluation.

During data sorting and filtering, images with severe blurring, high redundancy, or inadequate acquisition quality were excluded. Ultimately, 538 high-quality original pomelo fruit images were retained to form the pomelo image dataset, as shown in Figure 1. These images were collected from approximately 300 pomelo trees.

2.2. Dataset Construction

In this study, all images were initially annotated by one researcher using Labelme software (version 5.9.1), with two categories defined: Pomelo and Stem. To improve annotation reliability, the completed annotations were subsequently reviewed by another researcher familiar with orchard target annotation. During the review process, particular attention was paid to slender stem regions, heavily occluded targets, and challenging samples in which stems were visually similar to branches or petioles. For ambiguous samples, the two researchers jointly examined and corrected the polygon boundaries according to unified annotation criteria. Specifically, for each pomelo fruit with a visible and distinguishable stem, the corresponding stem region was annotated in association with the fruit instance. Only the visible contours of pomelo fruits and stems were delineated using polygon annotations, as illustrated in Figure 2, while fully occluded or indistinguishable stems were not annotated as valid Stem instances. After annotation and review, the label files were converted into TXT format to meet the input requirements of the instance segmentation models.

During image acquisition, variations in illumination, background interference, and environmental noise may affect the visual appearance of pomelo fruits and stems. To increase the diversity of the training samples and improve the model’s adaptability to visual variations within the collected orchard environment, data augmentation was performed, as illustrated in Figure 3.

To avoid source-level overlap among different data subsets, the collected original images were first assigned to the training, validation, and test sets before data augmentation. Augmented images derived from the same original image were retained within their corresponding predefined subsets, thereby preventing samples from a common source from being distributed across different subsets. Following augmentation and quality screening, during which blurred, duplicate, and highly similar images were removed, the final dataset comprised 2152 training images, 269 validation images, and 269 test images. This construction procedure increased sample diversity while maintaining source-level independence among the three subsets, thereby providing a reliable basis for subsequent model training and performance evaluation.

After annotation and data processing, the dataset contained 13,574 annotated Pomelo instances and 13,024 annotated Stem instances, yielding a total of 26,598 annotated objects. Specifically, the training set contained 8812 Pomelo instances and 8524 Stem instances; the validation set contained 2260 Pomelo instances and 2270 Stem instances; and the test set contained 2502 Pomelo instances and 2230 Stem instances. The number of Stem instances was slightly lower than that of Pomelo instances because only visible and reliably distinguishable stem regions were annotated, whereas fully occluded or indistinguishable stems were excluded from valid Stem annotations.

To further characterize the difficulty of the Stem segmentation task, the 13,024 annotated Stem instances were categorized according to their visibility level and dominant occlusion source. Among them, 5985 instances were fully visible, accounting for 46.0% of all Stem instances, while 7039 instances were partially visible due to occlusion, accounting for 54.0%. Among the partially visible instances, 2085 were primarily affected by leaf occlusion, 1998 by branch or petiole occlusion, and 2956 by fruit occlusion. These statistics demonstrate that the dataset contains a substantial proportion of slender stem targets affected by natural occlusion, providing representative samples for evaluating the robustness of stem segmentation in complex orchard environments.

2.3. Experimental Environment and Evaluation Metrics

This section presents the experimental environment and evaluation metrics from two aspects. First, the hardware setup, software framework, and experimental parameters are described to ensure reproducibility. Subsequently, the metrics and decision-making criteria utilized for assessing model performance in this research are detailed.

2.3.1. Experimental Platform and Parameter Settings

This study was conducted in a deep learning experimental environment. The experimental hardware included 32 GB of RAM, an Intel Core i5-14600KF CPU(Intel Corporation, Santa Clara, CA, USA), and an NVIDIA RTX 4060 GPU(NVIDIA Corporation, Santa Clara, CA, USA). The software environment comprised the Windows 10 operating system(Microsoft Corporation, Redmond, WA, USA), Python 3.11.4, PyTorch 2.0.0, and CUDA 11.8. The relevant experimental parameter settings are shown in

Table 1. Experimental Parameter Settings.

Parameter Category	Parameter Setting
Epochs	300
Picture size	640 × 640
Batch size	8
workers	8
Momentum	0.937
Optimizer	SGD

.

To ensure fair comparison among different models, all experiments were conducted under consistent training conditions, including the same dataset split, training epochs, and learning rate scheduling strategy. Meanwhile, the optimizer and its hyperparameters, such as learning rate and weight decay, remained unchanged across all experimental groups. By adopting the control variable method, performance differences between models can more truly reflect the effects brought by network structure improvements, thereby enhancing the reliability of experimental conclusions.

2.3.2. Evaluation Metrics

To comprehensively evaluate the instance segmentation performance of the model for pomelo fruits and stems, this study used metrics for both object detection and mask segmentation. The evaluation metrics are derived from the alignment between the predictions made by the model and the actual annotations in the dataset, thereby guaranteeing objectivity in the evaluation outcomes.

(1) Basic Evaluation Metrics

In this paper, the basic performance of the model is first evaluated using Parameters (Params), GFLOPs, Precision (P), Recall (R), F1-score [20,21,22], FPS and mDSC. Their calculation formulas are given as follows:

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(1)

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(2)

$$F1-\mathrm{sc}ore={\displaystyle \frac{2PR}{P+R}}$$

(3)

$$FPS={\displaystyle \frac{1}{Average inference time per frame}}$$

(4)

$$mDSC={\displaystyle \frac{2TP}{2TP+FP+FN}}$$

(5)

where TP, FP, and FN denote true positives, false positives, and false negatives, respectively. The mean Dice coefficient across all evaluated instances is denoted as mDSC.

(2) Evaluation Metrics for Object Detection

For object detection evaluation, this study focuses on the accuracy of the model’s bounding box predictions. Mean Intersection over Union (mIoU) is a key metric measuring the overlap between predicted boxes and ground-truth boxes, defined as:

$$mIoU={\displaystyle \frac{Area\left(B_{pred}\cap B_{gt}\right)}{Area\left(B_{pred}\cup B_{gt}\right)}}$$

(6)

where B_pred and B_gt denote the predicted bounding box and the ground-truth bounding box, respectively.

Boundary F-score was calculated as the harmonic mean of boundary precision and boundary recall, and Stem mBF-score was obtained by averaging the BF-scores over all evaluated Stem instances.

$$B{F}_i={\displaystyle \frac{2P_{b,i}{R}_{b,i } }{P_{b,i}+R_{b,i }}}$$

(7)

where P_b,i and R_b,i denote the boundary precision and boundary recall of the $i$-th Stem instance, respectively, and N_s represents the total number of Stem instances.

$$Stem mBF-score={\displaystyle \frac{1}{N_s}}{\displaystyle \sum _{i=1}^{N_s}B{F}_i}$$

(8)

On this basis, mean Average Precision (mAP) is adopted to evaluate the overall detection performance of the model. mAP@50 refers to the mean of Average Precision (AP) across all categories at an IoU threshold of 0.5, expressed as:

$$mAP@50={\displaystyle \frac{1}{N}}{\displaystyle \sum _{N=1}^N{\displaystyle {\int }_0^1{P}_N(R)dR}}$$

(9)

$$mAP@0.5:0.95={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^{N}A{P}_{IoU=0.5:0.95}}$$

(10)

where N is the total number of categories, and P_N(R) represents the precision-recall curve of the N-th category under different recall rates. $A{P}_{IoU=0.5:0.95}$ denotes the average precision of each category across mIoU thresholds ranging from 0.5 to 0.95.

(3) Evaluation Metrics for Mask Segmentation

For instance segmentation tasks, in addition to bounding box detection accuracy, the pixel-level quality of the model’s predicted masks must be further evaluated. Therefore, Mask IoU and Mask mAP@50 are adopted as the main segmentation evaluation metrics. Mask mIoU measures the pixel-wise overlap between predicted masks and ground-truth masks, using the same formulation as Equation (4) but with mask regions instead of bounding box regions. This metric directly reflects the spatial consistency of segmentation results for individual objects. Mask mAP@50 represents the mean of mask average precision across all categories at a Mask mIoU threshold of 0.5. In contrast, Mask mAP@0.5:0.95 calculates the mean mask average precision over mIoU thresholds ranging from 0.5 to 0.95 with an interval of 0.05, thereby providing a stricter evaluation of mask segmentation quality under different overlap requirements.

In the mask matching process, this paper adopts a one-to-one matching strategy based on maximum mIoU priority. Specifically, each ground-truth mask corresponds to only one optimal predicted mask. When multiple predicted masks are matched to the same ground-truth mask, only the prediction with the largest Mask mIoU that satisfies the threshold is recorded as a TP, while the remaining predictions are regarded as FP. If a single predicted mask corresponds to several ground-truth masks at the same time, it is matched only with the ground-truth mask that has the highest mIoU, while the other unmatched ground-truth masks are categorized as false negatives (FN). Conversely, a predicted mask is deemed a false positive (FP) when its mIoU with all ground-truth masks is below the specified threshold. A ground-truth mask is designated as FN if it fails to be matched by any predicted mask.

(4) Role of Evaluation Metrics

Precision evaluates how accurately the model predicts, indicating the ratio of predicted target regions that actually correspond to the targets. Recall represents the model’s capability to identify genuine targets, signifying the ratio of actual targets that are recognized accurately. As the harmonic mean of Precision and Recall, F1-score comprehensively characterizes the balance between detection accuracy and complete target detection. mAP@0.5 evaluates the overall detection and segmentation performance of the model at an mIoU threshold of 0.5, and can intuitively reflect the model’s ability to recognize pomelo fruits and stems. Compared with mAP@0.5, mAP@0.5:0.95 adopts multiple mIoU thresholds from 0.5 to 0.95 and therefore provides a more stringent assessment of localization and mask segmentation quality. In this study, pomelo stems are characterized by being slender, small-scale, easily occluded, and having blurred boundaries. The model therefore needs not only to accurately locate targets but also to maintain the integrity and boundary quality of segmented regions as much as possible. Thus, by integrating Precision, Recall, F1-score, mAP@0.5, and mAP@0.5:0.95, a more comprehensive assessment of the model’s performance in segmenting pomelo fruits and stems is achieved.

3. Principle and Composition of the YOLO-MOD Algorithm

To clarify the structural design and improvement principles of YOLO-MOD, this section is presented from two aspects: baseline model analysis and improved algorithm construction. First, the overall architecture of the YOLOv11-seg baseline model, as well as its feature extraction, feature fusion, and segmentation output mechanisms, is explained to provide a foundation for subsequent model improvements. Then, based on this, the improvement ideas and component structure of the YOLO-MOD network are described in detail, with emphasis on the mechanisms of key modules such as ODConv, MSDA, and DySample in extracting local differential features, multi-scale semantic modeling, and boundary detail recovery.

3.1. Architecture of the YOLOv11-Seg Baseline Model

The launch of YOLOv11 by Ultralytics in 2024 represents a significant advancement in real-time visual models. This model successfully achieves a blend of rapid inference speed, computational efficiency, and elevated detection accuracy, while also accommodating a range of tasks such as object detection, instance segmentation, and classification [23,24]. Relative to earlier versions, YOLOv11 adopts an improved backbone and neck structure to enhance feature extraction capability. Meanwhile, through optimized network design and training procedures, it attains an improved equilibrium between precision and efficiency, rendering it more appropriate as a foundational reference for lightweight visual models used in intricate agricultural environments [25]. To meet the requirements of different application scenarios for model complexity and performance, YOLOv11 provides five model scales: n, s, m, l, and x. Structurally, it mainly consists of an input module, a backbone network, a neck network, and a segmentation head. The preprocessing of raw images is mainly carried out by the input module, while the backbone network plays a crucial role in extracting multi-scale semantic features from the images in a hierarchical manner. Additionally, the neck network merges and improves features at various levels through the use of a feature pyramid mechanism; finally, the segmentation head outputs object detection results along with corresponding instance segmentation masks.

In comparison to earlier models, YOLOv11 showcases significant enhancements primarily in the areas of feature extraction and representation. To begin with, the original C2f module has been substituted with the C3k2 module, enhancing computational efficiency by integrating a CSP-based bottleneck architecture. Additionally, while the SPPF module remains intact, the integration of the C2PSA module has been implemented to bolster the model’s capacity to capture essential spatial information. Furthermore, several C3k2 modules are added in the head section to further enhance the refinement of feature maps. Moreover, the internal configuration of the module can be adjusted using the parameter C3k: if C3k is set to False, a conventional bottleneck structure is utilized; conversely, when C3k is set to True, it transitions to the C3 module, facilitating deeper feature extraction. The YOLOv11-seg baseline model is shown in Figure 4 below.

3.2. YOLO-MOD Network Model

To address the specific challenges of joint instance segmentation of pomelo fruits and stems in natural orchard environments, this study develops YOLO-MOD as a task-oriented improvement framework based on the YOLOv11n-seg model. Rather than introducing a new general-purpose segmentation paradigm, the network is designed through the targeted integration of functionally complementary components according to the visual characteristics of the studied task. Specifically, three major difficulties are considered. First, stems are easily confused with branches and petioles because of their similar local appearance, particularly under severe occlusion and complex background interference. Second, their slender geometry, small scale, and partial visibility make it difficult for the baseline model to obtain sufficient multi-scale contextual information. Third, weak and discontinuous stem boundaries are prone to further degradation during upsampling and mask reconstruction, resulting in blurred or fragmented segmentation masks. Accordingly, ODConv is introduced into the C3k2 module to enhance adaptive local structural feature representation; MSDA is incorporated to strengthen multi-scale contextual modeling of small and occluded stem regions; and DySample is employed to improve feature reconstruction and preserve boundary continuity during mask generation. Through these task-specific modifications, YOLO-MOD is expected to achieve a more favorable balance between segmentation performance and model complexity for pomelo fruit and stem segmentation in complex orchard scenes. The overall architecture of the proposed network is illustrated in Figure 5, and the individual and combined effects of the three modifications are further evaluated through the ablation experiments in Section 4.3. The methodological novelty of YOLO-MOD lies in its task-oriented integration of three complementary mechanisms for pomelo fruit and stem segmentation in natural orchard environments. Rather than simply stacking existing modules, the proposed strategy follows a progressive enhancement pathway from local structural discrimination to multi-scale contextual representation and boundary-detail recovery. Specifically, ODConv improves the adaptive discrimination of stems, branches, petioles, and fruit regions; MSDA enhances the representation of small and partially occluded stems under complex backgrounds; and DySample improves fine-grained feature reconstruction and mask boundary continuity during upsampling. The single-module and pairwise ablation experiments further confirm that these components contribute at different stages and provide complementary benefits for segmenting slender and easily occluded stem targets.

3.2.1. ODConv Module

In intricate orchard settings, the capacity to identify and segment pomelo fruits and stems depends not only on the general accuracy of the model but also heavily on the effectiveness of feature extraction and the ability to represent structural details. To address issues such as leaf and branch occlusion, densely distributed targets, and scale variations, insufficient extraction of shallow and middle-layer features can easily lead to blurred boundaries between fruits and stems, missing local details, as well as complications in later feature fusion. To achieve this objective, the current study utilizes full-dimensional dynamic convolution ODConv to refine the C3k2 module, thereby improving the network’s capacity for representation in intricate backgrounds. Figure 6 illustrates the structural distinctions from conventional dynamic convolution to ODConv. Compared with traditional dynamic convolution, ODConv mainly adapts to different inputs by weighted combination of multiple convolution kernels, further extending the attention mechanism to multiple key dimensions of the convolution kernels [26,27,28,29].

In addition, ODConv learns the adaptive weight allocation of convolution kernels in parallel across dimensions of kernel number, spatial size, input channel, and output channel, thereby achieving more refined and sufficient dynamic regulation [30]. This mechanism improves the response of convolution kernels to target regions while maintaining high computational efficiency, allowing the network to more effectively concentrate on the distinguishing features of pomelo fruits and slender stems. A schematic representation of the four categories of attention mechanisms and their convolutional processes in ODConv is illustrated in Figure 7. Especially in scenarios with severe occlusion and dense targets, ODConv helps improve the accuracy of feature extraction, enhance semantic representation, and reduce feature conflicts during cross-scale information fusion, providing a more stable foundation for subsequent segmentation accuracy improvements.

Among them, the formulas for the four attention weights in ODConv, namely spatial dimension α_si, input channel dimension α_ci, output channel dimension α_fi, and kernel dimension α_wi, are shown in Equations (11)–(14):

$${\alpha }_{si}=\sigma (C{\mathrm{onv}}_{1\times 1}(X))$$

(11)

$${\alpha }_{ci}=\sigma (MLP(GAP(X)))$$

(12)

$${\alpha }_{fi}=\sigma (MLP(GAP(X)))$$

(13)

$${\alpha }_{wi}=Soft\max (MLP(GAP(X)))$$

(14)

where σ denotes the Sigmoid activation function, X denotes the input feature map, MLP denotes the fully connected network with dimension reduction, GAP denotes global average pooling, and Softmax denotes the normalized exponential function.

In addition, the calculation method of ODConv is as follows:

$$W_{ODConv}=(W_1\odot {\alpha }_{s1}\odot {\alpha }_{c1}\odot {\alpha }_{fi}\odot {\alpha }_{w1}+\cdots +W_n\odot {\alpha }_{sn}\odot {\alpha }_{cn}\odot {\alpha }_{fn}\odot {\alpha }_{wn})\ast X$$

(15)

3.2.2. MSDA Module

Significant differences exist between pomelo fruits and stems in size, shape, and texture features. Meanwhile, the orchard environment is accompanied by problems such as branch and leaf occlusion, uneven illumination, and background interference. The traditional convolutional network, which relies on a fixed receptive field for feature extraction, is difficult to balance the overall representation of large objects and the local detail recognition of finer details in smaller ones [31]. Especially for slender, low-contrast targets such as fruit stems, relying solely on conventional convolutions tends to result in blurred boundaries, breakages, or missed segmentation. To improve the model’s ability to adapt to targets of various scales, this study incorporates an MSDA module into the network. By combining dilated convolutions with local self-attention, it strengthens feature representation and semantic segmentation [32]. The configuration of MSDA is shown in Figure 8 below, and its core improvement mechanisms are as follows.

(1) Multi-scale Feature Extraction and Dilated Convolution

MSDA first performs linear mapping on the input feature map $X\in R^{H\times W\times C}$ to obtain the query matrix Q, key matrix K, and value matrix V. The channel dimension is then divided into multiple attention heads, with each head processing its respective sub-feature space. To achieve multi-scale perception, each head employs a different dilation rate for feature sampling, forming parallel multi-scale representation branches. The dilated convolution can be expressed as:

$$DConv{(X,r)}_i={\displaystyle \sum _k{G}_k}•X_{i+r•k}$$

(16)

where G_k denotes the convolution kernel weight and r denotes the dilation rate. Compared with standard convolution, dilated convolution allows for the expansion of the receptive field without significantly increasing the number of parameters, thereby balancing local texture details and a wider range of contextual information. For the segmentation of pomelo fruits and stems, a smaller dilation rate helps preserve edge details, connection points, and fine structural features of stems, while a larger dilation rate is more suitable for capturing the overall contour of fruits and their spatial relationships with surrounding branches and leaves. This effectively improves the model’s ability to recognize targets with varying scales.

(2) Sparse Self-Attention within Sliding Windows

To reduce the overhead associated with global self-attention, the MSDA approach employs the Sliding Window Dilated Attention (SWDA) technique, which calculates attention weights exclusively within localized windows. For a query at position (i, j) on the original feature map, SWDA performs sparse sampling within a local window of size W × W; $r\in N^{+}$ denotes the dilation rate, which controls the sparsity of sampling. The component corresponding to position (i, j) in the output X is further defined as:

$$X_{ij}=Attention(Q_{ij},K_r,V_r)=Soft\max ({\displaystyle \frac{Q_{ij}{K}_r^T}{\sqrt{d_k}}})V_r,1\le i\le W,1\le j\le H$$

(17)

where H and W are the height and width of the feature map respectively; K_r and V_r are the keys and values sparsely selected from K and V; and d_k denotes the dimension of the key vector.

(3) Channel-Spatial Dual-Path Attention Enhancement Mechanism

To further enhance feature representation, MSDA implements a Dual-Path Attention enhancement mechanism (DPA) that operates on both channel and spatial dimensions, building upon the initial attention calculation. By jointly analyzing the significance distribution across these two dimensions, it dynamically amplifies important feature areas. This approach includes both a channel attention branch and a spatial attention branch, which collaboratively process the input features to enhance the model’s ability to represent targets in complex environments. In detail, the channel attention branch first utilizes Global Average Pooling (GAP) to derive global statistics for each channel from the input feature X, after which it generates corresponding channel weights using a multi-layer perceptron. This process enhances features relevant to the target while diminishing responses that are redundant or not applicable. The formulation is as follows:

$$W_c=\sigma (MLP(GAP(X)))$$

(18)

The spatial attention branch uses dilated convolution to model spatial dependencies of input features, highlighting salient responses in target regions and their neighborhoods, and improving the model’s perception of key spatial positions. Its calculation is formulated as:

$$W_s=\sigma (DConv(X,r))$$

(19)

Once the channel weights and spatial weights have been acquired, the initial input features are recalibrated together through element-wise multiplication, leading to the final output being represented as:

$$X_{out}=(W_c\odot W_s)\odot X$$

(20)

where σ denotes the Sigmoid activation function and $\odot$ signifies element-wise multiplication. This mechanism markedly enhances the network’s ability to focus on essential information while diminishing the disturbance caused by intricate backgrounds and non-essential areas in the feature extraction process by simultaneously refining and improving features from both channel and spatial dimensions. Particularly in tasks related to object detection and segmentation, this strategy effectively boosts the model’s robustness when faced with occlusion, cluttered backgrounds, and low-contrast environments.

(4) Multi-Head Feature Fusion

In the process of multi-head feature aggregation, the initial feature map X undergoes a linear projection to generate Q, K, and V. Subsequently, the channels are partitioned into n attention heads, and multi-scale SWDA is conducted using varying dilation rates across the different heads:

$$h_i=SWDA(Q_i,K_i,V_i,r_i),1\le \mathrm{i}\le n$$

(21)

$$X=Linear(Concat[h_1\cdots h_n])$$

(22)

where r_i denotes the dilation rate of the i-th head, and Q_i, K_i, V_i are the feature slices of the i-th head.

3.2.3. DySample Module

To address the inherent constraints associated with the static interpolation found in the YOLOv11’s original upsampling module, this study presents the DySample dynamic upsampling operator aimed at enhancing the model’s capacity to recover and express intricate features of pomelo fruits and their stems [33,34,35,36]. When processing low-resolution feature maps, traditional upsampling methods struggle to adaptively adjust sampling positions according to target structures, often introducing noise and causing loss of detailed information. In the case of segmenting instances of fruits and their stems, the slender form, faint edges, and diminutive size of the fruit stems result in the original upsampling method being ineffective at accurately reconstructing the outlines and boundary details of the targets. This easily damages the structural integrity of feature maps, resulting in serrated, blurred, or locally broken segmentation boundaries, thereby degrading the final segmentation accuracy.

The sampling procedure for DySample is illustrated in Figure 9. Initially, a feature map X of dimensions C × H × W is introduced into the module. On one hand, a sampling set S of size 2g × sH × sW is generated via the sampling point generator, where 2 denotes the coordinate dimension (each sampling point is located by x and y coordinates), g represents the number of channel groups, and s is the upsampling scale factor. On the other hand, the feature map is sent to the built-in function Grid Sample, which uses the coordinate positions in sampling set S to resample feature map X through bilinear interpolation, obtaining a high-resolution feature map of size C × sH × sW.

DySample adopts two strategies in the sampling point generation process: static scope factor and dynamic scope factor, as shown in Figure 10. In the static range factor method, the input feature X first undergoes linear mapping to obtain offset features. Pixel Shuffle is then used to convert channel information into spatial offsets, which are scaled by a predefined range factor to form the initial offset O. At the same time, a basic sampling grid G is established according to the dimensions of the input feature map and the rate of upsampling. The ultimate set of sampling points S is acquired by incorporating O into G. Although this method features a straightforward structure, the predetermined offset range poses challenges in accommodating feature variations across different areas.

To improve the adaptability of sampling positions, DySample introduces a dynamic range factor based on the static method. This strategy adds a dynamic adjustment branch and uses the Sigmoid function to constrain the range of the adjustment coefficient for adaptive correction of offsets, resulting in a more reasonable distribution of sampling points and reducing boundary artifacts caused by overlapping sampling points [37]. Compared with the static method, the dynamic range factor strategy better preserves detailed information, including fruit contours, stem edges, and occluded regions, thereby improving segmentation accuracy while controlling computational overhead.

4. Experimental Results

Experiments in this section are conducted in three parts: attention module selection, comparison with representative segmentation models, and ablation analysis. By evaluating the effects of different attention modules on model performance, the most appropriate attention module for this task is identified. Comparisons with several representative YOLO-based segmentation methods show that YOLO-MOD achieves a better balance between segmentation accuracy and computational cost. In addition, ablation experiments are performed to examine the individual contributions of each improved module to the overall performance gain. Validation on an independent test set further shows that YOLO-MOD achieves better instance segmentation performance in real orchard scenes with complex illumination, branch and leaf occlusion, fruit overlap, and local shadows. This improvement is particularly evident in the detection and segmentation of slender and easily occluded stem targets. Combined with heatmap visualization results, the differences in feature responses and robustness under complex scenarios are analyzed, providing comprehensive evidence for the effectiveness of the proposed model and its application potential in complex orchard environments.

4.1. Comparative Experiments of Different Attention Mechanisms

To evaluate the effect of different attention modules on the joint segmentation of pomelo fruits and stems, five attention modules were separately integrated after the C2PSA module in the YOLOv11-seg baseline framework. All other network structures were kept unchanged. The compared modules included Efficient Multi-Scale Attention (EMA) [38], Large Separable Kernel Attention (LSKA) [39], Mixed Local Channel Attention (MLCA) [40], Multi-Scale Convolutional Attention (MSCA) [41], and Multi-Scale Dilated Attention (MSDA) [42]. The experimental results are presented in

Table 2. Experimental Results of Different Attention Mechanisms.

Models	Params (M)	GFLOPs (G)	Box			Mask			Box mAP@50 (%)			Mask mAP@50 (%)
			P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)	Pomelo	Stem	Overall	Pomelo	Stem	Overall
YOLOv11-seg	2.8	10.2	89.9	79.0	84.1	88.3	77.6	82.6	99.4	71.9	85.7	99.4	68.4	83.9
EMA	3.9	11.2	89.2	82.7	85.8	91.4	80.3	85.5	99.5	77.0	88.2	99.5	74.1	86.8
LSKA	3.9	11.2	90.9	81.2	85.8	89.9	80.3	84.8	99.4	74.5	86.9	99.4	72.5	86.0
MLCA	3.8	11.2	90.9	81.4	85.9	88.4	81.6	84.9	99.4	75.6	87.5	99.4	73.5	86.5
MSCA	3.9	11.2	91.5	81.2	86.0	89.9	79.9	84.6	99.4	74.4	86.9	99.4	71.3	85.3
MSDA	4.1	11.3	92.7	82.6	87.4	90.5	82.9	86.5	99.5	77.8	88.6	99.5	76.2	87.8

.

As shown in Table 2, the introduced attention mechanisms generally improved model performance, but the degree of improvement varied across these modules. Among them, MSDA achieved the most notable overall improvement. Compared with the baseline YOLOv11-seg model, the model equipped with MSDA obtained better detection and segmentation performance with only a limited increase in model complexity. Specifically, the number of parameters and GFLOPs increased from 2.8 M and 10.2 G to 4.1 M and 11.3 G, respectively. Meanwhile, Box Precision increased from 89.9% to 92.7%, Mask Precision increased from 88.3% to 90.5%, overall Box mAP@50 increased from 85.7% to 88.6%, and overall Mask mAP@50 increased from 83.9% to 87.8%. For the stem class, the advantage of MSDA was more pronounced. Its Mask mAP@50 increased from 68.4% to 76.2%, corresponding to a gain of 7.8%. This result indicates that MSDA can effectively enhance the feature representation of slender targets and occluded regions.

Further analysis of Figure 11 shows that MSDA achieved the highest Mask mAP@50 and Mask Recall among all compared attention mechanisms, reaching 87.8% and 82.9%, respectively. Although its Mask Precision was slightly lower than that of EMA, its Mask mAP@50 and Mask Recall were both higher. This suggests that MSDA improves target recall and overall segmentation performance while maintaining competitive precision.

Overall, EMA, LSKA, MLCA, and MSCA also improved model performance to different degrees. However, MSDA showed more pronounced advantages in segmentation accuracy, recall capability, and stem segmentation performance. Therefore, MSDA was adopted as the attention module in the proposed YOLO-MOD algorithm.

4.2. Comparative Experiments of Different Models

In this experiment, several representative segmentation models from the YOLO series were selected as benchmarks. Under uniform experimental conditions, differences among models in segmentation precision, recall performance, parameters, and computational complexity were compared to comprehensively evaluate the performance of YOLO-MOD in terms of complex-scene adaptability, fruit–stem segmentation capability, and the balance between accuracy and efficiency. Meanwhile, in the visualization analysis, P-R curves of YOLO-MOD and other mainstream YOLO models for pomelo stem segmentation and overall segmentation were plotted.

4.2.1. Experimental Data

To evaluate the segmentation performance of the YOLO-MOD model for segmenting pomelo fruits and their stems, comparative experiments were conducted using instance segmentation models such as YOLOv5-seg [43], YOLOv6-seg [44], YOLOv7-seg [45], YOLOv8-seg [44], YOLOv9-seg [46], YOLOv10-seg [47], and YOLOv12-seg [48], while YOLOv11-seg acted as the reference model. The results from these experiments can be found in

Table 3. Experimental Results of Different YOLO Models.

Models	Params (M)	GFLOPs (G)	Box			Mask			Box mAP@50 (%)			Mask mAP@50 (%)
			P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)	Pomelo	Stem	Overall	Pomelo	Stem	Overall
YOLOv5-seg	53.8	177.4	72.0	66.5	69.1	73.3	66.3	69.6	97.7	38.7	68.2	97.7	38.6	68.1
YOLOv6-seg	42.8	123.8	60.6	62.6	61.6	54.6	50.7	52.6	96.6	27.9	62.2	96.6	5.0	50.8
YOLOv7-seg	37.9	142.7	92.6	90.5	91.5	88.5	82.7	85.5	99.4	87.3	93.3	99.4	72.6	86.0
YOLOv8-seg	3.3	12.1	93.0	79.0	85.4	87.3	75.1	80.7	99.1	71.9	85.5	99.1	61.9	80.5
YOLOv9-seg	27.8	149.0	93.3	88.0	90.6	89.4	83.8	86.5	99.1	84.6	91.9	99.1	75.8	87.5
YOLOv10-seg	42.1	192.1	75.3	70.7	72.9	71.7	68.6	70.1	97.8	47.1	72.5	97.8	42.7	70.3
YOLOv11-seg	2.8	10.2	89.9	79.0	84.1	88.3	77.6	82.6	99.4	71.9	85.7	99.4	68.4	83.9
YOLOv12-seg	2.8	9.8	88.7	77.1	82.5	84.3	73.9	78.8	99.0	67.4	83.2	99.0	59.5	79.3
YOLO-MOD	4.1	11.3	92.7	82.6	87.4	90.5	82.9	86.5	99.5	77.8	88.6	99.5	76.2	87.8

.

As shown in Table 3, YOLO-MOD achieves competitive segmentation performance while maintaining relatively low model complexity under the current experimental setting. The proposed model contains 4.1 M parameters and requires 11.3 GFLOPs, with overall Box mAP@50 and Mask mAP@50 values of 88.6% and 87.8%, respectively. These values are higher than those of the lightweight baseline models YOLOv11-seg, YOLOv8-seg, and YOLOv12-seg in the present experiment. For the Stem class, YOLO-MOD achieves a Mask mAP@50 of 76.2%, which is the highest numerical value among the evaluated models in Table 3. Compared with the baseline YOLOv11-seg model, the overall Mask mAP@50 and Stem-class Mask mAP@50 values increase by 3.9% and 7.8%, respectively. These results suggest that the proposed modifications are beneficial for the segmentation of slender stem structures under the evaluated orchard conditions.

YOLOv7-seg achieves higher values than YOLO-MOD in several detection-related metrics, including overall Box mAP@50. However, YOLOv7-seg contains 37.9 M parameters and requires 142.7 GFLOPs, whereas YOLO-MOD contains 4.1 M parameters and requires 11.3 GFLOPs. Under the current experimental setting, YOLO-MOD obtains higher overall and Stem-class Mask mAP@50 values than YOLOv7-seg, while requiring substantially lower model complexity.

YOLOv9-seg also achieves competitive results, with an overall Mask mAP@50 of 87.5% and a Stem-class Mask mAP@50 of 75.8%, which are close to the corresponding values of YOLO-MOD at 87.8% and 76.2%. Therefore, without statistical testing, these small numerical differences should be interpreted cautiously. Overall, the results indicate that YOLO-MOD provides a favorable descriptive trade-off between segmentation performance and model complexity under the current experimental conditions, particularly for the Stem class.

4.2.2. Visualization Analysis

To visually present the segmentation performance of YOLO-MOD relative to representative YOLO-based segmentation models, the Precision–Recall (P–R) curves were compared, as shown in Figure 12. The results are interpreted as descriptive comparisons under the current experimental setting, rather than as evidence of statistically significant superiority over all evaluated models.

Specifically, in the Stem segmentation task shown in Figure 12a, YOLO-MOD achieves a Mask mAP@0.5 value of 0.762. This value is higher than those of YOLOv5-seg (0.386), YOLOv6-seg (0.050), YOLOv7-seg (0.726), YOLOv8-seg (0.619), YOLOv10-seg (0.427), YOLOv11-seg (0.684), and YOLOv12-seg (0.595), while it is close to that of YOLOv9-seg (0.758). In the overall segmentation task shown in Figure 12b, YOLO-MOD achieves a Mask mAP@0.5 value of 0.878, which is also close to that of YOLOv9-seg (0.875), while exceeding the numerical values obtained by the other evaluated models under the same experimental conditions.

These results indicate that YOLO-MOD achieves competitive mask segmentation performance in the current experiment, particularly for the challenging Stem class. However, because repeated experiments and statistical significance tests were not conducted in the present study, the small numerical differences between YOLO-MOD and models such as YOLOv9-seg should be interpreted cautiously.

4.3. Ablation Experiments

To clarify the individual contributions and combined effects of the proposed modifications, a systematic ablation study was conducted on the basis of YOLOv11-seg. The evaluated components included the MSDA module, the ODConv-integrated C3k2 module, and the DySample upsampling operator. As shown in

Table 4. Results of Ablation Experiments.

Experiment Number	MS DA	OD Conv	Dy Sample	Para ms (M)	GF LOPs (G)	Box			Mask			Box mAP@50 (%)			Mask mAP@50 (%)
						P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)	Pomelo	Stem	Overall	Pomelo	Stem	Overall
1	×	×	×	2.8	10.2	89.9	79.0	84.1	88.3	77.6	82.6	99.4	71.9	85.7	99.4	68.4	83.9
2	√	×	×	4.7	19.4	89.8	79.1	84.1	90.8	77.9	83.9	99.3	70.7	85.0	99.3	70.4	84.8
3	×	√	×	3.8	11.1	91.2	80.1	85.3	90.1	80.0	84.8	99.4	74.1	86.8	99.4	72.1	85.8
4	×	×	√	2.9	10.3	92.6	81.9	86.9	90.2	81.6	85.7	99.5	75.9	87.7	99.5	74.0	86.7
5	√	√	×	4.1	11.3	88.2	83.5	85.8	88.1	83.9	85.9	99.5	75.4	87.4	99.5	74.7	87.1
6	×	√	√	3.8	11.1	93.0	81.9	87.1	91.2	81.1	85.9	99.5	75.4	87.4	99.5	73.6	86.5
7	√	×	√	3.1	10.5	90.6	81.7	85.9	90.2	81.3	85.5	99.3	75.8	87.5	99.3	74.6	86.9
8	√	√	√	4.1	11.3	92.7	82.6	87.4	90.5	82.9	86.5	99.5	77.8	88.6	99.5	76.2	87.8

, Experiment 1 represents the baseline model; Experiments 2–4 evaluate the individual effects of MSDA, ODConv, and DySample, respectively; Experiments 5–7 examine the pairwise combinations of these components; and Experiment 8 evaluates the complete YOLO-MOD model integrating all three modifications. This experimental design enables the respective roles of feature representation, multi-scale contextual modeling, and boundary reconstruction to be analyzed separately and jointly.

Experiment 1 uses the original YOLOv11-seg model as the baseline. The baseline model contains 2.8 M parameters and requires 10.2 GFLOPs. Its overall Mask mAP@50 is 83.9%, while the Mask mAP@50 of the Stem class is 68.4%. These results provide the reference for evaluating the contribution of each subsequent modification.

Experiments 2–4 evaluate the individual contribution of each introduced component. In Experiment 2, MSDA is introduced after the C2PSA block to enhance multi-scale contextual modeling. Compared with the baseline model, its overall Mask mAP@50 increases from 83.9% to 84.8%, and the Mask mAP@50 of the Stem class increases from 68.4% to 70.4%, corresponding to improvements of 0.9% and 2.0%, respectively. Meanwhile, Mask Precision increases from 88.3% to 90.8%. However, the model complexity increases to 4.7 M parameters and 19.4 GFLOPs, and its overall Box mAP@50 decreases slightly from 85.7% to 85.0%. These results indicate that MSDA alone mainly contributes to mask-oriented contextual representation of stem regions under complex backgrounds, but its individual contribution is accompanied by relatively high computational cost in the present configuration.

In Experiment 3, ODConv is introduced into the C3k2 module to enhance adaptive local structural feature representation. Compared with the baseline model, the parameters and GFLOPs increase moderately to 3.8 M and 11.1 GFLOPs, respectively. The overall Mask mAP@50 increases from 83.9% to 85.8%, while the Mask mAP@50 of the Stem class increases from 68.4% to 72.1%, corresponding to improvements of 1.9% and 3.7%, respectively. In addition, the overall Box mAP@50 increases from 85.7% to 86.8%. These results suggest that ODConv improves the discrimination of local structural differences between stem regions and background structures, thereby benefiting both object localization and mask segmentation.

In Experiment 4, DySample replaces the original upsampling operator to evaluate its effect on feature reconstruction and boundary recovery. This modification increases the parameters and GFLOPs only slightly, from 2.8 M and 10.2 G to 2.9 M and 10.3 G, respectively. Nevertheless, it provides the largest performance improvement among the three single-component configurations. Specifically, the overall Mask mAP@50 increases from 83.9% to 86.7%, and the Mask mAP@50 of the Stem class increases from 68.4% to 74.0%, corresponding to improvements of 2.8% and 5.6%, respectively. This result indicates that, for slender stem targets with weak and discontinuous boundaries, improving the upsampling-based reconstruction process is particularly effective for alleviating boundary blurring and mask fragmentation. Although DySample achieved the greatest single-module improvement with minimal additional complexity, stem segmentation requires more than boundary reconstruction. Similar appearances between stems and branches or petioles, together with small-scale and occluded targets, also require enhanced structural discrimination and contextual representation. Therefore, ODConv, MSDA, and DySample complement one another by improving local feature discrimination, multi-scale contextual modeling, and boundary reconstruction, respectively. Compared with DySample alone, the complete YOLO-MOD model further increased the overall and Stem-class Mask mAP@50 from 86.7% to 87.8% and from 74.0% to 76.2%, respectively. Thus, the complete configuration was selected because it achieved the best mask segmentation performance among the evaluated settings, particularly for the challenging Stem class.

Experiments 5–7 further evaluate the combined effects of two components in order to determine whether the proposed modifications provide complementary benefits. In Experiment 5, MSDA and ODConv are introduced simultaneously. This configuration achieves an overall Mask mAP@50 of 87.1% and a Stem-class Mask mAP@50 of 74.7%, corresponding to improvements of 3.2% and 6.3% over the baseline model, respectively. Among the three pairwise configurations, this combination achieves the highest overall Mask mAP@50 and Stem-class Mask mAP@50. Compared with introducing MSDA or ODConv alone, the results indicate that multi-scale contextual modeling and adaptive local structural representation can jointly improve the segmentation of stem regions under complex background interference.

In Experiment 6, ODConv and DySample are combined to enhance local structural representation and boundary reconstruction, respectively. The resulting model contains 3.8 M parameters and requires 11.1 GFLOPs. Compared with the baseline model, its overall Mask mAP@50 increases from 83.9% to 86.5%, and the Stem-class Mask mAP@50 increases from 68.4% to 73.6%, corresponding to improvements of 2.6% and 5.2%, respectively. This configuration also achieves the highest Box Precision among all ablation settings, reaching 93.0%. However, its Mask mAP@50 values are slightly lower than those obtained by DySample alone, which reaches 86.7% overall and 74.0% for the Stem class. This result indicates that the combination of ODConv and DySample improves detection-side feature discrimination, but does not produce an additional mask segmentation gain beyond DySample alone under the current experimental configuration.

In Experiment 7, MSDA and DySample are combined to enhance multi-scale contextual perception and boundary-detail reconstruction. With 3.1 M parameters and 10.5 GFLOPs, this configuration achieves an overall Mask mAP@50 of 86.9% and a Stem-class Mask mAP@50 of 74.6%, representing improvements of 3.0% and 6.2% over the baseline model, respectively. Compared with DySample alone, the addition of MSDA further increases overall Mask mAP@50 by 0.2% and Stem-class Mask mAP@50 by 0.6%. Although the improvement is modest, the results suggest that contextual enhancement can provide supplementary benefits to boundary reconstruction when segmenting small-scale and partially occluded stem regions.

Finally, in Experiment 8, the joint introduction of MSDA, ODConv, and DySample achieved the best overall mask segmentation performance among the evaluated configurations. The resulting model contained 4.1 M parameters and required 11.3 GFLOPs. Its overall Box Precision, Recall, and F1-score reached 92.7%, 82.6%, and 87.4%, respectively, while the corresponding Mask Precision, Recall, and F1-score reached 90.5%, 82.9%, and 86.5%, respectively. Compared with the baseline model, the overall Mask mAP@50 increased from 83.9% to 87.8%, corresponding to an improvement of 3.9%. More importantly, the Mask mAP@50 for the challenging stem class increased from 68.4% to 76.2%, representing an improvement of 7.8%.

Combined with the single-module and pairwise-module experiments, these results suggest that the performance gain of YOLO-MOD is derived from the complementary contributions of the three task-oriented modifications at different stages of the network. ODConv mainly contributes to the adaptive representation of local structural differences between fruits, stems, and background branches; MSDA enhances multi-scale contextual modeling for small and partially occluded stem regions; and DySample improves the reconstruction of fine boundaries during mask generation. Therefore, the proposed YOLO-MOD should be regarded as an application-oriented enhancement of YOLOv11-seg for pomelo fruit and stem segmentation in complex orchard environments, rather than as a newly proposed general-purpose segmentation mechanism.

4.4. Performance Evaluation and Feature Attention Analysis

To further evaluate the performance of the improved model within the sampled orchard domain, a held-out test subset was used. Before data augmentation, the collected original images were first divided into training, validation, and test subsets. Data augmentation was subsequently conducted within each predefined subset, and all augmented images derived from the original images assigned to the test subset were retained exclusively in the test subset. Therefore, neither the original test-source images nor their augmented variants were used for model training or model selection, thereby preventing source-level overlap between the training and test subsets. YOLO-MOD was compared with the baseline YOLOv11-seg model on this test subset. The test images included representative challenging conditions observed in the studied orchard, such as different illumination conditions, shooting angles, branch and leaf occlusion, fruit overlap, and local shadows. Therefore, this evaluation reflects the segmentation performance of the two models under challenging conditions within the collected orchard environment, rather than demonstrating broad generalization across different orchards, seasons, or pomelo varieties. In addition to segmentation accuracy, frames per second (FPS) was introduced to evaluate the processing efficiency of the two models. The FPS tests were conducted under the same hardware and software conditions described in Section 2.3.1. The quantitative comparison results are presented in

Table 5. Comparison of Segmentation Results of YOLO-MOD and YOLOv11-seg on the Test Set.

Models	Para ms (M)	GFLO Ps (G)	FPS (Frames/s)	Box			Mask			Box mAP@50 (%)			Mask mAP@50 (%)
				P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)	Pomelo	Stem	Overall	Pomelo	Stem	Overall
YOLOv11 -seg	2.8	10.3	42.2	88.1	80.7	84.2	88.3	80.3	84.1	99.4	73.1	86.3	99.4	71.4	85.4
YOLO -MOD	4.1	11.3	34.8	89.3	85.6	87.4	88.5	85.0	86.7	99.4	79.9	89.7	99.4	77.2	88.3

. In addition, Grad-CAM++ was used to visualize the feature response regions of the two models, as shown in Figure 13.

As shown in Table 5, YOLO-MOD outperforms the baseline YOLOv11-seg model in overall segmentation performance. For all categories, YOLO-MOD achieves Box Precision, Recall, and mAP@50 values of 89.3%, 85.6%, and 89.7%, respectively, which are higher than those of YOLOv11-seg at 88.1%, 80.7%, and 86.3%. For mask segmentation, YOLO-MOD achieves Mask Precision, Recall, and mAP@50 values of 88.5%, 85.0%, and 88.3%, respectively, also exceeding the baseline values of 88.3%, 80.3%, and 85.4%. These results indicate that the improved model performs better in object localization, target recall, and overall instance segmentation quality.

In terms of processing efficiency, YOLOv11-seg achieves a processing speed of 42.2 FPS, whereas YOLO-MOD achieves 34.8 FPS under the same experimental conditions. The decrease of 7.4 FPS is associated with the additional feature representation, multi-scale attention modeling, and dynamic upsampling operations introduced in YOLO-MOD. Nevertheless, YOLO-MOD maintains a processing speed of 34.8 FPS on the computer platform used in this study, while its overall Mask mAP@50 increases from 85.4% to 88.3%, corresponding to an improvement of 2.9%. These results indicate that the proposed model introduces a measurable computational overhead, but achieves improved segmentation accuracy under complex orchard conditions.

Further category-level analysis shows that the performance gain of YOLO-MOD is mainly reflected in the more challenging stem class. For the Stem class, Box mAP@50 increases from 73.1% to 79.9%, with an improvement of 6.8%. Mask mAP@50 increases from 71.4% to 77.2%, with an improvement of 5.8%. This indicates that the improved model has stronger detection and segmentation capability for slender, small-scale, and easily occluded stem targets. In contrast, both models achieve 99.4% Box mAP@50 and Mask mAP@50 for the Pomelo class. This suggests that the baseline model already performs well in recognizing large fruit targets, while the advantage of YOLO-MOD is mainly reflected in improving the segmentation of complex targets such as stems. Combined with the FPS results, these findings show that YOLO-MOD improves stem segmentation performance at a quantifiable cost in processing efficiency.

To further evaluate the segmentation performance of YOLO-MOD under stricter overlap requirements, as well as the region-level and boundary-sensitive segmentation quality of slender stem targets, additional evaluation results are reported in

Table 6. Extended Evaluation Results on the Test Set.

Models	Stem mIoU (%)	Stem mDSC (%)	Stem mBF-Score (%)	Overall Box mAP@50:95 (%)	Overall Mask mAP@50:95 (%)	Stem Mask mAP@50:95 (%)
YOLOv11-seg	36.4	45.2	52.3	71.2	65.7	34.1
YOLO-MOD	40.7	50.3	59.1	74.0	69.0	39.9

. Under the mAP@50:95 criterion, YOLOv11-seg achieves an overall Box mAP@50:95 of 71.2% and an overall Mask mAP@50:95 of 65.7%, whereas YOLO-MOD achieves 74.0% and 69.0%, respectively, representing numerical increases of 2.8% and 3.3%. For the challenging Stem class, the Mask mAP@50:95 increases from 34.1% to 39.9%, with an improvement of 5.8%. Furthermore, YOLOv11-seg achieves a Stem Mean Mask mIoU of 36.4%, a Stem Mean Dice of 45.2%, and a Stem Mean BF-score of 52.3%, whereas YOLO-MOD achieves 40.7%, 50.3%, and 59.1%, respectively, corresponding to improvements of 4.3%, 5.1%, and 6.8%. The improvements in Mask mAP@50:95, Mask mIoU, and Dice indicate that YOLO-MOD provides more accurate mask segmentation for stem targets under stricter overlap requirements, while the increase in BF-score demonstrates better boundary consistency under complex orchard conditions. These results complement the Mask mAP@50 evaluation and provide additional quantitative evidence that the proposed model improves the segmentation quality of slender and easily occluded stems.

The Grad-CAM++ visualization results further support the performance differences between the two models from the perspective of feature responses. In the multi-target scene, YOLOv11-seg can respond to some fruit regions. However, its high-activation regions are relatively scattered and are easily affected by nearby branches, leaves, and background information, resulting in insufficient responses in some target regions. In contrast, YOLO-MOD forms relatively more complete and continuous high-response regions for multiple pomelo targets and focuses more accurately on the fruit bodies.

In the occlusion scene, the high responses of YOLOv11-seg are mainly concentrated on local visible regions, indicating limited perception of the overall structure of occluded targets. By comparison, YOLO-MOD maintains stronger responses on the fruit body and its visible contour regions. This suggests that the improved model has better representation ability for target structural features under occlusion. Under front-light conditions, the activation responses of YOLOv11-seg are mainly concentrated on the lower adjacent fruit regions, while its response to the single pomelo target in the upper right is weak. This indicates that this target is not sufficiently focused on and may increase the risk of missed detection. In addition, some responses spread to surrounding branches, leaves, and background regions, showing that the baseline model still has limited target localization ability under strong illumination and complex background interference. In contrast, YOLO-MOD maintains clear high-response distributions in both the lower overlapping fruit regions and the upper-right single fruit region. No obvious target response loss is observed in the heatmap. This indicates that the improved model can more stably focus on fruit targets at different positions and shows better multi-target perception and adaptability to complex illumination.

In the back-light scene, insufficient illumination and low target–background contrast make the activation responses of YOLOv11-seg relatively weak and scattered. By contrast, YOLO-MOD still maintains clear activation distributions in the main target regions. This indicates that the improved model has better target localization ability and stronger robustness against interference under low illumination and complex background conditions.

Overall, the held-out test subset results, FPS evaluation results, and Grad-CAM++ visualization results demonstrate that YOLO-MOD achieves better instance segmentation performance and stronger target-region localization ability in complex orchard environments. The introduction of ODConv enhances the adaptive extraction of local differential features between pomelo fruits and stems. MSDA improves the multi-scale semantic representation of key regions under complex backgrounds. DySample optimizes feature reconstruction and strengthens boundary detail recovery. The synergy of these three modules enables YOLO-MOD to reduce interference from branches, leaves, shadows, and cluttered backgrounds, while improving the representation of fruit bodies and slender stem structures. Although the processing speed of YOLO-MOD decreases from 42.2 FPS to 34.8 FPS compared with YOLOv11-seg, the improved model achieves higher overall and stem-specific mask segmentation accuracy. Therefore, YOLO-MOD presents a measurable trade-off between processing efficiency and segmentation performance on the computer platform used in this study. Overall, with an acceptable increase in model complexity, YOLO-MOD improves the segmentation stability of pomelo fruits and stems under the orchard conditions represented in the current test set. The proposed method shows potential as a front-end visual perception component for subsequent studies on picking-point localization and robotic harvesting, while its practical performance on embedded devices and integrated robotic harvesting systems still requires further validation.

5. Discussion

Compared with previous YOLO-based agricultural vision studies on apple, tomato, citrus, mango, strawberry, and grape, YOLO-MOD also adopts task-oriented improvements, but focuses on the joint instance segmentation of pomelo fruits and slender stems. Compared with fruit targets, stem regions are smaller, more easily occluded, and more difficult to distinguish from branches and petioles. The improved Stem segmentation performance is mainly attributed to the complementary effects of ODConv, MSDA, and DySample, which enhance structural discrimination, multi-scale contextual representation, and boundary reconstruction, respectively. However, direct comparisons with previous studies should be interpreted cautiously because of differences in crops, datasets, imaging conditions, model scales, and evaluation protocols.

Although the proposed YOLO-MOD achieved improved segmentation performance for pomelo fruits and stems under the evaluated orchard conditions, the generalization scope of the present study remains limited by the dataset composition. Specifically, the images used in this study were collected from one orchard in Xindu District, Chengdu, Sichuan Province, during a single maturity period, and only two local pomelo varieties, namely Xindu Pomelo No. 1 and No. 2, were included. Although the dataset contains representative challenging scenes involving occlusion, fruit overlap, uneven illumination, shadows, and complex backgrounds, it cannot fully represent the variations that may occur across different geographic regions, seasons, orchard management conditions, canopy structures, pomelo varieties, or other fruit types. Therefore, the results obtained on the current test set should be interpreted as evidence of model performance within the sampled orchard domain, rather than as confirmation of broad cross-domain generalization. Future work will expand the dataset using images collected from multiple orchards, seasons, growth stages, and pomelo varieties, and will conduct external validation and cross-domain testing to systematically evaluate the transferability and stability of the proposed model in open orchard environments. In addition, the comparative and ablation experiments in this study were not repeated using multiple random seeds. Therefore, the reported performance differences should be interpreted as descriptive evidence under the current experimental setting rather than as statistically validated improvements. Future work will conduct repeated experiments with multiple random seeds under consistent dataset splits and training configurations, and will report mean values, standard deviations, confidence intervals, and appropriate statistical comparisons to further assess the stability and reproducibility of YOLO-MOD.

In addition, stems are usually slender and small-scale targets with blurred boundaries, and they are easily occluded by branches and leaves. Therefore, the segmentation quality of stems is affected not only by the model structure, but also by annotation accuracy and target visibility. In real orchard images, stems often show similar colors and textures to branches, petioles, and shadow regions, and their boundaries are sometimes difficult to distinguish. Even manual annotation may involve a certain degree of subjectivity. Therefore, future research should not only focus on improving the model architecture, but also further analyze the influence of stem annotation consistency, boundary uncertainty, and hard sample distribution on model training. Boundary supervision, fine-grained structural constraints, uncertainty modeling, or active learning strategies can be introduced to improve the recognition ability of the model for weak-boundary, small-scale, and partially occluded stem regions.

Moreover, this study mainly performed instance segmentation of fruits and stems based on single-frame RGB images, whereas real harvesting robots usually operate in dynamic observation environments. The movement of the robotic arm, camera, or mobile platform may cause viewpoint changes, motion blur, and short-term target occlusion. Relying only on single-frame images may make it difficult to ensure stable and continuous recognition. Future studies can further combine video sequences, multi-view images, or RGB-D data to enhance the continuous perception of stem regions by using temporal information and spatial geometric constraints. For example, multi-frame feature fusion, target tracking, 3D point-cloud-assisted segmentation, or visual servo feedback mechanisms can be adopted, so that the model can not only recognize targets in static images, but also maintain stable perceptual outputs during robot motion.

In addition, there remains a task gap between instance segmentation results and actual harvesting actions. For pomelo harvesting robots, fruit and stem segmentation is only the front-end part of visual perception. The ultimate goal is to determine reliable picking points, plan robotic arm motion paths, and complete stable harvesting operations. Therefore, future research should further focus on how segmentation results affect harvesting decisions. For example, small errors in stem mask boundaries may influence stem centerline extraction, picking-point localization, and end-effector pose calculation. Mis-segmentation near the fruit–stem connection region may also lead to picking-point deviation. In future work, an error propagation model from instance segmentation errors to picking-point localization errors can be established. Combined with 3D pose estimation, stem skeleton extraction, and robotic arm motion constraints, a closed-loop validation system from visual segmentation to harvesting control can be developed.

From the perspective of engineering deployment, agricultural robots are usually constrained by computing resources, power consumption, environmental stability, and real-time response requirements. Although the processing efficiency of YOLO-MOD has been evaluated on the computer platform used in this study, its runtime performance, resource consumption, power consumption, and long-term operational stability on edge computing devices, embedded platforms, or real harvesting robots have not yet been systematically verified. Future work will focus on deployment-oriented evaluation, including tests of inference speed, memory usage, power consumption, and long-term stability on platforms such as Jetson devices, industrial computers, or mobile robotic systems. Meanwhile, model pruning, knowledge distillation, quantization acceleration, and TensorRT-based deployment optimization can be adopted to reduce computational costs while maintaining stem segmentation accuracy, thereby improving the deployability of the model in real orchard environments.

Finally, research on intelligent pomelo harvesting vision systems should not be limited to improving the accuracy of a single model. It should gradually move toward multi-task collaborative perception and robotic system-level validation. In the future, fruit detection, stem segmentation, maturity assessment, obstacle recognition, picking-point localization, and 3D pose estimation can be integrated into a unified visual framework to build a more complete robotic harvesting perception system. In addition, real harvesting experiments should be further conducted to quantitatively analyze the relationships between model outputs and harvesting success rate, picking time, fruit damage rate, and system stability. This will provide a more comprehensive evaluation of the practical value of visual algorithms in agricultural robotic applications.

6. Conclusions

This study proposes YOLO-MOD, an improved instance segmentation algorithm based on YOLOv11-seg, to enhance the front-end visual segmentation performance of pomelo fruits and stems under complex natural orchard conditions. By introducing ODConv into the C3k2 module, embedding the MSDA module, and replacing the original upsampling operator with DySample, the proposed method enhances the representation ability of the model for local structural differences, multi-scale semantic information, and fine boundary features. As a result, the joint instance segmentation performance for pomelo fruits and stems is improved.

The experimental results showed that YOLO-MOD achieved Box mAP@50 and Mask mAP@50 values of 88.6% and 87.8%, respectively, with GFLOPs and Params of 11.3 G and 4.1 M. Compared with the baseline YOLOv11-seg model, the overall Box mAP@50 and Mask mAP@50 increased by 2.9% and 3.9%, respectively. For the more challenging stem class, Box mAP@50 and Mask mAP@50 increased by 5.9% and 7.8%, respectively. These results indicate that the proposed method can effectively improve the instance segmentation accuracy of pomelo fruits and stems with only a moderate increase in model complexity. In particular, it improves the Mask overlap quality and boundary consistency of slender, small-scale, and easily occluded stem targets.

Overall, YOLO-MOD shows better comprehensive performance and scene adaptability than the baseline model. The proposed method shows potential as a front-end visual perception component for subsequent studies on pomelo picking-point localization and robotic harvesting. However, its effectiveness in 3D localization, robotic arm planning, and complete harvesting operations still requires further validation on an integrated robotic platform. Future work will further expand the dataset with pomelo fruit and stem images collected from multiple regions, varieties, and growth stages, and deployment tests will be conducted on edge devices, harvesting robots, or intelligent terminals. Meanwhile, instance segmentation will be further combined with picking-point localization, 3D pose estimation, and robotic arm cooperative control, so that the segmentation results can provide more accurate visual guidance for robotic harvesting and further improve the practical value of the proposed method in intelligent pomelo harvesting.