Instance Segmentation Method for ‘Yuluxiang’ Pear at the Fruit Thinning Stage Based on Improved YOLOv8n-seg Model

Abstract

Accurate detection and segmentation of young ‘Yuluxiang’ pear fruits at the fruit thinning stage are crucial for the development of intelligent fruit thinning robots. To address the challenges in recognition and segmentation of young ‘Yuluxiang’ pears in natural environments characterized by occlusion, overlap, and small targets, this paper proposes an improved instance segmentation model based on YOLOv8n-seg, named YOLOv8n-DSW. Firstly, the C2f modules were optimized by introducing DualConv to construct C2f-Dual modules, which enhanced feature extraction capability while reducing the number of parameters. Secondly, a Spatial-Channel Synergistic Attention (SCSA) mechanism was embedded ahead of the small-object detection head to improve detection accuracy for small targets. Finally, the original CIoU loss function was replaced with the WIoU v3 loss function to accelerate model convergence and improve accuracy. Deployment on a Firefly ROC-RK3588S-PC development board confirmed the model’s suitability for edge devices. Experimental results demonstrated that YOLOv8n-DSW achieved excellent performance. The mAP₅₀, mAP₇₅, and mAP_50:95 for detection reached 95.6%, 83.2%, and 70.3%, respectively, and those for segmentation were 94.8%, 78.2%, and 65.3%. The proposed model outperformed its baseline, YOLOv8n-seg, as well as other classic models such as YOLOv5n-seg, YOLOv11n-seg, and YOLOv12n-seg. These results demonstrate that YOLOv8n-DSW provides accurate and efficient segmentation of young ‘Yuluxiang’ pear fruits.

1. Introduction

The ‘Yuluxiang’ pear is a high-quality, mid-maturing, and storage-resistant new cultivar of the Korla fragrant pear, characterized by its large fruit size, juicy flesh, and high yield. It was developed through hybridization using the Korla fragrant pear as the female parent and the Snowflake pear as the male parent [1]. ‘Yuluxiang’ pear trees tend to form flower buds easily and exhibit a high fruit set rate. Timely and effective flower and fruit thinning is crucial for ensuring final fruit quality [2,3]. Current primary fruit thinning methods include chemical thinning and manual thinning. Chemical thinning is cost-effective but raises concerns about pesticide residues. While manual thinning allows selective fruit removal by experienced workers to ensure quality, it suffers from low efficiency, high labor demands, and significant time costs [4]. To balance efficiency and cost, developing intelligent fruit thinning equipment has been increasingly adopted in modern orchard management [5], for which accurate identification is a prerequisite.

Currently, existing traditional machine learning methods for fruit recognition include Fuzzy C-Means [6,7], K-means clustering [8], and Support Vector Machines [9,10]. While these methods can achieve fruit recognition, they are susceptible to environmental interference such as varying illumination and leaf occlusion, leading to insufficient recognition accuracy in complex orchard environments. In contrast, deep learning algorithms have demonstrated notable advantages in agricultural detection [11,12,13] by automating feature extraction and offering superior generalization, accuracy, and processing speed. The YOLO series models feature impressive real-time detection performance and high accuracy, which lend them broad application value in agricultural scenarios. For accurate detection of young apple fruits, Song et al. [14] proposed a target detection method based on the YOLOv4-SENL model, which integrated SE Block and NL Block visual attention mechanisms to recalibrate high-level features from both channel and non-local perspectives. The improved model achieved an mAP of 96.9% for young apples in natural environments, demonstrating excellent detection performance. To address the issues of dense grape clusters and occlusion in natural environments, Zhang et al. [15] incorporated CBAM into YOLOv5 to produce YOLOv5-GAP, which achieved an mAP of 95.13%. For multi-task segmentation in pitaya orchards, Zhao et al. [16] enhanced the YOLOP backbone by embedding a Focus and ACmix (FAC) module, integrating C2f and SPPF structures, and optimizing the loss function. Similarly, Li et al. [17] integrated residual modules with gated convolution and a spatial pyramid structure, achieving a segmentation accuracy of 95.11% for green apples. Despite the progress made by existing studies in fruit recognition and segmentation, research, specifically on young fruits, remains relatively scarce, especially when considering the challenges posed by factors like occlusion, overlap, and small targets.

In response to the challenges outlined above, this study introduced the YOLOv8n-DSW model. The main contributions include the following:

(1) This study proposed an instance segmentation model capable of accurately identifying and segmenting young ‘Yuluxiang’ pear fruits under natural conditions. The model incorporates three key improvements: the C2f-Dual module to strengthen feature extraction while achieving model lightweighting; the SCSA mechanism to improve the detection capability for young fruits; and the WIoU v3 loss function to optimize bounding-box regression and accelerate convergence.

(2) Deployment on a Firefly ROC-RK3588S-PC edge computing platform demonstrated that YOLOv8n-DSW maintained high detection and segmentation accuracy while achieving low inference latency on the tested hardware. These results suggest the model’s feasibility for integration into real-time, resource-constrained intelligent thinning equipment and support further development toward automated orchard management.

2. Materials and Methods

2.1. Dataset

2.1.1. Data Acquisition

The images were acquired in early May 2025 at the ‘Yuluxiang’ pear experimental base of the Pomology Institute at Shanxi Agricultural University, Jinzhong City, Shanxi Province. Image acquisition was performed using an Intel^® RealSense™ Depth Camera D435i (Intel Corporation, Santa Clara, CA, USA). To ensure data diversity, the images were captured under both frontlighting and backlighting conditions. After removing duplicate and blurred images, a final dataset of 1715 images of young ‘Yuluxiang’ pear fruits was established. The dataset was then randomly split into training, validation, and test sets at a ratio of 7:2:1. The dataset encompassed various scenarios, including frontlighting, backlighting, fruit overlap, and occlusion by branches or leaves, as shown in Figure 1.

Figure 1. Images of young ‘Yuluxiang’ pear: (a) frontlight; (b) backlight; (c) fruit overlap; (d) branch occlusion; (e) leaf occlusion.

2.1.2. Data Processing

To mitigate overfitting caused by a limited number of training images, the training set was augmented, increasing its size from the original 1200 to 6000 images. The augmentation techniques employed included Gaussian noise addition, salt-and-pepper noise addition, brightness adjustment, and horizontal flipping. Examples of the augmented images are shown in Figure 2. The detailed distribution of the dataset is presented in Table 1.

Figure 2. Data augmentation process: (a) original image; (b) Gaussian noise; (c) salt-and-pepper noise; (d) brightness adjustment; (e) horizontal flip.

Table 1. Dataset of young ‘Yuluxiang’ pear.

Data Set	Training Set	Validation Set	Test Set	Total Quantity
Original image	1200	343	172	1715
Augmented image	4800	0	0	4800
Total quantity	6000	343	172	6515

The images of young fruits were manually annotated using the LabelMe software (Version 5.6.1). Each fruit instance was assigned the category label “pear”, and its contour was delineated with polygons, resulting in JSON files containing the respective target information [18,19]. The annotation examples are shown in Figure 3.

Figure 3. Examples of dataset annotation details.

2.2. Research Method

2.2.1. YOLOv8 Architecture

YOLOv8 [20], developed by Ultralytics, is a real-time object detection architecture comprising a backbone, a neck, and a detection head. It supports various computer vision tasks, including object detection, instance segmentation, and pose estimation, catering to diverse application scenarios. The backbone, based on CSPDarknet, efficiently extracts multi-scale features from input images, supplying both detailed and high-level semantic information for subsequent processing. The neck employs a Path Aggregation Network with Feature Pyramid Network (PAN-FPN) architecture, which enhances multi-scale feature integration through cross-level connections and effectively balances semantic and spatial details. Furthermore, the architecture employs an anchor-free, decoupled detection head that directly predicts class probabilities and bounding-box coordinates from feature map grids, and a segmentation (mask) head that generates per-instance masks. This design reduces computational complexity while improving localization and segmentation accuracy, enabling efficient real-time instance segmentation. The YOLOv8 framework encompasses multiple versions. Among them, the lightweight YOLOv8n-seg model was chosen as the baseline for this study because it offers a good balance of accuracy and speed and has been widely applied in computer vision.

2.2.2. Improved YOLOv8n-Seg Network Architecture

This study proposed an improved version named YOLOv8n-DSW, based on the YOLOv8n-seg model, and its architecture is depicted in Figure 4. The improvements encompassed three key aspects. First, a DualConv structure was introduced into the original C2f module, forming a C2f-Dual module. This module processed the same input feature map channels simultaneously with dual convolutional kernels, which enhanced spatial feature extraction while reducing both the parameter count and computational cost. Second, to address the challenges of small target size and feature loss in young ‘Yuluxiang’ pears, an SCSA mechanism was embedded before the small-object detection head. This integration enhanced the model’s focus on local details and feature extraction. Third, the WIoU v3 loss function was employed, which strategically focused on anchor boxes of ordinary quality. This replacement accelerated model convergence and enhanced model performance.

Figure 4. Architecture of the proposed YOLOv8n-DSW model. Modifications relative to the original YOLOv8n-seg are indicated below: all C2f modules in the backbone and neck are replaced with the proposed C2f-Dual modules (green blocks); an SCSA attention module is inserted before the small-object detection head (outlined by a red dashed box); and the bounding-box regression loss is replaced with WIoU v3.

2.2.3. C2f-Dual Module

DualConv [21] is a novel convolutional module that processes the same input feature map channels in parallel using two distinct kernel sizes: 3 × 3 and 1 × 1. The 3 × 3 kernels capture spatial features, while the 1 × 1 kernels perform feature fusion and reduce parameters. It also employs group convolution, which partitions the input and output feature maps into multiple independent groups. The convolutional filters within each group process only their corresponding partitioned input channels. This design markedly reduces model complexity and enhances the efficiency of information flow and feature extraction, while maintaining the network’s representational power. The architecture of the DualConv module is depicted in Figure 5.

Figure 5. DualConv network structure.

Assume that the size of the output feature map is D_o × D_o × N, where D_o is the width and height dimension of the output feature map, and the total number of FLOPs performed in a standard convolutional layer FL_SC is as follows:

$${\mathit{FL}}_{\mathit{SC}}={ D}_o^2\times K^2\times M\times N$$

(1)

where K × K is the convolutional kernel size, M is the number of input channels, and N is the number of convolutional filters and also the number of output channels.

In DualConv, the number of convolutional filter groups G is used to control the proportion of K × K convolutional kernels in a convolutional filter. For a given G, the proportion of combined simultaneous convolutional kernels with size of (K × K + 1 × 1) is 1/G of all channels, while the proportion of the remaining 1 × 1 convolutional kernels is (1 − 1/G). Therefore, in a dual convolutional layer composed of G convolutional filter groups, the total number of FLOPs is as follows:

$${\mathit{FL}}_{\mathit{CC}}={\displaystyle \frac{D_o^2\times K^2\times M\times N+D_o^2\times M\times N}{G}}$$

(2)

$${\mathit{FL}}_{\mathit{PC}}=\left(D_o^2\times M\times N \right)\times \left( 1 - 1/G \right)$$

(3)

$${\mathit{FL}}_{\mathit{DC}}={\mathit{FL}}_{\mathit{CC}}+{\mathit{FL}}_{\mathit{PC}}={\displaystyle \frac{D_o^2\times K^2\times M\times N}{G}}+D_o^2\times M\times N$$

(4)

The computational reduction ratio R_DC/SC of the dual convolution layer compared with the standard convolution layer is as follows:

$$R_{\mathit{DC}/\mathit{SC}}={\displaystyle \frac{{\mathit{FL}}_{\mathit{DC}}}{{\mathit{FL}}_{\mathit{SC}}}}={\displaystyle \frac{1}{G}}+{\displaystyle \frac{1}{K^2}}$$

(5)

According to Formula (5), given that K = 3 in the DualConv design, the speedup can reach 8 to 9 times when G is large. Thus, DualConv achieves higher efficiency while maintaining accuracy.

The C2f-Dual module proposed in this study embeds DualConv into the original C2f structure. The architecture of the C2f-Dual module is illustrated in Figure 6, and its workflow is as follows: First, the number of feature map channels is doubled through a 1 × 1 convolution. The input is then split into two parts. One part is processed through several DualBottleneck modules to extract deep features. Subsequently, the output from these modules is concatenated with the other part, which was held as a residual connection. Finally, the concatenated features are passed through a convolutional layer to compress them to the required output channel size.

Figure 6. C2f-Dual structure diagram.

The DualBottleneck module is illustrated in Figure 7. It first reduces the channel dimension by half using a 3 × 3 convolution and then processes the features through the DualConv layer. This structure enhances the performance of the feature extraction network while maintaining computational efficiency and training stability. Furthermore, by incorporating a cross-stage partial connection mechanism, the C2f-Dual module improves contextual information extraction and synergistically enhances multi-scale feature representation. This design enhances model performance, enabling accurate identification and segmentation of young fruits in natural orchard environments.

Figure 7. DualBottleneck structure diagram.

2.2.4. SCSA Module

The SCSA module comprises two key components: Shared Multi-Semantic Spatial Attention (SMSA) and Progressive Channel-wise Self-Attention (PCSA) [22]. It integrates the spatial attention’s ability to focus on target locations with the channel attention’s capability to screen critical features, thereby enhancing performance in visual tasks. Specifically, the SMSA extracts multi-semantic spatial information and injects this discriminative spatial information into the PCSA through a progressive compression strategy, which guides the subsequent channel recalibration. The PCSA then utilizes a self-attention mechanism to compute channel-wise similarities, thereby reducing the semantic discrepancies among the different sub-features produced by the SMSA.

As illustrated in Figure 8, the operational workflow is as follows: The SMSA first decomposes the input feature map along the height and width dimensions. After applying global average pooling, it partitions the representation into multiple independent sub-features. These sub-features are subsequently processed by multi-scale MS-DWConv1d, normalized via GroupNorm, and activated by the Sigmoid function to generate the spatial attention map. For the PCSA, it first performs average pooling and GroupNorm normalization on the features processed by the SMSA. It then generates the Query, Key, and Value through a multi-branch depthwise convolution (DWConv). Following feature aggregation via Channel-wise Single-Head Self-Attention (CA-SHSA), the output is passed through an average pooling layer and a sigmoid activation function to produce the channel attention map. Finally, this channel attention map is multiplied by the SMSA-processed features to yield the module’s final output.

Figure 8. SCSA structure diagram.

The SCSA module is placed before the small-object detection head, as the feature maps for this head have the highest spatial resolution and the smallest receptive fields, making them well-suited to the SCSA mechanism for refining fine-grained features.

2.2.5. WIoU Loss Function

The original YOLOv8n-seg model employs the CIoU loss for bounding-box regression. Although CIoU improves regression accuracy by incorporating aspect ratio, it often leads to unstable gradients and slow convergence when predicted boxes have low overlap with ground truths and large aspect ratio differences. To overcome these limitations, this study adopts WIoU v3 [23]. WIoU v3 incorporates a dynamic non-monotonic focusing mechanism and a rational gradient gain allocation strategy. It assigns smaller gradient gains to anchor boxes with high outlierness, thereby preserving anchor box quality and reducing the impact of harmful gradients.

$$R_{\mathit{WIoU}}=\mathit{exp}\left({\displaystyle \frac{{\left( x-x_{\mathit{gt}} \right)}^2+{\left( y-y_{\mathit{gt}} \right)}^2}{{\left( W_g^2+H_g^2 \right)}^{*}}}\right)$$

(6)

$$L_{\mathit{WIoUv}1}=R_{\mathit{WIoU}}{L}_{\mathit{IoU}}$$

(7)

$$\beta ={\displaystyle \frac{L_{\mathit{IoU}}^{*} }{\overline{L_{\mathit{IoU}}}}}\in [0,+\infty )$$

(8)

$$r={\displaystyle \frac{\beta }{\delta {\alpha }^{\beta -\delta }}}$$

(9)

$$L_{\mathit{WIoUv}3}=r{L}_{\mathit{WIoUv}1}$$

(10)

where L_IoU is the bounding-box loss, R_WIoU is the distance loss, x and y are the coordinates of the predicted bounding-box center, x_gt and y_gt are the coordinates of the ground truth bounding-box center, W_g and H_g are the width and height of the smallest enclosing rectangle covering both the predicted and ground truth boxes. The $\ast$ operator indicates that W_g and H_g are detached from the computation graph, r represents the gradient gain, β denotes the outlier measure, and α and δ are hyperparameters. $L_{\mathit{IoU}}^{*}$ indicates that L_IoU is detached from the computation graph, and $\overline{L_{\mathit{IoU}}}$ is the exponential running average.

3. Experiments and Results

3.1. Experimental Setup

3.1.1. Implementation Environment and Training Parameters

All experiments were conducted using PyTorch 2.6.0 with Python 3.9.23, employing the Ultralytics YOLO library version 8.3.91 for model implementation. Development was performed in PyCharm 2024.1 on a Windows 11 workstation equipped with an NVIDIA RTX A6000 GPU (48 GB VRAM) and an Intel Xeon W-2295 CPU @ 3.00 GHz.

The training hyperparameters are summarized in Table 2. The input image size was set to 640 × 640 pixels. The model was trained for 200 epochs with a batch size of 128. The optimizer was Stochastic Gradient Descent (SGD) with an initial learning rate of 0.01 and a weight decay coefficient of 0.0005. The random seed was set to 0. For evaluation and inference, the confidence and Non-Maximum Suppression (NMS) Intersection over Union (IoU) thresholds were set to 0.25 and 0.45, respectively.

Table 2. Training parameters.

Training Parameter	Value
Input Image Size	640 × 640
Epochs	200
Batch Size	128
Optimizer	SGD
Initial Learning Rate	0.01
Weight Decay Coefficient	0.0005
Augment	True
Hue Shift (hsv_h)	0.015
Saturation Shift (hsv_s)	0.7
Brightness Shift (hsv_v)	0.4
Translation (translate)	0.1
Scale (scale)	0.5
Horizontal Flip (fliplr)	0.5
Mosaic (mosaic)	1.0
Seed	0

3.1.2. Evaluation Metrics

To evaluate the model’s performance on the ‘Yuluxiang’ pear segmentation task, this study employed precision (P), recall (R), mean average precision at an IoU of 0.5 (mAP₅₀), mAP at 0.75 (mAP₇₅), mAP averaged over IoUs from 0.5 to 0.95 (mAP_50:95), giga floating-point operations per second (GFLOPs), parameters (Params), inference time, and frames per second (FPS) as evaluation metrics. The formulas for the primary evaluation metrics are defined as follows:

$$P = {\displaystyle \frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}}\times 100\%$$

(11)

$$R={\displaystyle \frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}}\times 100\%$$

(12)

$${\mathit{mAP}}_{50}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathit{AP}}_i\left(\mathit{IoU}=0.5\right)$$

(13)

$${\mathit{mAP}}_{75}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathit{AP}}_i\left(\mathit{IoU}=0.75\right)$$

(14)

$${\mathit{mAP}}_{50:95}={\displaystyle \frac{1}{N}}\sum _{i=1}^{\mathrm{N}}\left({\displaystyle \frac{1}{10}}\sum _{k=0}^9{\mathit{AP}}_i\left(\mathit{IoU}=0.5+0.05k\right)\right)$$

(15)

where TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively.

3.2. Experiments

3.2.1. Impact of Integrating the C2f-Dual Module at Different Locations

To validate the effectiveness of the C2f-Dual module, it was used to replace the original C2f modules in both the backbone and neck networks. Three comparative model configurations were established: YOLOv8n-Db (replacement only in the backbone), YOLOv8n-Dn (replacement only in the neck), and YOLOv8n-Dbn (replacement in both). The results are presented in Table 3.

Table 3. Results of applying C2f-Dual to different positions.

Models	Detection					Segmentation					GFLOPs	Params
	P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%	P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%
YOLOv8n-seg	93.4	87.7	93.0	78.5	67.4	92.5	86.5	92.1	72.5	62.1	12.0	3.26 × 106
YOLOv8n-Db	93.9	88.9	94.6	82.4	69.6	93.1	88.2	93.6	75.0	64.4	11.6	3.10 × 106
YOLOv8n-Dn	94.1	88.7	94.7	81.2	69.1	93.4	88.7	93.9	75.1	64.1	11.7	3.11 × 106
YOLOv8n-Dbn	94.1	89.1	94.8	82.5	69.7	93.7	88.6	94.0	75.3	64.5	11.3	2.96 × 106

The YOLOv8n-Db model outperformed the baseline model. In terms of detection performance, its mAP₅₀, mAP₇₅, and mAP_50:95 were increased by 1.6%, 3.9%, and 2.2%, respectively. For segmentation performance, the corresponding metrics were improved by 1.5%, 2.5%, and 2.3%, respectively. Furthermore, both GFLOPs and Params were reduced. These results indicate that replacing the C2f modules in the backbone effectively enhanced the feature extraction capability, achieved model lightweighting, and improved detection and segmentation precision simultaneously. The YOLOv8n-Dn model, which only modified the C2f modules in the neck, also exhibited better performance than the baseline model in all metrics. The YOLOv8n-Dbn model achieved outstanding performance. Its detection mAP₅₀, mAP₇₅, and mAP_50:95 reached 94.8%, 82.5%, and 69.7%, while its segmentation mAP₅₀, mAP₇₅, and mAP_50:95 reached 94.0%, 75.3%, and 64.5%, all of which were higher than those of the other two models and the baseline model. Meanwhile, its GFLOPs and Params stood at 11.3 and 2.96 M, corresponding to the lowest computational complexity and parameter count among all models. In summary, the comprehensive replacement of C2f modules in both the backbone and the neck achieved the optimal balance between accuracy improvement and model lightweighting.

3.2.2. Analysis of SCSA Module Integration Effects at Different Detection Layers

To analyze the impact of integrating the SCSA module at different positions within the detection head, the SCSA module was individually added to the large, medium, and small-object detection layers of the YOLOv8n-seg model. The resulting models are denoted as YOLOv8n-L, YOLOv8n-M, and YOLOv8n-S, respectively. A model incorporating the SCSA module across all three detection layers is denoted as YOLOv8n-all. Table 4 presents the performance metrics of these five models.

Table 4. Performance indicators of five models.

Models	Detection					Segmentation
	P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%	P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%
YOLOv8n-seg	93.4	87.7	93.0	78.5	67.4	92.5	86.5	92.1	72.5	62.1
YOLOv8n-L	92.5	89.2	94.6	81.2	69.2	93.7	87.5	94.2	75.5	64.5
YOLOv8n-M	93.5	89.0	95.0	81.3	69.1	93.5	87.9	94.3	75.1	64.3
YOLOv8n-S	94.1	89.4	94.7	81.6	69.3	93.7	88.5	94.3	76.0	64.7
YOLOv8n-all	93.9	88.9	94.7	81.5	69.0	93.9	88.1	94.2	75.6	64.7

As shown in Table 4, integrating the SCSA module individually into the large, medium, or small-object detection layers improved most key metrics for both detection and segmentation tasks compared with the original YOLOv8n-seg model. The most noticeable enhancements were observed when the module was added to the small-object detection layer. For the detection task, precision, recall, mAP₅₀, mAP₇₅, and mAP_50:95 increased by 0.7%, 1.7%, 1.7%, 3.1%, and 1.9%, respectively. In the segmentation task, the corresponding metrics improved by 1.2%, 2.0%, 2.2%, 3.5%, and 2.6%, indicating a more pronounced benefit for segmentation performance. This can be attributed to the small-object detection layer processing shallow features from the backbone network, characterized by larger feature map resolutions and smaller receptive fields. Furthermore, the overall performance of the YOLOv8n-all model was slightly inferior to that of YOLOv8n-S. Consequently, the final model configuration integrates the SCSA module solely into the small-object detection layer.

3.2.3. Comparative Experiments with Different Loss Functions

To validate the performance of the WIoU loss function in the detection and segmentation tasks of young ‘Yuluxiang’ pear fruits, both CIoU and WIoU were, respectively, used as the loss functions for training the YOLOv8n-seg model. Figure 9 illustrates the decrease in loss values for both loss functions on the validation set.

Figure 9. Loss curves of different loss functions: (a) box loss; (b) mask loss.

Both box loss curves drop rapidly in the initial training phase, indicating quick weight adaptation to reduce bounding-box errors. The CIoU loss is higher than the WIoU loss in the early epochs, showing that WIoU optimizes bounding-box regression more quickly; over the full run, WIoU remains lower and stabilizes, suggesting better validation performance for bounding-box regression. The mask loss curves for both losses decline and converge to nearly identical values, implying that WIoU’s main benefit is improved bounding-box regression rather than a direct effect on segmentation; more accurate candidate regions produced by WIoU indirectly favor segmentation and yield observable metric gains.

WIoU also exhibits faster convergence than CIoU in the box loss curves, enabling quicker parameter optimization and lower loss values. These properties suggest that WIoU can more effectively guide training, enhancing stability and reducing overfitting. In summary, WIoU accelerates convergence and improves regression accuracy, leading to better overall model performance.

3.2.4. Ablation Experiment and Result Analysis

Ablation experiments were conducted to assess the impact of the proposed improvements. The results are summarized in Table 5.

Table 5. Ablation experiments’ results.

Models	A	B	C	Detection					Segmentation					GFLOPs	Params
				P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%	P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%
1	×	×	×	93.4	87.7	93.0	78.5	67.4	92.5	86.5	92.1	72.5	62.1	12.0	3.26 × 106
2	√	×	×	94.1	89.1	94.8	82.5	69.7	93.7	88.6	94.0	75.3	64.5	11.3	2.96 × 106
3	×	√	×	94.1	89.4	94.7	81.6	69.3	93.7	88.5	94.3	76.0	64.7	12.0	3.27 × 106
4	×	×	√	94.6	89.2	95.3	82.2	69.7	93.0	89.1	94.6	77.1	65.0	12.0	3.26 × 106
5	√	√	×	93.9	88.7	94.9	79.7	69.5	94.4	88.9	93.8	74.5	64.3	11.3	2.97 × 106
6	√	×	√	94.4	88.9	94.6	80.9	67.9	94.6	88.5	94.4	75.6	64.2	11.7	3.11 × 106
7	×	√	√	93.8	89.9	94.8	79.3	68.3	93.3	89.9	94.2	76.9	64.9	12.0	3.27 × 106
8	√	√	√	94.7	90.2	95.6	83.2	70.3	95.0	89.9	94.8	78.2	65.3	11.3	2.97 × 106

Note: A, B, C denote C2f-Dual, SCSA, WIoU. “√” indicates that the module is included, “×” indicates that the module is not included.

Incorporating the DualConv architecture into the baseline YOLOv8n-seg improved both detection and segmentation while reducing model complexity. Detection mAP₅₀, mAP₇₅, and mAP_50:95 reached 94.8%, 82.5%, and 69.7%, corresponding to gains of 1.8%, 4.0%, and 2.3% over the baseline; segmentation mAP₅₀, mAP₇₅, and mAP_50:95 increased by 1.9%, 2.8%, and 2.4%, respectively. Params and GFLOPs also decreased, indicating that DualConv both strengthens feature extraction and achieves lightweighting by using dual convolutional kernels. Adding the SCSA module before the small-object detection layer improved detection mAP₅₀ by 1.7% and segmentation mAP₅₀ by 2.2% relative to the baseline. The adoption of the WIoU enhanced model performance. For the detection task, mAP₅₀, mAP₇₅, and mAP_50:95 reached 95.3%, 82.2%, and 69.7%, respectively. Segmentation metrics also showed substantial improvement, while Params and GFLOPs remained unchanged.

When any two improvements were combined, the model’s overall performance remained better than the baseline. The simultaneous integration of all three enhancements yielded optimal performance. The final model achieved detection precision, recall, mAP₅₀, mAP₇₅, and mAP_50:95 of 94.7%, 90.2%, 95.6%, 83.2%, and 70.3%, respectively, representing gains of 1.3%, 2.5%, 2.6%, 4.7%, and 2.9% over the baseline. For segmentation, it attained precision, recall, mAP₅₀, mAP₇₅, and mAP_50:95 of 95.0%, 89.9%, 94.8%, 78.2%, and 65.3%, improving by 2.5%, 3.4%, 2.7%, 5.7%, and 3.2%, respectively. These results validate the synergistic value of the three strategies: the lightweight feature extraction of DualConv, the small-target focus of SCSA, and the loss optimization of WIoU collectively enhance the model’s adaptability to growth scenarios of young ‘Yuluxiang’ pear fruits.

Figure 10 compares the segmentation results of the baseline model and the proposed YOLOv8n-DSW model to illustrate the improvement in performance. The baseline model exhibits several limitations: a high rate of missed detections, particularly for small or heavily occluded young fruits (Figure 10b–d); false positives caused by leaves being misclassified as young fruits (Figure 10c,d); and insufficient segmentation accuracy resulting in coarse delineation of fruit boundaries (Figure 10d). In contrast, the proposed YOLOv8n-DSW model, incorporating all three improvements, accurately segments occluded fruits and more precisely localizes small targets. It better captures fine-grained features and improves boundary segmentation, producing clear and well-defined contours. Overall, the proposed model demonstrates improved robustness and generalization in handling occluded and overlapping targets.

Figure 10. Examples of model segmentation results: (a) frontlight; (b) fruit overlap; (c) branch occlusion; (d) leaf occlusion. Note: Red circles indicate missed detections of young fruits, yellow circles indicate false positive detections, and green circles highlight instances where the model exhibits inaccurate segmentation.

The experimental results confirm that the proposed method effectively improves model performance, yielding favorable results in the detection and segmentation of young ‘Yuluxiang’ pear fruits.

3.2.5. Heatmap Analysis

To better visualize the enhancement in the model’s recognition capability, this study utilizes Gradient-weighted Class Activation Mapping (Grad-CAM) to generate heatmaps [24], enabling detailed examination of the network’s attention distribution. Grad-CAM computes the gradients of the target class score relative to the feature maps in the final convolutional layer. These gradients undergo global average pooling to obtain weights for each feature map. The weights are then used for a weighted summation with their corresponding feature maps, followed by ReLU activation. Finally, the result is upsampled to the original image dimensions, producing a heatmap that highlights regions most critical for the model’s predictions.

Figure 11 compares heatmaps generated by the baseline and improved models for young ‘Yuluxiang’ pear images. In Figure 11a,b, the enhanced YOLOv8n-DSW model shows more intense activation in the actual fruit regions, indicating better focus on the targets themselves and demonstrating improved localization precision. In Figure 11c, while the baseline YOLOv8n-seg model fails to activate significantly in the occluded fruit area, the improved model produces distinct heatmap values on the exposed portions of occluded fruits. Additionally, in Figure 11d,e, the baseline model exhibits noticeable activation in background regions without fruits, whereas such background activation is clearly reduced in the improved model. These observations demonstrate that the enhanced YOLOv8n-DSW model possesses superior capability in distinguishing targets from background and identifying occluded fruits, thereby reducing false positives and missed detections. The proposed model exhibits stronger robustness and stability in complex scenarios.

Figure 11. The heatmap comparison of the model before and after improvement: (a) frontlight; (b) backlight; (c) fruit overlap; (d) branch occlusion; (e) leaf occlusion.

3.2.6. Comparative Analysis of Various Models

To further evaluate the effectiveness of the improved model, YOLOv5n-seg [25], YOLOv9c-seg [26], YOLO11n-seg [27], YOLO12n-seg [28], and the original and improved YOLOv8n-seg models were compared under identical experimental conditions. The comparative results of these six models are summarized in Table 6.

Table 6. Comparison results of different network models.

Models	Detection					Segmentation					GFLOPs	Params
	P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%	P /%	R /%	mAP50 /%	mAP75 /%	mAP50:95 /%
YOLOv5n-seg	90.9	78.5	85.9	72.1	62.4	90.8	77.6	84.3	68.5	57.5	6.7	1.88 × 106
YOLOv8n-seg	93.4	87.7	93.0	78.5	67.4	92.5	86.5	92.1	72.5	62.1	12.0	3.26 × 106
YOLOv9c-seg	95.4	90.3	96.1	90.8	79.2	95.5	90.5	96.2	85.5	70.9	157.6	2.76 × 107
YOLO11n-seg	92.5	90.4	94.4	82.6	69.3	93.4	89.1	94.4	77.8	64.7	10.4	2.84 × 106
YOLO12n-seg	93.6	90.7	95.2	83.1	69.9	93.3	89.9	94.2	77.6	64.5	10.4	2.82 × 106
YOLOv8n-DSW	94.7	90.2	95.6	83.2	70.3	95.0	89.9	94.8	78.2	65.3	11.3	2.97 × 106

As shown in Table 6, the proposed YOLOv8n-DSW model demonstrates superior overall performance. For object detection, it attains a precision of 94.7%, a recall of 90.2%, a mAP₅₀ of 95.6%, a mAP₇₅ of 83.2%, and a mAP_50:95 of 70.3%. These represent clear improvements over its baseline. For instance, in segmentation, it achieves a precision of 95.0%, a recall of 89.9%, a mAP₅₀ of 94.8%, a mAP₇₅ of 78.2%, and a mAP_50:95 of 65.3%, also showing consistent gains across all segmentation metrics compared with the baseline.

Notably, compared to newer models like YOLOv11n-seg and YOLOv12n-seg, it delivers the best performance across nearly all key metrics for both detection and segmentation, with only a slight increase in computational cost. Among all the compared models, YOLOv8n-DSW achieves the best balance between performance and efficiency. In terms of accuracy, it is only slightly behind the highest-performing model, YOLOv9c-seg. However, YOLOv9c-seg requires substantial computational resources of 157.6 GFLOPs and 27.6 million parameters. Such a massive computational overhead makes it difficult to deploy in practical scenarios. In contrast, YOLOv8n-DSW maintains this high performance with a computational cost of only 11.3 GFLOPs and 2.97 million parameters.

This confirms that the proposed improvements effectively enhance the model’s capability without introducing extra efficiency overhead. Therefore, YOLOv8n-DSW represents an excellent choice, effectively balancing high precision with practical deployment efficiency.

3.2.7. Visual System Design and Edge Deployment

This study developed a visualization system and verified model deployment on an edge computing platform. The visualization system features a split-panel interface: the left panel shows original input images while the right panel displays model-generated segmentation results. Young fruit targets are annotated with bounding boxes, category labels, and confidence scores. The system supports multiple operational modes, including model weight selection, static image detection, and real-time camera detection, enabling flexible switching between offline analysis and online detection to accommodate various application scenarios. The visual system design and deployment architecture are illustrated in Figure 12.

Figure 12. Model deployment on ROC-RK3588S-PC.

The implementation adopts the Firefly ROC-RK3588S-PC development board equipped with the Ubuntu 20.04 operating system. To enable the deep learning model to be compatible with the RK3588 AI chip and achieve efficient accelerated neural network inference using the NPU, this study completed the conversion of the GPU-trained model to an NPU-compatible format through model conversion, quantization, and other techniques. Edge-side inference is implemented by calling the underlying RKNN inference engine via RKNN-Toolkit-Lite. The quantization process was as follows: first, the trained best.pt file was converted to the ONNX format; subsequently, the weights and activation values of the model were quantized from 32-bit floating-point (FP32) to 8-bit integer (INT8) using the RKNN Toolkit with a batch size of 1; finally, the RKNN model suitable for NPU-accelerated inference was obtained.

The test results are shown in Table 7. The improved YOLOv8n-DSW model reduces the inference time by approximately 5.7% compared to the original model. As seen in the segmentation results in Figure 13, the improved model produces fewer missed and false detections than the baseline. Deployment on edge devices verifies the practicality of YOLOv8n-DSW in practical agricultural environments. This work provides crucial technical support for the development of fruit thinning robots.

Table 7. Performance comparison of models on the edge device.

Models	Preprocess Time/ms	Inference Time/ms	Postprocess Time/ms	FPS	mAP50 (box)/%	mAP50:95 (box)/%	mAP50 (Mask)/%	mAP50:95 (Mask)/%
YOLOv8n-seg	14.31	26.70	28.03	14.48	91.0	65.9	90.1	60.7
YOLOv8n-DSW	13.75	25.17	27.12	15.14	93.5	68.8	92.7	63.9

Figure 13. Segmentation results on RK3588S: (a) YOLOv8n-seg; (b) YOLOv8n-DSW.

4. Discussion

Experimental results show that the proposed model notably outperforms the baseline YOLOv8n-seg model and other lightweight segmentation models. This advantage is mainly attributed to the synergistic effect of the three improved components. By integrating the C2f-Dual module, the SCSA attention mechanism, and the WIoU v3 loss function, the improved model significantly enhances the accuracy of instance segmentation for young ‘Yuluxiang’ pear fruits in natural orchard environments. Concurrently, the model was deployed on the Firefly ROC-RK3588S-PC edge device, validating its feasibility for real-time perception in resource-constrained environments.

Despite advances in the detection and segmentation of young fruits, occlusion remains a persistent challenge. Severe occlusion of young ‘Yuluxiang’ pears still presents considerable difficulty for detection, often leaving them largely or entirely invisible in captured images.

Future research will focus on three directions. First, the model architecture will be further refined to additionally reduce complexity while maintaining high accuracy. Second, the proposed model will be deployed on an intelligent fruit thinning robotic system to adapt to actual production environments. Finally, the framework will be extended to other small-target products, such as apples during the thinning period, to assess its generalizability and promote wider application in agricultural automation.

5. Conclusions

To address the challenges of occlusion, overlap, and small targets in detecting and segmenting young ‘Yuluxiang’ pears under natural orchard conditions, this study proposed YOLOv8n-DSW, an improved instance segmentation model. The model integrated a lightweight C2f-Dual module for enhanced feature extraction, incorporated an SCSA attention mechanism to improve the recognition capability of small-sized fruits, and used the WIoU v3 loss to optimize bounding-box regression. These enhancements effectively improved segmentation accuracy and addressed challenges posed by occlusion and overlap. Comprehensive experiments showed that the proposed YOLOv8n-DSW outperformed the baseline YOLOv8n-seg and other lightweight models, including YOLOv5n-seg, YOLOv11n-seg, and YOLOv12n-seg, across key performance metrics. Deployment on an edge computing platform validated the model’s efficiency and practical suitability for field applications. This work offers a reliable technical solution for intelligent fruit thinning robots and contributes to the advancement of automated, precision orchard management.