Visual Understanding of Intelligent Apple Picking: Detection-Segmentation Joint Architecture Based on Improved YOLOv11

Abstract

Achieving precise fruit localization and fine branch segmentation simultaneously in unstructured orchard environments remains challenging due to variable lighting, occlusion, and complex backgrounds. This study proposed a joint detection–segmentation architecture based on an improved YOLOv11 network for collaborative perception of apples and tree branches. First, a dual-task dataset of spindle-type apple orchards was constructed with bounding-box annotations for fruits and pixel-level polygon masks for branches, encompassing diverse illumination and occlusion conditions. Second, Convolutional Block Attention Modules (CBAMs) are strategically embedded into the YOLOv11 backbone to enhance feature discrimination for slender branch structures while preserving high fruit detection accuracy. The enhanced model achieves precision of 0.981, recall of 0.986, and F1-score of 0.983 for apple detection, and precision of 0.803, recall of 0.715, mAP of 0.698, and IoU of 0.6066 for branch segmentation on the validation set. Comparative experiments against YOLOv8 and baseline YOLOv11 confirm improved segmentation continuity and finer branch delineation. The proposed integrated perception framework provides reliable visual guidance for collision-avoidance robotic harvesting and offers a practical reference for multi-task agricultural vision systems.

1. Introduction

Smart agriculture is reshaping traditional production models with unprecedented depth and breadth. In the face of demographic changes, continuously rising labor costs, and increasing consumer demand for agricultural product quality and safety, promoting the transformation of agricultural production towards automation and intelligence has become a necessary path to ensure food security and enhance the core competitiveness of the industry. In this process, agricultural robots [1,2,3], especially agricultural picking robots with autonomous operation capabilities [4,5], integrate cutting-edge technologies such as mechanical electronics, sensing technology, and artificial intelligence, representing the cutting-edge direction of modern agricultural equipment technology. As the “smart eye” of robot systems, the performance of the machine vision perception module directly determines whether it can understand scenes, recognize targets, and make accurate decisions in complex and unstructured agricultural environments.

As one of the fruits with a wide cultivation area and high economic value worldwide, the demand for intelligent harvesting of apples is particularly urgent. The conventional approach to hand harvesting is characterized by both substantial labor demands and limited productivity, while also grappling with critical issues such as seasonal workforce deficits and a scarcity of skilled laborers [6]. To achieve intelligent and precise apple picking, the robot vision system must be able to quickly, accurately, and collaboratively perceive two key targets in the orchard scene. One is the object that the picking robot grabs: apple fruits, and the other is the object that the robotic arm picks and avoids obstacles: tree branches and trunks. This requires the system not only to accurately identify and locate the position of each fruit, but also to segment the morphological regions of the branches, in order to provide a decision-making basis for the robotic arm to plan a collision-free motion trajectory that can successfully grasp the fruit and effectively avoid branch obstacles.

Currently, the cultivation model utilizing dwarf rootstock and high-density planting [6] has emerged as the predominant approach in modern apple farming. This system offers several benefits, including compact tree size, simplified management, early fruiting, high yield, superior fruit quality, and compatibility with intelligent mechanized operations. These advantages contribute to the advancement of standardized and large-scale orchard management. Widely adopted in leading apple-producing countries, this model reflects the future direction of the modern apple industry. Among the various training systems, the spindle-shaped canopy is especially favored in contemporary standardized orchards due to its abundant fruiting branches and strong yield performance. Typical examples of spindle-shaped apple trees found in modern orchards [7,8,9] are shown in Figure 1.

Figure 1. Spindle-shaped apple trees.

Looking back at the research process of orchard target perception technology, it has gone through two stages: from traditional image processing to modern computer vision based on deep learning. Although traditional image processing methods have intuitive principles and are effective in controlled environments, their essence is to regularize the description of visual features, with weak generalization ability. They are sensitive to changes in lighting, complex backgrounds, and occlusions, making it difficult to adapt to the variability of real orchards.

The rise of deep learning, especially deep convolutional neural networks (DCNNs) [10,11,12,13,14], has brought revolutionary breakthroughs to this field. DCNN can automatically extract hierarchical feature representations from massive data through end-to-end learning, significantly improving the robustness and accuracy of the model. In the DCNN-based object perception framework, there are mainly two types of paradigms: two-stage and single-stage. The two-stage method first generates region suggestions, and then classifies, locates, and segments each suggestion box. This type of method has high accuracy, especially in segmentation tasks, and has been successfully applied to the detection and segmentation of various fruits. However, its multi-stage process results in high computational costs, making it difficult to meet the real-time requirements of harvesting robots. In contrast, single-stage methods represented by the YOLO series [15,16,17,18,19] reconstruct detection and segmentation tasks into a unified regression problem, directly outputting the bounding box, category, and mask of the target in a single forward propagation, which has an advantage in speed. The recently released YOLOv11 network [20,21,22,23,24] has evolved into a multitask platform that supports object detection and semantic segmentation, providing a foundation for building integrated perception systems. Therefore, the core research objective of this article is to construct an efficient, accurate, and integrated intelligent visual perception model that integrates apple and tree branch information. This model can achieve accurate recognition of apple fruits and precise segmentation of tree branches in a unified deep learning framework, providing a decision-making basis for apple picking robots.

The main innovations of the paper: (1) the construction of a dual-task dataset for spindle-shaped apple tree orchards, (2) the architectural innovation of integrating CBAM into YOLOv11 for joint detection and segmentation, and (3) the comprehensive experimental validation in complex orchard environments.

2. Materials and Methods

2.1. Data Collection and Annotation

2.1.1. Image Data Acquisition

The images utilized in this study were captured from actual modern spindle-type apple orchards, focusing on both fruits and branches. A Canon PowerShot G16 camera (Canon, Tokyo, Japan) was employed to photograph the trees during morning, noon, and afternoon sessions, under diverse weather conditions including sunny and overcast skies. The dataset encompasses a range of complex scenarios, featuring instances of occlusion, fruit overlap, and mutual obstruction between fruits and branches. The apple image in a complex orchard scene is shown in Figure 2. For the collected images, there are mainly two types of annotated objects: apple fruits (used for object detection tasks), whose position and category information need to be annotated, using rectangular box annotation, and apple tree branches (used for semantic segmentation tasks), which require precise annotation of their contour shape, using polygon annotation.

Figure 2. Various apple orchard scenes captured under different conditions: (a) fruit partially concealed by foliage, (b) fruit hidden behind limbs, (c) clusters with overlapping apples, (d) lighting from the front, (e) lighting from the side, and (f) backlighting conditions.

2.1.2. Annotation of the Dataset

This project uses the Labelme tool for image annotation. Labelme (version: 5.0.0) is a visual annotation software that supports multiple annotation types (rectangles, polygons, line segments, etc.), suitable for object detection and image segmentation tasks. In this study, Labelme’s rectangular box function was used to label the bounding boxes of apple fruits, and the label was set to “apple”; The annotation of fruit tree branches is done by using polygon function to annotate points along the contour of the branches, and setting the label as branch. After each image is annotated, Labelme will generate a corresponding. json file, which contains the category, coordinate information, and other content of the annotated object. The default annotation file for Labelme is in JSON format and cannot be directly used for training YOLOv11. Therefore, it needs to be converted into the. txt format annotation file required by YOLO model. During the conversion process, based on the different annotation types (rectangular or polygonal), the coordinate information is parsed and normalized separately, and converted into label data that is compatible with the YOLOv11 network training format.

2.2. Preprocessing of Datasets

2.2.1. Enhancement and Amplification of Data

In computer vision tasks, data augmentation and augmentation techniques are key means to improve model generalization ability and robustness, especially in the case of limited annotated data. Data augmentation refers to the dynamic transformation of the original image during model training, such as rotation, flipping, scaling, cropping, color transformation, etc. Data augmentation refers to extracting, combining, and modifying existing samples into new sample data to increase sample size and enhance the model’s generalization ability. This study used methods such as random cropping, rotation, brightness modification, and Gaussian noise addition on 300 original image data to expand the training set samples to over 2000. An example of amplification effect is shown in Figure 3.

Figure 3. Original image (left) and image with Gaussian noise added (right).

The specific amplification method and parameter values for the training set samples are shown in Table 1.

Table 1. Amplification methods and values of apple fruit and branch image samples.

Amplification Method	Value
Rotate	90°~270°
Add Gaussian noise	Variance 0.03
Modify brightness	0.5~1.5
Adjust contrast	0, 1
Random cropping	0.6~1.0

2.2.2. Partition of Dataset

The purpose of dataset partitioning is to split the data into training, validation, and testing subsets, allowing for effective monitoring of the model’s learning progress during training and an unbiased assessment of its generalization ability afterward. In this study, a fixed random seed was applied to shuffle the sample order, after which all data, together with the original images, were allocated into training, validation, and test sets following an 8:1:1 ratio. The detailed distribution of these sets is presented in Table 2.

Table 2. Sample distribution of training, validation and test sets.

Data Set	Number of Samples
Training set	2380
Validation set	297
Test set	297

2.3. YOLOv11 Network Architecture

The architectural design of the YOLOv11 model [25,26,27,28] comprises three integral modules: the backbone, the neck, and the head module. Within this framework, the backbone assumes the role of the principal feature extractor, leveraging convolutional operations to generate hierarchical feature maps from the initial image data. The extracted feature maps are then fed into the neck network; the neck network serves as an intermediate processing layer to integrate and enhance multi-scale features, allowing the model to simultaneously consider spatial details and semantic context; and the head network serves as a prediction mechanism to perform target prediction tasks based on optimized feature maps transmitted from the neck network.

The specific network structure of YOLOv11 is shown in Figure 4. Among them, d is the depth factor (depthwultiple), which controls the number of repeated modules in the network; W is the width factor (width_maultiple), which controls the scaling ratio of the number of convolutional channels in each layer; MC is the upper limit of the number of channels (max_channel), which limits the maximum number of output channels for a single layer; K is the kernel size, which controls the kernel size used in convolution operations; S is the stride, which controls the image reduction factor during convolution; Concat means concatenating feature maps from different layers; Upsample stands for upsampling operation, which upsamples low resolution feature maps to higher resolution; SPPF (Spatial Pyramid Pooling Fast) is an improved spatial pyramid pooling layer; C2PSA stands for “C2 improvement module+PSA attention mechanism”, which is usually used to extract advanced semantic features and enhance the model’s perceptual ability.

Figure 4. YOLOv11 network architecture.

2.3.1. Backbone Layer

The backbone network of YOLOv11 is responsible for extracting multi-scale features from input images and generating feature maps of different resolutions through stacked convolutional layers and dedicated modules. YOLOv11 uses initial convolutional layers for image downsampling, but replaces YOLOv8’s C2f module with C3k2. C3k2 and C2f are both based on the Cross Stage Partial Network (CSP) structure, which divides the input features into two parts within the module. One part is directly passed to the subsequent layer, while the other part is processed by several computing units and fused with the former. C3k2 uses two small kernel convolutions instead of a single large kernel convolution to improve processing speed while maintaining performance. The comparison between C3k2 and C2f is shown in Figure 5.

Figure 5. Structure comparison between C2F module and C3k2 module.

2.3.2. Neck Layer

Within the YOLOv11 framework, the Fast Spatial Pyramid Pooling (SPPF) module is preserved, followed by the integration of a Cross Stage Partial with Spatial Attention (C2PSA) module. The C2PSA module incorporates two Partial Spatial Attention (PSA) modules that operate on distinct branches of the feature map before being concatenated, mirroring the structural design of the C2f module. This configuration enables the model to emphasize spatial features while maintaining an equilibrium between computational load and detection precision. By applying spatial attention to the extracted features, the C2PSA module improves the model’s capacity to concentrate on relevant regions. The architecture of the C2PSA module is illustrated in Figure 6.

Figure 6. Structure comparison between PSA module and C2PSA module.

2.3.3. Head Layer

YOLOv11 adopts a decoupled detection head structure, which separates regression and classification tasks and uses independent sub networks for feature extraction and prediction. This structure helps to reduce interference between tasks and improve their predictive performance; The YOLOv11 network introduces Depthwise Convolution (DWConv) to replace traditional standard convolution operations. Depthwise By factorizing a standard convolution into depthwise and pointwise stages, the model achieves a substantial reduction in both parameter count and computational complexity, all while preserving strong feature extraction performance. The architectural design of the YOLOv11 detection head is illustrated in Figure 7.

Figure 7. Head layer structure of YOLOv11 network.

2.4. Embedded Design of CBAM Module

To enhance the accuracy of the integrated perception model for apple and branch detection and segmentation, this study incorporates the CBAM (Convolutional Block Attention Module) [29,30,31,32]. As a lightweight attention mechanism, CBAM refines feature representations by operating along both channel and spatial dimensions, thereby boosting the feature extraction capability of convolutional neural networks. The module comprises two core components: the channel attention module (CAM) and the spatial attention module (SAM). These components can be flexibly integrated into various layers of a CNN to strengthen its feature expression. In this study, considering that the C3k2 module is mainly used for feature extraction and fusion in the YOLOv11 architecture, embedding the CBAM module before the C3k2 module in the backbone network can help the C3k2 module focus more on meaningful features and improve the efficiency and effectiveness of feature extraction. There are also several studies where CBAM attention mechanism was utilized for agricultural tasks: A progressive learning scheme for tomato plant phenotype analysis was proposed, and was implemented based on the YOLOv11 architecture. The detection performance and processing speed was improved [33]. Another study designed a lightweight and efficient object detection model YOLO-Berry which integrated CBAM attention mechanism, specifically for real-time discrimination of blueberry phenology. The detection results can be converted into growth stage statistical data, which can then drive automated irrigation decision-making systems and support early yield estimation. The study has laid a solid technological foundation for precision irrigation management based on visual perception in modern smart agriculture [34]. The benefits of improving the embedded CBAM attention mechanism module are: Firstly, enhancing feature discrimination ability. CBAM uses a dual attention mechanism to enable the network to dynamically focus on key areas. In apple detection tasks, this means that the perception model pays more attention to the unique colors and circular contours of the fruit; in the task of tree branch segmentation, the perception model focuses more on the texture and direction of linear structures. Secondly, suppress complex background interference. In the natural apple orchard environment, the high proportion of background pixels can easily interfere with the recognition and perception of targets. The attention mechanism of CBAM can effectively suppress irrelevant information, which is particularly important for accurate segmentation of branches.

2.4.1. Channel Attention Module

The channel attention module in CBAM is designed to improve the representational capacity of individual channels. Its implementation begins by applying both global max pooling and global average pooling to each channel of the input feature map, capturing the maximum and average values per channel. These operations yield two vectors with the same length as the number of channels, reflecting the global maximum and average characteristics of each channel. These two pooled feature vectors are then passed through a shared fully connected network, specifically a Multi-Layer Perceptron (MLP) with one hidden layer, to produce the channel attention map. The MLP is composed of two fully connected layers: the first reduces the dimensionality to decrease computational load and introduce non-linearity, while the second restores the original channel dimension, enabling the network to learn which channels are more task-relevant. After being processed separately by the shared MLP, the two output vectors are combined through element-wise summation to form the final channel attention weights. A sigmoid activation function is then applied to constrain these weights to a range between 0 and 1. The resulting attention coefficients are multiplied with the corresponding channels of the input feature map, producing a refined output that highlights informative channels and suppresses less useful ones. This mechanism enhances task-relevant features and improves overall model performance. The architecture of the CBAM channel attention module is illustrated in Figure 8.

Figure 8. Structure of channel attention module of CBAM.

2.4.2. Spatial Attention Module

In contrast to the channel attention mechanism, the spatial attention module primarily identifies which regions of an image hold meaningful information, offering a complementary function to channel attention. The configuration of the spatial attention module is presented in Figure 9.

Figure 9. Structure of spatial attention module in CBAM.

Starting from the input feature map, two separate two-dimensional maps are produced via global max pooling and global average pooling along the width and height dimensions, respectively. These pooled outputs are then concatenated along the channel axis, resulting in a composite feature map of size 2 × H × W. Subsequently, a convolutional layer with a 7 × 7 kernel is applied to this concatenated representation to generate a spatial attention map, which indicates the regions to be emphasized or suppressed, thereby forming the spatial attention weights. In a manner analogous to the channel attention branch, a sigmoid activation function is employed on these weights to scale them into the [0, 1] range. The YOLOv11 architecture, enhanced by the integration of the CBAM module as developed by our institute, is depicted in Figure 10.

Figure 10. Structure of the improved YOLOv11 network.

2.5. Model Training Platform and Training Parameters

In this study, the training of the fusion intelligent perception model was based on the Windows 11 operating system, with a CPU configuration of 12th Gen Intel (R) Core (TM) i5-12500H, a GPU configuration of NVDIA GForce RTX 3050, a deep learning framework of Pytorch 2.0.0+CUDA 11.8, and a Python version of 3.10 used. This study used RTX 3050 GPU for model training, and the entire process of model training was recorded in real time in the running terminal, including changes in loss values and accuracy indicators. The training configuration of the perception model is mainly achieved by setting hyperparameters in the model configuration file. In this study, some key model training parameter configurations are shown in Table 3.

Table 3. Training parameter configuration information of fusion perception model.

Parameter	Value	Function
imgsz	640	Image size (zoom to square). YOLOv11 will uniformly adjust the image to imgsz × imgsz.
patience	10	Set tolerance of early shutdown system.
cos_lr	True	The learning rate is dynamically updated throughout training.
warmup_epochs	5	Control learning rate preheating rounds.
lr0	1 × 10−3	Control initial learning rate.
workers	2	Set the number of threads used to load data.
optimizer	AdamW	Use the adamw optimizer.
amp	True	Training with mixed accuracy.

Among them, the parameters warmup_cpochs, lr0, and cos_1r are all used to control the learning rate and make the training smoother. The warmup_ipochs parameter is used to control the learning rate of the first few epochs to gradually increase from a small value to the set initial learning rate (lr0), avoiding gradient explosion at the beginning of training and ensuring the stability of the model training process. In the training process, the Cosine Learning Rate Scheduler is employed to progressively reduce the learning rate following a cosine decay pattern, ensuring stable convergence and mitigating fluctuations during the later stages of optimization.

2.6. Evaluation Indicators of the Model

In order to comprehensively and objectively evaluate the performance of the fusion intelligent perception model in apple tree fruit object detection and branch segmentation tasks, the main evaluation indicators used in this study include: precision (P), recall (R), F1 value, IoU value, mean average precision (mAP), etc.

Precision (P) represents the proportion of targets predicted as positive samples by the model that are truly positive samples. The higher the accuracy, the fewer erroneous targets predicted by the model. This indicator can be calculated using Equation (1).

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(1)

In this context, True Positive (TP) refers to the number of samples correctly classified as belonging to the positive class, while False Positive (FP) denotes the count of negative samples that are mistakenly predicted as positive.

Recall (R) evaluates the model’s capability to detect all actual positive instances, reflecting the proportion of true positives successfully identified. This metric is computed using Equation (2).

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(2)

False Negative (FN) denotes the count of actual positive instances that the model fails to detect. A higher recall rate indicates that fewer positive targets are overlooked by the model.

During the training process of the perception model, the accuracy (P) and recall (R) will be dynamically monitored, and a balance point will be found between the two through optimization of the loss function. If the model has high accuracy but low recall, it indicates that it is relatively conservative and prone to missed targets; if the recall is high but the accuracy is low, it indicates that the model is prone to false positives. Therefore, the mAP index has also become an important criterion for comprehensive evaluation of model performance, which can reflect the overall situation of accuracy and recall at the same time.

The F1-score, defined as the harmonic mean of precision and recall, serves as a comprehensive metric for assessing the overall effectiveness of a detection model in object recognition tasks. Its mathematical formulation is provided in Equation (3).

$$F1=2\times {\displaystyle \frac{P\times R}{P+R}}$$

(3)

Intersection over Union (IoU) is a key performance metric used to evaluate the alignment between predicted outputs and ground truth labels in object detection and segmentation tasks. It is calculated as the ratio of the overlapping area between the predicted bounding box and the actual bounding box to their combined area, as expressed in Equation (4).

$$IoU={\displaystyle \frac{A_{pred}\cap A_{gt}}{A_{pred}\cup A_{gt}}}$$

(4)

Among them, A_pred represents the predicted area, and A_gt represents the true annotated area. When IoU is greater than a certain threshold, the prediction result is considered correct. A higher IoU threshold requires the model to be more accurate.

In the task of branch segmentation, it is not only necessary to detect the position of the target, but also to predict its accurate edge contour information. Therefore, this study introduced the IoU value index in the segmentation task, which is calculated similarly to the detection task and evaluated based on the degree of overlap between the predicted mask and the real mask. The calculation formula for IoU value is shown in Equation (5).

$$IoU={\displaystyle \frac{TP}{TP+FP+FN}}={\displaystyle \frac{PR}{P+R-PR}}$$

(5)

3. Results and Discussion

3.1. Results and Analysis of Model Training

After the training of the fusion perception model was completed, the loss values of each iteration was recorded in the training log, and a loss curve was drawn, as shown in Figure 11.

Figure 11. Loss curve of the improved YOLOv11 network on training set and verification set.

By observing the changes in various loss curves, the learning dynamics and convergence of the model can be clearly understood. As shown in Figure 11, the loss values of both the training set and the validation set decrease rapidly in the early stages of training, and then gradually stabilize. The loss values decrease to a lower level, indicating that the training process gradually reaches a stable stage without oscillation or overfitting.

The F1 curves of the original YOLOv11 and the improved model are shown in Figure 12 and Figure 13, respectively. From the figure, it can be seen that the models before and after improvement have high recognition and detection capabilities for apple fruit categories. The F1 value curve is smooth and steep, indicating that the model can maintain a high F1 value within a large threshold range, and the performance fluctuation is small, indicating that the perception model has stable and robust recognition performance for apple fruit categories. The visual differentiation between the original and improved YOLOv11 F1-score curves may appear subtle at first glance, but these figures serve an essential quantitative role in demonstrating the stability and robustness of the model’s detection confidence across a continuous range of thresholds; notably, the improved model exhibits an upward shift in the F1 curve specifically for the “branch” category, a modest yet meaningful improvement that validates the enhanced feature discrimination provided by the CBAM module for slender, texturally complex structures.

Figure 12. F1 value curve of original YOLOv11 model.

Figure 13. F1 value curve of the improved YOLOv11 model.

3.2. Analysis of the Perception Effect of Improving YOLOv11 Model

After the training of the fusion perception model, the performance indicators of the original YOLOv11 and the improved YOLOv11 models on the validation set are shown in Table 4.

Table 4. Performance indexes of original and improved YOLOv11 network on verification set.

Model	Perceived Object	P (B)	R (B)	F1 (B)	P (M)	R (M)	mAP (M)	IoU
YOLOv11	apple	0.975	0.983	0.979	--	--	--	--
	branch	0.929	0.832	0.878	0.796	0.713	0.694	0.6017
Improved YOLOv11	apple	0.981	0.986	0.983	--	--	--	--
	branch	0.939	0.841	0.887	0.803	0.715	0.698	0.6066

Among them, B represents the detection box, and M represents the mask. For the apple category, since no mask is generated, only the box index of apple fruit detection task needs to be focused on, where the F1 value index is used to comprehensively evaluate the effectiveness of apple object detection. For tree branch segmentation tasks, due to the simultaneous generation of masks and detection boxes during the segmentation process, it is necessary to comprehensively focus on Mask and Box metrics. Based on the F1 value, mAP, and IoU indicators of the detection box and mask for tree branches, comprehensively evaluate the segmentation performance of the model on tree branches. From Table 4, it can be seen that for the apple object detection task, the improved YOLOv11 model can achieve a p-value of 0.981, an R-value of 0.986, and an F1 value of 0.983. For the tree branch segmentation task, the improved model has an IoU value of 0.6066, mAP value of 0.698, and F1 value of 0.887. Figure 14 and Figure 15 showcase the performance of the enhanced model in apple fruit detection and branch segmentation under varying weather conditions. It is evident that the proposed approach reliably identifies fruit targets and accurately delineates branch regions of apple trees in both sunny and overcast conditions.

Figure 14. Perception results of improved YOLOv11 model on fruits and branches in cloudy environment.

Figure 15. Perception results of improved YOLOv11 model on fruits and branches in sunny environment.

3.3. Comparative Analysis of Segmentation Effects of Different Models

In the evolution of the YOLO series models [35], YOLOv8 is a native support for multitask parallelism such as object detection and instance segmentation officially launched by Ultralytics. With its excellent performance and ease of use, it quickly became a widely recognized benchmark model in both academia and industry. As a subsequent version, YOLOv11 has undergone targeted optimizations in backbone networks, feature fusion, and training strategies, but the overall framework still follows the design philosophy of YOLOv8. Therefore, using YOLOv8 as a benchmark for comparison can more intuitively reflect the effectiveness of the improved model. Furthermore, due to the limited room for performance improvement based on object detection metrics, it is difficult to fully validate the optimization potential of the model. In contrast, segmentation tasks require the model to predict the target contour at the pixel level, which requires higher precision in feature extraction and contextual understanding ability, and can better reflect the differences in model structure. Therefore, this study compares and analyzes the performance of segmentation tasks of three perception models, systematically evaluates the segmentation accuracy of YOLOv8, YOLOv11, and improved models on the validation set, and verifies the improvement effect of the improvement strategy on the fine-grained perception ability of the models. The specific performance indicators for performing segmentation tasks on the validation set based on three network models are shown in Table 5.

Table 5. Comparison of performance indicators for segmentation tasks among three models.

Perception Model	Evaluation Metrics
	Precision	Recall	mAP	IoU
YOLOv8	0.808	0.689	0.679	0.5452
YOLOv11	0.796	0.713	0.694	0.6017
Improved YOLOv11	0.803	0.715	0.698	0.6066

From the data in the table, it can be seen that for the tree branch segmentation task, compared with the YOLOv8 network and the original YOLOv11 network, the improved YOLOv11 network has increased the segmentation recall (R) by 0.026 and 0.002, the segmentation IoU by 0.0614 and 0.0049, and the segmentation mAP by 0.019 and 0.004, respectively. Given that apple tree branches belong to the category of targets with slender structures and complex textures, the improvement of these performance indicators indicates that the improved model has more comprehensive feature learning for tree branch targets with slender structures and complex textures, demonstrating stronger segmentation capabilities.

The comparison of fruit detection and branch segmentation effects using YOLOv8, YOLOv11, and an improved YOLOv11 network model on the test set of apple tree images is shown in Figure 16. As can be seen from the figure, there is not much difference in the perception effect of the three models for detecting apple targets. For the segmentation of tree branches, the improved YOLOv11 model can perceive and segment smaller areas of tree branches, and the segmentation precision is better than that of YOLOv8 and the original YOLOv11 model. Based on the result images of intelligent perception in Figure 16, it can be seen that the proposed improved YOLOv11 model can achieve higher accuracy in branch segmentation, and can segment the branch regions in the image as much as possible. Compared to the improved YOLOv11, both the original YOLOv11 and YOLOv8 have more missing segmentation areas. Indicating that the proposed fusion perception algorithm has superior performance.

Figure 16. Comparison of detection and segmentation effects in cloudy and sunny of three perception models.

In agricultural robotics perception, even marginal improvements in the continuity and precision of slender branch masks translate directly into tangible functional advantages for downstream path planning, as discontinuous or fragmented branch segmentation is a primary cause of false-negative obstacle representations that lead to robotic arm collisions or inefficient trajectories. The proposed integration of the lightweight Convolutional Block Attention Module is deliberately designed not to revolutionize the underlying detection paradigm, but rather to address a specific issue in multi-task agricultural vision: the challenge of simultaneously maintaining high recall for fruit targets while achieving coherent pixel-level segmentation of elongated, texturally subtle branch structures against a cluttered foliar background.

A deep learning perception model for real-time recognition of immature persim-mons in complex field scenarios was proposed based on YOLOv12n model with CBAM attention mechanism integrated [36]. Scale-Dependent Performance Analysis of YOLO26 for PPE Detection was also conducted, indicating that YOLO26 is preferred for accuracy-critical applications at large/X-Large scales with limited training data [37]. An excavator posture recognition method based on an improved Real-Time Detection Transformer (RT-DETR) [38] was proposed, providing an efficient and feasible method for real-time excavator posture perception. The improved YOLOv11 model proposed in our study provides a unique and balanced multitasking framework with segmentation support. Through the strategic integration of lightweight Convolutional Block Attention Modules, efficient apple detection and branch segmentation are achieved. The test results of the model show that it can meet the basic needs of robot visual perception.

The comparison of perception performance among different models is shown in Table 6.

Table 6. Comparison of perception performance among different apple detection models.

Perception Model	Evaluation Metrics			Apple-Branch Fusion Perception	Reference
	Precision (%)	Recall (%)	Model Size (MB)
Rep-Vision-GCN	92.5	85	12.4	No	[39]
Improved YOLOv5s	91.2	94	11.1	No	[35]
Ours	98.1	98.6	6.34	Yes	-
YOLOv8+Mask R-CNN	98.9	92.8	-	No	[40]
DABAMNet	98.13	-	-	No	[41]

As shown in Table 6, our proposed improved YOLOv11 model achieved a precision of 98.1% and a recall of 98.6% for apple detection, outperforming alternative approaches such as Rep-Vision-GCN (92.5% precision, 85% recall) [39], improved YOLOv5s (91.2% precision, 94.0% recall) [35], YOLOv8+Mask R-CNN (98.9% precision but only 92.8% recall) [40], and DABAMNet (98.13% precision) [41], while simultaneously offering the unique advantage of joint apple–branch fusion perception, a capability absent in other compared models. Furthermore, our model is considerably more lightweight, with a size of only 6.34 MB compared to 12.4 MB for Rep-Vision-GCN and 11.1 MB for Improved YOLOv5s, which is a critical factor for real-time deployment on resource-constrained edge devices in the field. It is important to note that these aggregated performance metrics were derived from a comprehensive dataset encompassing diverse illumination conditions (sunny, overcast, and backlighting) and occlusion scenarios (foliage obstruction, fruit clustering, and branch overlap), thereby providing a holistic assessment of model robustness rather than isolated condition-specific benchmarking. The higher precision and recall of our model, combined with its integrated detection–segmentation architecture, underscore its capability to maintain reliable visual perception across the complex orchard environments encountered in practical harvesting operations.

3.4. Analysis of Limitations

(1)

In the study, a RTX3050 graphics card was used to complete the training and verification of perception model, but the actual agricultural operations often need to be deployed in the field, usually using edge computing equipment with limited computing power. Although lightweight is considered in the design of the improved yolov11 model, the simultaneous operation of multitask learning (detection + segmentation) still has high requirements for computing resources. If it is directly transplanted to mobile devices, it may be necessary to reduce the image resolution to ensure the efficient implementation of the model.

(2)

For the branch and trunk segmentation task, occlusion will lead to discontinuous segmentation results, which may affect the subsequent operation decision, so the introduction of multi perspective fusion is the future research direction. The core idea of multi view fusion is to use multiple images taken from different angles to map the two-dimensional segmentation results from each perspective to a unified three-dimensional coordinate system, so as to fill in the missing information of the occluded area, reconstruct the continuous shape of branches in three-dimensional space, and provide a more reliable basis for the picking robot.

(3)

The data of this study are mainly from apple tree images in a specific orchard environment. Although different lighting conditions such as sunny and cloudy days are considered in the acquisition, it does not cover a wider range of scene changes: there are certain differences in branch morphology, bark texture, fruit color and other aspects of apple trees in different regions and varieties. As a result, the generalization performance of the current model on other fruit tree species remains to be validated. To facilitate its broader deployment, it is necessary to gather a wider variety of training data encompassing greater diversity.

3.5. Analysis of Future Work

In the future, multimodal fusion and extreme environment perception strategies will be explored to create all-weather harvesting robots that can integrate information from multiple sensor sources. More diverse multimodal fusion strategies will be explored in the future. One method is to integrate thermal infrared imaging. Mature apples may have slight differences in temperature with their leaves under sunlight, especially in the evening or early morning, and this difference may be more pronounced. The dual stream network combining visible light and thermal infrared images can utilize temperature features for auxiliary recognition and segmentation when color information is lost (such as green apples), improving detection rates under low light and complex lighting conditions. Another approach is to integrate hyperspectral imaging. The spectral reflectance curves of different varieties of apples and branches of different apple trees are unique. Although hyperspectral cameras currently have high costs and complex data processing, their potential in the field of precision agriculture is enormous. In the future, research can be conducted on how to reduce the dimensionality of hyperspectral data and deeply integrate it with RGB visual features. At the same time, adaptive sensor fusion mechanisms can be explored by designing a gating fusion module that allows the network to dynamically adjust the weights of different modal inputs based on current environmental conditions, such as light intensity sensor readings. When the lighting is good, RGB feature information is the main focus; when the lighting environment is poor, more reliance is placed on depth or infrared information. This dynamic and adaptive fusion strategy can maximize the advantages of multiple sensors and ensure that the perception system remains robust in various environments.

Continuing research based on large-scale models and embodied intelligence technology is also an important research direction for the future, which includes the following analysis: A new paradigm of embodied intelligent perception integrating vision language action fusion; tailored decision-making and autonomous planning driven by large models; and tailored interactive learning and continuous optimization based on large models.

4. Conclusions

For the intelligent fusion perception task involving apple tree fruits and branches, a multi-task object detection and segmentation model based on artificial intelligence and deep learning was developed and refined. An improved YOLOv11-based multi-task fusion perception framework was established to enhance the accuracy and robustness of apple target recognition and branch segmentation in orchard environments, mainly completing the following tasks:

(1)

The image data containing apple fruits and tree branches in a real orchard environment were collected and annotated. Through the system’s data augmentation strategy (rotation, noise, brightness adjustment, etc.), the model training dataset was augmented. This dataset includes scenes such as occlusion and lighting changes, providing a high-quality data foundation for model robustness training.

(2)

Building upon the YOLOv11 architecture, lightweight Convolutional Block Attention Modules (CBAMs) were incorporated into the backbone network to formulate a multi-task perception model capable of jointly performing fruit detection and branch segmentation. By embedding this attention mechanism, the model enhances salient information along both channel and spatial dimensions during feature extraction. This improvement strengthens the model’s capacity to capture fine-grained features of slender and complex branch structures, thereby boosting its overall performance in detecting apple fruits and segmenting branches within challenging orchard environments.

(3)

The improved YOLOv11 model has a p-value of 0.981, an R-value of 0.986, and an F1 value of 0.983 for apple target detection, indicating that the model can reliably detect fruit targets. For the segmentation task of tree branches, the improved model has a p-value of 0.803, an R-value of 0.715, an mAP value of 0.698, and an IoU value of 0.6066. The perception performance of YOLOv8, YOLOv11, and the improved model was compared. The experimental results showed that the improved model can segment smaller branch regions with finer segmentation, and has better performance in fruit recognition and branch segmentation.