Vision-Based Perception and Execution Decision-Making for Fruit Picking Robots Using Generative AI Models

Abstract

At present, fruit picking mainly relies on manual operation. Taking the litchi (litchi chinensis Sonn.)-picking robot as an example, visual perception is often affected by illumination variations, low recognition accuracy, complex maturity judgment, and occlusion, which lead to inaccurate fruit localization. This study aims to establish an embodied perception mechanism based on “perception-reasoning-execution” to enhance the visual perception and decision-making capability of the robot in complex orchard environments. First, a Y-LitchiC instance segmentation method is proposed to achieve high-precision segmentation of litchi clusters. Second, a generative artificial intelligence model is introduced to intelligently assess fruit maturity and occlusion, providing auxiliary support for automatic picking. Based on the auxiliary judgments provided by the generative AI model, two types of dynamic harvesting decisions are formulated for subsequent operations. For unoccluded main fruit-bearing branches, a skeleton thinning algorithm is applied within the segmented region to extract the skeleton line, and the midpoint of the skeleton is used to perform the first type of localization and harvesting decision. In contrast, for main fruit-bearing branches occluded by leaves, threshold-based segmentation combined with maximum connected component extraction is employed to obtain the target region, followed by skeleton thinning, thereby completing the second type of dynamic picking decision. Experimental results show that the Y-LitchiC model improves the mean average precision (mAP) by 1.6% compared with the YOLOv11s-seg model, achieving higher accuracy in litchi cluster segmentation and recognition. The generative artificial intelligence model provides higher-level reasoning and decision-making capabilities for automatic picking. Overall, the proposed embodied perception mechanism and dynamic picking strategies effectively enhance the autonomous perception and decision-making of the picking robot in complex orchard environments, providing a reliable theoretical basis and technical support for accurate fruit localization and precision picking.

1. Introduction

Litchi is native to southern China, and its short harvesting period makes timely picking essential for ensuring a bountiful harvest. Litchi clusters are randomly distributed and grow irregularly on fruit trees, which poses significant challenges for efficient picking [1]. Currently, the picking process relies predominantly on manual labor, which is not only time-consuming and labor-intensive but also leads to substantial increases in labor costs [2,3]. In recent years, researchers have achieved high-precision fruit recognition and localization in complex environments through deep learning methods. However, most existing studies focus on the detection of litchi or individual fruits [4], while fruit maturity assessment still largely relies on manual experience, image processing, and deep learning-based methods [5]. To address the occlusion of fruit by branches and leaves in orchards, researchers have attempted to combine image processing and deep learning approaches to mitigate the impact of occlusion [6,7]. To realize intelligent picking, many studies have focused on developing vision-based picking robots. Embodied perception robots, integrating artificial intelligence, mechanical automation, visual perception, image processing, and the cross-disciplinary technologies of agricultural machinery and agronomy, are designed to automate the fruit-picking process. This not only alleviates the physical labor intensity for workers but also addresses issues related to safety and cost.

Embodied Intelligence (EI) [8] refers to the ability of an intelligent agent [9] to perceive, make decisions, and take actions through real-time interaction with the environment via a physical body, thereby better adapting to complex scenarios. In fruit-picking operations, intelligent perception primarily relies on information obtained from sensors. The decision-making process assesses whether the target meets the picking criteria based on visual perception results, while the execution phase is carried out by a robotic arm that implements the picking decision. The embodiment perception mechanism enables robots to perceive their own status and environmental information in real time, providing accurate feedback for posture adjustment and picking decisions, thereby significantly enhancing their adaptability and operational efficiency in complex orchard environments. Liang et al. designed a visual servo control strategy that integrates spatial asynchronous visual positioning and embodiment perception, constructing a visual servo system for tomato picking robots [10]. This system effectively reduces positioning errors and harvesting failure rates in complex environments. Zhou et al. proposed a road extraction and roadside fruit recognition algorithm suitable for unstructured orchards, achieving synchronous extraction of road and fruit information, which provides references for robot path planning and picking decision-making [11]. Chen et al. focused on bananas as the research object. By combining deep semantic segmentation with a designed adaptive stereo matching strategy, they achieved automatic response of the stereo matching process to different target depths, thereby enhancing the robustness of the algorithm. Meanwhile, the fusion of multi-view information from multi-camera vision and hand-eye vision, along with high-precision point cloud stitching methods, addressed occlusion issues and improved the robustness of local scene target perception [12]. Tang et al. improved the visual detection algorithm for camellia fruits under varying lighting conditions. The proposed method maintained high localization stability and robustness even when illumination changed [13]. Meanwhile, Tang et al. explored the application of visual and quantitative methods in localization, object recognition, 3D reconstruction, and fault tolerance within complex agricultural environments [14]. Sun et al. combined deep learning with active perception techniques, leveraging fruit localization information and occlusion rates to guide the robotic manipulator in dynamically adjusting the camera pose, thereby achieving coordinated active perception and picking [15,16]. Embodied agricultural robots based on active perception and decision-making mechanisms can enable more efficient and accurate localization of target fruits [17], providing strong support for intelligent harvesting in complex environments.

Generative Artificial Intelligence (Generative AI) [18] is an advanced form of artificial intelligence that employs intricate algorithms and learning techniques inspired by the human brain. It is capable of generating new content by identifying patterns in existing data. The primary goal of Generative AI [18] is to produce text, code, images, and audio, utilizing technologies such as Generative Adversarial Networks (GANs) and large language models. As generative AI continues to advance, its research paradigm has gradually shifted from unimodal to multimodal approaches. Multimodal large-scale models [19,20] integrate heterogeneous sources of information, such as images and text, to achieve cross-modal feature alignment and semantic understanding, thereby providing novel technical approaches for research directions, including pest and disease detection, task planning, and decision-making. Zhu et al. [21] developed an artificial intelligence and large language model-based diagnostic and control platform for potato disease detection by integrating text and image-based multimodal artificial intelligence. This holds significant importance for intelligent disease detection. Qing et al. [22] have achieved agricultural diagnostics by combining the visual understanding capabilities of the YOLO (You Only Look Once) network with the deep logical reasoning abilities of the large model GPT-4. Zhou et al. [23] developed a crop disease recognition and semantic interpretation method based on multimodal deep learning, and established a knowledge graph for specific crop diseases to improve the intelligence level of crop disease recognition. Zhang et al. [24] proposed the IPM-AgriGPT model, which integrates expertise in pest and disease management to offer intelligent and practical solutions for agricultural pests. Huang et al. [25] constructed a multimodal agricultural dataset and proposed a novel entity-level cross-modal fusion architecture, providing robust technical support for agricultural knowledge graph construction and intelligent question-answering systems. Wang et al. [26] introduced a multimodal GPT-4V framework that enhances embodied task planning by integrating natural language instructions with robotic visual perception. In light of recent advances in multimodal large-scale models, this study introduces generative AI into the field of fruit-picking robots to provide auxiliary support for intelligent harvesting decision-making.

Most existing studies focus on embodied intelligence, large models, visual perception, and decision-making. However, research that integrates embodied perception with generative AI for fruit-picking decision-making remains relatively scarce. To address this gap, this study establishes a “perception-reasoning-execution” embodied perception framework to enhance the overall sensing and decision-making ability of picking robots in complex orchard environments. Within this framework, a generative artificial intelligence model is introduced to perform high-level semantic reasoning on visual perception results, assisting in the assessment of litchi fruit maturity and occlusion conditions. Based on the inference results provided by the generative AI model, decision-making support is offered for the selection of harvesting strategies, thereby enhancing the adaptability of the system in complex orchard environments and further exploring the application potential of embodied perception combined with generative AI in agricultural robotics.

Compared with fruit localization approaches that rely solely on YOLO and other vision-based models, the proposed method in this study demonstrates certain advantages at the harvesting decision-making level. On the one hand, by introducing a generative artificial intelligence model to perform semantic reasoning on fruit maturity and occlusion conditions, the dependence on multi-category manual annotations, such as maturity levels and occlusion types, is reduced, thereby lowering the cost and time required for model training. On the other hand, the proposed approach enables higher-level reasoning and decision-making based on visual detection results, compensating for the limitations of conventional methods in complex harvesting judgment scenarios and, to some extent, improving adaptability to complex orchard environments. The main contributions of this study are summarized as follows:

(1)

A perception-reasoning-execution-based embodied perception mechanism is proposed.

(2)

A Y-LitchiC instance segmentation method is proposed to achieve high-precision segmentation of litchi clusters.

(3)

A Transformer-based multimodal generative artificial intelligence model is introduced to perform high-level semantic reasoning on visual information, assisting in the assessment of fruit maturity and occlusion conditions.

2. Materials and Methods

2.1. Dataset Acquisition Equipment

The data collection was conducted in Zengcheng District and Conghua District of Guangzhou. The data was acquired using multiple smartphones with varying resolutions (2560 × 1440, 2240 × 1280, 4032 × 3024, 1600 × 1200 and 1280 × 960 pixels) along with a ZED 2 stereo camera (2208 × 1242 pixels; Stereolabs Inc., San Francisco, CA, USA). To reflect real-world orchard growing conditions, a multi-scenario litchi image dataset was constructed in this study. Data acquisition covered different times of the day, including morning, noon, and afternoon, and encompassed various natural illumination conditions, such as front-lit and back-lit scenes. The images were collected from June to July, resulting in a total of 1075 litchi images. The litchi varieties collected were Guiwei, Nuomici, and Jingfu Hongnuo. The picking robot equipment is shown in Figure 1.

Figure 1. Picking robot.

Since the main fruit fruit-bearing branches occupy relatively few pixels in the images and have colors similar to non-pickable fruit stems, misjudgment can easily occur. Therefore, the main fruit-bearing branch, fruit-bearing sub-branches, and litchi fruits are treated as a unified target and collectively annotated under the label “litchi cluster”. In the orchard environment, some main fruit-bearing branches are partially occluded by leaves. Considering that leaf occlusion does not affect the picking decision, we still annotated the fruits of these occluded clusters. However, regions occluded by branches that could interfere with the picking process were excluded from annotation. Labeling was performed using the Labelme (v5.4.1) software. The litchi cluster segmentation dataset contains 770 training images, 193 validation images, and 112 testing images, while the main fruit-bearing branch dataset includes 941 training images, 236 validation images, and 187 testing images. The visual perception and execution decision-making process of the fruit-picking robot based on the generative AI model is illustrated in Figure 2.

Figure 2. Flow chart. The flowchart consists of four parts: (1) Data acquisition and labeling. (2) Visual perception system: an improved lightweight backbone network, a multi-scale feature fusion module, and a loss function module are employed for litchi cluster detection and localization in complex orchard environments. (3) Inference system: a Transformer-based multimodal generative AI model is employed to assist in decision-making for fruit maturity and occlusion assessment. (4) Decision system: guided by the visual perception and inference systems, the robot executes precise picking operations.

2.2. Generative AI

Generative AI [27] is an unsupervised or semi-supervised machine learning framework that perturbs the input distribution to refine sample features, explore intrinsic relationships, and generate target samples. Generative AI models can integrate visual and textual information to produce meaningful outputs. Representative architectures include Generative Adversarial Networks (GANs), Diffusion Models, Variational Autoencoders (VAEs), and large-scale Transformer-based models. Most neural sequence transduction models adopt an encoder-decoder architecture. The encoder [28] maps an input sequence of symbols (x₁, …, x_n) into a continuous representation z = (z₁, …, z_n). Given z, the decoder generates the output sequence (y₁, …, y_m) step by step. During generation, the model exhibits an autoregressive property-each new symbol is produced based on both the encoded representation and the previously generated symbols. This study introduces a multimodal generative AI model based on a Transformer architecture, which jointly leverages visual inputs and textual prompts to perform high-level semantic reasoning on fruit maturity and occlusion. The inferred semantic information is further integrated with object detection outputs to provide decision support for intelligent harvesting. The schematic diagram of the generative AI-based decision-making principle is shown in Figure 3.

Figure 3. Schematic diagram of the generative AI-based decision-making framework.

2.3. Principles of Image Processing

For the original litchi image, the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm is first applied to the luminance channel in the LAB color space to equalize lighting and reduce the influence of environmental illumination variations on the fruit. The image is then converted to the HSV color space, where the saturation is enhanced to increase the color contrast between the fruit and the leaves, making the target region more distinguishable. To obtain the main fruit-bearing branch region, appropriate threshold ranges are set in the HSV space to remove the fruit and leaves. The largest connected component extraction method is subsequently applied to eliminate small scattered areas, retaining only the main fruit-bearing branches and primary stems. The morphological opening operation is employed to eliminate isolated white regions that do not belong to the target area, while the morphological closing operation fills small holes within the binary image, resulting in a more accurate contour fitting [29]. Based on this, the Zhang-Suen thinning algorithm is applied to extract the skeleton of the main fruit-bearing branch, followed by the removal of excessively short branches to preserve the complete skeletal structure of the main fruit-bearing branch. The Zhang-Suen thinning algorithm [30] establishes criteria for identifying contour points in an image. Based on these criteria, it iteratively marks and removes contour points until no pixels meet the identification conditions. The remaining pixels after the iterations constitute the skeleton of the original image.

3. Picking Robot Perception System

YOLOv11 is an updated algorithm built upon the YOLO series. The model architecture consists of three main components: the Backbone, Neck, and Head. YOLOv11 offers five model scales (n, s, m, l, and x), with each scale featuring increased network depth and width. It supports multiple vision tasks, including object detection and tracking, instance segmentation, image classification, and keypoint detection. In the Backbone, YOLOv11 introduces the C3k2 and C2PSA mechanisms to enhance feature extraction performance under lightweight design constraints. In the Neck, it employs a multi-scale feature fusion strategy that integrates features bidirectionally across dual layers, effectively expanding the receptive field and improving the feature fusion capability of the model. [31]. The YOLOv11-seg instance segmentation model shares the same Backbone and Neck as the original YOLOv11 architecture. However, YOLOv11 faces certain limitations when applied to orchard environments with varying illumination. To enhance the segmentation accuracy of litchi cluster perception, this study adopts YOLOv11-seg as the baseline model for improvement. Based on the YOLOv11-seg instance segmentation framework, the original C2PSA module at the end of the Backbone is replaced with a CBAM module, the Neck is redesigned using a RepBiFPN structure, and the original CIoU loss function is substituted with an MPDIoU loss function. The improved architecture of the YOLOv11 model is illustrated in Figure 4.

Figure 4. Y-LitchiC network architecture. * indicates whether this parameter enables the residual connection.

3.1. Introducing the CBAM Attention Mechanism Module into the Backbone

Sanghyun Woo proposed the Convolutional Block Attention Module (CBAM), which is composed of two key components arranged in series: a channel attention module and a spatial attention module. The channel attention mechanism models the interrelationships among feature map channels, evaluating the importance of each channel to emphasize the most informative semantic features [32]. The spatial attention mechanism focuses on the spatial locations within feature maps, identifying the regions that have the greatest influence on the output results and assigning appropriate weights to these critical areas [33]. CBAM sequentially infers attention maps along two independent dimensions—channel and spatial—and performs adaptive feature refinement by multiplying these attention maps with the input features [34]. The structure of the CBAM module is illustrated in Figure 5.

Figure 5. CBAM module.

In scenarios with complex backgrounds, leaf occlusion, and similar visual characteristics between main fruit-bearing branches and non-harvestable fruit stalks, accurately identifying the main fruit-bearing branches becomes challenging. Therefore, the main fruit-bearing branches and litchis were annotated together as a single litchi cluster. The CBAM module focuses on detailed features within specific regions, enabling the model to capture finer local information and enhancing the representation of local features within the litchi clusters, thereby achieving more accurate recognition and localization. In this study, the C2PSA module in the backbone was replaced with the CBAM module, which expands the receptive field of the deep network and enables it to capture richer contextual information. This enhancement improves both the accuracy and robustness of litchi cluster segmentation.

3.2. Neck

In the YOLOv11-seg model, the Neck is primarily responsible for fusing multi-scale features, integrating low-level detailed information with high-level semantic information to enhance the performance of both object detection and instance segmentation. In the Neck, the RepBi-PAN structure is employed to aggregate multi-scale features derived from re-parameterized blocks. The BiFusion module enables the effective fusion of feature maps across different scales, allowing low-level and high-level features to be combined more efficiently, thereby enhancing feature representation [35]. The RepBiFPN structure further improves the accuracy of litchi cluster localization in complex environments. As shown in Figure 6, Figure 6a illustrates the overall architecture of the RepBiFPN, while Figure 6b presents the BiFusion, SimConv, RepBlock, and RepConv modules.

Figure 6. RepBiFPN module [36].

To develop a hardware-friendly network with faster detection speed and higher accuracy, RepBlock and SimConv are employed to enhance computational efficiency and memory bandwidth [36]. The SimConv module is composed of a convolutional layer, a batch normalization layer, and a ReLU activation function. The main difference between SimConv and the standard Conv layer lies in the activation function: SimConv employs ReLU, while Conv uses SiLU. The ReLU activation function is computationally simpler and faster, which contributes to more efficient training and inference of the neural network [36]. The ReoConv module consists of two branches with different kernel sizes: a 3 × 3 SimConv and a 1 × 1 SimConv. This design improves hardware computational efficiency [36]. The RepBlock is composed of multiple RepConv modules, which enhance feature extraction capabilities and are well-suited for lightweight models. By appropriately adjusting the depth and width of the network, the performance and efficiency of the model are further optimized. Consequently, the RepBiFPN structure enables higher detection accuracy and faster detection speed for litchi clusters in complex environments.

3.3. MPDIoU (Minimum Points Distance Intersection over Union) Loss Function Module

In orchard environments with complex lighting conditions and background interference, fruit segmentation faces significant challenges due to factors such as leaf occlusion. Although the original CIOU loss function [37] in YOLOv11-seg considers the overlap area, center point distance, and aspect ratio between the predicted and ground truth boxes, it may fail to effectively optimize the predicted box when the aspect ratios are similar, but the width and height differ substantially [38]. Therefore, the MPDIoU loss function is introduced to address this issue, thereby improving the recognition and localization accuracy of litchi clusters occluded by leaves. The schematic diagram of the MPDIoU loss function is shown in Figure 7.

Figure 7. Schematic diagram of MPDIoU loss function.

MPDIoU is a boundary box similarity metric based on the minimum point distance. It minimizes the distance between the top-left and bottom-right corners of the predicted box and the ground-truth box [39]. Here, A denotes the ground-truth box, and B represents the predicted box. The variables w and h indicate the image width and height, while d₁ and d₂ correspond to the distances between the top-left and bottom-right corners of the predicted and ground-truth boxes, respectively. $(x_1^{gt},y_1^{gt})$ and $(x_1^{prd},y_1^{prd})$ refer to the top-left coordinates of the ground-truth and predicted boxes, whereas $(x_2^{gt},y_2^{gt})$ and $(x_2^{pr\mathrm{d}},y_2^{prd})$ correspond to their bottom-right coordinates. The MPDIoU loss function [40] is formulated as follows:

$$L_{MPDI\mathrm{o}U}=1-MPDIoU$$

(1)

$$MPDIoU={\displaystyle \frac{A\cap B}{A\cup B}}-{\displaystyle \frac{{d_1}^2}{w^2+h^2}}-{\displaystyle \frac{{d_2}^2}{w^2+h^2}}$$

(2)

$$d_1^2={(x_1^{pr\mathrm{d}}-x_1^{gt})}^2+{(y_1^{prd}-y_1^{gt})}^2$$

(3)

$$d_2^2={(x_2^{pr\mathrm{d}}-x_2^{gt})}^2+{(y_2^{prd}-y_2^{gt})}^2$$

(4)

4. Experimental Results and Analysis

4.1. Evaluation Indicators

The model was trained on an Ubuntu 20.04 operating system with an x86_64 processor and an NVIDIA GeForce RTX 3080Ti GPU. The development environment included PyCharm 2024.3 and Visual Studio 2022, with Python 3.8 and C++14 as the programming languages. During model training, the number of epochs was set to 250, with a learning rate of 0.001 and a batch size of 8. In this study, Precision (P), Recall (R), and mean Average Precision (mAP) are utilized as evaluation metrics. The corresponding formulas [41,42] are provided in Equations (5)–(8).

$$\mathrm{P}={\displaystyle \frac{TP}{TP+FP}}$$

(5)

$$\mathrm{R}={\displaystyle \frac{TP}{TP+FN}}$$

(6)

$${\mathrm{AP}}_i={\displaystyle {\int }_0^{1}P(R)\mathrm{d}}(R)$$

(7)

$$\mathrm{mAP}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^{N}A{P}_i}$$

(8)

In these definitions, TP (True Positives) represents the number of positive samples correctly identified, FP (False Positives) refers to the number of negative samples incorrectly classified as positive, and FN (False Negatives) denotes the number of positive samples that were missed or incorrectly classified as negative.

4.2. Ablation Experiment on Litchi Clusters

Using the YOLOv11s-seg instance segmentation model as the baseline, ablation experiments were conducted with different combinations of backbone, neck, and loss functions. The results for the various models are presented in Table 1.

Table 1. Results of the ablation experiment.

Model	Backbone				Neck			Loss Function			Box			Mask			Model Size /M	Parameter /M	GFLOPs
	CBAM	EMA	S A	GAM	Gather- and- Distribute	EffQAFPN	Repbifpn	SIoU	DIOU	MPDIoU	P (%)	R (%)	mAP @0.5 (%)	P (%)	R (%)	mAP @0.5 (%)
√											78	83.7	86.9	79	84.7	87.5	20.5	10.08	35.6
√	√										81.4	82.7	88	81.7	83	88.7	18.7	9.16	34.9
√		√									80.3	82.7	86.9	80.6	83	87.4	19.8	9.75	36
√			√								82.5	79.4	86.8	83.5	80.4	88	18.5	9.09	34.8
√				√							79	81.4	87.1	80.6	82.7	88.6	25.5	12.6	37.6
√	√				√						79.5	81.4	85.4	80.8	82.6	86.8	34.9	17.2	52.7
√	√					√					76.1	85.4	86.6	76.1	85.4	86.6	25.2	12.39	42.4
√	√						√				84.3	78.7	88.2	83.8	80.8	89	24.5	12	39.8
√	√						√	√			79.6	80.6	86	79.6	82.7	87.3	24.5	12	39.8
√	√						√		√		82.8	78.7	87.2	82.8	78.7	87.5	24.5	12	39.8
√	√						√			√	82.2	83.1	89.2	82.2	83.1	89.1	24.5	12	39.8

To compare the effects of replacing the final C2PSA layer in the YOLOv11s-seg backbone with different attention mechanisms, including CBAM, EMA, SA, and GAM, Grad-CAM visualization was performed. The resulting heatmaps are shown in Figure 8. The heatmaps indicate that YOLOv11s-seg is easily disturbed in complex backgrounds, leading to false positives and missed detections of litchi clusters. Compared with EMA, SA, and GAM, the CBAM attention mechanism more effectively enhances the focus of the model on litchi clusters, highlighting critical semantic features and regions of interest, thereby improving detection performance. As shown in Table 1, replacing the last C2PSA layer in the YOLOv11s-seg backbone with CBAM improves both box and mask precision (P) and mAP compared to the original YOLOv11s-seg model. Specifically, box P and mAP increased by 3.4% and 1.1%, while mask P and mAP increased by 2.7% and 1.2%, with reductions in model size, parameter count, and computational cost. Although attention mechanisms such as EMA and SA reduced the parameter count and computational load compared to the original model, their mAP improvements were smaller than those achieved with CBAM. In contrast, mechanisms like GAM increased both parameters and computation, but the improvement in mask mAP was still less than that achieved by adding CBAM.

Figure 8. Attention mechanism heatmap.

Building on the replacement of the last C2PSA layer in the YOLOv11s-seg backbone with the CBAM attention mechanism, we further investigated the Neck network. Three Neck variants-Gather-and-Distribute, EffQAFPN, and RepBiFPN-were tested. Replacing the Neck with RepBiFPN led to a 0.3% increase in mask mAP compared to YOLOv11s-seg with CBAM, while the Gather-and-Distribute and EffQAFPN Neck variants did not improve mask mAP. Based on replacing the C2PSA layer in the backbone with the CBAM attention mechanism and adopting RepBiFPN for the Neck, we further improved the loss function. The enhanced MPDIoU loss function increased box mAP by 1% and mask mAP by 0.1% compared to other improved loss functions, thereby enhancing the segmentation and localization performance of litchi clusters. In contrast, the other two improved loss functions, SIoU and DIoU, did not yield any mAP improvement.

4.3. Comparison of Different Algorithms

To evaluate the improved Y-LitchiC model, we compared it with four other instance segmentation networks: YOLOv8s-seg, YOLOv11s-seg, Mask R-CNN, and YOLACT. The experimental results for the different models are presented in Table 2.

Table 2. Experimental results of different models.

Model	Parameter Quantity/M	Calculation Quantity/ GFLOPs	mAP@0.5 (%)
YOLOv8s-seg	11.80	42.70	85.90
YOLOv11s-seg	10.08	35.60	87.50
Mask R-CNN	44.34	239.32	78.37
YOLACT	50.14	82.74	77.59
Y-LitchiC	12.00	39.80	89.10

The proposed Y-LitchiC model achieves improvements in mask-level mAP of 3.2, 1.6, 10.73, and 11.51 percentage points over YOLOv8s-seg, YOLOv11s-seg, Mask R-CNN, and YOLACT, respectively. This indicates that the Y-LitchiC model demonstrates a significant performance advantage over these four models in the task of litchi cluster segmentation. YOLOv8s-seg is a relatively lightweight model with an mAP of 85.9%, featuring low parameter quantity and calculation quantity. YOLOv11s-seg reduces both parameter quantity and calculation quantity compared to YOLOv8s-seg, achieving a higher mAP of 87.5%. Mask R-CNN and YOLACT both have a comparatively large parameter quantity and calculation quantity, with mAPs of 78.37% and 77.59%, respectively. This demonstrates that Y-LitchiC attains optimal segmentation performance while maintaining low computational cost. In the task of litchi cluster instance segmentation, the Y-LitchiC model not only outperforms the other comparison models in mAP but also offers greater efficiency in terms of parameter quantity and calculation quantity, making it well-suited for practical applications.

By comparing the five algorithms-YOLOv8s-seg, YOLOv11s-seg, Mask R-CNN, YOLACT, and Y-LitchiC-for litchi cluster prediction, the results are shown in Figure 9. In the segmentation results, Y-LitchiC is able to accurately distinguish individual litchi clusters compared to the other four models, with clear and continuous boundaries and superior detail-capturing ability, significantly reducing missed detections. Notably, Y-LitchiC can precisely segment small or partially occluded litchi clusters. The Y-LitchiC model achieves an average inference speed of approximately 55 FPS on the test set.

Figure 9. Comparison test results of different models.

4.4. Dynamic Picking Decision-Making with Embodied Perception

Building on the proposed visual perception model, this study develops a visualization system integrated with a generative large model. By incorporating the KIMI model, the system provides auxiliary assessment of fruit images, including fruit maturity and occlusion types. The system can automatically determine whether a fruit is approaching maturity and further distinguish whether any occlusion is caused by leaves or branches. For mature fruits partially obscured by leaves, the system employs image processing algorithms to handle the occluded regions and extract the main fruit-bearing branch structure, providing a basis for subsequent picking decisions. In contrast, fruits occluded by branches are temporarily deferred from picking to prevent potential damage or incorrect picking. The generative AI inference interface is shown in Figure 10. The generative artificial intelligence reasoning system provides auxiliary judgments on fruit maturity and pickability based on visual perception results. It analyzes the image content to identify the fruit type, main nutritional components, maturity level, and degree of occlusion, and then offers a comprehensive recommendation on whether the fruit is suitable for picking.

Figure 10. Generative AI inference interface.

Generative AI was employed to assist in the assessment of litchi maturity and occlusion, and a total of 30 trials were conducted. The judgments provided by the generative AI were compared with the corresponding ground-truth observations, and the results are summarized in Table 3. Experimental results demonstrate that, on the constructed test samples, the proposed generative AI model achieves an accuracy of 93.3% in litchi maturity classification and attains 100% judgment consistency in occlusion assessment. In addition, the system can further infer the type of occlusion, such as branches or leaves. The generative AI reasoning system provides higher-level reasoning and decision-making capabilities for automated fruit picking, effectively addressing the limitations of traditional agricultural decision-making in fruit harvesting. The visual perception and generative AI inference framework adopted in this study can be deployed on existing general-purpose computing platforms without requiring additional dedicated hardware, which helps reduce the overall system cost.

Table 3. Evaluation results of the generative AI in determining litchi maturity and occlusion conditions.

Category	Subcategory	Number of Trials	Correct Judgments
Maturity determination	Mature	20	20
	Immature	10	8
Occlusion determination	Non-occluded	13	13
	Partially occluded	17	17

For unoccluded main fruit-bearing branches, litchi clusters are first localized, and a secondary localization strategy is then applied within the extracted litchi cluster recognition region to identify the corresponding main fruit-bearing branch. The localized main fruit-bearing branch is further refined using instance segmentation, followed by skeleton extraction. The midpoint of the skeleton is selected as the reference point to accomplish the first type of localization and harvesting decision. The YOLOv11s-seg network achieves a mask-level mAP of 54.5% in main fruit-bearing branch localization. As the main fruit-bearing branch occupies a relatively small proportion of pixels in the image, further optimization methods will be explored to improve localization accuracy.

For main fruit-bearing branches that are heavily occluded by leaves, the Y-LitchiC algorithm is employed to localize the litchi cluster. Within this region, illumination equalization, color enhancement, and threshold segmentation are sequentially applied to remove interference from fruits and leaves. Subsequently, the main fruit-bearing branch is extracted using maximum connected component analysis, and a thinning algorithm is utilized to obtain the skeleton of the main fruit-bearing branch. This process enables the second type of dynamic picking decision. The dynamic picking strategy is illustrated in Figure 11.

Figure 11. Schematic illustration of the dynamic picking strategy.

5. Conclusions

(1) The proposed Y-LitchiC model demonstrates superior mask-level mAP performance compared with existing methods, achieving improvements of 3.2, 1.6, 10.73, and 11.51 percentage points over YOLOv8s-seg, YOLOv11s-seg, Mask R-CNN, and YOLACT, respectively.

The proposed Y-LitchiC system is designed with full consideration of practical orchard deployment requirements, demonstrating strong engineering applicability. The model maintains stable detection performance under typical orchard environmental conditions, including varying illumination, partial occlusion, and complex backgrounds. In addition, the proposed approach is compatible with existing agricultural robotic platforms, providing a solid foundation for future real-world deployment and applications.

(2) This study introduces a multimodal generative AI model based on a Transformer architecture, which jointly leverages visual perception information and textual prompts to perform high-level semantic reasoning on fruit maturity and occlusion conditions, thereby providing auxiliary support for harvesting decision-making. Experimental results demonstrate that, on the constructed test samples, the proposed generative AI model achieves an accuracy of 93.3% in litchi maturity classification and attains 100% judgment consistency in occlusion assessment.

(3) For main fruit-bearing branches heavily occluded by leaves, the Y-LitchiC algorithm is used to obtain the detection boxes of litchi clusters. The target regions are then processed through illumination equalization and color enhancement, followed by threshold segmentation to remove fruits and leaves. Subsequently, methods such as maximum connected component extraction are employed to isolate the main fruit-bearing branches, and a thinning algorithm is applied to extract their skeleton lines. Finally, the center point of the skeleton line is determined as the pick point.

This study primarily focuses on the recognition of litchi clusters, and future work will further expand to the localization and reconstruction of individual fruits.