GPC-Frame: Bridging Scene Perception and Harvesting Cognition for Active Disocclusion and Harvesting-Point Localization in Trellised Table-Grape Vineyards

Abstract

In trellised table-grape vineyards, frequent occlusion and complex scene structure pose major challenges to harvesting robots in both target perception and executable decision-making. To address this, an integrated perception–cognition framework, GPC-Frame, was proposed. At the perception layer, a Transformer-based network with an InternImage backbone and DyHead was used for fine-grained multi-target segmentation of grape clusters, cluster stems, canes, petioles, and leaves. At the cognition layer, a Graph Convolutional Network (GCN) was used for structural and occlusion reasoning, categorizing targets into non-occluded, harvestable after disocclusion, and non-harvestable states. Skeleton extraction and node-degree analysis were then performed on cluster stem and petiole masks to enable coordinated localization of the disocclusion point (D-point) and harvesting point (H-point). The experimental results showed an mIoU of 83.64% for scene segmentation, with structural and occlusion reasoning accuracies of 96.36% and 95.90%, respectively. In field tests, harvesting success reached 89.19% for non-occluded targets, while D-point localization and subsequent harvesting success for occluded targets requiring disocclusion reached 86.52% and 85.11%, respectively. GPC-Frame also improved non-occluded H-point localization by 4.08–14.97 percentage points over state-of-the-art methods. These results indicate that GPC-Frame provides an executable perception–cognition framework for robotic table-grape harvesting in trellised vineyards.

1. Introduction

1.1. Background

Table grapes are an important horticultural crop, and trellis cultivation is one of the common production systems used for their cultivation. Compared with wine grapes and other types of grapes, table grapes require higher bunch integrity, visual quality, and market value during harvesting. Because table grapes usually mature within a relatively concentrated period and harvesting is labor-intensive, harvesting operations still rely heavily on manual labor, especially in trellised vineyards. In recent years, with increasing labor costs, shortages of seasonal workers, and the continued development of intelligent agricultural equipment, autonomous robotic harvesting has become an important direction for improving the production efficiency and sustainability of table grapes [1].

However, reliable robotic harvesting under real vineyard conditions remains difficult to achieve [2]. In typical row-oriented trellised table-grape systems, grape clusters are usually distributed within dense fruiting zones and are surrounded by various organs or structures, including cluster stems, leaves, petioles, canes, and neighboring grape clusters. These organs and structures spatially overlap with each other and frequently occlude the target grape clusters and their cluster stems [3]. In addition, outdoor illumination changes can lead to unstable target visibility and incomplete structural cues, resulting in a highly unstructured trellised vineyard working environment [4]. Therefore, grape-harvesting robots first need reliable scene perception of harvest-related targets, especially pixel-level segmentation of grape clusters, cluster stems, leaves, petioles, and canes [5].

For robotic harvesting, however, segmentation alone is not sufficient. The robot must further determine whether the segmented organs form a valid harvesting structure and whether the target grape cluster is directly harvestable, requires occlusion removal, or should be avoided [6]. In this study, the harvesting point, referred to as the H-point, denotes the executable point on the cluster stem for cutting and harvesting. The disocclusion point, referred to as the D-point, denotes the interaction point used to remove or separate a removable occluding organ before harvesting. Therefore, robotic harvesting in trellised vineyards requires not only multi-target scene perception but also harvestability reasoning, occlusion-state interpretation, and coordinated D/H-point localization.

Based on this consideration, this study regards robotic table-grape harvesting as a perception–cognition process that connects scene perception, structural and occlusion reasoning, and action-point localization. In this process, scene perception identifies harvest-related organs, structural and occlusion reasoning determines the target state and operation strategy, and D/H-point localization converts the reasoning result into executable harvesting or disocclusion actions. This perception–cognition chain provides the basis for robotic harvesting of occluded trellised table grapes.

1.2. Related Work

Accurate visual perception is a prerequisite for robotic grape harvesting. Early studies mainly focused on grape clusters in vineyard scenes and usually used color-thresholding methods to roughly locate grape regions [7]. In recent years, related studies have increasingly adopted deep learning models to identify and segment harvest-related targets, such as grape clusters and cluster stems [8,9]. To improve model adaptability in orchard harvesting scenes, researchers have further integrated multi-scale features [10,11], attention mechanisms [12,13], and CNN–Transformer or Transformer-based structures to enhance perception under occlusion and background interference [14,15,16]. For example, Peng et al. [17] used an improved YOLOv8s-seg model with DDFAM and EADHead to obtain segmentation results for grape clusters and cluster stems. Shen et al. [18] introduced an attention mechanism and an upsampling structure into the Mask R-CNN framework to improve grape instance segmentation. Jiang et al. [19] developed YOLOv8n-GP based on YOLOv8n-pose and introduced the SENetV2 module and CARAFE upsampling operator to identify grape clusters and cluster stems. Wu et al. [20] proposed a grape cluster stem segmentation model that integrates multiple attention mechanisms and multi-scale features to improve the segmentation of small peduncle targets. Wang et al. [21] achieved the segmentation of grape bunches, cluster stems, and occluded regions using a CNN–Transformer fusion structure. In addition, grape cluster stems, also referred to as peduncles in some related studies, are small and slender structures, and their segmentation is easily affected by low pixel proportion, weak boundaries, and annotation uncertainty, which may lead to incomplete masks or unstable boundary localization [20,21].

However, most existing visual perception methods mainly output target locations or segmentation masks [22]. These results cannot directly support the harvesting cognition required by grape-harvesting robots, namely determining whether a target is harvestable, where the robot should operate, and whether occlusion should be handled before harvesting. Therefore, visual perception results still need to be further transformed into harvestability states, action-point localization, and occlusion-handling strategies.

Harvestability judgment usually depends on the correspondence between the target grape cluster and key structures such as the cluster stem or peduncle. Existing studies have mostly inferred harvesting relationships based on manually designed geometric rules, such as bounding-box expansion and region of interest (ROI) constraints derived from segmentation masks. Zhou et al. [23] established the correspondence between the target grape cluster and the peduncle by upwardly expanding the ROI of the grape detection box. Lin et al. [24] established the correspondence between the target grape cluster and the peduncle using ROI intersection constraints based on grape segmentation results. Similarly, Hussain et al. [25] used geometric constraints to pair young apples with fruit stems in robotic apple thinning. Overall, these studies provide a basis for harvestability judgment by establishing the correspondence between the target fruit and its peduncle or stem. After the harvesting relationship is determined, the robot must further locate the picking point [26,27]. Existing grape picking-point localization methods mainly include keypoint estimation and post-segmentation geometric inference [28,29]. Keypoint estimation methods, such as those proposed by Chen et al. [30] and Jiang et al. [19], used YOLO-pose-based frameworks to predict picking points on cluster stems while detecting grape clusters. Post-segmentation geometric inference methods, such as those proposed by Lu et al. [31] and Wang et al. [32], were respectively based on PGSS-YOLOv11s segmentation results and Transformer semantic segmentation results, and then combined morphological processing, skeletonization, endpoint detection, ROI extraction, or contour rules to locate grape picking points.

Occlusion handling is another important issue in robotic grape harvesting. Existing studies mainly include active visual adjustment and active physical disocclusion. For active visual adjustment, Yi et al. [33] improved target visibility under occlusion through viewpoint planning, whereas Luo et al. [34] combined grape instance segmentation, three-dimensional voxel representation, and deep reinforcement learning to improve occluded peduncle detection. For active physical disocclusion, Xiong et al. [35,36] proposed a Push-and-Drag method for strawberry harvesting, in which obstacles around the target strawberry were removed through pushing and dragging operations. He et al. [37] used an end-effector equipped with a fan to blow away or lift occluding leaves before harvesting. Overall, occlusion-handling studies have progressed from improving observation viewpoints to actively removing occluding objects, providing useful strategies for robotic harvesting under occlusion.

In summary, existing studies have made progress in grape visual perception, harvesting-relation inference, picking-point localization, and occlusion handling. However, most studies have focused on a single stage and have not formed a complete chain from robotic visual perception to harvesting cognition. In terms of visual perception, most existing methods adopt end-to-end task architectures and mainly output target locations, segmentation masks, or keypoints for a limited number of target categories, such as grape clusters and cluster stems. These methods insufficiently consider the interlaced relationships among grape clusters, cluster stems, petioles, leaves, and canes, and they rarely further transform these visual outputs into harvestability states, action-point localization, and occlusion-handling strategies. For harvesting-relation inference and picking-point localization, existing methods still rely heavily on manually designed geometric rules and ROI constraints, making it difficult to stably determine the harvesting relationship between the target grape cluster and surrounding organs under target interlacing and occlusion interference. Meanwhile, existing occlusion-handling methods mainly focus on viewpoint adjustment or occluder removal and rarely consider occlusion type, removability judgment, disocclusion-point localization, and picking-point localization in a unified manner. For trellised table-grape harvesting, both the robot’s viewpoint adjustment and manipulator workspace are constrained. In addition, table-grape fruit is delicate and cluster stems are slender; therefore, disocclusion based on pushing, dragging, or airflow can easily cause fruit abrasion, cluster-stem pulling, and cane disturbance, making these methods difficult to directly transfer to this scene. Therefore, it is necessary to construct a decoupled perception–cognition framework for trellised table-grape harvesting that connects multi-target scene perception, harvesting-relation inference, occlusion-state judgment, and collaborative H-point/D-point localization.

1.3. Research Hypothesis and Objectives

Based on the above analysis, this study proposes the following research hypothesis: in trellised table-grape harvesting scenes, visual outputs such as target detection, segmentation masks, or keypoint localization alone are insufficient to support reliable robotic harvestability judgment, occlusion-handling decisions, and action-point localization. By further integrating organ-level multi-target perception with structural and occlusion reasoning, scene information can be more effectively transformed into target-state classification and coordinated D/H-point localization for robotic harvesting.

To verify this hypothesis, this study proposes GPC-Frame, a perception–cognition framework for robotic table-grape harvesting in trellised vineyards. The objectives of this study are as follows:

(1) To construct a multi-target perception layer for grape-harvesting scenes and achieve the segmentation of grape clusters, cluster stems, leaves, petioles, and canes;

(2) To develop a cognition layer based on multi-target perception results, which models the harvesting relationship and occlusion state between the target grape cluster and surrounding organs, and supports the classification of targets into directly harvestable, harvestable after disocclusion, and non-harvestable states;

(3) To establish a coordinated D/H-point localization strategy and validate the effectiveness of GPC-Frame through image-dataset-based evaluation, comparison with representative grape-harvesting localization methods, and robotic field-harvesting experiments in trellised table-grape vineyards.

1.4. Main Contributions

The main contributions of this study are as follows:

(1) A perception–cognition framework named GPC-Frame is proposed for trellised table-grape harvesting scenes. This framework connects multi-target visual perception, harvesting-relation inference, occlusion-state judgment, and H-point/D-point localization to form a structured processing flow for robotic harvesting decisions.

(2) A harvesting cognition method based on multi-target perception results is established. This method jointly considers grape cluster, cluster stem, leaf, petiole, and cane instances, explicitly infers the relationship between the target grape cluster and surrounding organs, and determines whether the target grape cluster can be directly harvested, requires disocclusion before harvesting, or should be avoided.

(3) A collaborative H-point/D-point localization strategy is constructed and validated through image-dataset-based perception–cognition experiments and robotic field-harvesting experiments in trellised table-grape vineyards. The results demonstrate the feasibility of GPC-Frame in supporting harvesting decisions in occluded trellised table-grape harvesting scenes.

2. Materials and Methods

2.1. Experimental Sites and Vineyard Characteristics

The experiments were conducted at two greenhouse trellised table-grape production bases in Northeast China: the Yushu table-grape production base in Harbin, Heilongjiang Province, and the Lvyuan table-grape production base in Changchun, Jilin Province. Two representative table-grape cultivars, ‘Lanfeng’ and ‘Xiahei’, were used in this study.

Both bases adopted row-oriented trellis cultivation. The grapevines were planted in furrow-based rows, with an approximate furrow depth of 25 cm. The vine spacing was approximately 0.75 m, the trellis height was approximately 2.2 m, and the planting density was approximately 337.5 vines per mu (approximately 5062.5 vines ha⁻¹). In these scenes, target grape clusters were distributed within the fruiting zone and were spatially interlaced with cluster stems, leaves, petioles, canes, and neighboring fruits. These scenes included both directly visible, non-occluded target grape clusters and target grape clusters affected by leaf/petiole occlusion, neighboring-fruit overlap, or cane interference, representing the main visual conditions encountered during greenhouse trellised table-grape harvesting.

Data were collected during two harvest seasons, from July to October in both 2023 and 2025. To cover the main operating conditions of robotic harvesting in trellised table-grape scenes, images were acquired under sunny daytime, cloudy daytime, and nighttime conditions with artificial supplementary lighting. In total, 4000 greenhouse trellised table-grape scene images were collected for dataset construction and experimental evaluation. Details of the robotic platform, image acquisition procedure, annotation protocol, and dataset split are provided in Section 2.2 and Section 2.3.

2.2. Robotic Platform and Image Acquisition

Image acquisition was performed using a grape-harvesting robotic platform to approximate the perception process of the robot before harvesting. As shown in Figure 1, the platform consisted of a Qicheng 4S tracked mobile chassis (Liubu Workshop, Beijing, China), a self-developed robotic manipulator, electric pruning shears, and a flexible gripper, fixed 6500 K cool-white LED light sources (Foshan Lighting, Foshan, China), a Hasee Z8 onboard control computer (Hasee, Shenzhen, China), a PMC NEC 12 ADDA 485 motion control card (Bopai, Dongguan, China), and two RGB-D vision sensors, including Kinect V2 (Microsoft Corporation, Redmond, WA, USA) and Intel RealSense L515 (Intel Corporation, Santa Clara, CA, USA). During image acquisition, an RGB-D vision sensor was mounted at the end of the robotic arm to acquire RGB-D images of the target grape clusters and their surrounding structures, including color images and corresponding depth maps. The acquired RGB-D images were standardized to a unified resolution of 680 $\times$ 480 pixels. Specifically, images acquired by Kinect V2 were cropped to this resolution, while images acquired by Intel RealSense L515 were processed using the same image size.

During data collection, the robotic arm moved the RGB-D vision sensor around the target grape cluster. Taking the center of the target grape cluster region as the reference, multi-view images were acquired from frontal to side-view directions along an approximately spherical trajectory with a radius of about 600 mm. This acquisition strategy was used to simulate the visual observation process of the robot before harvesting and to provide image inputs for subsequent multi-target perception, occlusion reasoning, and H-point/D-point localization.

2.3. Dataset Construction and Annotation

Based on the image acquisition procedure described above, 4000 greenhouse trellised table-grape RGB-D scene images with a unified resolution of 680 $\times$ 480 pixels were obtained. Images acquired using Kinect V2 and Intel RealSense L515 were pooled into a unified dataset after resolution standardization and annotation. The images were randomly divided into training and test sets at a ratio of 8:2, with 3200 images used for training and 800 images used for testing. Data augmentation, including illumination variation and random rotation, was applied only to the training set. After augmentation, the number of training images increased to 6400, while the test set remained unchanged.

Polygon-based instance annotation was performed using Labelme, and the annotations were converted into COCO-format files for model training and evaluation. The annotated categories included grape clusters, cluster stems, petioles, leaves, canes, and background. Each visible foreground instance was independently delineated and assigned a category label and an instance ID. Each target grape-cluster instance was further assigned an occlusion-state label, including non-occluded, leaf/petiole-occluded, overlap-occluded, and cane-occluded. An example of the instance and occlusion-state annotations is shown in Figure 2.

These annotations served as the basis for multi-target scene perception, harvesting-relation reasoning, occlusion-state inference, and H-point/D-point localization evaluation in GPC-Frame.

2.4. Overview of GPC-Frame

To support robotic harvesting in trellised table-grape scenes, this study proposes GPC-Frame, a visual perception–harvesting cognition framework. As shown in Figure 3, GPC-Frame consists of a perception layer and a cognition layer, which convert multi-target visual perception information in grape-harvesting scenes into executable robotic decisions for harvesting or disocclusion.

The perception layer performs multi-target segmentation on RGB-D images and outputs pixel-level masks, semantic categories, and instance information for harvesting-related targets, including target grape clusters, cluster stems, leaves, petioles, and canes. Taking the perception-layer outputs as input, the cognition layer further infers the harvesting relationships and occlusion states between target grape clusters and surrounding organs, and accordingly determines whether the robot should perform direct harvesting, disocclusion before harvesting, or operation avoidance.

Based on this perception–cognition process, GPC-Frame performs coordinated H-point and D-point localization, linking multi-target scene perception, harvesting-relation inference, occlusion-state reasoning, and action-point localization into a structured harvesting decision workflow.

2.5. Perception Layer: Multi-Target Scene Segmentation

Reliable scene perception is a prerequisite for subsequent structural reasoning and action-point localization in robotic grape harvesting. In the perception layer of GPC-Frame, the robot is required to segment multiple harvesting-relevant targets, including grape clusters, cluster stems, petioles, leaves, and canes, from complex trellis vineyard scenes. This task is challenging because of marked scale differences among targets, frequent leaf occlusion, and strong local visual similarity between cluster stems or petioles and surrounding leaves or canes. To address these difficulties, a multi-target segmentation network was developed within the perception layer using a Backbone–Fusion Head–Decoder architecture, as shown in Figure 4. Specifically, given an input RGB-D image $I\in R^{480\times 680\times 3}$, the InternImage backbone extracts five hierarchical feature maps {$F_1$, $F_2$, $F_3$, $F_4$, $F_5$}, which are fused by DyHead into $F_{fuse}$. The Transformer decoder then uses learnable queries $Q$ to interact with $F_{fuse}$ and generates semantic categories and instance masks through parallel class and mask heads. The resulting fine-grained segmentation outputs provide the direct input for the subsequent cognition layer.

2.5.1. Backbone for Irregular Morphological Feature Extraction

Traditional convolutional networks, constrained by fixed regular receptive fields, often struggle to capture the highly irregular and slender morphological features of cluster stems and canes. To address this, an InternImage backbone was employed, leveraging its deformable convolution (DCNv3) mechanism to adaptively adjust the receptive field [38]. Given an input image $I\in R^{480\times 680\times 3}$, the multi-level feature maps extracted by the backbone can be expressed as shown in Equation (1):

$$F_l={\mathrm{D}\mathrm{C}\mathrm{N}}_{\mathrm{V}3}\left(F_{l-1};P_l,M_l\right)$$

(1)

where $F_{l-1}$ and $F_l$ denote the input and output feature maps of the $l$-th feature-extraction layer, respectively; $P_l$ denotes the learnable spatial offsets; $M_l$ denotes the modulation weights.

The InternImage backbone consists of five hierarchical stages that progressively extract multi-scale features from the input image. For an input image with height $H$ and width $W$, the outputs of the five stages are denoted as $F_1$, $F_2$, $F_3$, $F_4$, and $F_5$, with feature dimensions of $H/4\times W/4\times 64$, $H/8\times W/8\times 128$, $H/16\times W/16\times 256$, $H/32\times W/32\times 512$, and $H/32\times W/32\times 512$, respectively. These feature maps form the multi-level backbone representation $F=\{F_1$, $F_2$, $F_3$, $F_4$, $F_5\}$, which is subsequently fed into the DyHead fusion module. Each stage consists of $L_i$ InternImage basicblocks. In each basicblock, DCNv3 performs adaptive spatial sampling through learnable offsets and modulation weights, LN stabilizes the intermediate feature distributions, and FFN enhances channel-wise nonlinear representation after spatial feature aggregation.

This enables the backbone network to flexibly adapt to irregular morphological features during feature extraction, thereby extracting highly accurate multi-level spatial semantics of these slender structures.

2.5.2. Dynamic Feature Fusion for Significant Scale Disparities

In trellised table-grape harvesting scenes, there are significant scale disparities among various targets: a background cane may span a large part of the field of view, while a tiny petiole or fine cluster stem may occupy only a few pixels. To address this, a Dynamic Head (DyHead) module was integrated into the feature fusion stage [39]. In DyHead, the fusion of multi-level features is dynamically adjusted through level-aware, spatial-aware, and task-aware attention, as summarized in Equation (2):

$$F_{fuse}=A_T\left(A_S\left(A_L\left(F\right)\right)\right)$$

(2)

where $F$ denotes the multi-level feature representation from the backbone; $A_L$, $A_S$, and $A_T$ denote the level-aware, spatial-aware, and task-aware attention operations, respectively; and $F_{fuse}$ denotes the fused feature representation.

Specifically, $F$ represents the five-stage backbone features $\{F_1$, $F_2$, $F_3$, $F_4$, $F_5\}$. These feature maps provide complementary representations at different spatial resolutions and receptive-field ranges. Lower-level features retain fine boundary information for small and slender organs, such as cluster stems and petioles, whereas higher-level features encode larger grape clusters, leaves, canes, and global scene context. In this module, level-aware attention adaptively weights different feature levels to balance targets with large size variations; spatial-aware attention enhances informative locations within each feature map, especially partially occluded or visually ambiguous regions; and task-aware attention recalibrates the fused representation for downstream category prediction and mask generation. The resulting $F_{fuse}$ is then used as the feature input to the Transformer-based query decoder.

2.5.3. Transformer-Based Decoder for Multi-Target Segmentation

To resolve the high local textural similarity among various targets in trellis environments and achieve robust multi-target segmentation, a Transformer-based query decoder was employed as the segmentation head [40]. Given the fused feature representation $F_{fuse}$ and a set of learnable queries $Q$, the decoder generates mask predictions and category-probability vectors for candidate targets, as shown in Equation (3):

$${\left\{\left(M_q,P_q\right)\right\}}_{q=1}^{N_q}=D_t\left(F_{fuse},Q\right)$$

(3)

where $Q$ denotes the set of learnable object queries;$N_q$ denotes the number of object queries; $M_q$ denotes the predicted mask corresponding to the $q$-th query; $P_q$ denotes the corresponding category-probability vector; and $D_t$ denotes the Transformer-based decoder.

In the Transformer decoder, the learnable object queries $Q$ represent candidate target instances in the grape-scene image. Through cross-attention, $Q$ attends to the fused multi-scale feature $F_{fuse}$, allowing each query to aggregate image features relevant to a potential object instance. Self-attention further models contextual relationships among different queries, thereby reducing duplicate responses and improving instance-level discrimination. The resulting query embeddings are then fed into two parallel prediction heads. The class head maps each query embedding to a category-probability vector $P_q$, while the mask head projects the query embedding onto the mask feature space to generate the corresponding pixel-level instance mask $M_q$. During inference, the semantic category of each valid query is determined from $P_q$, and its associated mask is obtained from $M_q$, producing the final multi-target segmentation results.

By integrating self-attention and cross-attention for global semantic modeling, the decoder helps alleviate the limitation of relying solely on local visual features. Instead, it incorporates the global context of the trellised scene, enabling the network to distinguish visually similar targets, including grape clusters, cluster stems, canes, petioles, and leaves, according to their semantic and structural roles. The resulting fine-grained segmentation outputs serve as the direct input for the subsequent cognition layer.

2.6. Cognition Layer: Reasoning and Localization

The fine-grained segmentation masks provided by the perception layer contain semantic and instance-level target information, but they lack the explicit structural and spatial logic required for robotic operation. To address this, a GCN was leveraged to explicitly model the biological topological relationships among grape clusters, cluster stems, petioles, leaves, and canes [41]. This approach compensates for the unstructured nature of raw segmentation masks, enabling the inference of occlusion states and structural hierarchies required for robotic harvesting. The cognition layer receives the instance masks and semantic categories from the perception layer and outputs the occlusion state, execution category, and D/H-point localization result for each target. This layer comprises two modules: (1) grape cluster structure and occlusion reasoning, and (2) D/H-point decision-making and localization.

2.6.1. Grape Cluster Structure and Occlusion Reasoning

This module was designed to convert isolated segmentation masks into a connected structural topology. A GCN was employed to perform joint reasoning on the structural hierarchy and occlusion states. The process involves node representation, biologically constrained graph construction, and GCN-based reasoning, as shown in Figure 5.

Node representation: Each valid instance mask was abstracted as a graph node. Initial node features were constructed by concatenating the semantic category, visual appearance vector, mask centroid, and main-axis orientation of each organ instance. These organ-level nodes and biologically constrained edges formed the structural graph for subsequent GCN-based reasoning.

Structural graph construction: Biological and geometric priors were imposed to build the adjacency matrix $A$, thereby preventing physically impossible associations, such as a grape cluster directly growing from a leaf. A candidate edge $A_{ij}$ between node $i$ and $j$ was established only when spatial and biological constraints were simultaneously satisfied, as shown in Equation (4):

$$A_{ij}=\left\{\begin{array}{c}1, d_{ij}\le {\tau }_d,{ \theta }_{ij}\le {\tau }_{\theta }, and \left(c_i,c_j\right)\in S_{bio} \\ 0, otherwise \end{array}\right.$$

(4)

where $d_{ij}$ and ${\theta }_{ij}$ are the Euclidean distance and the relative angle between the main axes of nodes $i$ and $j$, respectively. $c_i \mathrm{a}\mathrm{n}\mathrm{d} c_j$ are the semantic categories of the two nodes; ${\tau }_{\theta }$ and ${\tau }_d$ are the angular and distance thresholds, respectively; and $S_{bio}$ denotes the set of unordered biologically permissible adjacency pairs, defined as $\left\{\left(\mathrm{G}\mathrm{r}\mathrm{a}\mathrm{p}\mathrm{e},\mathrm{S}\mathrm{t}\mathrm{e}\mathrm{m}\right),\left(\mathrm{S}\mathrm{t}\mathrm{e}\mathrm{m},\mathrm{C}\mathrm{a}\mathrm{n}\mathrm{e}\right),\left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{f},\mathrm{P}\mathrm{e}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{l}\mathrm{e}\right),(\mathrm{P}\mathrm{e}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{l}\mathrm{e},\mathrm{C}\mathrm{a}\mathrm{n}\mathrm{e})\right\}$. Here, Grape and Stem denote grape-cluster and cluster-stem nodes, respectively. Any connection attempting to bridge nodes outside these biologically permissible pairs was strictly prohibited.

For the geometric thresholds, ${\tau }_d$ was empirically set to 36 pixels at an average RGB-D working distance of 500–600 mm, corresponding to a maximum tolerable physical disconnection gap of approximately 15–20 mm caused by occlusion. The angular threshold ${\tau }_{\theta }$ was set to 45°, reflecting the natural growth flexibility and bending limits of cluster stems under fruit weight. Finally, edge weights were dynamically learned to reflect the spatial relationships and mask intersections between adjacent nodes.

GCN-based reasoning: Using this initialized structural graph, a three-layer GCN integrating ChebConv operators and residual connections was constructed. By performing message passing across this biologically constrained topology, the network inferred the implicit structural attributes and occlusion states of each grape cluster, effectively mapping disconnected 2D masks into a coordinated occlusion hierarchy.

2.6.2. D/H-Point Decision-Making and Localization

To extract precise coordinates for robotic execution, skeleton extraction and topological node-degree analysis were performed on the segmented masks of cluster stems and petioles. Based on the GCN-derived occlusion states, a “single-view-priority, side-view-complementary” strategy was executed.

If the target was classified as leaf/petiole-occluded or overlap-occluded and no valid H-point could be localized in the initial view, the manipulator executed a lateral translation for side-view observation. To guarantee sufficient stereoscopic parallax without losing the target, the translation distance $T_x$ was determined based on the camera horizontal field of view (FOV) and the average local canopy depth $D_{avg}$, as shown in Equation (5):

$$T_x=2\cdot D_{avg}\cdot \tan \left({\displaystyle \frac{\mathrm{F}\mathrm{O}\mathrm{V}}{2}}\right)\cdot \gamma$$

(5)

where $\gamma$ is the visual-overlap safety coefficient. Based on the empirically adopted lateral translation distance of $T_x=600 \mathrm{m}\mathrm{m}$, $D_{avg}=600 \mathrm{m}\mathrm{m}$, and a horizontal camera FOV of approximately 70°, $\gamma$ was estimated as 0.70 in this study. This setting was used to preserve partial overlap between the initial and side-view images while providing sufficient lateral parallax for disocclusion observation.

To rigorously determine the target coordinates, the cluster-stem and petiole masks were skeletonized into a spatial graph ${\mathrm{G}}_{\mathrm{s}\mathrm{k}\mathrm{l}}=({\mathrm{V}}_{\mathrm{s}\mathrm{k}\mathrm{l}},{\mathrm{E}}_{\mathrm{s}\mathrm{k}\mathrm{l}})$. Node degrees were calculated within a localized spatial search radius $r_{search}$, corresponding to the empirical 7 × 7 pixel window, to identify endpoints and junction nodes, as shown in Equation (6):

$$\left\{{\displaystyle \genfrac{}{}{0pt}{}{V_{end}=\left\{v\in V_{skl}Deg\left(v\right)=1\right\}}{V_{juc}=\left\{v\in V_{skl}Deg\left(v\right)\ge 3\right\}}}\right.$$

(6)

where $V_{end}$ and $V_{juc}$ denote the endpoint set and junction-node set of the skeleton graph, respectively.

By tracing the continuous path between valid endpoints $V_{end}$ and junction nodes $V_{juc}$, the main structural axis $V_{main}$ was isolated. The execution coordinate $P$ was then defined as the geometric centroid of $V_{main}$, as shown in Equation (7):

$$P={\displaystyle \frac{1}{\left|V_{main}\right|}}\cdot \sum _{v\in V_{main}}{p}_v$$

(7)

where $V_{main}$ is the isolated main structural axis, $\left|V_{main}\right|$ is the number of skeleton nodes on this axis, and $p_v$ is the coordinate of node $v$. In the final robotic field experiments, the Intel RealSense L515 was used for RGB-D sensing and 3D localization. After the image-space D- or H-point was obtained, its corresponding depth value was extracted from the aligned RGB-D data. The 2D image coordinate and depth value were then projected into the camera coordinate system using the intrinsic parameters of the L515 and further transformed into the robot base coordinate system through hand–eye calibration, thereby providing an executable 3D point for the manipulator.

According to the reasoning hierarchy, harvesting targets were topologically classified into three execution categories, as shown in Figure 6:

(i) Non-occluded grape clusters (harvestable): The H-point was calculated from the main axis $V_{main}$ of the cluster stem using Equation (7).

(ii) Occlusion by leaves/petioles or overlapping clusters (active disocclusion): For leaf/petiole occlusions, the skeletonization and centroid calculation described in Equations (6) and (7) were applied to the petiole mask to define the D-point. For overlapping grape clusters, the H-point of the foreground occluding cluster was computed and used as the initial D-point for safe removal.

(iii) Occlusion by canes (non-harvestable): Canes function as rigid structural supports. Targets occluded by canes were strictly classified as non-removable obstacles to prevent mechanical damage to the trellis system.

An iterative visual-servoing scheme was employed. Following each interaction, including harvesting or disocclusion, the scene topology was dynamically updated until the target was successfully harvested or confirmed as non-harvestable.

2.7. Framework Loss Function and Two-Stage Optimization Strategy

Staged training strategies have been proven effective in decoupling complex multi-task learning processes, as they prevent mutual interference between tasks [42]. The perception and cognition layers of GPC-Frame had distinct optimization objectives. The perception layer aimed to obtain accurate pixel-wise segmentation masks, whereas the cognition layer aimed to infer structural attributes and occlusion states from the organ-level graph. Consequently, a two-stage optimization strategy was adopted to address their respective task requirements. In Stage 1, the perception layer was optimized for multi-target segmentation. In Stage 2, the trained perception layer was frozen, and the cognition layer was optimized for graph-based structural and occlusion reasoning. This strategy ensured that the perception layer converged on segmentation accuracy before the cognition layer optimized structural logic, thereby reducing gradient conflicts between heterogeneous tasks.

2.7.1. Perception Layer Optimization

The perception layer aims to provide accurate pixel-wise masks for five foreground categories: grape clusters, cluster stems, canes, leaves, and petioles. Given the marked size disparity between large targets such as grape clusters and fine structures such as cluster stems and petioles, relying solely on standard cross-entropy loss may cause the network to under-represent slender targets because of their small pixel proportions. To address this, the network was trained using a combination of weighted cross-entropy loss $L_{CE}$ and Dice-coefficient loss $L_{DC}$ [43]. Specifically, $L_{CE}$ was used to supervise pixel-wise semantic classification, whereas $L_{DC}$ was used to improve the structural integrity of small and slender target masks. The two loss terms are defined in Equations (8) and (9):

$$L_{CE}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{c=1}^C}{{\alpha }_{c}E}_{i,c}\log \left(f_{i,c}\right)$$

(8)

$$L_{DC}=1-{\displaystyle \frac{2{\sum }_{i=1}^n{\alpha }_i{e}_i{E}_i+\epsilon }{{\sum }_{i=1}^n{{\alpha }_{i}e}_i+{\sum }_{i=1}^n{\alpha }_i{E}_i+\epsilon }}$$

(9)

where $N$ is the number of pixels used for multi-class classification; $C$ is the number of categories; $f_{i,c}$ is the predicted probability of pixel $i$ belonging to category $c$; $E_{i,c}$ is the corresponding one-hot ground-truth label; $e_i$ is the predicted value of pixel $i$; $n$ is the number of pixels used for Dice-loss calculation; $E_i$ is the corresponding ground-truth value; ${\alpha }_c \mathrm{a}\mathrm{n}\mathrm{d} {\alpha }_i$ are weighting coefficients; and $\epsilon$ is a smoothing term used to avoid division by zero.

The Stage 1 segmentation loss was defined as the weighted sum of these two terms, as shown in Equation (10):

$$L_{seg}=L_{CE}+L_{DC}$$

(10)

where $L_{seg}$ denotes the perception-layer segmentation loss.

2.7.2. Cognition Layer Optimization

After the perception layer converged, its weights were frozen. During Stage 2, the GCN cognition layer was trained on the organ-level structural graphs described above using grape-cluster-level structural-attribute and occlusion-state labels. The structural-attribute label indicated whether the target grape cluster had a valid association with its corresponding cluster stem for H-point localization. The optimization of the cognition layer consisted of two objectives: graph-embedding optimization and supervised readout optimization.

A graph embedding loss $L_f$ was employed to embed the biologically constrained graph topology into a latent feature space [44]. This loss constrained the feature-space relationships between adjacent nodes by reducing the discrepancy between learned and reference feature distances. The graph embedding loss is defined in Equation (11):

$$L_f=\sum _{(i,j)\in G}{\left[d\left({\widehat{h}}_i,{\widehat{h}}_j\right)-d\left(h_i,h_j\right)\right]}^2$$

(11)

where $G$ denotes the set of valid adjacent node pairs in the constructed graph; $i$ and $j$ are adjacent nodes; ${\widehat{h}}_i \mathrm{a}\mathrm{n}\mathrm{d} {\widehat{h}}_j$ are the node embeddings updated by the GCN; $h_i \mathrm{a}\mathrm{n}\mathrm{d} h_j$ are the reference node embeddings derived from the initial node features before GCN message passing; and $d\left({\widehat{h}}_i,{\widehat{h}}_j\right) \mathrm{a}\mathrm{n}\mathrm{d} d(h_i,h_j)$ denote the learned and reference feature distances between adjacent nodes, respectively.

For each grape-cluster node, the final node embedding $z_i$ obtained after GCN message passing was decoded by a fully connected readout layer to obtain the probability distributions of structural attributes and occlusion states, as shown in Equation (12):

$$\left\{{\displaystyle \genfrac{}{}{0pt}{}{{\mathrm{r}}_i^{\mathrm{s}\mathrm{t}\mathrm{r}}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{W}}_{\mathrm{s}\mathrm{t}\mathrm{r}}{z}_i+{\mathrm{b}}_{\mathrm{s}\mathrm{t}\mathrm{r}}\right)}{{\mathrm{r}}_i^{\mathrm{o}\mathrm{c}\mathrm{c}}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{W}}_{\mathrm{o}\mathrm{c}\mathrm{c}}{z}_i+{\mathrm{b}}_{\mathrm{o}\mathrm{c}\mathrm{c}}\right)}}\right.$$

(12)

where $z_i$ denotes the final embedding of grape-cluster node $i$ after GCN message passing; ${\mathrm{r}}_i^{\mathrm{s}\mathrm{t}\mathrm{r}}$ and ${\mathrm{r}}_i^{\mathrm{o}\mathrm{c}\mathrm{c}}$ are the readout probability vectors for structural attributes and occlusion states, respectively; and ${\mathrm{W}}_{\mathrm{s}\mathrm{t}\mathrm{r}}$, ${\mathrm{W}}_{\mathrm{o}\mathrm{c}\mathrm{c}}$, ${\mathrm{b}}_{\mathrm{s}\mathrm{t}\mathrm{r}},$ and ${\mathrm{b}}_{\mathrm{o}\mathrm{c}\mathrm{c}}$ are the parameters of the fully connected readout layer. The final structural attribute and occlusion state were determined by the classes with the highest readout probabilities.

To supervise the structural-attribute and occlusion-state predictions, a classification loss $L_{cls}$ was calculated, as shown in Equation (13):

$$L_{cls}=CE\left(r^{str},y^{str}\right)+CE\left(r^{occ},y^{occ}\right)$$

(13)

where $CE$ denotes the cross-entropy loss; $r^{str}$ and $r^{occ}$ denote the predicted probability vectors for structural attributes and occlusion states, respectively; and $y^{str}$ and $y^{occ}$ denote the corresponding ground-truth labels.

The Stage 2 cognition-layer loss was then defined as shown in Equation (14):

$$L_{cog}=L_f+L_{cls}$$

(14)

where $L_{cog}$ denotes the cognition-layer optimization objective.

2.8. Training Parameters and Evaluation Metrics

2.8.1. Implementation and Training Parameters

The GPC-Frame framework was implemented on an Ubuntu 20.04 operating system using Python 3.8 and the PyTorch 2.0 deep learning library with CUDA 11.8 acceleration. The hardware platform consisted of an NVIDIA GeForce RTX 4070 Super GPU with 12 GB memory. To reduce interference between the segmentation and reasoning tasks, a two-stage supervised training strategy was adopted.

In Stage 1, the perception layer was trained independently for 500 epochs using a multi-step decay learning-rate strategy. The learning rate and weight decay were both set to 1 × 10⁻⁴, and the momentum was set to 0.9. This stage optimized target segmentation and feature extraction.

In Stage 2, the trained perception layer was frozen, and only the cognition layer was updated. The cognition layer was trained for 500 epochs using a cosine annealing learning-rate strategy. The learning rate and weight decay were both set to 5 × 10⁻⁴. This stage optimized graph-structural reasoning and occlusion-state inference.

2.8.2. Evaluation Metrics

To comprehensively assess GPC-Frame, nine metrics were used to evaluate the framework from three aspects: perception performance, cognition-layer reasoning, and D/H-point localization. For the perception layer, Precision, Recall, F1-score, Intersection over Union (IoU), and Average Precision (AP) were used to evaluate multi-target segmentation performance. For the cognition layer, structural reasoning accuracy and occlusion reasoning accuracy were used to evaluate the GCN-based reasoning results. The occlusion states included non-occluded, leaf/petiole-occluded, overlap-occluded, and cane-occluded grape clusters. For D/H-point localization, H-point localization success was defined as the successful localization of an executable harvesting point on the corresponding cluster stem of the target grape cluster. D-point localization success was defined as the successful localization of an interaction point on a removable occluding structure, such as an occluding leaf, petiole, or foreground overlapping grape cluster. H-point localization was evaluated for non-occluded grape clusters, whereas D-point localization was evaluated for leaf/petiole-occluded and overlap-occluded grape clusters. Cane-occluded grape clusters were recorded as not applicable for D/H-point localization. Localization success was calculated as the percentage of target grape clusters for which the corresponding H-point or D-point was correctly localized.

2.9. Implementation of Comparative Methods

To evaluate the performance of the proposed GPC-Frame, five representative agricultural vision methods related to fruit harvesting were re-implemented and evaluated using the same dataset and data split as those used in this study. To ensure a fair comparison, all comparative models were initialized with publicly available pre-trained weights where applicable, re-trained on the constructed dataset for 1000 epochs, and tested on the same hardware platform. A cosine annealing learning-rate strategy was used during training, and inference time was measured under the same runtime environment.

The comparative methods covered three paradigms. For the keypoint-detection paradigm, the methods of Chen et al. [30] and Jiang et al. [19] were re-implemented using YOLOv8-pose to predict one and three keypoints, respectively. For the morphological paradigm, the method of Lu et al. [31] was reconstructed using YOLOv11-seg together with its morphological bounding-box labeling strategy. For the geometric-reasoning paradigm, the methods of Zhou et al. [23] and Lin et al. [24] were implemented using YOLACT and SegFormer-B2 backbones, respectively. Their corresponding geometric constraints, including the GJK algorithm in Zhou et al. [23] and semantic rules in Lin et al. [24], were applied as post-processing steps.

3. Results

To evaluate whether GPC-Frame meets the perception, cognition, and robotic execution requirements of grape harvesting, the experimental results are presented in three parts. Section 3.1 evaluates the perception–cognition performance of GPC-Frame on the constructed grape dataset. Section 3.2 compares GPC-Frame with representative agricultural vision methods. Section 3.3 further validates the framework through field robotic harvesting experiments.

3.1. Perception–Cognition Performance Evaluation

To evaluate the perception–cognition performance of GPC-Frame, a comprehensive evaluation was conducted using the grape dataset constructed in this study. The evaluation covered multi-target scene segmentation, grape cluster structure and occlusion reasoning, D/H-point localization, and performance under different lighting conditions.

3.1.1. Multi-Target Scene Segmentation Performance

As shown in

Table 1. Performance evaluation of grape scene segmentation.

Category	Instance Number	Precision/%	Recall/%	F1-Score/%	AP/%	IoU/%
Grape clusters	1080	96.72	95.58	96.15	96.24	92.58
Cluster stems	522	90.45	87.92	89.17	89.65	80.46
Petioles	2237	87.15	84.68	85.90	86.32	75.28
Leaves	4141	94.39	92.85	93.61	93.92	87.99
Canes	2150	91.05	89.05	90.04	90.51	81.88
Mean	N/A	91.95	90.02	90.97	91.33	83.64

Note: N/A indicates not applicable.

, GPC-Frame achieved an mIoU of 83.64% and an mAP of 91.33% on the test set. The Precision values for grape clusters, cluster stems, petioles, leaves, and canes were 96.72%, 90.45%, 87.15%, 94.39%, and 91.05%, respectively.

Grape clusters and leaves obtained higher IoU values, reaching 92.58% and 87.99%, respectively. These two categories usually occupied larger image regions and had more continuous visual boundaries, which made them easier to segment. By contrast, cluster stems and petioles had lower Recall values of 87.92% and 84.68%, respectively. Their lower Recall was mainly related to their small pixel proportions, weak boundary continuity, and local visual similarity to surrounding leaves or canes.

In Figure 7, the masks of grape clusters and leaves are generally complete under sunny, cloudy, and nighttime conditions. Small artifacts and under-segmentation mainly appear around cluster stems and petioles, especially when these fine structures are partially occluded or visually similar to the surrounding background.

3.1.2. Grape Cluster Structure and Occlusion Reasoning with D/H-Point Localization

Table 2. Performance evaluation of grape cluster structural reasoning and occlusion reasoning, and D/H-point localization.

Grape Cluster Status	Grape Cluster Number	Structural Reasoning Accuracy/%	Occlusion Reasoning Accuracy/%	D/H-Point Localization Success/%	Reasoning and Localization Time/s
Non-occluded	294	98.64	98.30	96.60	0.04
Leaf/petiole-occluded	381	96.59	96.06	95.01	0.05
Overlap-occluded	228	94.74	94.30	93.86	0.04
Cane-occluded	177	95.48	94.92	N/A	0.03
Mean	N/A	96.36	95.90	95.16	0.04

Note: N/A indicates not applicable.

summarizes the cognition-layer performance in structural reasoning, occlusion reasoning, and D/H-point localization under different grape-cluster states. The evaluated states included non-occluded, leaf/petiole-occluded, overlap-occluded, and cane-occluded clusters, with 294, 381, 228, and 177 instances, respectively. The mean structural reasoning accuracy and mean occlusion reasoning accuracy reached 96.36% and 95.90%, respectively. For non-occluded clusters, the occlusion reasoning accuracy reached 98.30%, and the H-point localization success rate was 96.60%. For leaf/petiole-occluded and overlap-occluded clusters, the D-point localization success rates were 95.01% and 93.86%, respectively. Across the cluster states with D/H-point localization, the mean localization success rate was 95.16%, with an average reasoning and localization time of 0.04 s. For cane-occluded clusters, D/H-point localization was recorded as N/A because no executable D/H point was generated for this state.

The cognition-layer outputs in Figure 7g,h illustrate two cases of graph-based structural correction. In Figure 7g, a petiole misidentified as a cluster stem was removed from the grape-cluster structure during graph reasoning, and the target was classified as leaf/petiole-occluded. The corresponding D-point was then localized on the occluding structure. In Figure 7h, a cluster stem misidentified as a petiole was corrected, and fragmented structures belonging to the same grape cluster were aggregated. The target was consequently classified as non-occluded, and the H-point was localized on the corrected cluster stem.

3.1.3. Performance Under Different Lighting Conditions

Table 3. Performance evaluation of GPC-Frame under different lighting conditions.

Lighting Condition	Grape Cluster Number	mIoU/%	Structural Reasoning Accuracy/%	D-Point Localization Success/%	H-Point Localization Success/%	Perception Time/s	Cognition Time/s
Sunny	368	81.88	95.38	93.70	94.32	0.21	0.04
Cloudy	364	84.35	96.98	95.33	95.86	0.20	0.05
Night (Fixed LED)	348	86.23	97.70	96.80	96.79	0.20	0.04
Mean	N/A	84.15	96.69	95.28	95.66	0.20	0.04

Note: N/A indicates not applicable.

presents the performance of GPC-Frame under sunny, cloudy, and nighttime fixed-LED illumination conditions. Among the three illumination settings, nighttime fixed-LED illumination produced the best overall results, with an mIoU of 86.23%, structural reasoning accuracy of 97.70%, D-point localization success rate of 96.80%, and H-point localization success rate of 96.79%. Cloudy illumination yielded slightly lower but still stable performance, with an mIoU of 84.35%, structural reasoning accuracy of 96.98%, D-point localization success rate of 95.33%, and H-point localization success rate of 95.86%.

Performance declined under sunny conditions, with the mIoU decreasing to 81.88%, while the D-point and H-point localization success rates were 93.70% and 94.32%, respectively. Across the three lighting conditions, the mean mIoU, structural reasoning accuracy, D-point localization success rate, and H-point localization success rate were 84.15%, 96.69%, 95.28%, and 95.66%, respectively. The mean perception and cognition times were 0.20 s and 0.04 s, indicating that the perception–cognition pipeline maintained stable processing efficiency under different illumination conditions.

3.2. Comparison with Representative Grape-Harvesting Localization Methods

Table 4. Performance comparison with representative grape-harvesting localization methods.

Method	Core Technique	Occluded D-Point Localization	Non-Occluded H-Point Localization/%	Lead of GPC-Frame	Inference Time/s
Chen et al. [30]	YOLOv8-pose + One keypoint	N/A	81.63	+14.97	0.08
Jiang et al. [19]	YOLOv8-pose + Three keypoints	N/A	84.35	+12.25	0.06
Lu et al. [31]	YOLOv11-seg + Morphological label	N/A	87.41	+9.19	0.13
Zhou et al. [23]	YOLACT + Box constraints	N/A	90.48	+6.12	0.16
Lin et al. [24]	SegFormerB2 + Geometric rules	N/A	92.52	+4.08	0.37
GPC-Frame	Perception layer + Cognition layer	Yes	96.60	N/A	0.24

Note: N/A indicates not applicable.

compares GPC-Frame with five representative grape-harvesting localization methods. For non-occluded grape clusters, GPC-Frame achieved an H-point localization success rate of 96.60%, outperforming the best baseline method, Lin et al. [24], by 4.08 percentage points and the single-keypoint method of Chen et al. [30] by 14.97 percentage points.

The five comparative methods were designed to estimate harvesting points using keypoint prediction, morphological labeling, or geometric constraints; therefore, D-point localization for occluded clusters was not supported. In contrast, GPC-Frame generated D-points for leaf/petiole-occluded and overlap-occluded clusters through cognition-layer occlusion reasoning. The inference time of GPC-Frame was 0.24 s, which was shorter than that of the SegFormer-B2-based method of Lin et al. [24] but longer than that of the keypoint-based methods.

Figure 8 shows the localization outputs of different methods under five typical grape-harvesting scenes. Columns (b)–(f) show the H-point predictions of the five comparative methods, whereas column (g) shows the H-point or D-point generated by GPC-Frame according to the inferred occlusion state. In non-occluded and bifurcated cluster-stem scenes, GPC-Frame localized the H-point on the valid cluster-stem structure. In overlap-occluded and leaf/petiole-occluded scenes, the comparative methods still produced H-point predictions, whereas GPC-Frame generated D-points on the foreground occluding cluster or occluding petiole for disocclusion. In the detection-error case, GPC-Frame corrected the misidentified structure through graph reasoning and generated a D-point on the occluding petiole.

3.3. Field Robotic Validation

3.3.1. Field Perception–Cognition Performance Across Time Windows

Field perception–cognition experiments were conducted over a full day–night operation period to evaluate GPC-Frame under changing field illumination. A total of 261 grape clusters were evaluated across six time windows: 06:00–08:00, 08:00–11:00, 11:00–14:00, 14:00–18:00, 18:00–20:00, and 20:00–06:00. The field images used for this evaluation were manually annotated and used as the ground truth for calculating field mIoU and localization success. These windows covered early morning, morning, midday, afternoon, evening, and nighttime conditions, as summarized in

Table 5. Field perception–cognition performance of GPC-Frame across six time windows.

Time Periods	Time Window	Lighting Condition	Grape Cluster Number	mIoU/%	Structural Reasoning Accuracy/%	D/H-Point Localization Success/%	Perception–Cognition Time/s
Early morning	06:00–08:00	Natural + Fixed LED	41	81.25	87.80	85.37	0.24
Morning	08:00–11:00	Natural (Side light)	45	78.18	84.44	80.00	0.24
Midday	11:00–14:00	Natural (Top light)	42	75.45	78.57	73.81	0.25
Afternoon	14:00–18:00	Natural (Side light)	44	77.50	81.82	77.27	0.24
Evening	18:00–20:00	Natural + Fixed LED	43	80.53	88.37	86.05	0.24
Night	20:00–06:00	Fixed LED	46	82.48	93.48	91.30	0.25
Mean	N/A	N/A	N/A	79.23	85.75	82.30	0.24

Note: N/A indicates not applicable.

. Figure 9 shows example perception–cognition outputs for the six time windows in the same order.

Table 5 summarizes the field perception–cognition performance of GPC-Frame across the six time windows. The strongest field performance was observed at night under fixed LED illumination, with an mIoU of 82.48%, structural reasoning accuracy of 93.48%, and D/H-point localization success rate of 91.30%. In contrast, midday top-light conditions resulted in the lowest performance, with an mIoU of 75.45%, structural reasoning accuracy of 78.57%, and D/H-point localization success rate of 73.81%. Across the six time windows, the mean mIoU, structural reasoning accuracy, and D/H-point localization success rate were 79.23%, 85.75%, and 82.30%, respectively. The mean perception–cognition time was 0.24 s, indicating stable processing efficiency under different field illumination conditions.

3.3.2. Autonomous Harvesting Operations

Autonomous harvesting operations driven by GPC-Frame are shown in Figure 10, and the statistical results are summarized in

Table 6. Field harvesting performance of GPC-Frame under two operation modes.

Scene Type	Mode	Grape Cluster Number	D-Point Localization Success/%	Harvesting Success/%	Mean Harvesting Time per Cluster/s
Non-occluded	Direct harvesting	37	N/A	89.19	23
Occluded	Side-view/disocclusion harvesting	141	86.52	85.11	75
Mean	N/A	N/A	N/A	87.15	49

Note: N/A indicates not applicable.

. Among the 261 targets evaluated in the field perception–cognition test, 178 targets were used for robotic harvesting, including 37 non-occluded targets and 141 occluded targets requiring side-view observation or disocclusion. The remaining 83 targets were excluded from harvesting because the localized H- or D-points were outside the reachable workspace of the manipulator, or because the targets were cane-occluded.

Figure 10 illustrates three robotic execution processes: direct harvesting of a non-occluded grape cluster, disocclusion-assisted harvesting of a leaf/petiole-occluded grape cluster, and side-view/disocclusion-assisted harvesting of an occluded grape cluster. In these processes, GPC-Frame provided the H-point for direct harvesting or the D-point for disocclusion before subsequent H-point localization and harvesting.

The harvesting statistics in Table 6 further show the execution performance under the two operation modes. Direct harvesting was applied to 37 non-occluded targets, resulting in a harvesting success rate of 89.19% and a mean operation time of 23 s per cluster. For the 141 occluded targets, side-view/disocclusion-assisted harvesting achieved a D-point localization success rate of 86.52% and a harvesting success rate of 85.11%, with a mean operation time of 75 s per cluster. The mode-averaged harvesting success rate and operation time were 87.15% and 49 s, respectively.

4. Discussion

4.1. From Point Localization to State-Aware Harvesting Decisions

The results of this study show that the main contribution of GPC-Frame is not only improving the localization accuracy of harvesting points, but also introducing a reasoning step between scene perception and robotic action selection. In the cognition-layer evaluation, GPC-Frame achieved high structural reasoning accuracy, occlusion reasoning accuracy, and D/H-point localization success across different grape-cluster states. In the comparison experiment, GPC-Frame also achieved the highest H-point localization success rate for non-occluded grape clusters and, more importantly, provided D-point outputs for occluded grape clusters. These results indicate that the framework extends robotic grape harvesting from direct harvesting-point localization to state-aware action-point selection.

This distinction is important in trellised table-grape scenes because the scene contains not only grape clusters and cluster stems, but also leaves, petioles, canes, and neighboring clusters that may change the appropriate harvesting action. For a non-occluded target, the robot should directly localize the H-point on the cluster stem. For a leaf/petiole-occluded or overlap-occluded target, directly estimating an H-point may be insufficient because the target first requires disocclusion. For a cane-occluded target, the target should be avoided because the occluding structure is not removable. Therefore, the key problem is not only where the harvesting point is located, but also which action point should be generated under the current target state.

The comparison with existing methods further clarifies this difference. Keypoint-based methods directly estimate picking points or cluster-stem-related keypoints from images [19,30]. Post-segmentation methods use segmentation masks, skeletonization, contour extraction, or morphological rules to estimate harvesting points [31,32]. Geometric or semantic rule-based methods establish grape-cluster–stem correspondence through box constraints, ROI intersection, or semantic association rules [23,24]. These methods provide useful solutions for H-point localization, especially when the target cluster and its stem are clearly visible. However, they mainly rely on local visual or geometric evidence and do not explicitly determine whether the target should be directly harvested, disoccluded before harvesting, or excluded from harvesting.

GPC-Frame addresses this limitation by organizing grape clusters, cluster stems, leaves, petioles, and canes into a graph-based structural representation. The cognition layer uses GCN-based reasoning to evaluate harvest-related relationships among these components before D/H-point selection. The visual comparison in Figure 8 further illustrates this difference. In overlap-occluded and leaf/petiole-occluded scenes, the five comparative methods still generated H-point predictions because they did not define a separate D-point for disocclusion, whereas GPC-Frame generated D-points on the foreground overlapping cluster or the occluding petiole. In the detection-error case, a petiole was incorrectly identified as a cluster-stem candidate. The methods of Chen et al. [30], Jiang et al. [19], Zhou et al. [23], and Lin et al. [24] still generated H-point predictions on this incorrect structure, while Lu et al. [31] failed to provide a valid H-point. In contrast, GPC-Frame revised the structural relationship through graph reasoning and selected the D-point on the occluding petiole.

Therefore, the advantage of GPC-Frame lies in connecting multi-target perception with harvesting-state reasoning, rather than simply refining a point location. However, this reasoning ability still depends on the quality of the perception layer. When fine structures such as cluster stems and petioles are missed, fragmented, or misclassified, the graph constructed by the cognition layer becomes incomplete or incorrect. In such cases, the cognition layer may still fail to infer the correct target state or generate the correct D/H point. Therefore, fine-structure segmentation remains a key prerequisite for reliable harvesting cognition in trellised table-grape scenes.

4.2. Illumination Effects and Failure Propagation from Perception to Cognition

The field perception–cognition results show that illumination variation directly affected the performance of GPC-Frame. As shown in Table 5, the best field performance was obtained at night under fixed LED illumination, whereas the lowest performance occurred at midday under strong top lighting. This pattern was also observed in Figure 9. Under early morning, evening, and nighttime conditions, grape clusters, cluster stems, and petioles were more distinguishable from the background. By contrast, the midday scene showed local overexposure and reduced visibility of fine structures.

This performance difference is closely related to the visual characteristics of cluster stems and petioles. These structures are small, slender, and often partially visible in trellised grape scenes. They may also show weak boundaries or local similarity to leaves, canes, and background branches. Similar difficulties have been reported in studies involving small grape peduncle segmentation and complex vineyard-scene segmentation [20,21]. In this study, these difficulties were further amplified by field illumination variation. Strong top light at midday caused local overexposure and texture loss, making cluster stems and petioles more difficult to segment.

The failure cases in Figure 11 further show that perception errors can propagate to the cognition layer. In Figure 11a, overexposure reduced the contrast among cluster stems, petioles, leaves, and grape clusters. Once these fine structures were missed or incorrectly segmented, the graph representation used by the cognition layer became less reliable. Consequently, harvesting-relation reasoning, occlusion-state inference, and D/H-point localization were all affected. This explains why the midday time window produced lower mIoU, structural reasoning accuracy, and D/H-point localization success in Table 5.

RGB-D sensing further increases the complexity of field deployment. In this study, Kinect V2 was mainly used for relatively long-distance RGB-D image acquisition and served as an auxiliary supplementary source in the training dataset, whereas Intel RealSense L515 was used for near-range RGB-D sensing and final robotic field experiments. After resolution standardization, images acquired by both sensors were pooled into a unified dataset for model training and testing, which increased the diversity of observation distances and canopy appearances. The final robotic localization and execution were conducted using the L515 because table-grape harvesting is a near-range manipulation task and requires stable depth sensing for thin structures such as cluster stems and petioles. This sensor configuration is consistent with previous depth-camera evaluations for proximal grape sensing. Parr et al. [45] evaluated several depth cameras, including Kinect V2 and Intel L515, for proximal grape sensing, supporting the relevance of LiDAR-based RGB-D sensing in close-range grape-scene perception. Sensor-dependent uncertainty may still affect field performance. Kinect V2, based on time-of-flight sensing, may suffer from depth instability under strong ambient illumination or complex canopy backgrounds, whereas the L515 may be affected by LiDAR-related depth noise, weak returns from reflective surfaces, and depth fluctuations around object boundaries. Therefore, although Kinect V2 data improved dataset diversity during training, the final D/H-point localization accuracy in field execution was mainly influenced by the near-range depth quality of the L515.

These findings indicate that illumination control and sensor configuration should be treated as part of the robotic harvesting system. Future work should improve adaptive exposure control, controllable supplementary lighting, and robust RGB-D fusion under variable field illumination. This is especially important for trellised grape scenes, where small structural components such as cluster stems and petioles strongly influence both cognition-layer reasoning and final action-point localization.

4.3. Robotic Execution Constraints After Perception–Cognition

The field harvesting experiment further shows that successful perception–cognition does not necessarily guarantee successful robotic execution. Among the 261 grape clusters evaluated in the field perception–cognition test, 178 were used for autonomous harvesting. The remaining targets were excluded mainly because the localized H- or D-points were outside the reachable workspace of the manipulator, or because the targets were cane-occluded. This result indicates that field deployment is constrained not only by visual reasoning accuracy, but also by manipulator reachability, camera–arm coordination, and end-effector accessibility.

This distinction is important for interpreting the engineering value of GPC-Frame. In image-based evaluation, a target may be correctly segmented, reasoned, and assigned an H-point or D-point. In robotic field operation, however, the same point must also be reachable by the manipulator and compatible with the approach direction of the end-effector. Therefore, the transition from perception–cognition output to executable robotic action introduces additional constraints that are not fully reflected by image-based metrics.

The operation time in Table 6 further reflects the execution cost of active disocclusion. Direct harvesting of non-occluded targets required an average of 23 s per cluster, whereas side-view/disocclusion-assisted harvesting required 75 s per cluster. The longer operation time resulted from the additional execution chain required for occluded targets, including side-view observation when needed, D-point localization, occlusion removal, scene updating, and subsequent H-point localization. Thus, active disocclusion can convert some targets from not directly harvestable to harvestable after interaction, but it also increases operation time and execution complexity.

This trade-off should be interpreted as an engineering limitation rather than a failure of the perception–cognition framework. Existing studies have explored active viewpoint adjustment to improve the observation of occluded grape stems [33,34], and physical disocclusion or obstacle-removal strategies have been reported in clustered fruit and strawberry harvesting [35,36,37]. Compared with these studies, GPC-Frame emphasizes the reasoning step before interaction: the framework first determines whether the target should be directly harvested, disoccluded, or avoided, and then generates the corresponding H-point or D-point. In this sense, the D-point is not a replacement for the H-point, but an additional interaction point generated when the target requires disocclusion before harvesting.

Future improvements should focus on the execution chain after perception–cognition. First, manipulator reachability should be incorporated into action-point selection so that unreachable H/D points can be filtered or adjusted before execution. Second, camera–arm coordination should be improved to reduce spatial mismatch between visual localization and end-effector motion. Third, motion planning and end-effector design should be optimized to shorten the cycle time of side-view/disocclusion-assisted harvesting. Adaptive trajectory-planning strategies may also be introduced for active disocclusion, using cognition-layer outputs such as target state, D-point location, and structural confidence as inputs for online motion adjustment [46].

4.4. Implications for Harvesting Cognition in Horticultural Fruit Crops

Although this study focused on trellised table-grape harvesting, the perception–cognition strategy of GPC-Frame may provide a reference for robotic harvesting of other horticultural fruit crops with repeated fruit–stem–leaf structures. In horticultural fruit crops, such as apple, mango, and strawberry, the target fruit is often surrounded by pedicels, leaves, petioles, branches, shoots, or neighboring fruits [4,25,37]. These structures may occlude the target fruit or interfere with end-effector access, making it difficult to determine the harvesting action from fruit detection alone. Similar challenges have motivated active viewpoint adjustment [33,34] and physical disocclusion or obstacle-removal strategies [35,36,37] in robotic fruit harvesting.

The main implication of GPC-Frame is that robotic harvesting can benefit from reasoning about the structural relationships between the target fruit and surrounding components. Instead of treating fruit detection, occlusion judgment, and action-point selection as separate steps, a cognition layer can organize multi-target perception results into a task-oriented structural representation. Such a representation may help robots distinguish directly harvestable targets, targets requiring occlusion removal, and targets that should be avoided because of non-removable supporting structures or poor accessibility.

For other fruit crops, this strategy cannot be transferred directly without modification. The target categories, structural connections, occlusion types, action-point definitions, and execution rules may differ across crops and cultivation systems. For example, apple or mango harvesting may require reasoning about fruit–stem correspondence, fruit detachment direction, and end-effector approach path, whereas strawberry harvesting may require different reasoning about peduncle visibility, fruit orientation, and leaf occlusion. Therefore, future studies should redesign crop-specific structural representations and validate them in corresponding harvesting environments.

From this perspective, the contribution of GPC-Frame is not only a grape-specific localization method, but also a perception–cognition paradigm for linking scene understanding with executable robotic harvesting decisions. The key idea is to move robotic harvesting from isolated target detection toward state-aware reasoning over fruits, stems or peduncles, leaves, petioles, canes, and neighboring structures. This paradigm may be useful for developing more adaptive robotic harvesting systems in complex horticultural environments.

5. Conclusions

This study proposed GPC-Frame, a perception–cognition framework for robotic harvesting in trellised table-grape vineyards. The framework integrates multi-target scene perception, graph-based structural and occlusion reasoning, and coordinated D/H-point localization. By modeling the structural relationships among grape clusters, cluster stems, leaves, petioles, and canes, GPC-Frame transforms multi-organ perception results into state-aware harvesting decisions. This enables the robot to distinguish directly harvestable targets, targets requiring disocclusion before harvesting, and targets that should be avoided.

On the constructed grape dataset, GPC-Frame achieved a structural reasoning accuracy of 96.36%, an occlusion reasoning accuracy of 95.90%, and a mean D/H-point localization success rate of 95.16%. For non-occluded grape clusters, the H-point localization success rate reached 96.60%. Compared with representative grape-harvesting localization methods, GPC-Frame achieved higher H-point localization performance and further provided D-point localization for occluded targets requiring disocclusion. These results indicate that the cognition layer contributes not only to structural interpretation, but also to harvesting-oriented action-point selection.

Field robotic validation further evaluated the framework under six time windows and practical harvesting conditions. A total of 261 targets were included in the field perception–cognition test, and 178 targets were used for autonomous harvesting operations. The best field perception–cognition performance was obtained at night under fixed LED illumination, with an mIoU of 82.48%, a structural reasoning accuracy of 93.48%, and a D/H-point localization success rate of 91.30%. In the harvesting trials, direct harvesting of non-occluded targets achieved a success rate of 89.19%, while side-view/disocclusion-assisted harvesting of occluded targets achieved a success rate of 85.11%. The mode-averaged harvesting success rate was 87.15%.

The results show that robotic table-grape harvesting should not be treated only as a target detection or picking-point localization problem. Instead, harvest-related organs, occlusion states, and action-point selection need to be considered jointly. By introducing a cognition layer between visual perception and robotic execution, GPC-Frame provides a feasible approach for converting complex trellised grape scenes into executable harvesting decisions. Nevertheless, field performance was still affected by illumination variation, RGB-D sensing noise, manipulator reachability, and the increased operation time required for disocclusion-assisted harvesting. Future work will focus on adaptive illumination control, more robust RGB-D fusion, workspace-aware action planning, improved camera–arm coordination, and faster disocclusion–harvesting execution under field conditions.