Green Pepper Harvesting Robot System Based on Multi-Target Tracking with Filtering and Intelligent Scheduling

Abstract

To address the challenges of unstable target localization and poor multi-module coordination in automated green pepper harvesting—caused by occlusions from branches and leaves, as well as varying lighting conditions—this paper presents the design and implementation of a modular robotic picking system. At the perception level, the system integrates a YOLOv8 detector with a RealSense D435i camera to identify and locate the calyx–ectocarp junctions of green peppers. An integrated multi-target tracking and filtering framework is proposed, which fuses multi-feature association, trajectory smoothing and coordinate denoising strategies to suppress depth noise and trajectory jitter, thereby enhancing the stability and accuracy of 3D localization. At the control and execution level, a depth-first picking sequence strategy with ID freeze-state management is implemented within a multithreaded software–hardware co-design architecture. This approach avoids task conflicts and duplicate operations while supporting continuous multi-fruit harvesting. Field experiments under natural outdoor lighting and varying occlusion levels demonstrate that the proposed system achieves recognition rates of 91.57% and 80.29% and harvesting success rates of 82.85% and 77.68% for non-occluded and lightly occluded fruits, respectively. The average picking cycle per pepper fruit is 9.8 s. This system provides an effective technical solution for addressing stability control challenges in the automated harvesting process of green peppers.

1. Introduction

Pepper is a globally important economic crop, playing a critical role in agricultural production and the food industry. According to statistics from the United Nations Food and Agriculture Organization [1], China’s green pepper production in 2024 reached approximately 17.3309 million tons, accounting for about 38.71% of the global total output. This solidifies China’s position as the world’s largest producer and consumer of green peppers. Against this backdrop, the efficient and non-destructive harvesting of mature green peppers is crucial for enhancing their economic value. Currently, harvesting operations still rely heavily on manual labor, which, amid an aging population and continuously rising labor costs [2], has become a significant bottleneck constraining the development of the pepper industry. Consequently, there is an increasingly urgent need for the development of intelligent harvesting equipment and robotic technologies tailored for green peppers [3,4].

In recent years, substantial progress has been made in fruit and vegetable harvesting robots. Regarding perception, researchers have employed and optimized various deep learning architectures and image processing algorithms to tackle challenges in fruits and vegetables detection, localization, and picking point identification in complex agricultural environments [5,6,7,8,9,10,11]. In the development of harvesting robot systems, Pan et al. designed an end-effector equipped with tactile sensors, achieving a harvesting success rate of 79.17% through visual servoing and grasp posture control algorithms, with a single-fruit picking time of approximately 15 s [12]. Alam et al. developed a low-cost Cartesian robot integrated with a GRBL controller and a Kinect V2 sensor. By optimizing the linear movement speed to 0.83 cm/s, they achieved a harvesting efficiency of 86% and a fruit damage rate of only 5% [13]. Ravuri et al. applied Mask R-CNN and a 6-degree-of-freedom robotic arm to green pepper harvesting [14]. Although the fruit localization accuracy reached 90.7%, the harvesting success rate was only 31.4% with a damage rate of 6.97%, primarily due to the need for further improvement in the adaptability of the end-effector to dynamic environments.

Nevertheless, despite the progress in visual perception and system integration of fruit and vegetable harvesting robots, the stability and continuity of object localization and tracking in the picking task is still a problem to be further improved. Li et al. integrated YOLOv8 with ByteTrack to track falling maize kernels, achieving over 99% counting accuracy in dynamic scenes [15], yet their method primarily focused on 2D trajectory association without explicit handling of 3D coordinate noise. Paul et al. developed a comprehensive capsicum harvesting system that employed YOLOv8 for detection, segmentation, and growth stage classification, and integrated a RealSense D455 camera for 3D peduncle localization [16]. Its tracking module relied primarily on ByteTrack without dedicated filtering mechanisms to suppress depth noise or trajectory jitter in continuous harvesting sequences. Liang et al. developed a tomato-picking robotic system based on the fusion of global and hand-level vision. Through spatial asynchronous localization and vision-based servo control, it significantly improved positioning accuracy and picking success rates in complex environments [17]. However, maintaining stable servo tracking under dynamic occlusion conditions remains challenging. Rapado-Rincón et al. proposed MinkSORT, a 3D sparse convolutional network for tomato tracking in greenhouses [18], which improved tracking accuracy under occlusion but required extensive 3D data and computational resources, limiting its real-time applicability in field environments. Similarly, Arlotta et al. employed an Extended Kalman Filter with RGB-D data for grape bunch tracking on a mobile robot, demonstrating robustness in handling intermittent measurements [19], yet their framework did not incorporate trajectory smoothing or ID consistency management for continuous multi-fruit harvesting tasks.

Collectively, most existing methods still suffer from two critical limitations in real agricultural harvesting scenarios: on one hand, depth noise and trajectory jitter in 3D fruit localization caused by leaf occlusion and varying illumination are rarely suppressed systematically; on the other hand, there is a lack of tight coordination between target tracking consistency, sequential picking execution control, and system-level multi-module collaboration, which easily leads to frequent ID switches, task conflicts, repeated operations, and discontinuous harvesting. To overcome these challenges and improve the perception robustness, spatial positioning stability, scheduling intelligence, and collaborative efficiency of automated green pepper harvesting systems in complex natural environments, this paper designs and implements a modular robotic picking system for green pepper harvesting tasks. Specifically, at the perception level, the integration of multi-object tracking and a three-dimensional filtering algorithm significantly improves the accuracy and output stability of fruit spatial localization. At the decision-making and execution level, optimized picking sequence scheduling, ID consistency management, and a closed-loop communication mechanism between the end-effector and the control system effectively enhance the reliability, real-time performance, and operational efficiency of multi-module collaboration.

The main contributions of this paper are as follows:

(1)

System-level innovation: A fully integrated modular robotic harvesting system for green peppers is proposed and implemented, which integrates YOLOv8-based visual recognition, RealSense depth camera-based 3D localization, a 6-DOF robotic arm, and a custom cutting–gripping end-effector. Different from most open-loop systems in previous studies, this system adopts a closed-loop perception–decision–action workflow to enhance operational reliability.

(2)

Algorithmic novelty: A multi-feature fusion tracking and 3D coordinate smoothing algorithm is developed to address target loss and severe coordinate fluctuation in complex harvesting environments. Unlike IoU-only or ByteTrack-based methods that lack dedicated depth noise suppression, the proposed framework combines IoU-based association, Mahalanobis-distance-based matching, the Hungarian algorithm, Kalman filtering, the 3σ criterion, and single exponential smoothing to effectively suppress outliers and jitter in localization results.

(3)

Methodological novelty: A depth-first picking sequence scheduling strategy with ID freeze-state management is designed together with a multithreaded software–hardware cooperative control architecture, which resolves task conflicts and resource competition during multi-target harvesting and ensures continuous, stable, and efficient system operation.

(4)

System validation: Extensive field experiments under natural outdoor lighting and different occlusion levels are conducted to comprehensively evaluate the system performance, verifying the effectiveness of the proposed methods in terms of recognition accuracy, harvesting success rate, and operational stability, and providing guidance for future research and practical application.

2. Materials and Methods

2.1. Robot System Settings

The intelligent green pepper harvesting system consists of an Intel RealSense D435i camera for RGB-D sensing, an upper computer (Ubuntu 18.04) for vision and planning algorithms, a 6-DOF industrial robotic arm with its dedicated controller, a custom-designed cutting–gripping end-effector, and an STM32-based control unit (Figure 1). The RealSense D435i camera is mounted on the flange of the 6-DOF robotic arm in a rigid eye-in-hand configuration, transmitting real-time synchronized RGB and depth image streams to the upper computer via USB 3.0; its shooting height and fixed viewing angle are calibrated to match the green pepper canopy height in the experimental scenario, ensuring optimal target detection and localization. The camera streams data to the upper computer via USB, which processes it to generate motion commands. These commands are then sent via Ethernet to the robot controller for execution.

Figure 1. A robotic system for automated harvesting of green peppers.

The hardware connection and control circuit architecture of the system is shown in Figure 2. The crawler chassis supplies 220 V AC power to the upper computer and robot controller. The robot controller controls the motion of the 6-DOF robotic arm and provides 24 V DC power. Part of the 24 V DC is used to power the 24 V relay module and motor driver, and the other part is converted to 3.3 V DC for the 3.3 V relay module and STM32 microcontroller core. The motor driver is connected to the microcontroller in a common-anode configuration. The STM32 generates PWM signals through the motor driver to regulate the cutting, and clamping actions of the end-effector.

Figure 2. An overall block diagram of the system hardware and circuit connection.

The end-effector assembly consists of a self-designed linear actuator (including a push rod and mounting base) and a commercial cutting–gripping head. Driven by the STM32 control circuit, the actuator drives the head through the push rod. The control circuit integrates power conversion, motor driving, and relay modules, and realizes real-time status synchronization with the upper computer through I/O communication to ensure coordinated and stable operation.

The overall system follows a closed-loop “perception–decision–action” workflow (Figure 3).

Figure 3. Green pepper picking system workflow diagram.

Key hardware specifications are listed in Table 1.

Table 1. Hardware system main parameters.

Module Name	Model/Control Method	Main Parameters
Upper computer	Hasee z7-ct7na	Intel i7-9750H, 16 GB RAM, GTX1660Ti, Ubuntu 18.04
RGB-D camera	RealSense D435i	1280 × 720 @ 30FPS, depth accuracy ± 2%
Robot arm	FR5-SPARK	6-DOF, Load 5 kg, Repeat positioning accuracy ± 0.02 mm, Fiber optic Ethernet communication
End-effector	Self-designed actuator module, commercial cutting–gripping head	24V DC motor drive, actuator stroke 40 mm, maximum clamping width 50 mm, motor torque 2 N·m

The 6-DOF robotic arm serves as the main motion actuator, with a repeat positioning accuracy of ±0.02 mm and a rated load of 5 kg, ensuring stable and precise movement in unstructured field environments. The end-effector employs a linear motor-driven hinged linkage mechanism for integrated clamping and cutting, as illustrated in Figure 4. It is mainly composed of a linear motor, an I-type joint, a hinged push rod, and an integrated clamping–cutting head. The linear motor is connected to the hinged push rod via the I-type joint, converting the linear reciprocating motion of the motor into the opening and closing motion of the cutting–gripping head.

Figure 4. Overall structure diagram of the end-effector.

The adoption of a hinged push rod instead of a rigid straight rod is critical for efficient power transmission. Due to the offset between the direction of the motor’s thrust and the ideal motion path of the push rod, a rigid straight connection would introduce significant lateral forces and mechanical resistance, leading to increased wear and reduced transmission efficiency. In contrast, the hinged structure allows rotational compensation, effectively releasing lateral constraints. This design ensures that the thrust of the linear motor is transmitted along the optimal direction of the push rod, minimizing the motor load and mechanical resistance while providing sufficient torque to drive the stable opening and closing of the clamping–cutting head for low-damage harvesting. The linear motor is driven by a 24 V DC motor, with a maximum stroke of 40 mm and an output torque of 2 N·m. The cutting–gripping head provides a maximum opening width of 50 mm to adapt to different diameters of green pepper peduncles. During operation, the linear motor drives the cutting–gripping head to perform closing and opening actions, realizing synchronous clamping and cutting of the peduncle. The end-effector is rigidly mounted on the flange of the robotic arm, ensuring high alignment accuracy between the cutter and the target picking point.

2.2. Image Acquisition and Dataset Annotation

The pepper image dataset used in this study was collected in June 2025, with the acquisition objects being self-cultivated potted pepper plants under outdoor natural lighting conditions (Figure 5). The image acquisition device was an Intel RealSense D435i camera. To enhance the robustness and generalization ability of the model under different lighting conditions, image collection covered typical time periods, including morning (7:00–9:00), noon (11:00–13:00), and afternoon (15:00–17:00), encompassing diverse natural light intensities and weather conditions. The collected images had a resolution of 1280 × 720 and were saved in JPG format. The shooting distance remained stable between 0.5 and 0.8 m. To comprehensively capture the performance of peppers under varying occlusion levels, poses, and sizes, and to enhance viewpoint and target diversity, each potted plant was rotated at 10° intervals in the horizontal direction and randomly photographed at eye-level, upward, and downward angles until full 360° coverage was achieved, before proceeding to the next plant. A total of 3871 images were collected, and after screening, 3450 images were retained for subsequent annotation. The screening criteria involved removing images of poor quality, such as those with blurring or ghosting caused by movement or changes in lighting during the rotation-based shooting process.

Figure 5. Potted green peppers.

Based on the overall structural diagram of green peppers (Figure 6), the LabelImg 1.8.6 tool was used to manually annotate bounding boxes by precisely locating the junction between the calyx and the outer pericarp of each pepper [20], as shown in Figure 7.

Figure 6. The overall structure of green peppers.

Figure 7. Labeling of data sets.

The annotated targets were categorized into four occlusion levels based on visual assessment of the bounding box area containing the calyx and the ectocarp junction: no occlusion, where the target region is fully visible without any obstruction; light occlusion, where a small portion of the bounding box is covered but the majority of the junction remains clearly discernible; moderate occlusion, where a significant portion is obscured, yet key characteristics of the junction are still partially visible; and heavy occlusion, where over half of the bounding box is covered, substantially limiting the visibility of the junction features. Representative examples of each occlusion level are provided in Figure 8. All annotations were saved under the unified label “Stem”.

Figure 8. Four occlusion levels of green peppers.

2.3. Feature Detection and 3D Coordinate Extraction

2.3.1. YOLOv8 Target Detection Model Construction

For robust real-time detection of green peppers, the YOLOv8 object detection architecture was employed. To address the specific challenge of accurately locating small calyxes under occluded conditions, which demands high feature extraction capability, the YOLOv8l variant was selected. This model provides an optimal balance between detection precision and computational efficiency for our complex agricultural scenario.

After partitioning the annotated dataset into training, testing, and validation sets at a ratio of 7:2:1, the model was trained on a platform equipped with an Intel i5-14600KF processor, 32 GB RAM, an NVIDIA GeForce RTX 5060TI GPU (16 GB VRAM), and a Windows 10 operating system, following the training parameters detailed in Table 2. The evaluation was performed on the test set using the saved optimal weight file. The recognition results are shown in Figure 9. The overall Precision, Recall, F1-score, and mAP reached 92.51%, 84.17%, 88.14%, and 89.65%, respectively. These metrics fully demonstrate that the model can reliably achieve high-precision recognition of small targets, meeting the practical requirements of intelligent harvesting robots in terms of detection accuracy.

Table 2. Model training parameters.

Training Parameter	Value
weight	Yolov8l.pt
batch-size	8
epochs	300
imgsz	640 × 640
momentum	0.937
warmup_bias_lr	0.1
lr0	0.01
lrf	0.01
optimizer	SGD
weight_decay	0.0005
mosaic	0.5
cos_lr	True
seed	0
dropout	0.0

Figure 9. Recognition results of green pepper.

2.3.2. 3D Coordinate Extraction

This study employs an Intel RealSense D435i RGB-D camera to achieve three-dimensional localization of harvesting points. The system simultaneously captures depth and color streams to generate spatially aligned RGB-D frames in real-time, while loading camera intrinsic parameters and distortion coefficients. Prior to depth mapping, geometric correction is first applied to RGB images using distortion coefficients to eliminate lens distortion effects on spatial alignment. Subsequently, the target pixel coordinates detected by the YOLOv8l model are matched with the corrected depth map, and the 3D spatial coordinates of targets are calculated through the camera coordinate system transformation model. This method ensures pixel-level accurate registration of color images and depth data, providing high-precision input for subsequent 3D positioning of the robotic arm.

The YOLOv8l model outputs the pixel coordinate center point (u, v) of each target. By retrieving the corresponding depth value Z (unit: mm) at this point in the depth map, the pixel coordinates can be converted to 3D coordinates (X, Y, Z) in the camera coordinate system using Equations (1)–(3), as illustrated in Figure 10:

$$\mathrm{X} = {\displaystyle \frac{\left(\mathrm{u}-{\mathrm{c}}_{\mathrm{x}}\right)}{{\mathrm{f}}_{\mathrm{x}}}}\ast \mathrm{Z}$$

(1)

$$\mathrm{Y}={\displaystyle \frac{\left(\mathrm{v}-{\mathrm{c}}_{\mathrm{y}}\right)}{{\mathrm{f}}_{\mathrm{y}}}}\ast \mathrm{Z}$$

(2)

$$\mathrm{Z}=\mathrm{depth}\left(\mathrm{u}, \mathrm{v}\right)$$

(3)

where $(\mathrm{u}, \mathrm{v})$ represents the target’s pixel position in the RGB image coordinate system, $\left({\mathrm{f}}_{\mathrm{x}}, {\mathrm{f}}_{\mathrm{y}}\right)$ denotes the camera’s focal length (in pixels), $({\mathrm{c}}_{\mathrm{x}},{ \mathrm{c}}_{\mathrm{y}})$ indicates the principal point coordinates (i.e., the optical center’s position in the pixel coordinate system), and Z represents the depth value obtained from the depth map for the corresponding pixel, measured in millimeters.

Figure 10. 3D Localization results based on YOLO detection.

2.4. Multi-Target State Management and Coordinate Smoothing Filter

In continuous robotic harvesting of green peppers, relying only on single-frame detection results suffers from critical limitations in practical field scenarios. Under natural outdoor conditions, factors such as leaf shading, fluctuations in light intensity, and specular reflection from the fruit surface can easily lead to unstable detection results (in some cases, the coordinates of the same target may be normal in one frame but jump directly to (0, 0, 0) in the next frame, as shown in Figure 11); in particular, the depth values captured by RGB-D cameras are prone to outliers or fluctuations (as shown in Figure 12).

Figure 11. Abnormal jump of 3D coordinates.

Figure 12. Fluctuations in depth coordinates.

These unstable and abnormal coordinates can directly lead to positioning errors even premature closure of the end-effector before reaching the actual green pepper stem, resulting in a failed harvest (Figure 13). Meanwhile, for multi-target continuous operation, maintaining consistent target identity (ID) across frames is essential. Conventional single-frame detection cannot realize robust inter-frame target association, leading to frequent ID switches or track loss, especially when fruits are clustered or occluded. Without effective ID management, the system may repeatedly attempt to harvest the same fruit or miss valid targets, causing task conflicts and reduced operational efficiency.

Figure 13. Depth coordinate instability causes the end-effector to close prematurely.

To overcome these inherent drawbacks of single-frame detection and achieve stable, continuous, and accurate 3D localization and multi-target tracking, a multi-level filtering and state management framework is proposed. This framework is designed to systematically eliminate coordinate jitter, remove depth outliers, maintain ID consistency, and provide reliable smooth positioning for robotic arm control. It consists of four sequential modules: multi-feature fusion data association for robust tracking, Kalman filtering for 2D trajectory smoothing, 3σ criterion for outlier rejection, and single exponential smoothing for stabilizing 3D coordinates, especially the depth channel. Through this layered processing pipeline, noisy raw detection and depth data are converted into stable, consistent, and executable picking points, ensuring reliable continuous harvesting. The workflow of the proposed multi-target state management and decision system is illustrated in Figure 14.

Figure 14. Flowchart of the harvesting decision system based on multi-level filtering.

2.4.1. Multi-Feature Fusion Data Association and Target Tracking

Based on the object bounding box information output by the YOLOv8 detection model for the current frame, spatial overlap consistency (IoU) and motion consistency (Mahalanobis distance [21]) are integrated into a unified cost matrix and globally optimized using the Hungarian algorithm [22].

The IoU between current detection ${\mathrm{B}}_{\mathrm{current}}$ and previous track ${\mathrm{B}}_{\mathrm{prev}}^{\mathrm{i}}$ is defined as:

$$\mathrm{IOU}\left({\mathrm{B}}_{\mathrm{current}},{ \mathrm{B}}_{\mathrm{prev}}^{\mathrm{i}}\right) = {\displaystyle \frac{\left|{\mathrm{B}}_{\mathrm{current}}\cap {\mathrm{B}}_{\mathrm{prev}}^{\mathrm{i}}\right|}{\left|{\mathrm{B}}_{\mathrm{current}}\cup {\mathrm{B}}_{\mathrm{prev}}^{\mathrm{i}}\right|}}$$

(4)

where $\left|{\mathrm{B}}_{\mathrm{current}}\cap {\mathrm{B}}_{\mathrm{prev}}^{\mathrm{i}}\right|$ represents the intersection area of the two bounding boxes, and $\left|{\mathrm{B}}_{\mathrm{current}}\cup {\mathrm{B}}_{\mathrm{prev}}^{\mathrm{i}}\right|$ denotes their union area.

The Mahalanobis distance ${\mathrm{D}}_{\mathrm{M}}$ evaluates motion consistency between observation ${\mathrm{z}}_{\mathrm{k}}$ and Kalman-predicted state ${\widehat{\mathrm{x}}}_{\mathrm{k}\mid \mathrm{k}-1}$:

$${\mathrm{D}}_{\mathrm{M}}^2 = {\left({\mathrm{z}}_{\mathrm{k}}-\mathrm{H}{\widehat{\mathrm{x}}}_{\mathrm{k}\mid \mathrm{k}-1}\right)}^{\mathrm{T}}{\mathrm{S}}_{\mathrm{k}}^{-1}\left({\mathrm{z}}_{\mathrm{k}}-\mathrm{H}{\widehat{\mathrm{x}}}_{\mathrm{k}\mid \mathrm{k}-1}\right)$$

(5)

where ${\mathrm{S}}_{\mathrm{k}} = \mathrm{H}{\mathrm{P}}_{\mathrm{k}\mid \mathrm{k}-1}{\mathrm{H}}^{\mathrm{T}} + \mathrm{R}$ is the innovation covariance matrix. The Mahalanobis distance threshold ${\theta }_{\mathrm{M}} = 9.210$ corresponds to the 99% confidence interval of the chi-squared distribution with 2 degrees of freedom, which is used to exclude unreasonable matching.

The final matching cost integrates both cues:

$${\mathrm{C}}_{\mathrm{i},\mathrm{j}} = {\lambda }_{\mathrm{IoU}}\cdot \left(1-{\mathrm{IoU}}_{\mathrm{i},\mathrm{j}}\right) + {\lambda }_{\mathrm{M}}\cdot \min \left({\displaystyle \frac{{\mathrm{D}}_{\mathrm{M},\mathrm{i},\mathrm{j}}}{{\theta }_{\mathrm{M}}}},1\right)$$

(6)

where ${\lambda }_{\mathrm{IoU}} = 0.7$ and ${\lambda }_{\mathrm{M}} = 0.3$ are empirically determined weighting coefficients that balance spatial and motion cues.

2.4.2. Kalman Filter Modeling and State Update

State-Space Model Formulation

To smooth the target bounding box center coordinates (x, y), a Kalman filter [23] is initialized for each newly detected target. The state vector is defined as:

$${\mathrm{x}}_{\mathrm{k}} = {\left[\mathrm{x}, \mathrm{y},{ \mathrm{v}}_{\mathrm{x}}, {\mathrm{v}}_{\mathrm{y}}\right]}^{\mathrm{T}}$$

(7)

where x and y represent the center coordinates of the target at time k, and ${\mathrm{v}}_{\mathrm{x}},{ \mathrm{v}}_{\mathrm{y}}$ denote the corresponding velocity components. The state transition equation and the observation model equation are expressed as follows:

$${\mathrm{x}}_{\mathrm{k}} = \mathrm{F}{\mathrm{x}}_{\mathrm{k}-1} + {\mathrm{w}}_{\mathrm{k}-1}$$

(8)

$${\mathrm{z}}_{\mathrm{k}}=\mathrm{H}{\mathrm{x}}_{\mathrm{k}}+{\mathrm{v}}_{\mathrm{k}}$$

(9)

where ${\mathrm{w}}_{\mathrm{k}-1}$ and ${\mathrm{v}}_{\mathrm{k}}$ are zero-mean Gaussian process and observation noises with covariances Q and R. The state transition matrix F and observation matrix H are:

$$\mathrm{F} = \left[\begin{array}{cccc}1& 0& \Delta \mathrm{t}& 0\\ 0& 1& 0& \Delta \mathrm{t}\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right], \mathrm{H} = \left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right]$$

(10)

Recursive Formulas of the Kalman Filter

First, prediction:

$${\widehat{\mathrm{x}}}_{\mathrm{k}}^{-} = \mathrm{F}{\widehat{\mathrm{x}}}_{\mathrm{k}-1} + {\mathrm{w}}_{\mathrm{k}}$$

(11)

$${\mathrm{P}}_{\mathrm{k}}^{-}=\mathrm{F}{\mathrm{P}}_{\mathrm{k}-1}{\mathrm{F}}^{\mathrm{T}}+\mathrm{Q}$$

(12)

where ${\widehat{\mathrm{x}}}_{\mathrm{k}}^{-}$ is the prior state estimate at time k, and ${\mathrm{P}}_{\mathrm{k}}^{-}$ is the prior error covariance matrix.

Subsequently, update:

$${\mathrm{K}}_{\mathrm{k}} = {{\mathrm{P}}_{\mathrm{k}}^{-}{\mathrm{H}}^{\mathrm{T}}\left(\mathrm{H}{\mathrm{P}}_{\mathrm{k}}^{-}{\mathrm{H}}^{\mathrm{T}} + \mathrm{R}\right)}^{-1}$$

(13)

$${\widehat{\mathrm{x}}}_{\mathrm{k}}={\widehat{\mathrm{x}}}_{\mathrm{k}}^{-}+\mathrm{K}\left({\mathrm{Z}}_{\mathrm{k}}-\mathrm{H}{\widehat{\mathrm{x}}}_{\mathrm{k}}^{-}\right)$$

(14)

$${\mathrm{P}}_{\mathrm{k}}=\left(\mathrm{I}-\mathrm{KH}\right){\mathrm{P}}_{\mathrm{k}}^{-}$$

(15)

where ${\mathrm{K}}_{\mathrm{k}}$ is the Kalman gain, ${\widehat{\mathrm{x}}}_{\mathrm{k}}$ is the posterior state estimate, and ${\mathrm{P}}_{\mathrm{k}}$ is the updated posterior error covariance matrix.

Parameter Initialization and Tuning Strategy

Upon YOLO detection of a new target, the Kalman filter state is initialized as:

$${\widehat{\mathrm{x}}}_0 = \left[\begin{array}{c}\begin{array}{c}{\mathrm{x}}_0\\ {\mathrm{y}}_0\end{array}\\ \begin{array}{c}0\\ 0\end{array}\end{array}\right]$$

(16)

$${\mathrm{P}}_0=\mathrm{Diag}\left(\right[10, 10, 1000, 1000\left]\right)$$

(17)

where $\left({\mathrm{x}}_0, {\mathrm{y}}_0\right)$ is the first-frame detection coordinate. The diagonal values reflect higher initial uncertainty in velocity components.

The process noise covariance is defined as $\mathrm{Q} = 0.01·{\mathrm{I}}_4$, and the observation noise covariance is defined as $\mathrm{R} = {\mathrm{I}}_2$, which are determined via parameter tuning experiments to balance noise suppression and tracking fidelity.

The Kalman filter effectively reduces high-frequency jitter in 2D coordinates while closely following the true motion trends. The smoothed 2D coordinates are back-projected to 3D space using the RealSense depth camera. A historical 3D position queue is maintained for each target ID to support subsequent outlier rejection and smoothing.

2.4.3. 3σ Outlier Rejection and Single Exponential Smoothing

Raw 3D coordinates converted from depth images still contain disturbances such as instantaneous depth noise, outliers, or mild fluctuations, which are critical for harvesting precision. In practical operation, even a small depth deviation of several millimeters may lead to positioning error and premature clamping of the end-effector. To enhance the reliability of 3D coordinate sequences, a two-step smoothing strategy is applied to the historical position queue $\left\{{\mathrm{P}}_{\mathrm{i}}\right\}$ of each tracked target.

First, statistical outliers are removed using the 3σ criterion [24]. The mean μ and standard deviation σ of the sequence are computed:

$$\mu = {\displaystyle \frac{1}{\mathrm{N}}}\sum _{\mathrm{i}= 1}^{\mathrm{N}}{\mathrm{P}}_{\mathrm{i}}$$

(18)

$$\sigma =\sqrt{{\displaystyle \frac{1}{\mathrm{N}}}\sum _{\mathrm{i}=1}^{\mathrm{N}}{\left({\mathrm{P}}_{\mathrm{i}}-\mu \right)}^2}$$

(19)

Any point satisfying $\mid {\mathrm{P}}_{\mathrm{i}}-\mu \mid >3\sigma$ is discarded. The remaining inliers are averaged to obtain the fitted 3D point ${\mathrm{P}}_{\mathrm{t}}$ for the current frame:

$${\mathrm{P}}_{\mathrm{t}} = {\displaystyle \frac{1}{\left|\left\{{\mathrm{P}}_{\mathrm{i}}^{\prime }\right\}\right|}}\sum _{{\mathrm{P}}_{\mathrm{i}}^{\prime }\in \left\{{\mathrm{P}}_{\mathrm{i}}^{\prime }\right\}}{\mathrm{P}}_{\mathrm{i}}^{\prime }$$

(20)

Subsequently, single exponential smoothing [25] is applied to further suppress single-frame jitter:

$${\mathrm{P}}_{\mathrm{t}}^{\mathrm{smooth}} = \alpha \times {\mathrm{P}}_{\mathrm{t}} + \left(1-\alpha \right) \times {\mathrm{P}}_{\mathrm{t}-1}$$

(21)

where α = 0.7 is the smoothing coefficient.

When the standard deviation of the target’s smooth coordinates (X, Y, Z) in the last two consecutive frames is below 5 mm, the target is deemed spatially stable and added to the sampling queue. This multi-level filtering strategy ensures that planar jitter in 2D images is first suppressed by the Kalman filter, while instabilities in the depth domain are further mitigated through exponential smoothing. These two methods operate in different domains and address distinct sources of error; when combined, they provide sufficiently stable and precise localization for reliable robotic grasping, thereby preventing grasping failures caused by coordinate fluctuations.

The overall algorithm flow for the aforementioned multi-target state management and coordinate smoothing filter is presented in the pseudocode in Algorithm 1.

Algorithm 1: Multi-feature fusion tracking and 3D Coordinate smoothing

Input: color_image, depth_frame, depth_intri Output: Smoothed 3D coordinates of tracked peppers 1:  Initialize prev_boxes, current_boxes, kalman_filters, track_missing_count, xyz_histories, smoothed_history 2:  Set IOU_TH, MAHALANOBIS_TH, LAMBDA_IOU, LAMBDA_M, max_frames_missing, window_len, α 3:  while camera stream is available do 4:     Run YOLOv8 on color_image, obtain detections D = { (box_k, conf_k) |    conf_k ≥ IOU_TH } in (x, y, w, h) format 5:     Prune tracks whose missing count > max_frames_missing; let existing_tracks be keys of prev_boxes 6:     if existing_tracks ≠ ∅ and D ≠ ∅ then 7:         For each track_i ∈ existing_tracks and det_j ∈ D, compute IoU(track_i, det_j) in image plane 8:         Use Kalman prediction to get predicted center of track_i and Mahalanobis distance to det_j center 9:         If Mahalanobis distance > MAHALANOBIS_TH, set cost_{i,j} = +∞; else cost_{i,j} = LAMBDA_IOU·(1 − IoU) + LAMBDA_M·Mahalanobis 10:           Apply Hungarian algorithm to the cost matrix to obtain globally optimal one-to-one matches 11:           For each matched pair (track_id, det_box) do Kalman prediction + update with det_box center, obtain (x_kf, y_kf, w, h) 12:              Set current_boxes[track_id] ← (x_kf, y_kf, w, h), track_missing_count[track_id] ← 0 13:           For unmatched tracks do Kalman prediction only, update current_boxes and increment track_missing_count 14:           For unmatched detections do create new track_id, initialize its Kalman filter with det_box    center, add to current_boxes 15:     else if existing_tracks = ∅ and D ≠ ∅ then 16:           Initialize a new track and Kalman filter for each det_box ∈ D and fill current_boxes 17:     end if 18:     For each (track_id, box) ∈ current_boxes do 19:         Obtain (x_kf, y_kf) from box and query median_depth at (x_kf, y_kf) in depth_frame 20:         if median_depth is valid then 21:           Back-project (x_kf, y_kf, median_depth) via depth_intri to camera_xyz and append to    xyz_histories[track_id] (kept at window_len) 22:           if |xyz_histories[track_id]| ≥ N then 23:              Compute mean μ and std σ of xyz_histories[track_id], discard points violating the 3σ rule to obtain inliers 24:              Let filtered_xyz be mean of inliers; apply single exponential smoothing 25:                 smoothed_xyz = α·filtered_xyz + (1 − α)·previous smoothed_xyz (or filtered_xyz if no history) 26:              Update smoothed_history[track_id] with smoothed_xyz 27:           end if 28:         end if 29:     end for 30:     Set prev_boxes ← current_boxes; clear current_boxes 31:  end while

2.5. Hand-Eye Calibration

To achieve vision-based grasping tasks, it is necessary to accurately transform the 3D coordinates of the target point from the camera coordinate system to the robot base coordinate system. In this experiment, the hand-eye calibration was performed in the “Eye-in-Hand” configuration to determine the transformation between the camera and the robot flange. After calibrating the tool center point, multiple robot end-effector poses and their corresponding ArUco calibration plate poses were recorded (Figure 15). A total of 8 groups of valid pose data were collected for calibration to ensure the robustness of the transformation matrix, and the classical AX = XB model was solved using both the Horaud [26] and Park [27] methods.

Figure 15. The field of view of the ArUco board in the camera under different robot poses during the calibration process.

The accuracy of hand-eye calibration was validated by comparing the position of the vision-guided end-effector with ground truth measurements [28], where the Park method demonstrated slightly superior performance. Notably, the error along the Z-axis was consistently higher, attributable to the inherent depth measurement characteristics of the Intel RealSense D435i camera [29]. Consequently, the transformation matrix obtained via the Park method was selected for all subsequent experiments.

2.6. Picking Task Planning and System Coordination

After converting the smoothed target 3D coordinates into the robotic arm’s base coordinate system, it is essential to further plan the picking sequence and generate safe and efficient motion trajectories. This paper proposes a depth-first-based picking sequence planning strategy that effectively reduces the risk of collisions between the robotic arm and overhead branches and foliage during movement.

To ensure overall system responsiveness and real-time performance, a multi-threaded software architecture is adopted to decouple visual perception from motion control. During robotic arm movement, the visual detection window remains active, but visual processing is temporarily suspended to avoid localization errors induced by arm motion. Furthermore, a target ID freezing mechanism is implemented to prevent repeated operations on fruits that have been successfully picked or have previously failed, thereby ensuring continuous and efficient task progression.

Precise execution is achieved through a segmented motion planning strategy integrated with coordinate compensation. This strategy defines a preset approach point P, positioned at a compensated offset (Z-axis: −100 mm, Y-axis: −20 mm) relative to the centroid of the visually estimated target bounding box, as illustrated in Figure 16. Upon reaching point P, the end-effector opens to a predefined width of 25 mm to accommodate variations in fruit stem dimensions.

Figure 16. Schematic of the picking point determination strategy.

The harvesting cycle is controlled by a closed-loop event-driven protocol between the host computer and the STM32 microcontroller. During each harvesting operation cycle, the robotic arm first moves to the preset approach point P. Upon receiving the “robotic arm in position” signal, the microcontroller sends a command to drive the end effector to extend to the specified width. The robotic arm then advances along the negative Z-axis, allowing the extended end effector to wrap around the green pepper stem. After confirming the “robotic arm arrived” signal, the end effector closes to complete the grasping action. The robotic arm then retracts with the green pepper, transfers it to the unloading point for release (defined as the point 15 cm directly above the center of the green pepper basket), and finally returns to its initial position. This cycle continues until no new targets are detected for 20 consecutive frames, at which point the task is deemed complete (Figure 17).

Figure 17. Multi-segment motion planning process for robotic systems. The arm is in the initial state (T0). The arm moves to the approach point (T1). It advances to the gripping point (T2). The end-effector cuts and grips the green pepper peduncle (T3). After gripping is complete it moves back (T4). The robotic arm moves to the unloading point and releases the green pepper (T5).

2.7. Experimental Setup

All harvesting experiments were conducted in the afternoon (14:00–17:00) of sunny days in June 2025 at the outdoor experimental field of Hunan Agricultural University, using self-cultivated potted green pepper plants. The potted plants were arranged in a single, compact row according to pot size, and the leaf distribution was not artificially adjusted, to ensure that the degree of shading matched the natural growth conditions of green pepper plants in actual field cultivation. The entire test was conducted under a unified, clear and windless natural outdoor lighting condition.

No additional obstacles around the test site. All harvesting experiments were performed using a 6-DOF robotic arm equipped with the integrated cutting–gripping end-effector. The vision system consisted of an Intel RealSense D435i camera mounted in an eye-in-hand configuration. All perception and control algorithms were implemented in Python 3.8.20 on an upper computer running Ubuntu 18.04, which communicated with the robotic arm via TCP/IP. Coordination with the STM32-based end-effector controller was achieved through relay-mediated I/O signaling for real-time status synchronization. An overview of the outdoor harvesting scenario with coordinate annotations is provided in Figure 18.

Figure 18. Harvesting test on potted pepper plants with annotated coordinate systems.

3. Results and Discussion

3.1. Performance of Multi-Target Tracking and Filtering

To verify the effectiveness of the proposed multi-feature fusion tracking and coordinate smoothing strategy, performance tests under different IoU thresholds and comparison between different matching methods were carried out. All statistical indicators for multi-target tracking performance (ID matching rate, association accuracy, miss rate) were calculated based on the number of valid matching pairs and total annotated test frames, with valid tracking defined as continuous target ID consistency and stable 3D coordinates for more than 10 consecutive frames.

The tracking performance for multiple green pepper targets was statistically evaluated by applying different IoU thresholds (θ). The results are presented in Table 3. The selection of θ = 0.5 represents an optimal trade-off between correct matching rate and ID stability.

Table 3. Performance test comparison with different thresholds.

θ	Total Matching Pairs	ID Matches	ID Switches	ID Matching Rate
0.1	400	400	0	100%
0.3	400	400	0	100%
0.5	400	376	24	94%
0.7	400	260	140	65%
0.9	400	41	359	10.25%

Performance evaluation on 300 annotated frames shows that the fusion method reduces ID switches and improves association accuracy compared to IoU-only matching, demonstrating superior robustness in occluded and overlapping scenarios (Table 4).

Table 4. Performance comparison between IoU-only and multi-feature fusion matching methods.

Matching Method	Total Pairs	ID Switches	Association Accuracy (%)	Miss Rate (%)
IoU-only (θ = 0.5)	300	40	84.47	19.5
Fusion Method	300	18	92.48	22.67

3.2. Overall Harvesting Performance

To evaluate the overall performance of the designed pepper-harvesting robot, a total of 999 picking trials were conducted under clear and wind-free outdoor conditions using self-cultivated potted pepper plants. Before each picking operation, the system determined the occlusion level of the target pepper through visual assessment based on real-time im-age analysis, and the trial was then categorized into the corresponding occlusion class for subsequent statistical analysis. The experiment focused on the system’s recognition and picking performance under different occlusion conditions. Detailed statistical results for each occlusion level are presented in Table 5.

Table 5. Performance metrics under different occlusion levels.

Occlusion Level	Count (pcs)	Detection Correct Rate (%)	Harvesting Success Rate (%)	Plant Damage Rate (%)
No occlusion	261	91.57	82.85	11.11
Light occlusion	279	80.29	77.68	34.48
Moderate occlusion	244	72.13	57.39	51.49
Severe occlusion	215	46.51	40.00	75.00
Average	249.75	72.63	64.34	56.85

Each occlusion category was evaluated according to three metrics:

(1)

Detection correct rate: the percentage of peppers correctly detected within that occlusion level.

(2)

Harvesting success rate: the percentage of successfully picked peppers among those correctly detected.

(3)

Plant damage rate: the percentage of successfully picked peppers that caused plant damage during the harvesting process.

For example, in the no occlusion case, 239 out of 261 green peppers were correctly detected (91.57%). Among these 239 green peppers, 198 were harvested successfully (82.85%), and 22 of those 198 caused plant damage (11.11%). Data for other occlusion levels were calculated using the same method.

The experimental results indicate that as the level of occlusion increases, both the recognition and picking performance of the system decline. Under no and light occlusion conditions, the detection rates were 91.57% and 80.29%, the picking success rates were 82.85% and 77.68%, and the plant damage rates were 11.11% and 34.48%, respectively. Under moderate and heavy occlusion conditions, the detection rates decreased to 72.13% and 46.51%, the picking success rates dropped to 57.39% and 40.00%, while the plant damage rates increased to 51.49% and 75.00%, respectively.

In order to further explore the reasons for harvesting failure of the system under different occlusion conditions, especially when the target is in severe occlusion, the harvesting failure rate will rise sharply, the acquisition failure trials are further classified, and statistical analysis is performed according to the root cause of failure. Taking the 175 failed harvesting experiments (from 215 severe occlusion tests, Table 5) involving severe obstruction as an example, the fault cases can be divided into two categories according to the system operation stage: perception-level fault and execution-level fault. The statistical results of failure type distribution and their respective proportions are presented in Table 6, and representative examples of each failure type are shown in Figure 19.

Table 6. Quantitative distribution of severe occlusion failure cases.

Failure Type	Number of Cases	Proportion (%)
Perception-level failure	115	53.49
Execution-level failure	60	27.91

Figure 19. Typical example of failed harvesting due to severe occlusion. Severe occlusion of the calyx-ectocarp junction by leaves and stem results in the target not being detected or coordinate anomalies (A). From different perspectives, the end-effector collides with the stem while cutting the same target, causing interference and resulting in a failed harvest. (B).

Perception-level failure is the dominant cause of harvesting failure under severe occlusion, accounting for more than 50% of all failed cases. Due to the lack of multi-view information, the system cannot effectively reconstruct the complete spatial feature of the green pepper calyx-ectocarp junction when over half of the target region is covered, leading to missed detection or 3D coordinate anomalies. Execution-level failure is the secondary cause, which is a derivative problem of perception limitation. Even if the target is detected, the dense surrounding vegetation easily interferes with the end-effector’s clamping and cutting actions.

In terms of operating efficiency, the average single-pepper harvesting cycle of the system is 9.8 s (as shown in Figure 20). The time consumption of each stage is specifically distributed as follows: coordinate convergence and smoothing take 0.22 s, robotic arm movement to the pre-harvesting point takes 1.95 s, cutting and grasping actions take 2.35 s, and fruit placement plus arm returning takes 5.28 s. The overall efficiency meets the basic requirements of automated green pepper harvesting.

Figure 20. Time distribution of pepper harvesting stages.

Compared with existing green pepper harvesting robotic systems, our proposed system achieves a higher detection and harvesting success rate under low-to-moderate occlusion conditions, and the single-fruit harvesting cycle is significantly shortened (9.8 s), which is superior to most existing systems. The key reason for these performance improvements is the integrated multi-target tracking and filtering framework proposed in this study, which effectively suppresses depth noise and trajectory jitter in 3D localization, and the depth-first picking sequence strategy with ID freeze-state management that ensures continuous multi-fruit harvesting without task conflicts. While Ravuri et al. [14] achieved a high detection rate, the extremely low harvesting success rate is mainly due to the lack of an optimized end-effector and motion planning strategy, which verifies the importance of the integrated hardware and software design in our system. Paul et al. [16] adopted a similar YOLOv8 and RealSense depth camera scheme but lacked a dedicated filtering mechanism for depth noise, leading to lower harvesting success rate and longer cycle time than our system. The comparison fully demonstrates that our proposed technical framework effectively makes up for the deficiencies of existing systems in 3D localization stability and multi-module coordination, providing a more practical technical solution for automated green pepper harvesting.

3.3. Discussion

The proposed system presents clear engineering practicability and environmental robustness for green pepper harvesting based on potted plant experiments. The multi-target tracking with filtering framework significantly improves the stability of 3D localization, and the intelligent scheduling strategy ensures continuous and efficient multi-fruit harvesting. The modular hardware and software architecture also facilitates system integration, maintenance and secondary development, which is conducive to the transformation and application of the technology in actual agricultural production. The system demonstrates reliable recognition and picking performance under no occlusion conditions. However, performance degrades significantly as occlusion increases, revealing limitations of the system.

3.3.1. Perception Limitations

The quantitative statistical analysis of severe occlusion failure cases (Table 6) shows that perception-level failure caused by single-view information deficiency accounts for 53.49% of all failed trials, which directly confirms the core limitation of the current single RGB-D camera perception scheme. The single view restricts the system’s ability to capture the complete morphological features of the green pepper calyx-ectocarp junction under dense foliage occlusion, leading to frequent missed detection, ID mismatch, and depth coordinate deviation. Such a restricted field of perception faces difficulties in achieving reliable identification in densely growing crops such as calyx–ectocarp junction, where heavy foliage frequently results in severe occlusion of green pepper. Moreover, the similarities in color and texture between green peppers and their surrounding stems and leaves complicate the reliable identification of key morphological features, leading to a high rate of missed detections under occluded conditions.

To overcome these limitations, future perception systems should transition from single-view perception to multi-view or multimodal global 3D sensing frameworks [30,31,32,33,34]. Such approaches can reconstruct a more complete three-dimensional crop scene model by collecting information from multiple perspectives, thereby enhancing feature visibility and localization accuracy under occluded conditions. Drawing on the concepts of global 3D modeling and multi-view fusion, multi-angle data acquisition and reconstruction of canopy spatial structure can enable robust perception of occluded green peppers, thereby improving the overall recognition success rate and positioning accuracy of the harvesting system.

3.3.2. Execution Limitations

Execution limitations under occlusion primarily stem from the current end-effector design and motion planning strategy. The relatively bulky and mechanically coarse structure of the end-effector, combined with its fixed opening width, often leads to entanglement with adjacent vegetation when multiple green peppers or stems are in close proximity (Figure 21). This not only causes collateral plant damage but also increases the risk of mis-cutting the target stem. Even under light occlusion, the damage rate remains as high as 34.48%. Improvements should be made in both the structural design of the end-effector and the optimization of operational parameters to enhance its selectivity and precision under occluded conditions. Based on the development of a miniaturized and refined end-effector, on one hand, the design of gripping units with adaptive deformation capability and real-time force sensing can be achieved by drawing on bio-inspired flexible structures and triboelectric sensing feedback technology. This would enable precise, low-damage grasping and separation in complex clustered environments [35]. On the other hand, by integrating a cutting dynamics model of multilayer composite materials with multi-objective parameter optimization methods, cutting parameters such as angle and speed of the integrated cutting-grasping mechanism can be systematically matched and dynamically adjusted. This approach helps reduce operational energy consumption and minimize plant damage [36]. Furthermore, during the approach phase to an occluded green pepper, unintended collisions with stem can displace the target stem or the green pepper itself (Figure 22), leading to harvesting failure even after successful initial localization. Future work could focus on enhancing the intelligent obstacle avoidance and path planning capabilities of picking robots in dynamic and unstructured environments [37]. On the one hand, sampling-based path planning methods could be further optimized by introducing target gravity mechanisms to improve search efficiency, combined with genetic algorithms for path post-processing [38]. On the other hand, strategies based on deep reinforcement learning could be explored, particularly by integrating recurrent neural networks to memorize and utilize historical state information, thereby achieving more efficient and robust obstacle avoidance decisions [39].

Figure 21. Damaged harvesting.

Figure 22. Harvesting failure.

Moreover, recent research on multi-arm coordination provides promising directions for overcoming occlusion-related challenges. Lammers et al. demonstrated successful apple harvesting using two highly coordinated robotic arms [40]. In contrast, the current single-arm system shows clear limitations in highly occluded and spatially constrained scenarios. Inspired by such dual-arm collaboration, a potential solution could involve deploying one robotic arm equipped with a compliant gripping mechanism to intelligently move obstructing branches and leaves, while another arm focuses on precise localization and harvesting of the target green pepper. Such cooperative task division is expected to significantly improve harvesting success rates and plant preservation in densely occluded environments.

3.3.3. Strategies for Improving Harvesting Efficiency

The average harvesting cycle of the system for a single green pepper is 9.8 s. Among all stages of the picking cycle, the longest time consumption (5.28 s) is attributed to the robotic arm’s process of placing the harvested pepper and returning to its initial position, which is mainly caused by the conservative uniform-speed motion trajectory adopted in the current system to ensure operational safety. A feasible improvement is to install a small loading net at the end-effector: after a target pepper is picked, the end-effector opens directly to let the pepper fall into the net by gravity. The robotic arm only transports the net to the loading point for unified fruit release when the net is fully loaded, instead of placing a single pepper immediately after each picking. This batch harvesting and placement mode can save a substantial amount of time consumed by repeated single fruit transportation and return movements.

The cutting and grasping stage takes 2.35 s, which is mainly limited by the fixed opening and closing width of the current end-effector. The opening and closing stroke of the end-effector can be adaptively adjusted according to the diameter of the green pepper peduncle detected in real time, which avoids the redundant stroke of the current fixed stroke and further improves the action efficiency of the end-effector.

Although the current perception stage only takes 0.22 s, further lightweight optimization can free up more computing resources for the real-time motion control of the system and lay a foundation for the parallel processing of multi-target perception and motion planning. The YOLOv8l model can be replaced with a lightweight model (e.g., YOLOv8n/s) optimized by knowledge distillation and pruning. This optimized model can reduce the model inference time while maintaining the detection accuracy of the calyx–ectocarp junction, thus further compressing the time consumption of the perception stage.

The current system adopts a single 6-DOF robotic arm to harvest single targets sequentially, which is an inherent limitation of the current hardware architecture for efficiency improvement. For large-scale practical agricultural production, a multi-arm collaborative harvesting architecture can be constructed, where each robotic arm is responsible for harvesting targets in a small independent area and multiple arms perform picking operations simultaneously. This architecture can break through the efficiency bottleneck of single-arm sequential harvesting. Combined with the above-mentioned motion planning and end-effector optimization strategies, the multi-arm collaborative system is expected to reduce the average harvesting cycle for a single fruit to less than 5 s, which is close to the efficiency of manual harvesting.

4. Conclusions

This paper presents an integrated robotic system for automated green pepper harvesting, combining YOLOv8-based recognition, RealSense depth sensing, and a 6-DOF robotic arm with a cutting–gripping end-effector. A multi-stage filtering and tracking framework is proposed, which integrates IoU-Mahalanobis matching with the Hungarian algorithm for robust data association, Kalman filtering for trajectory smoothing, 3σ outlier rejection, and exponential smoothing for stable 3D coordinate estimation. Experimental results show that under no occlusion, the system achieves detection and picking success rates above 80%, with an average cycle time of 9.8 s per pepper fruit. While performance declines in highly occluded scenarios, the system demonstrates reliable coordination and continuous operation over extended periods. This work provides a practical modular solution for automated pepper harvesting and lays a foundation for future improvements in perception, execution, and multi-arm collaboration in complex agricultural environments.

This work not only provides an effective technical scheme for stable and continuous automated harvesting of green peppers in potted testing environments but also offers a referable integration framework for vision-guided agricultural robots with high robustness and practicality.