Key Technologies for Autonomous Fruit- and Vegetable-Picking Robots: A Review

Abstract

With the rapid pace of urbanization, a significant number of rural laborers are migrating to cities, leading to a severe shortage of agricultural labor. Consequently, the modernization of agriculture has become a priority. Autonomous picking robots represent a crucial component of agricultural technological innovation, and their development drives progress across the entire agricultural sector. This paper reviews the current state of research on fruit- and vegetable-picking robots, focusing on key aspects such as the vision system sensors, target detection, localization, and the design of end-effectors. Commonly used target recognition algorithms, including image segmentation and deep learning-based neural networks, are introduced. The challenges of target recognition and localization in complex environments, such as those caused by branch and leaf obstruction, fruit overlap, and oscillation in natural settings, are analyzed. Additionally, the characteristics of the three main types of end-effectors—clamping, suction, and cutting—are discussed, along with an analysis of the advantages and disadvantages of each design. The limitations of current agricultural picking robots are summarized, taking into account the complexity of operation, research and development costs, as well as the efficiency and speed of picking. Finally, the paper offers a perspective on the future of picking robots, addressing aspects such as environmental adaptability, functional diversity, innovation and technological convergence, as well as policy and farm management.

1. Introduction

Agricultural labor is on the decline, particularly in China, where younger generations are increasingly reluctant to engage in farming, resulting in a significant labor shortage. As the economy continues to develop and labor costs rise annually, traditional manual harvesting becomes progressively more expensive, thereby reducing farmers’ profit margins. According to China Ministry of Agriculture and Rural Affairs, from 2013 to 2023, the average monthly wages of migrant laborers increased from approximately CNY 2609 to around CNY 4780, representing a rise of over 150% [1]. Labor costs are especially high for seasonal workers. Fruit- and vegetable-picking robots, as a key component of smart agriculture, offer a promising solution. By incorporating advanced robotics, these machines can facilitate the transition to intelligent and modernized farming, alleviate the burden of strenuous physical labor, reduce farmers’ workload, and enhance the overall working environment.

The autonomous harvesting robot for fruits and vegetables is composed of several key systems, including a vision system, robotic arm, control unit, mobility platform, end-effector, and power supply [2]. The robot utilizes a vision camera to capture images and applies image processing techniques along with machine learning algorithms to accurately identify the type, location, and ripeness of the produce. Based on this information, the robot calculates an optimal motion path and generates a trajectory for the robotic arm to reach the target produce. Upon approaching the fruit, the end-effector is activated and employs gentle gripping or suction mechanisms to carefully harvest the produce. Once the produce is successfully grasped, the robotic arm transfers it to a collection basket or conveyor belt, thereby completing the harvesting process [3].

2. Research Progress of Fruit- and Vegetable-Picking Robots

Agricultural picking robots are an important part of agricultural automation and intelligence, and significant progress has been made in terms of technology and application. For example, tomatoes, strawberries, apples, and other crops with large fruits and more fixed growth environments have begun to use picking robots for commercialization. Many countries have begun to introduce relevant policies to support the research and development and application of agricultural robotics to promote the process of agricultural modernization [4]. Since 1980s, developed countries such as the United States, the Netherlands, and Japan have begun to increase the research and development of picking robots, utilizing advanced technologies such as electrical automation technology, image processing technology, computer vision, and artificial intelligence to promote the functional improvement and performance enhancement of picking robots [5].

In 2017, a strawberry picking robot called Rubion had been invented by a company called Octinion in Leuven, Belgium [6]. The system’s independence and effectiveness are ensured by seven key components: a walking system, localization system, vision system, robotic arm, gripper, logistics handling module, and quality monitoring module, as illustrated in Figure 1a. Silwal et al. developed an apple-picking robot, shown in Figure 1b, which is equipped with a single CCD color camera to identify apples and determine their 3D coordinates, along with a three-jawed manipulator for grasping [7]. Zhao et al. designed a 5-degree-of-freedom articulated picking robot, depicted in Figure 1c [8,9]. This system features an open fruit tree-picking system controlled by an industrial computer, a serial communication converter, and a machine vision system. Henry et al. developed a kiwifruit-picking robot with a quadruple robotic arm, as shown in Figure 1d [10], capable of operating on the undulating terrain of hilly orchards. In response to the current reliance on manual labor for tomato harvesting in solar greenhouses, as shown in Figure 1e, which is both time-consuming and labor-intensive, Yu et al. have designed and developed a tomato harvesting robot specifically for use in such environments [11].

Boaz et al. developed a sweet pepper-picking robot with six degrees of freedom, as shown in Figure 1f [12]. This robot features a lifting platform that enables the end-effector to reach higher picking points. The robot gripper separates the branch stems from the main stem using a vibrating blade at the fixed end, thereby facilitating the bell pepper harvesting process. Feng et al. designed a four-armed apple-picking robot [13,14], depicted in Figure 1g, capable of performing integrated “picking–transporting–collecting” operations. Tevel, founded by Israel’s Yaniv Maor, has conducted in-depth research on picking drones and in 2020 launched a commercial solution for orchard harvesting centered on drone picking. It has a forward-extending robotic arm that recognizes the location and ripeness of the fruit through vision sensors, grabs the fruit, and then picks it by rotating and twisting it [15]. Rong et al. developed a mushroom-picking robot for greenhouse applications, as shown in Figure 1i [16]. This robot operates four picking units in parallel on a mobile platform, with the end-effector using pneumatic clamping. The current research lineage of typical fruit- and vegetable-picking robots is shown in

Table 1. Research lineage of typical fruit- and vegetable-picking robots.

Year	Nation	Species	Camera	Degrees of Freedom	End-Effector	Success Rate
2014	China [8,9]	Apple	Color camera	5	2-Finger Clamping	77%
2017	Belgium [6]	Strawberrie	RGB + 3D camera	5	Two-finger grip	—
2017	USA [7]	Apple	CCD + RGB-D	7	Three-finger twist	84%
2019	USA [10]	Kiwi	Four pairs of color camera cameras	4	Clamping type	51%
2022	China [11]	Tomato	Raspberry Pi 4B Camera	6	Flexible 3-Finger Clamping	—
2020	Britain [12]	Sweet pepper	RGB-D	6	Vibrating Blade + Clamping	61%
2021	China [16]	Mushroom	Depth camera	3	Pneumatic suction-held	86.8%
2023	China [13,14]	Apple	RGB-D	3	Three-Finger Grip Screw	82%

.

3. Key Technology

The picking robot utilizes its vision system to capture images and employs advanced image processing technology to accurately identify the type, location, and ripeness of fruits and vegetables. Based on this data, the robot uses positioning technology to obtain precise location information, allowing it to plan the optimal movement trajectory for the robotic arm to reach the target produce. As the robotic arm approaches the fruit, the end-effector is activated, carefully separating the produce without causing any damage. Once the targets are securely grasped, the robotic arm transfers them to a collection basket or conveyor belt, thereby completing the harvesting process. Thus, the effectiveness of fruit and vegetable harvesting relies on two critical technologies: vision systems and end-effector fruit separation mechanisms.

3.1. Vision System

A vision system typically comprises a camera, an image processing module, a localization device, and a central processing unit. The number of cameras commonly used is either one or two, corresponding to monocular or binocular setups [17]. The primary functions of the vision module include acquiring digitized images, processing, localizing the produce, and identifying obstacles such as stems and leaves. Based on the information provided by the vision system, the arm, gripper, and other components are controlled to execute precise picking actions. Vision systems are often paired with artificial light sources or optical filters to mitigate issues such as inaccurate recognition caused by shadows on the surface of fruits and vegetables in natural environments [18].

3.1.1. Sensors

The visual sensors of picking robots provide crucial visual information that enables effective recognition, localization, and decision making. These sensors form the foundation for the robot’s ability to perceive its external environment, identify target fruits, and plan the optimal picking path. By equipping picking robots with advanced visual sensor technology, they are endowed with “intelligent eyes”, allowing them to autonomously and efficiently complete picking tasks even in complex natural environments. Picking robots rely on various vision sensors, including color cameras for capturing fruit color information and near-infrared and multispectral cameras to differentiate fruits against complex backgrounds. Features such as color, texture, and shape in color images are commonly used for fruit recognition [19]. Researchers have designed real-time monocular vision systems for apple picking that utilize color CCD cameras to capture apple images, with image processing performed on a PC [20,21,22]. However, single-color cameras can only provide two-dimensional position and ripeness information, lacking the ability to obtain three-dimensional position data [23]. Currently, picking robots generally use binocular machine or monocular vision combined with range sensors to achieve fruit localization. Teruo et al. developed an apple-picking robot whose vision sensors used two color CCD cameras [24]. By simultaneously capturing images of the same target, these cameras enabled the synthesis of 3D information about the picking target. Wang et al. employed the binocular vision sensor, which includes both a color and an infrared sensor, the latter of which captures depth images corresponding to the color image [25]. Wei et al. designed intelligent picking robots based on depth binocular vision processing [26], which utilize binocular stereo vision to convert images into 3D stereograms. Yuan et al. implemented a binocular vision stereo system with a matching strategy combining grayscale correlation and polyline geometry to achieve stereo matching and 3D reconstruction of cucumber gripping points [27]. Song et al. developed a recognition and ranging method for eggplant-picking robots based on binocular vision [28], using a brightness-based threshold segmentation algorithm to segment the G-B grayscale image. The center of mass was then selected as the matching primitive, and the depth information of eggplants was calculated using the principle of similar triangles.

To better acquire depth information of fruits, some researchers utilize RGB-D depth cameras to capture images. An RGB-D camera combines the functions of a traditional RGB camera with depth perception capabilities, enabling it to simultaneously output both color and depth images [29]. Xie et al. developed a set of picking robots with an autonomous path-planning system based on a depth camera and SLAM navigation technology [30]. Abundant Robotics created an apple-picking robot that uses depth camera technology for localization and recognition [31]. Silwal et al. developed a depth camera-based apple-picking robot, primarily used in apple orchards employing a V trellis cultivation method [32]. This robot achieved picking time per fruit of 84%, with an average picking time of 6 s per apple. Additionally, Kondo [33], Zhang Qin [34], and Liu et al. [35] have developed tomato-picking robot vision systems based on depth camera technology. In addition to camera sensors, various other technologies are employed for fruit identification and localization, including laser, ultrasonic radar, and hybrid methods that combine cameras with additional sensors. Wang et al. developed an agricultural fruit-picking robot vision system utilizing automatic laser localization, incorporating a PID algorithm to enhance the system’s localization accuracy [36]. Zhang et al. introduced a novel laser vision recognition system for apple-picking robots, which directly captures depth images of hierarchical relationships and improves the system’s resistance to interference [37]. Feng et al. employed an LMS211 laser rangefinder, based on the time-of-flight principle, to measure target distances [38]. Qin et al. designed a picking robot using Raspberry Pi for deep learning, which was equipped with an HC-SR04 ultrasonic sensor for distance measurement, and integrated it with a depth camera for fruit localization [39].

Overall, 2D vision sensors are relatively cost-effective, easy to integrate, and benefit from a wide range of off-the-shelf algorithms for color and shape recognition. However, their performance can be significantly affected by ambient lighting conditions, which may impact recognition accuracy. In contrast, 3D depth cameras provide detailed depth information necessary for accurate 3D modeling, but come at a higher cost. Laser and ultrasonic ranging sensors, while useful for localization, are expensive and can exhibit slower system response times. Binocular vision systems, though effective, require complex algorithms for stereo matching, involve substantial computational resources, and demand high accuracy. It is essential to select the appropriate sensors based on specific needs, application scenarios, and optimizing performance through the integration of multiple sensors [40]. A classification comparison of each vision sensor is presented in

Table 2. Classification comparison of vision sensors.

Sensors	Dominance	Detection Distance	Defective	Positioning Accuracy/%
Monocular vision camera [41]	Simple structure, low cost, relatively simple image processing algorithms	400–1500 mm	Unable to directly determine the true size of the object, occlusion of the object will lead to loss of information, recognition accuracy is not high	81–91
Stereo visual matching [42]	Ability to provide rich depth and position information with high calibration accuracy	300–1500 mm	Susceptible to the influence of the target surface reflection requires a large amount of computational resources, complex algorithms, high hardware costs	83–95
RGB-D depth camera [43]	Small size, high integration, better performance in scenes with low light or lack of texture, high depth measurement accuracy, wide range of applications	400–1000 mm	High power consumption, low resolution, limited by depth of field and glare, high data processing requirements	86–94
Laser rangefinders [44]	High accuracy of distance measurement, fast ranging response, long distance, strong adaptability	1000–1500 mm	Affected by the foliage or tree branches blocking the impact of easy signal interference, the distance is too long, easily leads to out-of-focus results	87–90

.

3.1.2. Recognition Algorithm

When performing picking tasks, robots first segment and recognize the acquired images to identify fruit targets. Commonly used target recognition algorithms include traditional digital image processing techniques, which rely on image features such as color, shape, and texture; machine learning-based image segmentation methods; and neural network algorithms based on deep learning [45].

• Traditional Imaging Processing

Traditional digital image processing techniques rely on feature segmentation based on attributes such as color, shape, and texture to detect fruits like apples, tomatoes, and citrus, which have distinct and stable image features. These techniques encompass a range of methods and algorithms, including image enhancement, filtering, segmentation, and feature extraction. They are versatile and applicable to a broad spectrum of tasks and requirements. Fruits exhibit significant and stable color features, which are less dependent on image size. Yoshida et al. employed both color and point cloud features to identify tomato bunch picking points, with each frame taking approximately 1 s to process [46]. Lv et al. utilized the Otsu dynamic threshold segmentation method based on R-G color features to segment fruit images [47]. A comparison of the Otsu segmentation algorithm and a fixed-threshold segmentation algorithm is shown in Figure 2. The Otsu method demonstrates superior segmentation performance and greater adaptability to changes in lighting conditions. Additionally, they applied color feature methods to pick ripe tomatoes, using an improved Otsu segmentation algorithm based on chromatic aberration information [48]. This approach involved removing natural background noise and addressing issues with contour extraction by employing the GMM method.

In addition to color features, shape feature segmentation can be employed to detect fruits. Shape features are particularly useful for addressing challenges such as fruits being obscured by branches, leaves, or clusters, which can alter the geometric parameters of the targets and reduce recognition accuracy. Tokuda et al. utilized the spectral reflection differences between watermelons and leaves in the near-infrared band to binarize and label watermelon images, identifying the target regions based on fruit shape features [49]. Meng et al. extracted the boundary between the target fruit and the background in captured images using edge detection methods [50]. Specifically, the Canny edge detector and the Hough transform were employed for target contour detection. Texture features are crucial for distinguishing fruits from background interference and for segmenting the target fruit from the background image. Li et al. used texture features of cucumbers in conjunction with RGB coarse segmentation to identify cucumber fruits [51]. Rakun et al. combined texture and color features to identify apples, but the accuracy was suboptimal due to factors such as underdeveloped fruits, visual effects caused by natural light, and overlapping fruit occlusion [52]. Chaivivatrakul et al. proposed a method for detecting green fruits based on image texture feature analysis, which improved recognition accuracy to over 90% [53]. Figure 3 illustrates the recognition effect on bitter melon. While single-feature analysis methods can detect fruits in natural environments, they may struggle in complex orchard structures where shadows, branch and leaf occlusions, and clustered overlapping fruits adversely affect feature extraction. Research indicates that integrating multiple features, such as color, texture, and shape, can enhance both the accuracy and robustness of target detection [54]. The comparison of experimental results using traditional digital image processing techniques is shown in

Table 3. Experimental results of traditional digital image-processing techniques.

Recognition Algorithm	Advantages	Disadvantages	Accuracy/%
Based on color [46,47,48]	Can significantly distinguish fruit objects.	Significantly affected by lighting.	80–85
Based on shape [49,50]	Can acquire the contour information of fruit objects.	Clearly influenced by branch and leaf occlusion and fruit size.	80–87
Based on texture [51,52,53]	Can separate fruit and background information well.	Clearly influenced by environmental factors such as lighting and branch and leaf occlusion.	75–90

.

• Image segmentation

Machine learning can uncover patterns in datasets that are difficult to summarize through traditional analysis methods. This process involves using a training dataset to adjust the model parameters so it can effectively learn from the data. During training, a validation dataset is used to assess the model performance. Once the model is trained and validated, it can be applied to new data for prediction or decision making. Image segmentation algorithms leverage features such as color, shape, and texture to identify fruits. These algorithms include threshold segmentation, chromatic aberration, K-means clustering, region growing methods, and support vector machines [55]. Han et al. proposed a fiber image segmentation algorithm that integrates K-means clustering with the Gradient Vector Flow Snake model. This approach effectively addresses issues such as false edges and discontinuities that are common in traditional methods [56]. Chinchuluun et al. applied a Bayesian classifier for citrus image segmentation [57]. Jiang et al. employed a 2R-G-B color difference method and adaptive thresholding algorithm for apple image segmentation [58]. Lv et al. utilized Otsu dynamic thresholding for apple fruit identification and extraction [59]. Xu et al. used a 1.5R-G super-red image for threshold segmentation, extracting apple targets and defining contours with the Snake model, while a distance-based corner detection algorithm was used, as shown in Figure 4 [60]. Gongal applied histogram equalization in the HIS color space, combined with RGB color space methods, Otsu threshold segmentation, Hough transform, and Blob analysis for apple identification [61]. Support vector machine is a type of feedforward neural network and belongs to the category of supervised statistical learning algorithms. It is often used in combination with features such as the target’s color, texture, and current state. Ji et al. developed an apple image segmentation method based on region growth and color features, and also proposed an SVM-based classification algorithm for apple recognition [62]. Liao et al. used the Random Forest algorithm to classify and identify green apple fruits, performing Otsu thresholding and filtering based on the RGB color space. The average recognition accuracy achieved was 88% [63]. Due to discrepancies between training images and real-world conditions, Lv et al. suggested morphological operations combined with multi-class SVM for improved segmentation and recognition efficiency, achieving a recognition rate of 92.4% [64]. Zhang et al. focused on tea shoots against a blue background, extracting morphological, texture, and HOG features, and compared SVM, Random Forest, and K-Nearest Neighbors models for post-picking classification, finding Random Forest to be the most effective, with an accuracy of 97.06% [65].

In natural lighting environments, fruits can be obscured by branches and leaves or may overlap with each other, which adversely affects their recognition. To address these challenges, Tao et al. proposed a modified feature extraction method that combines RGB and HSI color components from point cloud data with 3D geometric features obtained using the FPFH descriptor [66]. This approach enhances the robustness of fruit characterization through color feature analysis and employs a genetic algorithm-optimized support vector machine to automatically recognize apples, branches, and leaves, making it effective in complex recognition environments. Wachs et al. introduced a method that integrates infrared and color images [67]. By leveraging the different temperature characteristics of apples and branches, the apple location is initially identified in the infrared image, which is then fused with the color image. Haar features are applied to both the color and infrared images to complete the identification and localization of the fruit. Niu utilized the Shape Context algorithm to match extracted apple contours and subsequently extracted the symmetry axis of the contour for apple localization [68]. Hu et al. developed a method to recognize and localize occluded fruit by determining local maxima through calculating the minimum distance from a point inside a circle to its edge, thereby identifying the circle center and radius [69]. This method demonstrated better performance when occlusion was minimal. The aforementioned image segmentation algorithms can effectively isolate target objects from the background, providing boundary and shape information that aids in subsequent recognition and processing. However, these algorithms are sensitive to variations in lighting, which can significantly impact segmentation accuracy, particularly in natural scenes where lighting conditions frequently change [70]. Additionally, image noise can distort segmentation results, leading to either mis-segmentation or missed segmentation. To mitigate these issues, preprocessing steps are necessary to reduce the impact of noise. Consequently, further optimization of image segmentation algorithms is required to enhance the efficiency of picking robots [71]. The comparison of machine learning image segmentation techniques and classifier experimental results is shown in

Table 4. Image segmentation techniques and their comparative characteristics.

Image Segmentation	Advantages	Disadvantages	Accuracy/%
Bayesian classifier algorithm [57]	Performs well on small-scale datasets and has a fast response time.	Significantly affected by the training set, and not well-suited for environments with strong lighting conditions.	81
SVM algorithm [62,63]	High accuracy, performs well in classifying data outside the training set, and has simple computations.	SVM is sensitive to noisy data and outliers, and has a longer training time.	90–93
KNN clustering algorithm [65]	High accuracy and insensitive to outliers.	Sensitive to irrelevant features, its effectiveness depends on the chosen distance metric, and it is computationally intensive and slow.	90
K-means clustering algorithm [56]	Short processing time, quick response, good clustering results, and capable of separating fruits from the background.	Sensitive to outlier data and requires the pre-setting of the K value.	90
Random Forest [63]	High accuracy, strong resistance to overfitting, robustness.	Slow prediction speed, high computational and overhead costs.	96

• Deep Learning

Deep learning-based target recognition technologies can autonomously learn and identify target features, demonstrating considerable robustness to occlusions and variations in lighting. Recent studies have explored various deep learning methods for fruit and vegetable recognition, with convolutional neural networks such as Faster R-CNN, Mask R-CNN, SSD, and YOLO proving effective in target identification [72,73]. Yu et al. introduced Mask-RCNN, utilizing ResNet-50 as a backbone and integrating a Feature Pyramid Network (FPN) architecture for strawberry feature extraction [74]. The Region Proposal Network (RPN) is trained end-to-end to generate region proposals for each feature map. This approach, which visually localizes strawberry picking points from images of ripe fruits, achieved an average detection accuracy of 95.78% in instance segmentation across 100 test images. Du et al. employed an enhanced Faster R-CNN model for recognizing anisotropic cotton fibers, addressing the wide variety in sizes and shapes of these fibers [75]. This model utilized ResNet-50 instead of VGG16 for feature extraction and improved candidate frame generation through the k-means++ clustering algorithm. Peng et al. developed an improved SSD deep convolutional neural network based on ResNet-101, achieving an average fruit recognition accuracy of 89.53% with notable effectiveness and robustness [76]. Tian et al. proposed an advanced YOLOv3 model for detecting apples at various growth stages in orchards, accommodating changes in fruit characteristics such as size, color, and growth density [77]. Wan et al. introduced a multi-fruit recognition method based on an enhanced Faster R-CNN, demonstrating superior accuracy and speed compared to traditional image segmentation algorithms [78].

In addition to the neural network algorithms mentioned above, the YOLO algorithm stands out for its real-time, high-precision, adaptive, and resource-efficient capabilities, making it an ideal choice for target detection in picking robots. YOLO operates as a single neural network that performs all target detection tasks simultaneously. With iterative updates to its various versions, its precision and recall have significantly improved. YOLO demonstrates significant advantages in evaluating the ripeness of fruits and vegetables. Anna et al. proposed a YOLOv3-based algorithm for recognizing apples at different maturity levels in complex environments [79]. This approach achieved an average detection time of 19 ms per apple, though 7.8% of objects were misidentified as apples and 9.2% of apples were not recognized. Miao et al. proposed an improved YOLO v7 network model for assessing apple ripeness, replacing the original CIoU loss function with WIoU. This modification not only enhances detection accuracy, but also accelerates model convergence [80]. Yan et al. developed a real-time recognition method for apple-picking robots based on YOLOv5, providing visual guidance for manipulators to adjust their orientation and avoid fruit obstructions caused by branches. The improved network model is shown in Figure 5 [81].

Li et al. proposed an enhanced YOLOv5 model to address challenges in detecting citrus fruits in natural environments, including issues with small, dense, and occluded targets [82]. The improved model was tested in various conditions, including sparse and dense environments, as well as backlighting, with the results depicted in Figure 6. Li et al. also introduced the I-Yolov8 model, which adds a small target detection layer to the YOLOv8 framework, improving detection accuracy by 16.33% in sunny conditions and 14.63% in cloudy conditions compared to YOLOv8 [83]. Du et al. enhanced the SSD-MobileNet algorithm by replacing the VGG16 network with a decomposable convolutional MobileNet network, resulting in improved model recognition speed [84]. Rahnemoonfar et al. optimized the Inception-ResNet network, enabling it to effectively recognize fruits even when obscured by shadows, branches, leaves, or overlapping clusters, achieving an average test accuracy of over 90% [85].

Deep learning algorithms effectively detect fruits and vegetables in complex environments, balancing efficiency and accuracy while remaining relatively unaffected by environmental conditions. Convolutional neural networks (CNN), a subset of deep learning, are currently the most suitable for fruit and vegetable detection in natural settings and are widely used in this domain. Two-stage detection algorithms, such as R-CNN and Faster R-CNN, offer high precision and recall in detection and identification tasks. In contrast, single-stage algorithms, like YOLO, SSD, and RetinaNet, provide faster processing speeds but generally exhibit lower precision and recall compared to two-stage methods. The comparison of experimental results for convolutional neural networks in deep learning is shown in

Table 5. Comparison of experimental results for deep learning networks.

Deep Learning	Advantages	Disadvantages	Accuracy/%
Residual Neural Network [75,85]	Increase the depth of the network while reducing the number of network parameters.	Training data may be overfitted; high complexity.	85–87
Mask R-CNN [74]	High recognition accuracy.	Complex to implement, limited real-time performance, and high computational cost.	89
SSD [76]	High recognition accuracy, strong generalization and robustness, and very fast detection speed.	Lower accuracy for small objects and poor performance in dense scenes.	90
YOLO [79,80]	Fast recognition speed and relatively high accuracy.	Complex training, lower accuracy for small object detection, and less robustness to object scale variations.	89–95

.

3.1.3. Targeting Algorithm

Once the fruit has been recognized, determining its size and position within the image and calculating its location in real space are crucial steps. This process involves using stereo vision techniques, depth cameras, or other sensors to measure the target distance and position. Previously, stereo vision methods, RGB-D cameras, and laser rangefinders were employed to acquire depth information for apples [86]. Key challenges in accurately locating the target fruit include the effects of adverse factors such as natural wind, machine vibrations, and fruit displacement or overlapping during the localization process.

Baeten et al. calculated the distance from the camera to the apple target using the relationship between focal length, pixel size in the image plane, and the center of the apple [87]. Ling et al. employed a structured light SR300 depth camera for brown mushroom harvesting [88]. By combining multiple soil surface depth measurements, this method enables adaptive threshold selection for generating a binary map of mushroom caps. Kong et al. introduced an apple image-matching method based on the absolute value of image gray-level differences, which reduces 3D coordinate error to less than 10 mm [89]. However, this approach has limitations in matching accuracy due to some unmatched feature points. Xiao et al. developed an algorithm integrating a BP neural network with the Hough transform for fruit localization [90]. This algorithm trains a color recognition model using the BP neural network, extracts apple contours through morphological operations, and locates the apple using the Hough transform to determine circles. Test results demonstrate that this method effectively localizes shaded apple targets under varying lighting conditions. To address issues with overlapping and occluded fruits, Tian et al. utilized an RGB-D camera to obtain gradient information from apple depth images [91]. By projecting the gradient vectors from 3D to 2D space and aligning them, Tian’s method identifies the center of the apple circle, overcoming traditional limitations of fruit identification and localization in the presence of occlusions. In real-world picking scenarios, disturbances such as natural wind or fruit collisions can cause oscillations, adversely affecting target localization. To mitigate this, Lv et al. proposed an improved localization method for oscillating fruits, which correlates front and back frames of apple images to enable fast tracking and identification [92]. This approach adjusts the robotic arm picking speed in real time according to the fruit oscillation period, allowing for picking during stable periods. Tanigaki et al. used infrared laser ranging technology to determine the 3D positions of target fruits and obstacles and used this data to plan the end-effector motion trajectory to avoid collisions [93].

In general, multi-camera stereo vision matching technology can effectively expand the perception range of vision systems and enhance localization accuracy. However, its performance is significantly impacted by variations in lighting and object texture, and it is ineffective in low-light environments. RGB-D depth cameras offer high measurement accuracy at close ranges and are less influenced by natural light and fruit texture. Nevertheless, they have a limited measurement range, lower image resolution, and reduced accuracy in detecting and localizing object edges. Laser ranging sensors provide high accuracy, fast measurement speed, simple structure, and strong resistance to interference for 3D positioning. However, they face limitations when the measurement distance is excessive, which may cause the laser to lose focus, and obstacles such as branches and leaves can obstruct the laser, preventing it from reaching the target.

3.2. End-Effector Structure

Utilizing data provided by the vision system, the robot precisely controls its arm, gripper, and other components to execute accurate harvesting actions. As the end-effector approaches the fruit, it is activated to gently harvest the produce using either gripping or suction techniques. Upon successfully securing the fruit or vegetable, the robotic arm transfers it to a collection basket or conveyor belt, thereby completing the harvesting process. The end-effector is a critical component of the picking robot system. Given that most fruits have delicate outer skins and grow in varying environments, it is essential to minimize damage during the picking process. The design of the end-effector significantly influences the extent of fruit damage, making it crucial to analyze the biological characteristics of the fruits to develop an end-effector best suited for the specific type of fruit being harvested. The actuation methods for the end-effector primarily include electric, pneumatic, and hydraulic drives. Electric drives are characterized by precise control and fast response, making them suitable for delicate operations involving lightweight fruits. Pneumatic drives are structurally simple and well-suited for large-scale, rapid harvesting tasks. Hydraulic drives, offering high output force, are ideal for handling heavier crops. Currently, common types of end-effectors are categorized into three main types: clamping, suction, and blade shear [94]. Their respective characteristics are detailed in

Table 6. Classification of end-effectors and their characteristics.

Kind	Outlined	Characteristic
Clamping [95,96,97,98]	The jaws close slowly to grip the fruit with appropriate force and separate the fruit by rotating or pulling with the gripper.	Consisting of two or more jaws, the jaws are usually made of flexible material in order to avoid damaging the fruit; this structure is designed to adapt to different types of fruit and has the widest range of applications.
Suction [99,100,101]	Negative pressure is generated by the air drive to suck the fruit; the process needs to be knotted, using shear or rotating and other mechanical auxiliary action.	The suction cups are usually made of flexible material, which reduces physical damage to the surface of the fruit, and the suction and release action is quick, which can significantly improve the picking efficiency.
Shear [102,103,104,105]	The cutting head is aimed at the stalk and cuts the stalk by means of a fast-closing electric or pneumatic drive system, usually equipped with a gripper to prevent the fruit from falling.	Simple design of the shearing mechanism, wide range of applications, able to maintain the integrity of the fruit.

.

3.2.1. Clamping

Hand claws can be categorized into two types based on their configuration: two-fingered and multi-fingered designs. These end-effectors are capable of mimicking human picking actions, such as twisting and pulling, and are among the most widely used in fruit harvesting. Dimeas et al. developed a strawberry-picking gripper and identified the lateral pinch gripper as the optimal end-effector through a mechanical analysis of the structural characteristics of strawberries [95]. Wei et al. designed an under-driven citrus picking robot end-effector featuring a dual-linkage parallel finger design [96], allowing it to pick citrus of varying sizes by either gripping or pinching, as illustrated in Figure 7. This design minimizes fruit damage. Bulanon et al. introduced a two-fingered twisting end-effector for apple picking, which uses a direct current motor to drive the fingers that grasp the fruit stalk, while a stepper motor rotates the drive arm to separate the fruit [97]. Xu et al. developed a composite end-effector combining air suction and clamping technologies [98], as shown in Figure 8. This design reduces collisions between the end-effector and the fruit, enabling safer and more effective picking of navel oranges. Clamping end-effectors can be applied to a variety of shapes and sizes of objects and are very flexible in different picking tasks, and this structure robot technology is relatively mature and easy to maintain and repair. However, the working scenario may contain multiple movable joints and sensors, which will increase the complexity and cost of the equipment, and in high-speed work, it will cause some damage to soft or fragile fruits. Clamping end-effectors are versatile and adaptable to various object shapes and sizes, making them highly flexible for different picking tasks. This technology is relatively mature, with straightforward maintenance and repair. However, the presence of multiple movable joints and sensors in the working scenario can increase the complexity and cost. Additionally, high-speed operation may cause damage to soft or fragile fruits.

3.2.2. Suction

Suction cups, typically made of soft rubber or silicone, are designed to adapt to various surface shapes, enabling the fruit to be lifted by the negative pressure generated by an air actuator. Baeten et al. developed a funnel-shaped flexible end-effector [99], as shown in Figure 9, that uses vacuum adsorption to pick apples with diameters ranging from 6 to 11 cm. This device effectively enables non-destructive apple picking. Liu et al. designed a tomato-picking robot equipped with a vacuum suction cup that detaches tomatoes from branches and stems using suction [100], followed by a finger gripping mechanism that secures the tomatoes to prevent them from falling during the picking process. Additionally, this system includes a sensing mechanism capable of real-time distance control relative to the branches and stems. Liu et al. also developed a brown mushroom-picking robot featuring a vacuum-supported end-effector, achieving a 90.5% success rate and a 6% fruit damage rate in 400 trials on fresh, ripe brown mushrooms [101]. The suction end-effector operates by generating a vacuum to adhere to objects without directly contacting their surfaces, which helps minimize physical damage to fragile or delicate items. The adsorption and release processes of suction cups are typically rapid and efficient. However, their effectiveness is limited when dealing with irregularly shaped or unevenly surfaced fruits.

3.2.3. Blade Shear

Shear manipulators are typically either scissor-type or monolithic, and are driven by electric, pneumatic, or hydraulic systems that control the opening, closing, and cutting actions. These manipulators require a control system to precisely manage the position and force of the shear, making them widely applicable across various tasks. Xu et al. proposed a shear-type apple-picking robot end-effector, consisting mainly of a shear actuator, a powered mechanical arm group, and a support base, which effectively reduces damage during apple harvesting [102]. Lv et al. designed a shear end-effector for picking oil camellia flowers, characterized by its simple structure, flexibility, and ability to smoothly cut the flower stems [103]. Zhang et al. developed an elliptical trajectory occlusal end-effector for the rapid shear picking of dragon fruit, using leaf ribs as a positioning reference [104]. They identified 0.5 MPa as the optimal air pressure for driving the end-shear hand, achieving a 90% picking success rate and an average single-fruit picking time of 0.46 s. Chen et al. designed a lychee-picking robot end-effector with a single-power-source drive and an integrated clamping and shearing system [105], as shown in Figure 10. This robot exhibits strong clamping capacity and can cut parent branches with diameters ranging from 3 mm to 7 mm, with an average clamping and shearing time of 2 s. In general, for fruits and vegetables with hard or thick stems, such as grapes, persimmons, and lychees, the shear end-effector is effective in ensuring that stems are cut cleanly without damaging the fruit or vegetable itself. However, the shear-type robot has certain disadvantages. Compared to clamping-type end-effectors, it demands higher precision in the identification and localization capabilities of the fruit-picking robot. Additionally, for fruits with short stems, the shear mechanism faces spatial limitations, increasing the risk of damaging the fruit.

4. Existing Deficiencies

Many scholars have conducted extensive research on picking robots and achieved notable results. However, picking technology remains largely in the research stage. Currently, most picking robots are significantly less efficient than manual labor and have not yet reached commercial viability. While fruit- and vegetable-picking robots show considerable potential in modern agriculture, they also encounter several technical and practical challenges. The following are some of the main shortcomings of current fruit- and vegetable-picking robots.

4.1. Deficiencies in the Visual System

4.1.1. Low Recognition Rate

The vision system’s recognition accuracy of current autonomous picking robots requires further improvement. The primary challenge is the difficulty of the machine vision system in accurately differentiating and recognizing target fruits and vegetables within complex environments. Variations in natural lighting conditions can affect image quality, which in turn impacts the performance of the recognition system. Additionally, shadows and reflections can interfere with accurate visual assessment. In field environments, the presence of plant branches, leaves, weeds, and soil creates a complex background, complicating the task of separating the target object from its surroundings. Furthermore, the diversity in shapes and colors among different fruits and vegetables due to variations in maturity, size, and color adds to the challenge of machine recognition. Occlusions caused by overlapping fruits, vegetables, or branches can render some targets invisible, further reducing the recognition rate.

4.1.2. Poor Positioning Accuracy

The significant deviation between the target localization and its actual position can be attributed to several factors, including errors in target detection, mechanical vibrations, and inaccurate distance measurement sensors. The physical structure and kinematic modeling of the robotic arm may lack precision, causing the end-effector to miss the desired position. This issue is particularly pronounced under long-distance or low-light conditions, where sensors such as cameras and LiDAR may exhibit measurement inaccuracies. Additionally, fruits and vegetables can oscillate due to wind or slight movements from gravity, which can cause the target fruit to shift and render pre-computed positional information invalid. Mechanical vibrations generated by the robot itself further exacerbate these discrepancies, leading to significant errors in localization and affecting the accuracy of the end-effector grasp. Furthermore, delays in the image capture, processing, and decision-making stages can result in changes to the target position during the robot’s actions, further impacting the overall positioning accuracy.

4.2. Fruit Injury

The maturity and variety of fruits and vegetables influence their hardness and fragility, making them susceptible to damage during mechanized picking operations, which can lead to a decline in fruit quality. The primary causes of fruit damage include: First, the robot recognition accuracy may be compromised by the complex and variable natural environment, such as changes in lighting, diverse fruit colors and shapes, and the presence of branches and leaves, leading to unintended contact between the end-effector and the fruit. Second, precisely controlling the grasping force of the end-effector is challenging. The robot must adjust this force carefully to avoid crushing or tearing the fruit, particularly when handling soft or exceptionally delicate varieties. Current technology may struggle to adapt perfectly to all fruit types. Additionally, the design of the robot arm and gripper must ensure that the picking action does not damage the fruit. Existing designs may lack the necessary flexibility and pliability, especially in densely planted or intricately branched environments, which can result in mechanical structures that are not sufficiently adaptable to avoid obstacles that could potentially damage the fruit.

4.3. Insufficient Efficiency and Speed

Compared to human pickers, current picking robots exhibit significantly slower speeds. This issue becomes particularly pronounced in high-density planting scenarios, where robots struggle to locate and harvest fruits efficiently, falling short of human-level performance. Additionally, picking robots typically rely on battery power, which limits their operational endurance. Batteries must be recharged or replaced once depleted, posing particular challenges for large-scale farms. Extended operation can also result in overheating and wear on the robot components, impairing continuous functionality. Moreover, long-term use of picking robots may lead to various malfunctions, such as mechanical wear, sensor failures, and software errors. These issues not only disrupt harvesting operations, but can also cause fruit damage or picking failures, adversely affecting crop yields.

4.4. Costs and Maintenance

Cost and maintenance represent the primary challenges for fruit- and vegetable-picking robots in real-world applications, significantly impacting their popularity and commercial viability. These robots integrate advanced sensors, complex mechanical systems, high-performance processors, and sophisticated software algorithms, all of which contribute to high development and manufacturing costs. Continuous research and development investments aim to enhance robot performance, including improving recognition rates, positioning accuracy, and picking efficiency. The intricate nature of these robots necessitates a dedicated maintenance team to conduct regular inspections, cleaning, lubrication, and calibration to ensure optimal operation. Over time, wear and tear on components, particularly non-standard or customized parts, is inevitable, and their replacement can be costly. Such factors contribute to the overall expense of deploying and maintaining picking robots.

5. Future Prospects

5.1. Optimization of Recognition Algorithms

Optimizing the vision algorithm of autonomous picking robots is crucial for achieving efficient and accurate fruit harvesting. The intelligent recognition and localization algorithms within the vision system must handle various challenges, including obstacles such as branches and trunks, diverse weather conditions, variable lighting, uneven colors, shadows, oscillations, overlap, and occlusion. Current algorithms still require improvement in the accuracy of fruit target recognition and localization. To address these challenges, there is a need for advanced image recognition and understanding capabilities that can manage complex lighting conditions, background interference, and morphological changes in fruit. Additionally, developing more lightweight and computationally efficient deep learning models is essential to reduce resource consumption and enhance real-time processing. Furthermore, adaptive algorithms should be developed to dynamically adjust parameters based on different crop types, growth stages, and environmental conditions, thereby improving picking success rates.

5.2. Farm Renovation

To effectively integrate harvesting robots, future farms will need to implement several modifications to create environments better suited for mechanized operations. These changes will not only enhance the efficiency and accuracy of the robots, but also contribute to the overall modernization of agricultural production. Optimizing row and plant spacing is crucial to ensure that robots can travel and operate smoothly, minimizing the risk of collisions or missed crops. Adopting linear and evenly distributed planting patterns, such as the trunk type, V trellis, T trellis, high spindle, and tree–wall configurations, along with constructing regular planting areas and durable, smooth access roads, will facilitate efficient path planning and navigation for the robots. These standardized layouts will help reduce energy consumption and minimize time wastage.

5.3. Cost Reduction

To comprehensively assess the economic feasibility of harvesting robots, a thorough comparison between the costs of manual labor and automated solutions is essential. Manual labor costs typically include wages, benefits, training expenses, and related labor management costs. With rising minimum wage levels and a tight labor market, these costs may continue to increase. Additionally, manual labor may face issues such as low production efficiency, inconsistent work quality, and high labor intensity, which can affect overall production effectiveness. In contrast, automated solutions like harvesting robots, while requiring a higher initial investment covering equipment purchase, installation, and calibration, generally have lower long-term operating costs. Robots can maintain high efficiency over continuous working cycles, reducing production fluctuations caused by human factors. Maintenance costs and power consumption are the main operational expenses, but these costs are relatively stable and predictable. Furthermore, robots can reduce dependence on seasonal labor and lower the risk of production interruptions due to labor shortages. To fully understand the cost differences between these two approaches, various factors need to be considered. For instance, manual labor costs are influenced by minimum wage standards, labor supply conditions, and the work environment, while robot costs are affected by technological advancements, equipment updates, and maintenance technologies. A detailed economic analysis of these factors can help to determine the payback period for investing in harvesting robots and assess their long-term economic benefits.

5.4. Environmental Adaptation and Functional Diversity

Future picking robots should exhibit enhanced environmental adaptability, enabling them to operate efficiently under various weather conditions. They must be designed to withstand strong light, rain, and wind, facilitating all-weather, uninterrupted harvesting operations. Beyond their primary picking functions, these robots should integrate additional agricultural tasks, such as plant protection, fertilization, and monitoring. Developing versatile robots that can be adapted to a wide range of crops will enhance their applicability across different farms. This increased versatility will significantly improve robot utilization, reduce farmers’ input costs, and advance the automation and intelligence of agricultural production.

5.5. Innovation and Technology Integration

Integrating picking robots with advanced technologies such as drones, 5G communication, blockchain, and new materials is crucial for improving agricultural automation. Drones can collaborate with robots to provide real-time aerial monitoring and data collection, optimizing harvesting efficiency, and 5G technology enables fast, stable communication, allowing seamless coordination between robots and drones. Blockchain can enhance traceability and food safety by tracking the production and distribution of crops. Meanwhile, new material technologies will drive the development of flexible robotic arms, improving adaptability and reducing damage to delicate produce. These innovations collectively enhance the precision, efficiency, and sustainability of modern farming. Moreover, these integrated technologies promote greater resource efficiency by reducing waste and optimizing the use of inputs like water, fertilizers, and energy. By enabling more precise and data-driven farming, they contribute to reducing the environmental impact of agriculture, helping to meet the growing demand for food in a more sustainable and responsible manner.

5.6. Proposal for Policy and Farm Management

To further promote the adoption of agricultural robots, governments should increase support for agricultural robotics technologies by providing more funding, favorable policies, and training programs. This support would include investments in research and development, the formulation of preferential policies, and educational programs for farmers, helping them to better understand and utilize picking robots. In the future, farms should adopt multiple intelligent robots and systems to achieve full automation in management. Robots will be interconnected through IoT technologies, working collaboratively to handle tasks such as planting, picking, transportation, and packaging. Such intelligent farm systems will significantly improve agricultural productivity and reduce dependence on manual labor.

6. Conclusions

Agricultural picking is labor-intensive and influenced by environmental factors, highlighting the need for modernization in this field. This paper reviews the current research on fruit- and vegetable-picking robots both domestically and internationally, focusing on key technologies such as vision system sensors, target recognition, localization, and end-effector structures. It introduces various target recognition algorithms, including traditional image feature-based methods, machine learning-based image segmentation algorithms, and deep learning-based neural network algorithms. The paper also analyzes target recognition and localization methods in complex environments characterized by branch and leaf obstructions, fruit overlap, and oscillation.

The paper further explores the three main types of end-effectors used in fruit- and vegetable-picking robots—gripping, suction, and shearing—detailing their characteristics and evaluating the advantages and disadvantages of each structure. It summarizes the current limitations of agricultural picking robots, including issues with vision system performance, fruit damage, research and development costs, and picking efficiency and speed. The paper concludes with an outlook on the future of picking robots, addressing potential improvements in recognition algorithms, farm renovation, cost reduction, environmental adaptation, functional diversity, technological innovation, and policy and farm management. Although picking robots currently face challenges in work efficiency and automation and have not yet reached widespread commercialization, advances in new technologies and ongoing research are expected to yield breakthroughs in key technologies and design. As the concept of agricultural modernization evolves, fruit- and vegetable-picking robots are anticipated to become a vital component of modern agriculture.

In light of the current limitations of picking robots, our research plan aims to optimize several key components. For the design of the machine, we will adopt a structure that is both simplified and operationally efficient, ensuring it aligns with practical field conditions. In the vision algorithm, we will implement higher-precision and more efficient recognition algorithms, allowing for accurate fruit identification and localization. Regarding the end-effector, we will focus on minimizing fruit damage by employing a structure specifically designed to reduce the rate of injury. For other hardware components, our objective is to enhance the power and efficiency of the picking process while maintaining cost-effectiveness.