Design of and Experiment with a Dual-Arm Apple Harvesting Robot System

Abstract

Robotic harvesting has become an urgent need for the development of the apple industry, due to the sharp decline in agricultural labor. At present, harvesting apples using robots in unstructured orchard environments remains a significant challenge. This paper focuses on addressing the challenges of perception, localization, and dual-arm coordination in harvesting robots and presents a dual-arm apple harvesting robot system. First, the paper introduces the integration of the robot’s hardware and software systems, as well as the control system architecture, and describes the robot’s workflow. Secondly, combining a dual-vision perception system, the paper adopts a fruit recognition method based on a multi-task network model and a frustum-based fruit localization approach to identify and localize fruits. Finally, to improve collaboration efficiency, a multi-arm task planning method based on a genetic algorithm is used to optimize the target harvesting sequence for each arm. Field experiments were conducted in an orchard to evaluate the overall performance of the robot system. The field trials demonstrated that the robot system achieved an overall harvest success rate of 76.97%, with an average fruit picking time of 7.29 s per fruit and a fruit damage rate of only 5.56%.

1. Introduction

According to statistics, global annual apple production has exceeded 80 million tons. With the increasing global demand for fresh fruit, apple production has been on an upward trend year by year [1]. In orchard production, harvesting is the most labor-intensive and the hardest task to mechanize, currently still relying mainly on manual labor. To further improve labor productivity and reduce labor intensity, automating the harvesting process has become an urgent need for the development of the fresh fruit industry [2,3,4]. With the rapid development of fundamental technologies such as computers, machine learning, and sensors, harvesting robots have made breakthroughs in precise target perception, task planning, intelligent control, and flexible grasping [5,6,7], but some challenges and technical difficulties remain. Traditional single-arm harvesting is inefficient, and considering the short harvesting period in orchards, it is necessary to adopt a multi-arm robot solution to improve efficiency. Multi-arm coordination can expand the working range and increase the harvesting density per unit space, thereby overcoming the efficiency limitations of single-arm harvesting [8]. However, multi-arm harvesting robots still face several technical challenges in practical applications. The increase in the number of operational units significantly raises the complexity of the overall system architecture and increases the likelihood of spatial interference between robotic arms. When multiple robotic arms perform harvesting tasks, planning conflicts and resource competition may arise, posing new challenges for task coordination optimization and resource allocation algorithms [9]. In the highly unstructured orchard environment, the fruits, leaves, and branches of plants are densely intertwined, randomly distributed, and mutually occluded, significantly increasing the difficulty of precise target perception and information acquisition [10]. Additionally, the complexity of lighting conditions sets higher standards for the robustness and accuracy of the robot’s perception system [11].

Silwal et al. developed a seven-degree-of-freedom apple harvesting robot for planar canopy apple trees [12]. The robot uses an open-loop, three-fingered end-effector to harvest apples, achieving an average harvest success rate of 84% and an average picking time of 6 s per apple. Abundant Robotics developed an apple harvesting robot that uses a parallel robotic arm to operate a vacuum-suction end-effector for fast fruit picking [13], achieving an average harvesting efficiency of 2 s per fruit on V-shaped canopy trees. Williams et al. designed a kiwifruit harvesting robot with four three-degree-of-freedom serial joint robotic arms [14]. The arms collaborate in harvesting operations and effectively avoid interference, with a harvesting efficiency of 2.78 s per fruit. FFRobotics developed an apple harvesting robot that uses 12 symmetrically arranged robotic arms for parallel picking [15], with an efficiency of about 1.8 s per fruit. Xiong et al. developed a strawberry harvesting robot that uses two Cartesian robotic arms and flexible grippers to work collaboratively [16], achieving a harvesting time of about 4.6 s per fruit, reducing the time by 1.5 s compared to the single-arm mode. Zhao et al. developed an apple harvesting robot with a pneumatic-electric combination-driven end-effector, using infrared photoelectric and pressure sensors for feedback during fruit harvesting tasks [17]. The prototype achieves a harvesting efficiency of 7.81 s per fruit and a success rate of 80%. Under the conditions of a dwarf-dense orchard, Li et al. designed a muti-arm parallel-harvesting robot [18], with each arm equipped with a stereo vision sensor and an end-effector using a curved three-fingered gripper. Experiments showed that the robot’s collaborative harvesting efficiency improved by 150% compared to single-arm picking, with a fruit damage rate of less than 8%. In highly unstructured orchard environments, occlusion by branches and leaves and overlapping fruits have the most significant impact on the accuracy of apple recognition and localization. To address this issue, Lin et al. used local shape matching and probabilistic Hough transform methods to detect and aggregate fruit segments under occlusion [19], achieving a fruit detection accuracy of 84.8%. Yan et al. used an RGB-D camera to acquire the image information of apples and developed a model for apple recognition and maximum diameter estimation by combining multiple regression analysis and nonlinear surface fitting methods [20]. Experimental results showed that the model’s average maximum relative error was ±4.1%. Jia et al. used a pulse-coupled neural network to extract typical fruit features and developed a new GA-Elman algorithm to recognize overlapping apples [21], achieving an accuracy rate of 88.67%. Zhao et al. proposed an improved model, YOLOv8s-CFB, based on YOLOv8s, to enhance the detection capability for occluded targets [22]. The model incorporates partial convolution (PConv) and a bidirectional feature pyramid network (BiFPN) and replaces the original SPPF module with focal modulation technology. Test results demonstrated that the model achieved an average recognition accuracy of 93.86%. Jing et al. proposed a VGG16-based Faster R-CNN recognition algorithm for occluded apples [23], achieving a recognition accuracy of 91%, though the recognition efficiency was relatively low. Wang et al. integrated an EMA attention mechanism and DCNv3 deformable convolution with the YOLOv8 model to propose an improved PAE-YOLO model for the recognition and detection of Yunnan Xiaomila [24]. Experimental results showed that the model achieved an accuracy of 88.8%, which was 1.3% higher than the original model. To achieve accurate fruit localization, it is necessary to rely on three-dimensional information such as fruit point clouds and depth. Xu et al. proposed a 3D reconstruction and angular fruit recognition and localization method for rapeseed branches based on a depth camera [25], achieving an average distance error of less than 0.48 mm by calculating the 3D point cloud obtained by the camera. Li et al. proposed a 3D fruit localization method based on a deep learning segmentation network and a new cone-based point cloud processing method [26], enabling the extraction of the position parameters and size of occluded fruits. Gene et al. used Structure from Motion (SfM) photogrammetry to generate 3D point clouds of detected apples and spatially locate the fruits [27,28]. Their multi-view method reduced the number of occluded fruits and improved the localization accuracy. Hu et al. proposed a method for apple object detection and localization based on improved YOLOX and RGB-D images [29]. Experimental results showed that the method achieved an average recognition accuracy of 94.09% and a detection speed of 167.43 FPS. To further improve picking efficiency, more robots are adopting multi-arm parallel collaborative operation modes. In multi-arm collaborative operations, one of the most challenging problems is how to reasonably plan and schedule the robotic arms according to the distribution of fruits [30]. Zhang designed a dual-arm kiwifruit picking platform that performs the partitioning of picking targets and clusters the detection of double-kiwifruit for damage-free picking [31], adopting an outward-to-inward picking strategy. Barnett et al. studied the task partitioning and reachability of multi-arm kiwifruit picking [32,33], developing a task planning evaluation and optimization method based on task allocation parameters in overlapping picking domains. This method balances proximity and arm workload, achieving a picking speed of 5.5 s per fruit. Mann et al. researched the combinatorial optimization strategy for Cartesian multi-arm fruit-picking robots [34], solving the multi-arm collaboration optimization problem using a team-oriented problem with time windows to determine the picking order of each arm. Li et al. described the multi-arm picking task planning problem as a multi-traveling salesman problem model [35], considering asynchronous overlapping constraints and solving the picking order using a dual-chromosome genetic algorithm, increasing the harvesting efficiency by 4.25 times. Furthermore, reinforcement learning methods offer new perspectives and solutions for multi-arm collaborative robot operations. Sun et al. studied an adaptive batching strategy based on deep Q-networks and double DQNs to address task allocation adaptation and dynamic batching issues [36]. Jiang designed a tea bud picking robot with four robotic arms based on arm specifications and the distribution of buds [37]. A KNN algorithm was used to build a picking point partition model to ensure balanced task allocation, and multi-agent reinforcement learning was applied to train the robotic arms for picking tasks.

How to accurately obtain the positional information of the fruit, how to optimize the design of the multi-arm workspace, and how to resolve spatial interference and task conflict issues are the key factors that determine the performance of multi-arm harvesting robots. In response to this, this paper proposes a novel dual-arm apple harvesting robot system. This system employs a Yolov8-based multi-task network architecture to achieve target detection and instance segmentation. The 3D positional coordinates of the fruit are calculated using a frustum-based fruit localization method. Additionally, the robot uses a genetic algorithm-based dual-arm collaborative operation planning method, which significantly improves the overall harvesting efficiency of the harvesting robot.

2. Materials and Methods

2.1. System Design

2.1.1. Hardware Design

As shown in Figure 1, the dual-arm robot developed in this study mainly consists of a wheeled mobile platform, a robotic harvester, and a computer control system. The wheeled mobile platform allows the robot to move between rows of apple trees, stopping at each tree for harvesting. The computer control system hosts all software algorithms and establishes communication links with all other components. The robotic harvester is the key component for performing the picking operations. In addition, the mobile platform is equipped with a 6 kW silent gasoline generator. Serving as the energy supply, this generator can continuously provide a 220 V power source for the robot’s operation.

As shown in Figure 2a, the robotic harvester mainly includes the picking actuator, visual perception system, and fruit transport and collection mechanism. The picking actuator consists mainly of two vertically aligned 4-DOF (degrees of freedom) robotic arms and two end-effectors. Figure 2b shows the relevant dimensions of the robotic harvester, where it is integrated into a wheeled mobile platform through eight M20 bolts. Each robotic arm comprises three prismatic joints and one revolute joint, featuring a compact structure that enables efficient operation within the workspace, as shown in Figure 3. The vertical moving component moves up and down, while the horizontal moving component moves laterally in the horizontal direction. The rotational component rotates around the central axis of the rotational joint, performing pitch movements. The arm’s telescopic joints use a combined electric and pneumatic dual-stage extension mechanism, enabling the frequent and efficient reciprocating movements of the end-effector between the canopy and the robot body. Pneumatic actuation is used for extension between the robot body and the canopy edge, while precise positioning within the canopy is achieved via electric servo-driven extension. The electric extension of the horizontal, rotational, and telescopic joints are driven by separated servo motors, while the vertical prismatic joints are powered by integrated servo motors. All motors are equipped with controllers and encoders, which precisely control the speed and displacement while providing real-time feedback on the position and speed, allowing the accurate measurement of the robotic arm’s positional and velocity changes during operation.

As shown in Figure 4, the end effector is a curved-surface, three-finger gripper with two degrees of freedom: rotation around the axis and opening/closing. It enables a “twist-and-pluck” method for harvesting target fruits. The main body of the gripper is made of aluminum alloy, which reduces its weight while maintaining stiffness. The fingers are made of 3D-printed nylon, a rigid material. During the picking process, although the clamping force does not exceed the threshold that would damage the fruit, the edges of the rigid fingers can still cause injury to the fruit. To prevent this, silicone finger sleeves, as shown in Figure 4b, are employed. These silicone sleeves are flexible materials that effectively protect the fruit’s surface during physical contact.

As shown in Figure 2, two RGB-D depth cameras are fixed above the picking robotic arms, allowing the vertical and horizontal movement of the arms. These two RGB-D depth cameras, together with the edge computing module, form the robot’s visual perception system, used for fruit detection and localization. The depth cameras capture perceptual visual data of the target fruit and its surrounding environment and transmit it to the edge computing module for inference, enabling detection and segmentation tasks during fruit localization. The fruit transport and collection mechanism consists of a directional elbow and a vertical conveyor, as shown in Figure 5. The rotating elbow is mounted on the vertical moving part and can move up and down synchronously with the arm. The vertical conveyor is driven by a servo motor, transporting the fruit delivered by the directional elbow to the outlet behind the conveyor. To prevent fruit damage during transport, flexible stops are attached to the conveyor belt surface, and the parts of the stops that contact the fruit are designed to be cylindrical. The lighting system includes four flexible LED strips, which are fixed to the vertical moving parts of both robotic arms and on both sides of the vertical conveyor to meet the needs of nighttime operations.

2.1.2. Control System Design

The computer control system consists of a high-performance industrial computer, an edge computing module, and several embedded controllers. Users can monitor and control the robot system through this setup.

As shown in Figure 6, this is the control system architecture of the harvesting robot. The RGB-D depth cameras acquire image data using various open-source software packages, such as OpenCV (version 4.7.0), Open3D (version 0.16), and PyTorch (version 1.10.0), and perform inference on the edge computing module. A task plan is implemented using reinforcement learning methods within the OpenAI-gym environment. The motion control of the robot’s picking operations is conducted using the Moveit! ( version 1.0) integrated development platform provided by the ROS system. The control and communication topology of the harvesting robot is shown in Figure 7. In terms of communication, the RGB-D depth camera and the computer communicate using a USB protocol, ensuring high compatibility. The edge computing module and the computer communicate via Ethernet, enabling efficient image data transmission. Communication between the computer and the actuator components is conducted using CAN signals, improving the stability and real-time performance of the data transmission. Components such as the cylinders, lighting, and vertical conveyor use relays compatible with the CAN communication protocol, combined with appropriate module I/O interfaces, to establish communication with the computer.

2.1.3. Workflow

The workflow diagram of the overall robot system is shown in Figure 8. First, the robot moves to an initial position to begin working, and the RGB-D depth cameras in the visual perception system capture the image data of the fruit distribution. The edge computing module receives the image data from the depth cameras and performs inference to determine the three-dimensional spatial coordinates of the target fruits. The visual perception system provides positional information for the robot’s dual-arm coordination and control, ensuring smooth collaboration between the robotic arms. Next, the task planning method based on a genetic algorithm assigns a corresponding picking task sequence to each robotic arm according to the fruit distribution. This will be further explained in subsequent sections. Then, the robot’s joints controller controls the servo motors to drive the respective picking arms, which pick each fruit in their assigned task sequence. After a successful pick, the fruit is placed into the directional elbow, and the next picking operation begins. If the picking operation fails, a second attempt is made. Each fruit can be attempted a maximum of two times. Successfully picked fruit is transferred via the directional elbow to the conveyor, where it is transported to the outlet at the back of the vertical conveyor and collected in a fruit crate. When all arms have completed their assigned task sequences, the picking task for the current position is complete. The robot then moves to the next position to continue harvesting.

2.2. Fruit Detection and Localization

2.2.1. Dual-Vision Fruit Information Acquisition System

Accurate fruit detection and localization are the primary prerequisites for the robot to perform fruit-picking tasks. The former aims to segment the fruit from a complex background. The latter calculates the 3D position of the detected target fruit to guide the end-effector to the picking target point. The dual-arm harvesting robot has a wide operational space, making it difficult for a single camera to cover the entire area. Therefore, this paper uses two depth cameras to achieve the full visual perception of the robot’s workspace. As shown in Figure 9, this is the dual-vision fruit information acquisition system. The RealSense D435i depth camera provides rich visual data for target detection and scene reconstruction. This camera is an active infrared stereo camera, consisting of two imagers and an infrared dot projector for depth measurement, capturing depth channel information. Additionally, it includes a color camera module that captures RGB channel data. The edge computing module chosen is the Jetson TX2, equipped with an NVIDIA Pascal GPU (with 256 CUDA cores), up to 8 GB of memory, 59.7 GB/s memory bandwidth, and various standard hardware interfaces, enabling the inference computation of visual data. The host computer directly connects to the depth camera to obtain depth and color image data, which is then transmitted to the edge computing module for inference. The edge computing module performs inference on the data and returns the results to the host computer.

2.2.2. Multi-Task Network Based on Improved YOLOv8

Object detection and instance segmentation are the foundation for achieving accurate localization. In our previous research, the team proposed a multi-task deep convolutional neural network with both detection and segmentation capabilities, called MultiTaskNet for Orchard (MTN-O) [18]. This network is an improvement upon the YOLOv8 architecture, designed to perform both the object detection of the entire fruit region and the instance segmentation of the visible fruit pixels simultaneously, as shown in Figure 10. A fully convolutional neural network branch was added at the bottom output of the standard YOLOv8’s FPN, applying continuous convolution and upsampling to finally output a segmentation mask of the original image size. When handling segmentation tasks, it effectively utilizes features and directly decodes outputs through the shared backbone and FPN. Through a series of convolution and upsampling operations, the network restores the feature map to the original image size and generates the segmentation mask using Conv2D layers. During the decoding phase, the segmentation branch improves segmentation accuracy by fusing shallow features from the backbone network.

This network can simultaneously perform object detection and instance segmentation tasks within a unified framework. It demonstrates excellent performance in detecting small objects and handling occlusions, making it particularly suitable for automated harvesting operations in complex orchard environments. Experimental results showed that the MTN-O network model achieved a detection success rate of 89.6%, which was 5.3% higher than the original model and significantly improved compared to traditional network models. Notably, it demonstrated superior performance in handling occluded and multi-scale targets, as illustrated in Figure 11.

2.2.3. Accurate 3D Localization of Fruit Based on Visual Cone Method

After completing the detection and segmentation of the RGB image, the camera’s imaging model is used to obtain the target’s 3D information. Using the calibrated stereo camera’s imaging parameters and depth map, a 3D point cloud for all pixels can be generated. However, due to factors such as complex and changing lighting conditions and fruit occlusion, the generated 3D points often exhibit distortion, holes, and outlier noise. Further processing is required to accurately and effectively calculate the 3D position of the target. To address this issue, this paper adopts a high-precision fruit localization method based on the frustum-based approach, with the following processing steps: The target detection branch of the MTN-O network can obtain and output the complete bounding box of the fruit. Combined with the camera’s internal imaging parameter matrix, a 3D visual cone can be constructed, with the camera focal point as the origin and the lines connecting the focal point to the bounding box vertices as the edges. Since the 2D image bounding box containing the complete outer contour of the fruit has been estimated, even in the case of partial occlusion, the generated point cloud of the target fruit in the RGB-D image will be fully contained within the visual cone. The symmetrical axis of the visual cone can be regarded as the central line passing through the centroid of the fruit. Meanwhile, the instance segmentation branch of the MTN-O network can extract the mask of the fruit. Therefore, based on the mask pixel area of the visible region of the fruit and combined with the camera’s internal parameters and aligned depth map information, a 3D point cloud of the mask can be generated, particularly the surface point cloud of the fruit. Then, using a density-based clustering method, the best point cloud cluster is selected from the generated point cloud to exclude interference from outlier noise and holes. Subsequently, the radius of the target fruit can be further determined using the camera’s intrinsic parameters and 2D bounding box information. Finally, combining this radius information with the known centerline of the fruit, the centroid position of the fruit can be calculated.

The complementary fields of view of the dual-vision sensors enable the complete coverage of the robot’s operational area. However, because there are overlapping regions in the images captured by the two cameras, the same fruit may be detected and localized multiple times. Therefore, it is necessary to effectively eliminate these redundant targets. First, the pose relationship of each depth camera relative to the robot base coordinate system is calculated using a multi-camera joint calibration method. Then, based on the transformation between the camera coordinate system and the robot base coordinate system, all 3D spatial coordinates of the fruits in the camera’s field of view can be transformed from the camera coordinate system to the robot base coordinate system. In the base coordinate system, this study uses a simple distance threshold to determine whether two targets are the same fruit. For fruits whose centroids are within a certain distance threshold, they are considered to be the same fruit, thereby eliminating redundantly localized fruits.

2.3. Genetic Algorithm Based on Dual-Arm Task Planning

During the operation of the dual-arm harvesting robot, efficient collaboration between the robotic arms is crucial for improving work efficiency. To reduce their picking of dead zones and ensure that the robotic arms fully cover their workspace, this paper designed the vertical movement of the upper and lower robotic arms to use a shared guide rail. The two robotic arms share a central area, forming a picking overlap zone, as shown in Figure 12. To reduce total time consumption, improve efficiency, and avoid interference between the robotic arms in the picking overlap zone, the picking tasks of each robotic arm must be planned carefully, following these rules: (1) The entry of both robotic arms into the overlap zone at the same time is limited. (2) Each fruit can only be picked by one robotic arm. (3) The picking order of the fruits is not restricted. (4) The initial fruit picked by each robotic arm is not restricted. (5) Tasks must be allocated reasonably to ensure a balanced workload between the two robotic arms.

According to the above task-planning rules, the task allocation problem for harvesting target fruits in the working area is similar to the traditional traveling salesman problem (TSP). The task planning for multi-arm harvesting can be abstracted as a multiple traveling salesmen problem (MTSP). For the dual-arm harvesting robot in this paper, the robotic arms can be seen as salesmen and the target fruits as cities. The robotic arm performing a harvesting action can be considered the salesman visiting a city. Since there is interference between the upper and lower robotic arms in the overlapping working area, simply combining the TSP and MTSP cannot construct a model that satisfies the above rules. This paper adopts an asynchronously overlapped multiple traveling salesman problem (AOMTSP). For example, assume that there are 30 fruits in the working area, distributed as shown in Figure 13. In the figure, TU and TD represent the two salesmen (i.e., robotic arms), and OC represents the shared area between the two salesmen (i.e., the overlapping working area). Numbers one to thirty represent cities (i.e., fruit numbers). To unify the terminology, fruits are referred to as cities and robotic arms as salesmen, with both robotic arms collaborating to perform harvesting tasks. The routes of the two salesmen are straight and unordered, and their speeds are identical throughout the process. Therefore, the goal is to obtain the city visit sequences of the two salesmen, under the above constraints, in a way that minimizes the total time required to visit all the cities.

For the AOMTSP, this paper adopts a genetic algorithm-based solution, hereafter referred to as the AOMTSP-GA. The key to this method in dual-arm harvesting task planning lies in designing encoding and decoding schemes that meet the constraint conditions. Then, based on the selection, crossover, and mutation operators of the classical genetic algorithm, the population is iteratively optimized, ultimately generating a task execution sequence that meets the expected requirements. A detailed description of the AOMTSP-GA is as follows.

This paper adopts a dual-chromosome encoding strategy: a set of sequentially arranged city numbers is constructed into a city chromosome, and a set of sequentially arranged salesman numbers is constructed into a salesman chromosome of equal length to the city chromosome. Through the correspondence between these two chromosomes, the visit sequence of each salesman to multiple targets can be determined, thus avoiding conflicts caused by overlapping visit domains in traditional MTSP schemes. In the design of the selection operator, this paper uses a combination of the best individual preservation and roulette wheel selection strategies. The individual with the highest fitness in each generation is retained and passed on to the next generation. The combination of these strategies improves the convergence speed of the algorithm and enhances its global search capability, reducing the risk of becoming stuck in a local optimum. The crossover operator adopts the partially matched crossover (PMX) method, performing a crossover on the city chromosomes of two parent individuals. First, two crossover points are randomly generated on the city chromosome, and the area between them is defined as the matching region. The matching regions of the two parent chromosomes are then exchanged. Finally, the elements outside the matching region are checked for duplicates with the elements inside the region. If duplicates exist, they are replaced one by one with the corresponding elements in the matching region. To maintain population diversity and prevent premature convergence, the mutation operator performs local modifications on the city and salesman chromosomes. First, two gene positions are randomly generated as the start and end positions of the mutation segment, and various operations such as inversion, swapping, and sliding are used to rearrange the city gene segment.

To prevent different salesmen from simultaneously visiting the overlapping area, asynchronous rules must be introduced. By adding queuing wait times, the waiting process when two salesmen attempt to visit the overlapping area during multi-arm task planning can be restored. The core of the asynchronous rule lies in constructing a timetable for the salesmen. The distances between cities in the dedicated city set are defined as follows:

$$\left\{\begin{array}{l}{d}^{TU}=\left[d_{2,1},d_{5,6},\dots d_{2,1}\right]\\ d^{TD}=\left[d_{2,1},d_{5,6},\dots d_{2,1}\right]\end{array}\right.$$

(1)

where $d^{TU}$and $d^{TD}$represent the dedicated city region distance vectors for salesmen TU and TD, respectively; $d_{\mathrm{2,1} }$ represents the distance from city 1 to city 2; and the shared city region distance vector $d^{OC}$can also be obtained.

Subsequently, the visit time for the salesmen is calculated based on the distance between cities. Assume salesman TU has successively visited the dedicated cities 1 to 22 and is now visiting the shared city 23. Starting from the initial point, the current time is as follows:

$${T_{23}}^{TU} ={\displaystyle \sum _{i=1}^{12}{d_i}^{TU} +12T_{hold}}$$

(2)

where ${T_{23}}^{TU}$is the time at which salesman TU visits city 23, in seconds. ${D_i}^{TU}$is the th element of vector. $T_{hold }$ is the duration for which salesman TU stays in each city, in seconds. Similarly, by calculating the visit times for each city, an initial travel schedule for the salesman is generated.

Next, it is necessary to check the visit times of each salesman in the shared city area to see if there are any overlaps. If overlaps exist, the salesman visiting later must be assigned a queuing wait time $T_{w }$, and their schedule must be updated. This process is repeated, with schedules being checked and adjusted in real-time. The process continues until there are no visit conflicts in the shared city area in the schedules of all salesmen. An increase in the queuing time will directly lead to an increase in the total travel time. Therefore, the queuing constraint in the asynchronous rule is essentially a penalizing constraint, designed to guide the algorithm toward optimizing by reducing the queuing time and overall travel duration.

In the optimization process of the AOMTSP, the objective is to minimize the total time for multiple salesmen to traverse all cities. To achieve this goal and meet the requirements of the above rules, the fitness function $F\left(x\right)$ is designed as follows:

$$F(x)={\displaystyle \frac{c}{f\left(x\right)+{cnT}_w}}$$

(3)

where $c$ is the speed of the salesmen, measured in meters per second. $f\left(x\right)$is the total travel distance for the current visit sequence, measured in meters. $n$ is the number of times multiple salesmen simultaneously visit the shared city set. From the above equation, the larger the fitness value is, the better the performance of the individual is. Therefore, the optimization objective of the AOMTSP is to maximize the value of the fitness function $F\left(x\right)$.

3. Robot Field Experiments and Results

3.1. Experimental Scenario

Numbered lists could be added as follows: To fully evaluate the performance of the developed dual-arm apple harvesting robot, a field experiment was carried out in a standard apple orchard in the Haidian District of Beijing, China, during the harvest season in September 2024. The aim of this experiment was to assess the overall performance of the harvesting robot. The apple trees are mainly characterized by a densely planted dwarf spindle tree shape. Vertical support poles are placed next to the main trunk to maintain upright growth. The main branches are fixed and constrained by vertically spaced steel wires installed along the vertical direction. As shown in Figure 14, the row spacing of the trees is 3500 mm, the tree spacing is 1000 mm, and the trunk height is 3000 mm. The robot moves back and forth between the rows of apple trees, picking fruit from both sides of the canopy. Due to the spindle tree shape, the fruit is mainly concentrated in the lower-middle part of the canopy. When picking fruit from one side of the canopy, the robot’s workspace is set to a region 1200 mm in length, 400 mm in width, and 800~2500 mm in height from the ground, with the trunk as the center, as shown in Figure 15.

3.2. Results Analysis

We selected 10 apple trees with similar canopy conditions and fruit quantities from the aforementioned orchard as the test subjects, as shown in Figure 16. The two main indicators used to evaluate the robot’s performance were the harvest success rate and harvest efficiency, which represent the accuracy and speed of harvesting, respectively. The number of harvestable fruits on each tree was recorded before harvesting. During harvesting, the number of successfully harvested fruits and the total time spent on each tree were recorded. After harvesting, the number of remaining harvestable fruits on each tree was recorded. Additionally, failed cases were recorded and analyzed to identify challenges, which could be attributed to various factors, ranging from the robot system to environmental conditions or even the apples themselves. During each experiment, if the first attempt failed, the robot made a second picking attempt. The robot having to make more than two attempts was considered unsuccessful. The experiment results were summarized and analyzed.

Table 1. Robot’s harvest success rate and efficiency.

Tree ID	Quantity of Fruits	Harvested Fruits	Harvest Success Rate (%)	Total Time Cost (s)	Average Time Cost per Fruit (s)
Tree 1	28	22	78.57	156	7.09
Tree 2	32	25	78.13	183	7.32
Tree 3	30	23	76.67	165	7.17
Tree 4	31	25	80.64	178	7.12
Tree 5	35	28	80.00	202	7.21
Tree 6	27	20	74.07	148	7.40
Tree 7	29	21	72.41	146	6.95
Tree 8	30	22	73.33	172	7.82
Tree 9	34	27	79.41	198	7.33
Tree 10	28	21	75.00	157	7.47
Totals	304	234	76.97	1705	7.29

shows the harvest success rates and efficiency for the 10 trees, while

Table 2. Analysis of robot harvesting failure cases.

Failed Fruits	Recognition and Localization Error	Obstacle Obstruction	Grasp Failure	Separation Failure
70	28 (30%)	24 (11.43%)	13 (37.14%)	5 (11.43%)

lists the failure cases and their causes.

The harvest success rate was the proportion of fruits successfully picked by the robot out of the total number of target fruits. This metric was intended to comprehensively assess the robot’s performance throughout the entire fruit harvesting cycle. The harvest efficiency was measured by the average harvesting time per fruit, calculated based on the timing during field experiment, which included the total time for perception operations, fruit picking, and placing the picked fruit. This metric was intended to measure the robot’s harvesting speed during the harvest. The results from Table 1 indicate that there were 304 harvestable fruits across the 10 trees in the experiment. Of these, 234 fruits were successfully harvested by the robot, with a total time of 1705 s. The robot’s overall harvest success rate was 76.97%, with an average harvesting time of 7.29 s per fruit. During the experiment, in a complex unstructured environment, the robot experienced various picking failures. To further clarify directions for future improvement, four types of picking failure cases were recorded in Table 2. These failures included the following: (1) Recognition and localization error. As shown in Figure 17a, lighting conditions could affect the accuracy of the fruit depth information, leading to mislocalization. Additionally, severe occlusion or overlapping made it difficult for the fruit to be correctly identified. (2) Obstacle obstruction. As shown in Figure 17b, the gripper collided with branches while approaching the fruit, preventing it from reaching the target fruit. (3) Grasp failure. As shown in Figure 17c, the gripper inadvertently caught a branch while grasping the fruit, preventing it from fully closing. (4) Separation failure. As shown in Figure 17d, the gripper reached the correct position, but due to insufficient contact area or gripping force, the apple slipped from the gripper. Additionally, some stems were too tough, and despite the twisting motion, the gripper was unable to successfully separate the stem.

After the field experiment, the harvested apples were categorized into four types of damage according to Chinese national standards: stem intact and undamaged, stem missing, stem intact but with slight skin damage, and stem intact but with severe skin damage. The damage assessment aimed to evaluate the quality of the robot’s harvesting performance during the operation. Each harvested apple was visually inspected for damage. Pre-existing bruises were marked to ensure that they were not attributed to the robot’s harvesting. All harvested fruits were stored at room temperature for 24 h and then visually inspected for damage. The damage assessment results are summarized in

Table 3. Fruit damage assessment results.

Damage Condition	Evaluation Criteria	Quantity	Percentage (%)
Intact and Undamaged	Stem Intact and Undamaged	221	94.44
Stem Missing	Stem Missing	9	3.85
Intact but Slightly Damaged Peel	Stem Intact and Damage Area ≤ 1 cm2	3	1.28
Intact but Severely Damaged Peel	Stem Intact and Damage Area > 1 cm2	1	0.43

. The results show that the overall damage rate of the robot’s harvested fruits was 5.56%. The majority of the damage was due to missing stems, which occurred because some stems were tough, while the connection between the stem and the fruit was weak, causing the stem to fall off when the gripper retracted after grasping the fruit. We found that the fruits with skin damage tended to be larger than other fruits. The range and gripping force of the gripper were fixed, and when the fruit was too large, the gripping force could cause damage to the skin of the fruit.

Table 4. Performance comparison of different versions of apple harvesting robots.

Version	Harvest Success Rate (%)	Average Time Cost per Fruit (s)	Damage Rate (%)
The latest dual-arm harvesting robot	76.97	7.29	5.56
The previous single-arm harvesting robot	65.85	9.78	10.28
The previous four-arm harvesting robot	72.69	8.06	9.85

presents a performance comparison between our latest research and previous versions of the harvesting robot. The results indicate that, compared to the single-arm and four-arm harvesting robot, the overall harvest success rate of our latest robot increased by 11.12% and 4.28%, respectively. The average time per fruit harvested decreased by 1.98 s and 0.77 s, respectively, while the comprehensive damage rate of the harvested fruit was reduced by 4.72% and 4.29%, respectively. These results demonstrate that our dual-arm harvesting robot achieved significant improvements in its overall performance.

4. Discussion

The field experiment results indicate that the robot system achieved a relatively superior performance. Based on various metrics, the robot was capable of performing faster and more accurate apple harvesting operations, while maintaining a low fruit damage rate. However, the current system still faces several challenges. Further improvements are required to enhance the robot’s harvest success rate and efficiency in order to meet the demands of commercial applications. In unstructured environments, the 3D localization accuracy of the RGB-D camera depth is greatly affected, especially due to the instability of its image fill rate and precision. This leads to significant discrepancies between the localization results and the actual position of the fruit. Overlapping fruits and branches or leaf occlusion can result in misidentification or missed detection. Due to the limitations of the picking path, the gripper may be blocked by branches or other obstacles during picking operations, preventing it from reaching the target fruit. Even if not initially blocked, contact with branches may cause the fruit to shift, moving it out of the gripper’s grasp. Current apple recognition and localization algorithms mostly determine the center of the apple without identifying its orientation. The gripper cannot rotate or pull the apple in relation to the stem in a specific manner, leading to stem breakage. Additionally, apple skin is relatively fragile and prone to damage during the picking process. When the gripper’s opening and closing range is small, grabbing larger fruits can easily cause skin damage. When the gripper’s opening and closing range is large, the fruit is more likely to fall during the picking process, increasing the risk of collision interference between the gripper and obstacles.

In this study, we focused on regions with open and flat terrains, such as the standard orchard layouts commonly found in northern China (e.g., Shandong Yantai, Shaanxi Luochuan, Gansu Qingyang, and Xinjiang Khorgos). These areas feature large-scale, standardized orchards spanning thousands of acres, where our robot can operate effectively. For hilly or mountainous regions, such as Guizhou and Sichuan, the robot has low adaptability. Moreover, the current robotic harvesting system primarily relies on a generator for power supply. However, in large-scale orchard operations, prolonged and high-intensity tasks impose higher demands on endurance and efficiency. In addition, this study primarily focuses on technological advancements, particularly the development of an efficient and reliable apple harvesting system. However, the high costs of its research, development, and deployment, as well as operational limitations, continue to hinder its large-scale adoption. Economic factors such as initial investment, operational costs, and potential returns on investment also warrant significant attention in future research.

To address the current challenges faced by the robot, future efforts should focus on the following aspects: (1) The integration of agricultural machinery and agronomy and the exploration of new planting patterns to reduce leaf occlusion and fruit overlap. (2) Optimizing fruit recognition and localization methods. Identifying obstacles around the apples, such as trunks, branches, and leaves, provides more comprehensive environmental perception. (3) Optimizing task planning and intelligent control strategies for the robotic arms and the further exploration of more flexible kinematic configurations for the arms to enhance the robot’s autonomous obstacle avoidance capabilities. (4) Research focusing on reducing the manufacturing and operating costs of the system and exploring efficient energy management systems and the integration of renewable energy sources to enhance the practical value of the robotic harvesting system.

5. Conclusions

To meet the high-efficiency picking needs of typical standard orchards in China, this paper proposes a new dual-arm robot system designed to improve apple harvesting efficiency and success rates in orchards, and a field experiment was conducted to evaluate its performance. The hardware system was fully integrated with a dual-vision perception system, two four-DOF robotic arms, two curved three-finger grippers, and a fruit transport and collection device as the main components. Efficient software algorithms were designed to work in coordination with the hardware system to enable autonomous apple harvesting. First, an improved YOLOv8-based multitask network model was used to enhance the recognition accuracy. Second, a 3D localization method based on the cone method was adopted to achieve precise fruit localization. Finally, a multi-arm coordinated task-planning method based on a genetic algorithm and an ROS-based control scheme further improved the overall performance of the robotic system. The field experiment showed that the robot system achieved a total harvesting success rate of 76.97%, with an average harvesting time of 7.29 s per fruit and a fruit damage rate of only 5.56%.

Compared to other related studies, this robotic system has achieved significant improvements in its harvesting success rate and efficiency. However, this study also identified new challenges that require the further refinement of the current system. Efforts will be directed toward integrating agricultural machinery and agronomy, optimizing fruit recognition and localization methods, and exploring task planning and flexible grasping for robotic arms to accelerate innovation in key technologies. Additionally, further research will focus on reducing the manufacturing and operating costs of the system and optimizing energy utilization based on practical application scenarios to enhance the practical value of the robotic harvesting system.