Dual-Arm Coordination of a Tomato Harvesting Robot with Subtask Decoupling and Synthesizing

Abstract

Robotic harvesters have the potential to substantially reduce the physical workload of agricultural laborers. However, in complex agricultural environments, traditional single-arm robot path planning methods often struggle to accomplish fruit harvesting tasks due to the presence of collision avoidance requirements and orientation constraints during grasping. In this work, we design a dual-arm tomato harvesting robot and propose a reinforcement learning-based cooperative control algorithm tailored to the dual-arm system. First, a deep learning-based semantic segmentation network is employed to extract the spatial locations of tomatoes and branches from sensory data. Building upon this perception module, we develop a reinforcement learning-based cooperative path planning approach to address inter-arm collision avoidance and end-effector orientation constraints during the harvesting process. Furthermore, a task-driven policy network architecture is introduced to decouple the complex harvesting task into structured subproblems, thereby enabling more efficient learning and improved performance. Simulation and experimental results demonstrate that the proposed method can generate collision-free harvesting trajectories that satisfy dual-arm orientation constraints, significantly improving the tomato harvesting success rate.

1. Introduction

Nowadays, robotic manipulators have effectively replaced skilled technicians in various industrial applications, such as welding, sorting, and assembly. However, in agricultural scenarios, automation still faces formidable challenges, particularly in the development of robotic harvesters.

For harvesting robots, a primary challenge lies in recognizing targets and obstacles within complex environments to obtain their precise spatial coordinates. While computer vision is typically employed for this task, conventional methods often struggle to distinguish targets from obstacles due to the unstructured nature of agricultural scenes and severe occlusions caused by dense branches and leaves [1]. Recent advancements in deep learning, particularly Convolutional Neural Networks (CNNs), have significantly enhanced the performance of fruit and branch detection [2]. However, these techniques require large-scale labeled datasets that are labor-intensive and expensive to collect. Furthermore, different harvesting targets necessitate specialized datasets. Regarding tomato harvesting, no comprehensive publicly available dataset currently exists for the simultaneous detection of tomatoes and branches, leaving the reliable recognition of these elements a persistent challenge.

Path planning presents another critical bottleneck for robotic harvesters, especially for dual-arm systems. Two major difficulties arise in this context. The first is the complex obstacle avoidance problem. In agricultural environments, robots must not only reach the target fruits but also navigate around unpredictable obstacles. Since fruits and branches are physically interconnected, any collision with a branch may shift the fruit’s position, thereby reducing the success rate of traditional open-loop planning. To mitigate this, we employ a dual-arm system to mimic the human harvesting process, where one arm stabilizes the branch while the other picks the fruit. This approach necessitates a collision-free path planning method that accounts for both environmental obstacles and self-collision between the two arms—a classic and complex motion planning problem. The second difficulty involves orientation constraints on the end-effector. The harvesting process often requires the end-effector to maintain a specific pose—such as remaining horizontal or holding a precise angle—to prevent pushing the fruit away before it can be captured. Currently, sampling-based algorithms like Rapidly-exploring Random Trees (RRT) [3] are widely used due to their probabilistic completeness and extensibility to high-dimensional problems. However, while RRT can ensure that the robot reaches a target pose, it struggles to enforce strict orientation constraints throughout the entire path [4]. Moreover, sampling-based methods suffer from high computational complexity as the state-space dimensionality increases; for a dual-arm system like ours, the sampling dimension reaches 12 degrees of freedom (DOF), which is exceptionally high.

The success of Artificial Intelligence has prompted researchers to explore deep reinforcement learning (DRL) for robot control. DRL provides a robust alternative for handling complex tasks that are difficult for conventional methods, as it maps environmental observations directly to control actions with superior adaptability [5]. Despite these advantages, DRL often suffers from poor sample efficiency, leading to long and difficult training processes. Consequently, high-fidelity simulation environments are typically utilized for model training. In this paper, we develop a custom dataset for tomato and branch detection and utilize a semantic segmentation network to identify these components. By integrating segmentation results with point clouds from anRGB-D camera, we accurately determine the spatial coordinates of the targets. To enhance harvesting success, a dual-arm strategy is implemented to mimic human-like coordination. We employ DRL to achieve collision-free path planning under orientation constraints, carefully defining the state-action space and reward functions. Furthermore, a novel policy network architecture is designed to improve learning performance by decoupling the task. We established a simulation environment based on real harvesting scenarios; the results from both simulation and real-world experiments demonstrate that our method generates effective, collision-free paths that strictly satisfy the orientation constraints for dual-arm manipulators. The remainder of this paper is structured as follows: Section 2 reviews the related works. Section 3 describes the materials and methods. Section 4 presents the experimental results and analysis. Section 5 discusses the findings, followed by our conclusions in Section 6.

2. Related Works

The perception technologies of robotic harvesters based on computer vision can be divided into two kinds: deep learning and conventional image processing technologies. Wang et al. [6] used the HSV color space to detect mature apples and utilized the highlight spots generated by camera flashes to distinguish green apples from leaves. Similarly, Lv et al. [7] and Liu et al. [8] successfully recognized tomatoes in RGB or HIS color spaces using the OSTU automatic threshold segmentation method. Qiang et al. [9] further extended this by employing Multi-class Support Vector Machines (SVMs) combined with morphological operations to simultaneously segment mature citrus fruits and branches, achieving a fruit recognition rate of 92.4% under various lighting conditions. However, in practice, methods based on color segmentation and traditional machine learning are highly sensitive to ambient light. Even with automatic thresholding, the relative simplicity of these models limits their adaptability to changing harvesting environments and their ability to handle occlusions caused by foliage. The development of deep learning has significantly enhanced the robustness of agricultural perception. Rahnemoonfar et al. [10] utilized CNNs for apple detection, achieving 91% accuracy on real-world images, while Fu et al. [11] applied similar techniques to kiwifruit. For tomato harvesting, Liu et al. [12] implemented an improved YOLOv3 model with an image pyramid structure for multi-scale feature fusion, achieving a 92.39% mAP for detecting small-scale pests and diseases in complex backgrounds. Beyond simple detection, the integrated perception of fruits and branches has become a research focus to support complex harvesting tasks. Kang et al. [1] proposed DaSNet-v2, a multi-functional network for real-time detection, instance segmentation of apples, and semantic segmentation of branches. Similarly, Majeed et al. [13] utilized SegNet to achieve semantic segmentation of apple trees by leveraging both RGB and depth information to filter background noise. To obtain the precise spatial coordinates required for collision-free planning, Lin et al. [14] proposed a pose estimation method using a low-cost RGB-D sensor, where an FCN segments fruits/branches and Euclidean clustering is applied to point clouds for 3D reconstruction. Furthermore, Gené-Mola et al. [15] introduced a mobile terrestrial laser scanning approach, reaching a recognition success rate of 82.4% in real orchards. However, deep learning-based techniques require large amounts of labeled images, which are expensive to collect. As for tomato harvesting, there is still a lack of comprehensive datasets that include both fruit and branch labels, meaning the simultaneous recognition of tomatoes and their supporting structures remains challenging.

Obstacle avoidance and end-effector orientation constraints are two major challenges to be addressed in path planning algorithms for robotic harvesters. Baur et al. [16] simplified environmental obstacles into geometric primitives, such as cylinders and spheres, and planned collision-free trajectories based on the artificial potential field method. Hemming et al. [17] adopted an RRT-based planning algorithm to achieve autonomous picking of peppers. However, these methods often overlook the orientation constraints of the end-effector during the harvesting process. Consequently, picking failures may occur when the robot approaches the target from an improper angle, inadvertently pushing the fruit away rather than capturing it. To address these constraints, several approaches have been proposed, including trajectory optimization-based methods [18] and workspace sampling-based methods [19]. Nevertheless, sampling-based algorithms suffer from high computational complexity as the state-space dimensionality increases. For dual-arm systems like the one used in this study, the sampling dimension reaches 12 degrees of freedom (DOFs), which poses a significant challenge for conventional sampling-based methods.

With the rapid development of deep learning, deep reinforcement learning (DRL) has emerged as a prominent research topic in the field of robot motion and path planning. Liu et al. [20] successfully trained a dual-arm system to perform cooperative tasks—such as reaching, pushing, and picking—in a simulated environment using the DADDPG algorithm. Similarly, Bojan et al. [21] combined reinforcement learning with intelligent control frameworks to enable an articulated robot to perform complex door-opening maneuvers, validating their approach through both simulation and real-world experiments. In [22], the authors employed PPO [23] and DDPG [24] to train a manipulator in a dynamic environment containing three randomly moving cubic obstacles. Their agent achieved an 85% safety success rate, effectively reaching target positions without collisions.

The advantage of DRL lies in its ability to learn complex control policies directly from high-dimensional sensory inputs, potentially overcoming the limitations of traditional planning methods in unstructured environments. This capability has led to its widespread adoption in agricultural applications in recent years. Ref. [25] formulated agricultural picking sequence planning as a three-dimensional traveling salesman problem (TSP) and solved it using a pointer network-based actor–critic DRL framework. Li et al. [26] developed a DRL-based harvesting strategy for clustered kiwifruits, in which fruit recognition and localization, as well as picking order planning, are first performed, followed by DRL-based optimization to generate an efficient harvesting strategy. Yi et al. [27] proposed a self-supervised DRL-based view planning method that employs a Self-Supervised Convolutional Network to evaluate the effectiveness of actions during training and dynamically adjust rewards to guide policy learning. Liu et al. [28] proposed an expert experience-based guided reinforcement learning strategy for high-DOF robotic arms in apple picking, outperforming RRT in planning time and path quality. Feng et al. [29] introduced an enhanced HER-SAC algorithm that incorporates a heuristic action fusion strategy during training to optimize cherry tomato picking. Despite the promising potential of deep reinforcement learning (DRL), its application to agricultural harvesting tasks remains constrained by factors such as reinforcement environment design, network architecture, and deployment methodologies, which often result in relatively low success rates when DRL is directly applied to train harvesting robots.

3. Materials and Methods

In this section, we first formulate robot path planning in tomato harvesting as an RL problem. The state, action, reward function, and network architecture designed by us will be introduced. Then the framework of our dual-arm path planning method for tomato harvesting is illustrated, including the tomato and branch detection method.

3.1. RL Problem Formulation

We model the interaction between our dual arms and harvesting environment as a Markov Decision Process (MDP), defined by the tuple $〈S;A;R;S_0〉$. The state $s_t\in S$ for the MDP is $s_t=\left[s_l^t,s_r^t,p_t,p_b\right]$. $s_l^t$ is the state of the left arm, which contains the robot’s current joint angles $q_l^t$, robot’s current end-effector position $p_{el}^t$, and robot’s current end-effector orientation $o_l^t=(r,p,y)$, where $r,p,y$ are Euler angles. Similarly, the state of the right arm is $s_r^t=(q_r^t,p_{er}^t,o_r^t)$. $p_t$ is the position of the target tomato and $p_b$ is the position of the branch.

In each episode, the dual arms begin at a fixed initial joint state. At each time step t, the two arms take an action $a_t$ according to the policy $\pi \left(a_t\right|s_t)$. The action space $\mathcal{A}$ in this paper is defined as a twelve-dimensional vector, corresponding to the delta joint positions at each time step. At each time step, each joint position will only change $1.5$ degrees at most. The step is small so that every action can be executed smoothly on our JAKA manipulator under servo control mode. The control frequency is about 30 FPS to make sure that the manipulator will not move too fast or too slow.

3.2. Reward Function

The agent receives a reward $r_t$ and transits to the next state $s_{t+1}$ after action $a_t$ is taken. The goal of the agent is to maximize the expected return from each state. Consequently, by editing the reward function, we can guide the agent to perform our expected actions. The reward function is defined as

$$\begin{array}{c}\hfill r=r_o+r_t+r_c\end{array}$$

(1)

To motivate the agent to discover the collision-free path, we design a reward function $r_c$ to guide the agent to avoid collisions. When a collision happens, i.e., [Collision = 1], a large value of the reward will be deducted, as shown in Equation (2).

$$\begin{array}{c}\hfill r_c=-5\ast [collision=1]\end{array}$$

(2)

On the other side, we also want the arms to move to the target at an appropriate orientation, as shown in Figure 1. Hence, we design a reward function $r_o$ to control the orientations of the dual arms.

$$\begin{array}{c}\hfill r_o=0.5\ast cos\left(\alpha \right)-0.5\ast d\end{array}$$

(3)

$\alpha$ is the angle between the target and the approaching vector of the end-effector, as shown in Figure 1a. $cos\left(\alpha \right)$ is added to the reward function to make the robot move right towards the target. A similar orientation constraint is also required for the arm that is to grasp the branch. d is the distance between the branch and the approaching vector of the end-effector, as shown in Figure 2.

$$\begin{array}{c}\hfill r_t=50\ast [d(p_t,p_{et}^t)<\delta ,d(p_o,p_{eb}^t)<\delta ]\\ \hfill -d(p_t,p_{et}^t)-d(p_b,p_{eb}^t)\end{array}$$

(4)

$r_t$ denotes the reward designed to guide the dual arms of the system toward their respective target positions, as defined in Equation (4). Specifically, the assignment of the manipulator for grasping the tomato and the branch is first determined based on the arms’ relative positions. From the robot’s egocentric view, if the tomato is located on the left side of the branch, the left arm is designated to grasp the tomato; otherwise, the right arm is selected. Here, $p_{et}^t$ represents the end-effector position of the arm assigned to grasp the tomato, while $p_{eb}^t$ denotes the end-effector position of the arm assigned to grasp the branch. The variables $p_t$ and $p_b$ correspond to the target grasping positions of the tomato and the branch, respectively, and d(·) measures the Euclidean distance between an end-effector and its corresponding target position. A smaller distance yields a higher reward value. When the distances of both arms to their respective targets are less than $\delta$, the task is considered completed, and a larger terminal reward is granted.

3.3. Network Architecture

The whole tomato harvesting task is very complex, as we can see in the last part, including reaching the target, collision avoidance, and orientation constraints for both arms. The input of the network is also very complex, including different kinds of information about two arms and targets. Conventional MLP policy networks prove inadequate in effectively analyzing input state and integrating coupled information to fulfill the three tasks of grasping, obstacle avoidance, and end orientation control. Consequently, we propose a novel network architecture to decouple the task, as shown in Figure 3.

The tomato grasping state includes the position of the tomato $p_t$ and the state of the robot that is to grasp the tomato $s_{et}^t$, which contains the joint angles, position, and orientation of the end-effector. Similarly, the branch grasping state includes the position of the branch $p_b$ and the state of the robot that is to grasp the tomato $s_{eb}^t$. Collision avoidance state includes the state of both arms. For the Q network, the action $a_t$ will also be added to each state. The three decoupling networks each take one part of the state $s_t$ as input and generate features about a specific relatively simple task, respectively.

The network is optimized for our harvesting task by reducing the complexity of feature extraction. Specifically, we use a decoupling network to extract features and an integration network to combine them and generate the output. To further enhance the network’s performance, we design an actor network that separates the calculation process of the action for each arm, thereby preventing distraction by irrelevant features.

Soft Actor–Critic (SAC) [30], a model-free policy gradient algorithm, is adopted for policy and Q function learning. It has better exploration capabilities and stronger robustness. The hyperparameters we use in the training are a discount factor $\gamma =0.97$, a decoupling network size of 512, and an integration network size of 1024 for the actor network and 1536 for the Q network.

3.4. Framework of Dual-Arm Robotic Tomato Harvesting

The framework of our dual-arm path planning method for tomato harvesting is shown in Figure 4.

The task-decoupling policy network is trained in a simulation environment using the SAC algorithm. Since the network does not require image input, the sim2real problem is not a concern in this case. When deployed on a real robot, the network obtains the robot status information from the JAKA API, while the positions of fruits and branches are estimated by our semantic segmentation network.

For RGB image segmentation, there have been many depth learning models with excellent performance, such as U-Net, Mask RCNN, etc. In previous work [31], we proposed a U-Net-like network for rice root image segmentation shown in Figure 5. In this study, we used this network for tomato fruit and branch segmentation. To this end, we took 1000 photos of tomatoes and branches and created a semantic segmentation dataset through manual annotation; some of the annotated pictures are shown in Figure 6. In the experiments, an individual tomato fruit can be approximately modeled as a sphere with a diameter of around 100 mm, while the branch can be approximated as a cylindrical structure with a diameter ranging from 10 to 15 mm and a length of approximately 300–400 mm. After training, the mIoU of the model is 80.6%, the accuracy rate is 96.3%, and the Kappa coefficient is 0.8545. The model performance meets the requirements.

4. Results

In this section, we first describe the simulation environment for training and present our experimental results in simulation. Then, we evaluate our trained model’s performance on JAKA dual arms in a real-world environment.

4.1. Simulation Environment

Our simulation environment was built using Pybullet 2.7.8 [32], as shown in Figure 7.

The dual arms are mounted firmly on a platform. The red sphere represents the tomato that needs to be picked. Since the tomato branch is usually not entwined, it is simplified as a thin blue cylinder. The branch will be placed randomly in front of the workspace of the dual arms. One arm is responsible for grasping the tomato, while the other arm is responsible for grasping the branch, selected based on the relative position of the tomato and the branch. If the tomato is on the left side of the branch, the left arm will pick it. Since the distance between tomato branches is relatively long and other branches are beyond the reach of the dual arms, only one branch is added to the environment.

4.2. Simulation Results

Soft Actor–Critic (SAC) is used to train our policy network in the environment. The learning curve is shown in Figure 8. Moreover, by decoupling obstacle avoidance, grasp execution, and end-effector pose control within the network architecture, the performance gap between our method and the conventional multilayer perceptron (MLP)-based policy becomes increasingly pronounced after approximately 1,500,000 training steps, when the agent has acquired a certain level of competence. Specifically, our method achieves a higher final cumulative reward, indicating superior performance in terms of grasping success rate, end-effector pose accuracy, and obstacle avoidance capability. The experimental results are summarized in

Table 1. Performance comparison between different path planning methods in the simulation environment.

Method	Orientation Control	End Position Error	Success Rate
RL	√	4.9 cm	52%
RRT	×	0.6 cm	83%
Ours	√	1.5 cm	96%

. When the dual arms move closer to their target positions, the risk of collision increases. For agents trained with the conventional network, the primary reason for the lower success rate is that the policy tends to remain in safe states rather than actively attempting to accomplish the task objectives. As a result, a larger grasping position error is observed, as illustrated in Figure 9. In contrast, the proposed network effectively alleviates this issue through the task-driven decoupling architecture, achieving a success rate of 96% and an end-effector position error of 1.5 cm.

We also compared our method with RRT, a sampling-based path planning method. The performance of RRT is shown in Figure 10. Although the position error is small, RRT cannot control the orientation of the end-effector during the grasping process. The tomato might be pushed away before being grasped in Figure 10a. In addition, sampling-based algorithms need to re-plan their path every time before the harvesting process begins. As for our dual arms, the 12-dimensional sampling space also makes the time complexity of the iterations high.

The three path planning methods were evaluated 100 times each in the simulation environment, and their average performances are reported in Table 1. The distance between the position of the end-effector and the target should be smaller than 2 cm to ensure that the gripper can successfully grasp the target. Since the main contribution of this work lies in the design of a novel actor–critic (A–C) network architecture rather than in the reinforcement learning algorithm itself, the Soft Actor–Critic (SAC) algorithm, which exhibits the best overall performance, was adopted for training in the experiments. We also experimented with other reinforcement learning algorithms, such as PPO and DDPG, under the same environment; however, their success rates were relatively lower compared to SAC. From the results, we can see that because of the large position error, the conventional RL agent has a relatively low success rate. RRT usually fails because of the high dimensionality of the sampling space. We considered it a failure if the path had not been found by RRT in ten seconds. The success rate for RRT is about 80%, but it cannot control orientation during the picking process. As for our method, we considered it a failure if the dual arms failed to reach the target position in 200 steps. Our method has the highest success rate and better performance than RRT and the conventional RL network.

4.3. Real-World Experiment Results

Our tomato harvesting system is shown in Figure 11. The dual arms are fixed on a movable platform. The platform can go up and down via the lifting mechanism and move left and right via the mobile chassis. KinectV2 (Microsoft Corporation, Shanghai, China), which is the RGBD camera, is fixed between two arms. We use two flexible three-finger grippers driven by a double-acting cylinder as the end-effectors.

We carried out the experiment in our lab with fake branches and tomatoes. The performance of our method can be seen in Figure 12. The experimental success rates are reported in

Table 2. Performance comparison between different path planning methods in the real-world environment.

Method	Orientation Control	Number of Fruits to Be Picked	Number of Fruits Picked	Success Rate	Average Time
RL	√	36	20	55.5%	12.4 s/piece
RRT	×	36	29	80.6%	18.4 s/piece
Ours	√	36	32	88.9%	9.5 s/piece

. Since it is difficult to accurately measure the distance between the end-effector and the target position in real-world scenarios, the experimental results are presented only in terms of success rate and task completion time.

A video demonstration is available at https://www.bilibili.com/video/BV1zD4y1k7vz/ (accessed on 18 January 2026), where the dual arms successfully grasp the target with the appropriate orientation, as we expected.

5. Discussion

The experimental results validate that the proposed subtask-decoupling framework significantly enhances the performance of dual-arm tomato harvesting. The marked difference in success rates between our method and the conventional MLP network (96% vs. 52%) indicates that directly mapping high-dimensional inputs to actions in a 12-DOF space often leads to local optima, where the agent prioritizes safety over task completion. By decoupling the state space into specific modules for grasping, stabilizing, and obstacle avoidance, our architecture reduces the complexity of feature extraction, allowing for more precise coordination.

While RRT achieved low position error, its failure to maintain continuous orientation constraints and its high computational cost in high-dimensional spaces make it less suitable for real-time agricultural tasks compared to our DRL-based approach. The “pushing” effect observed in RRT-planned paths (Figure 10) underscores the necessity of the orientation-aware reward function used in our model. Furthermore, the successful sim-to-real transfer demonstrates that a state-based policy can effectively bypass visual noise, though future work should address the flexibility of real tomato branches to further enhance robustness in dense orchard environments. Overall, this “stabilize-and-pick” strategy provides a reliable foundation for complex, multi-arm agricultural manipulation.

6. Conclusions

We first employ a semantic segmentation network to accurately detect and localize tomatoes and surrounding branches in the harvesting environment. Based on the obtained perception results, the harvesting process is formulated as a Markov Decision Process (MDP), enabling the application of reinforcement learning for coordinated path planning and control of the dual-arm system. To effectively address the complexity arising from collision avoidance, orientation constraints, and inter-arm coordination, we propose a novel policy network architecture that explicitly decouples the harvesting task into complementary submodules. Experimental evaluations demonstrate that the proposed approach consistently outperforms conventional path planning methods and exhibits robust and promising performance in real-world agricultural harvesting scenarios.