Development of Shunt Connection Communication and Bimanual Coordination-Based Smart Orchard Robot

Abstract

This research addresses the enhancement of operational efficiency in apple-picking robots through the design of a bimanual spatial configuration enabling obstacle avoidance in contemporary orchard environments. A parallel coordinated harvesting paradigm for dual-arm systems was introduced, leading to the construction and validation of a six-degree-of-freedom bimanual apple-harvesting robot. Leveraging the kinematic architecture of the AUBO-i5 manipulator, three spatial layout configurations for dual-arm systems were evaluated, culminating in the adoption of a “workspace-overlapping Type B” arrangement. A functional prototype of the bimanual apple-harvesting system was subsequently fabricated. The study further involved developing control architectures for two end-effector types: a compliant gripper and a vacuum-based suction mechanism, with corresponding operational protocols established. A networked communication framework for parallel arm coordination was implemented via Ethernet switching technology, enabling both independent and synchronized bimanual operation. Additionally, an intersystem communication protocol was formulated to integrate the robotic vision system with the dual-arm control architecture, establishing a modular parallel execution model between visual perception and motion control modules. A coordinated bimanual harvesting strategy was formulated, incorporating real-time trajectory and pose monitoring of the manipulators. Kinematic simulations were executed to validate the feasibility of this strategy. Field evaluations in modern Red Fuji apple orchards assessed multidimensional harvesting performance, revealing 85.6% and 80% success rates for the suction and gripper-based arms, respectively. Single-fruit retrieval averaged 7.5 s per arm, yielding an overall system efficiency of 3.75 s per fruit. These findings advance the technological foundation for intelligent apple-harvesting systems, offering methodologies for the evolution of precision agronomic automation.

1. Introduction

Against the backdrop of accelerating global agricultural intelligence development, the fruit industry [1,2], as the core component of the agricultural system, is facing dual pressures of rising production demand and structural shortage of labor resources. Apple-picking robots are an important symbol of achieving full mechanization, automation, and intelligence in apple production in the future and are also one of the research hotspots and difficulties in the field of intelligent agricultural machinery [3,4,5,6,7,8,9]. Conducting in-depth research on the core and key technologies related to apple-picking robots has important strategic value and practical significance for reducing the labor intensity of fruit farmers [10,11,12,13,14,15], improving picking efficiency, and promoting the efficient development of the apple industry [16,17]. An apple-picking robot based on a two-finger gripper was developed [18]. This robot is based on machine vision and laser ranging sensors to recognize and locate fruits. The picking method is to use a two-finger end-effector to clamp the apple stem and apply a rotating motion to achieve separation of the fruit by twisting the stem. The results of the experiment conducted in the orchard showed that the success rate of robot recognition was over 90%, and the picking speed of a single fruit was 7.1 s. However, the robot’s applicability is only for apple varieties with longer fruit stalks.

An apple-picking robot based on binocular CCD cameras was developed [19], which mainly consists of a power chassis, a 5-degree-of-freedom robotic arm, an end-effector, and a visual servo control system. The end-effector of the robot first clamps the fruit and then uses an electric blade to cut off the connection between the fruit and the branch to achieve apple–branch separation. The experimental results in the apple orchard showed that the success rate of harvesting was 77%, and the harvesting time for a single apple was 15.4 s. The main reason for the failure of harvesting was analyzed to be the influence of obstacles such as tree branches during the harvesting process. An apple-picking robot consisting of a robotic arm, a two-finger gripping end, a sliding mechanism, and a control system was designed and developed [20]. The results of the experiment conducted in the apple orchard showed that the robot’s recognition accuracy for fruits was 81.73%, the average picking time for a single fruit was 9.5 s, and the success rate of the picking operation was 86.92%. The main factor causing harvesting failure is that the robot’s vision system is easily affected by natural lighting and occlusion in the environment.

Additionally, an apple-harvesting robot was constructed, incorporating a three-digit end-effector and employing a ZED stereo vision system [21]. This robotic platform features an integrated manipulator possessing five degrees of freedom. Fruit detachment is accomplished through a technique involving grasping followed by separation. Performance evaluations conducted within an orchard setting yielded the following results: image recognition by the vision system required 0.026 s per frame. The average duration for harvesting a single fruit was measured at 12.53 s. A success rate of 81.6% was achieved for fruit retrieval. A six-degree-of-freedom single robotic arm apple-picking robot based on the D435 depth camera was also developed [22]. The robot’s vision system has an efficiency of 0.053 s/frame image for apple target recognition and positioning, and the apple-picking method is based on a three-finger grasping end-effector grasping separation method. The results of the single machine arm robot picking experiment in the orchard showed that the single fruit picking time was 8.4 s, and the success rate of fruit picking was 83.98%.

In addition, a harvesting robot for “wall-shaped” apple trees was developed [23], which mainly includes a machine vision system, an actuator, and a control system. The robot adopts a 7-degree-of-freedom robotic arm equipped with a three-finger picking end-effector. The results of the picking experiment in the apple orchard showed that the picking time for a single apple was 6 s, and the success rate of picking was about 84%. Furthermore, Hohimer et al. [24] improved and replaced the picking end-effector of the robot with a three-finger pneumatic soft picking hand in order to reduce the damage rate during apple picking. In addition, the original 7-degree-of-freedom robotic arm was replaced with a 5-degree-of-freedom robotic arm. The experimental results in the orchard showed that the picking time for a single apple was 7.3 s, and the success rate of picking was 67%.

A vacuum-based apple-harvesting system was created by Abundant Robotics [25], achieving detachment of individual fruits within 1.5 s. This technology’s utilization context, however, corresponds to the foreign “V”-trellised orchard configuration. In contrast, spindle-trained apple trees featuring intricate canopy structures represent the predominant cultivation system across Chinese orchards. Consequently, its effectiveness when directly implemented within China’s contemporary spindle-type orchards for automated harvesting requires assessment. Moreover, the pneumatic fruit removal method employed frequently results in calyx detachment. Such stem loss is considered detrimental to maintaining postharvest freshness during subsequent storage and marketing phases.

The apple-picking robot is the core and important symbol of achieving intelligent apple harvesting [26,27]. However, so far, there have been no reports on the research and development of the integrated machine for efficient operation of six-degree-of-freedom dual-arm apple-picking robots. Therefore, in order to improve the overall picking efficiency of robots and promote intelligent fruit harvesting in modern apple orchards (as shown in Figure 1), it is urgent to conduct in-depth research on the core technologies related to six-axis dual-arm apple-picking robots. Therefore, based on artificial intelligence and robot technology, this study designed a dual robotic arm spatial layout for picking and collision avoidance, built a control system for flexible gripper picking hand and suction cup picking hand, and proposed their control methods. A dual-arm parallel motion communication system was established, and a dual-arm parallel coherent picking strategy was proposed. Finally, a six-degree-of-freedom dual-arm apple-picking robot was integrated and developed, and picking experiments were conducted in orchards.

At present, the harvesting of apples still mainly relies on manual picking operations [28,29]. Picking apples is a labor-intensive and time-intensive task, and manually picking fruits from high places poses safety hazards [30]. The automation of apple harvesting is beneficial for reducing labor intensity, improving operational efficiency, and promoting the efficient development of the apple industry. It is of great practical significance for solving the problems of labor shortage, high labor intensity, high labor costs, and low harvesting efficiency in the apple-harvesting process [31,32,33], ensuring timely and efficient harvesting of mature apples, and enhancing the international competitiveness of the apple industry.

2. Methodology of Robot Prototype Construction

2.1. Dual-Arm Spatial Layout Design for Picking and Collision Avoidance

The spatial arrangement of dual manipulators within a harvesting robot critically affects the operational envelope and potential physical interference between arms during fruit retrieval tasks. Three primary spatial configurations exist for such dual-arm systems.

In the first configuration, designated as the non-overlapping workspace arrangement, the manipulators’ operational spheres remain entirely separate. Figure 2 illustrates this layout schematically from a frontal perspective. Points O₁ and O₂ denote the central locations of the left and right manipulator mounting platforms, while O represents the midpoint between them. Distances OO₁ and OO₂ are equivalent, each measuring 935.5 mm. Power connection interfaces for both units are oriented outward. Within the depicted posture, a 90° displacement is observed at the second articulation (J2) of the left manipulator, whereas the right manipulator’s J2 exhibits a −90° displacement. The remaining five articulations on each unit are maintained at zero displacement. Key dimensional characteristics for the AUBO-i5 robotic units [34,35,36,37,38] are provided in Figure 3.

V-REP (Virtual Robot Experiment Platform) is an open-source robot multi-body simulation platform designed specifically for modeling complex robot systems, verifying control algorithms, and simulating scenarios. It integrates dynamic engines, sensor simulation, path planning tools, and multi-robot collaboration frameworks; supports programming interfaces such as Lua/Python/C++/ROS; and provides high-fidelity physical interaction and environment rendering. Its distributed architecture allows users to independently control each object, making it suitable for full scenario development such as robotic arms, mobile robots, and agricultural harvesting equipment. It significantly reduces the cost of on-site testing and is a core tool for academic research and engineering deployment. This study used the V-REP software (Version 3.6.2) to analyze the impact of dual-arm spatial layout on collision avoidance during dual-arm motion. After importing the model of the AUBO-i5 robotic arm, a simulation diagram of the non-overlapping dual-arm layout in the workspace is constructed as shown in Figure 4. In the dual-arm posture shown in the figure, the rotation angle of the left robotic arm joint 2 is 90°, and the rotation angle of the right robotic arm joint 2 is −90°, which can provide a more intuitive understanding of the dual-arm spatial layout design.

Visualization of the workspace distribution for this configuration, generated computationally, is presented in Figure 5. Distinct red and blue spheres mark the base centers. The collective operational volume comprises two isolated hemispherical regions, each possessing an 886.5 mm radius. Physical contact between the manipulators is precluded due to this spatial separation.

On the other hand, due to the fact that the operating space of the dual robotic arms of the constructed robot is two non-overlapping hemispherical regions, it may lead to fruit leakage during actual picking operations. The top view schematic diagram is shown in Figure 6. Due to the independence and discontinuity of the dual-arm workspace, the actual working range of each arm is limited by the geometric structure, and there are work areas that are inaccessible to both arms. That is, the middle of the two hemispheres is the inaccessible space for the robotic arm. Therefore, the fruits in the middle area of the two workspaces cannot be picked by the robotic arm, as indicated by the blue dashed line in the middle area selected in the following figure.

A second configuration, termed Type A overlapping workspace, features partially shared operational volumes. Its frontal schematic appears in Figure 7. Again, O₁ and O₂ are the base centers, with O being their midpoint. Distances OO₁ and OO₂ are identical (300 mm), a value below 465 mm. Power interfaces face outward. In the illustrated posture, the left manipulator’s J2 articulation exceeds 0°, while the right manipulator’s J2 is less than 0°.

Using the V-REP software to analyze the impact of the overlapping A-type dual-arm layout in the workspace on collision avoidance during dual-arm motion, and after importing the model of the AUBO-i5 robotic arm, a simulation diagram of the overlapping A-type dual-arm layout in space was constructed as shown in Figure 8, which provides a more intuitive understanding of the dual-arm spatial layout design.

Since both manipulators perform apple harvesting within a common frontal space, collision avoidance is maximized by partitioning this region using a vertical plane intersecting the midpoint between base centers. The left unit is assigned the left partition, and the right unit handles the right partition. Consequently, during vision-guided harvesting operations, end-effector collisions are prevented.

However, due to the significant operational overlap and the proximity of the bases being less than the combined lengths of certain linkages, a collision hazard exists primarily at the third articulation (J3) and its associated linkage on both arms. Verification of this interference risk was conducted. The left manipulator’s J2 and J3 were each commanded to 90°, while the right manipulator’s J2 and J3 were moved to −90°. Axial positions of the other articulations were held at zero. Positional data for J3 on both units was monitored, revealing intersecting x-coordinate trajectories relative to the midpoint coordinate system (Figure 9). This intersection indicates inevitable contact during motion toward points within their respective partitions.

A schematic representation of the Type A overlapping workspace, produced computationally, is shown in Figure 10. The merged operational volume results from the partial overlap of two hemispherical regions (radius: 886.5 mm each).

A third configuration, designated Type B overlapping workspace, is depicted frontally in Figure 11. Base centers O₁ and O₂ have a midpoint O, with OO₁ and OO₂ both equaling 550 mm. Outward-facing power interfaces are used. The illustrated posture shows the angular relationship between J2 and J3 on the left manipulator at 90° and on the right at −90°. The remaining four articulations per unit are at zero.

The simulation diagram of the spatial overlapping B-type dual-arm layout based on the V-REP software is shown in Figure 12, which provides a more intuitive understanding of the dual-arm spatial layout design.

Operational responsibilities are divided using the same spatial partitioning method as Type A. Vision-based constraints ensure end-effector separation during harvesting. Crucially, the distance between the bases on the distal side (1210 mm) exceeds twice the critical linkage length (1040 mm). Therefore, collisions involving J3 and its connecting linkage are avoided. Despite workspace overlap, the Type B configuration achieves collision-free operation through partitioned task allocation.

To verify that the picking operation based on the overlapping B-type dual-arm layout in the workspace will not collide, the left robotic arm joint 2 and joint 3 were controlled to rotate 90° each, the right robotic arm joint 2 and joint 3 to rotate −90° each, and the remaining 5 joints to move at 0°. The position changes of joint 3 on the left and right robotic arms were monitored, and the real-time changes in the x-space coordinates of joint 3 were obtained as shown in Figure 13.

It can be seen that during the movement of the left and right robotic arms to two points within the left and right segmentation zones, there was no cross phenomenon in the x-values of the three-dimensional spatial coordinates of the dual-arm joint 3, indicating that there was no collision between the left and right robotic arms during the movement process. The Type B overlapping workspace is visualized computationally in Figure 14, showing the partial merger of two hemispherical operational zones.

Considering the need for operational continuity, parallel task safety, and chassis stability, the “Type B Overlapping Workspace” configuration was selected as the optimal spatial arrangement for the harvesting robot’s dual manipulators. A physical implementation of this layout on the robot platform is presented in Figure 15.

2.2. Analysis of Apple No-Missed Harvesting Operation Strategy Based on Dual-Arm Physical Spatial Distribution

Due to the six-axis mechanical arm of the dual-arm picking robot, its working space is formed by partially overlapping and merging the working intervals of two single-arm hemispherical bodies. Therefore, there may be fruit missing during picking operations due to the discontinuity of the parallel working space of the dual arms. The top view schematic diagram of the situation of fruit missing during double-arm picking is shown in Figure 16.

To ensure the continuity of the dual-arm parallel operation area, the physical parameters of the dual-arm picking space were analyzed, and combined with the distribution of the tree crown boundary line, the maximum distance d₁ between the robot and the apple tree required for the apple no-missed harvesting operation based on the dual-arm physical space distribution was analyzed. The top view of the spatial distribution of the dual-arm continuous no-missed harvesting operation is shown in Figure 17, where R is the working radius of the single arm, d₂ is the distance between the dual-arm base, L₂ is the outer edge interface of the apple tree crown, L₁ is the plane tangent to the overlapping area of the left and right arm picking spaces, which is parallel to the plane where O₁O₂ is located and has a distance of 695 mm (d₁) from it.

It can be seen that when the distance between the outer boundary surface L₂ of the apple tree crown and the plane where O₁O₂ is located is less than d₁, the double-arm parallel operation area inside the crown where the apples to be harvested are located is continuous. Therefore, in order to ensure the continuity of the dual-arm parallel picking space in both horizontal and vertical directions and to lay a theoretical foundation for the design of the subsequent adjustable six-axis dual arm-picking robot, the strategy for the double-arm apple picking operation is to make the apple to be picked as close as possible to the space between the L₁ plane and the plane passing through O₁O₂ and parallel to the L₁ plane, that is, the vertical distance between the fruit to be picked and the plane passing through O₁O₂ and parallel to the L₁ plane is less than d₁.

In order to analyze the continuous conditions of the dual-arm workspace more intuitively, a schematic diagram of the apple no-missed harvesting operation strategy based on the physical spatial distribution of the dual arms is drawn using the Python programming language, as shown in Figure 18. The workspace is formed by partially overlapping and merging two single-arm hemispherical operation intervals, and the continuous boundary plane of the workspace has been marked in the three-dimensional diagram.

2.3. Construction of a Prototype Double-Arm Picking Robot

Design the overall scheme of the dual-arm apple picking robot: The robot mainly consists of dual robotic arms and their control system, picking manipulator, visual perception system, robot control terminal, and robot tracked power chassis. The overall scheme design is shown in Figure 19.

The specific core components included in each component of the picking robot are as follows:

Dual robotic arms: Two AUBO-i5 lightweight 6-degree-of-freedom collaborative robotic arms. The AUBO robotic arm adopts a modular joint design and uses a developer-oriented robot system. Users can develop robot control systems that meet specific job requirements based on the application programming interfaces provided by the AUBO platform; In addition, the AUBO robotic arm is equipped with a dedicated programmable operation interface, through which users can observe the real-time operation status of the robot and perform various control settings on the robot.

Mechanical arm: ① Flexible gripper picking hand: The four-finger joint finger cavity structure of the flexible gripper is an isosceles trapezoid. The flexible gripping system is mainly composed of actuators, pneumatic controllers, and accessory air circuits. By connecting the air source of the SCB-18IAS active controller and the picking robotic arm, the construction of the flexible gripping system can be achieved. Based on the communication protocol of the active controller, the flexible gripper can autonomously control the opening, grasping, pressure relief and other actions, as well as the magnitude of pressure, to achieve collaborative picking work with the robotic arm. ② Suction cup type picking hand: This pneumatic biomimetic non-destructive picking hand draws inspiration from the principle of “octopus adsorbing prey”. The picking hand has a symmetrical center structure, with four suction nozzles evenly distributed on the palm. Under the action of vacuum negative pressure, the suction nozzles can produce concave deformation that fits the surface of the apple and drag the apple away from the branch to complete the picking operation through the suction force generated by the negative pressure in the four suction nozzles.

Visual perception system: mainly composed of an adjustable camera mounting bracket mounted on the robot control cabinet, an Intel Realsense D435 depth camera, and its USB 3.2 signal transmission cable. The D435 camera is equipped with a global shutter and a wide field of view, with a minimum sensing depth of about 0.11 m and a maximum sensing distance of up to 10 m. It is suitable for standardized modern apple orchards and has a depth resolution of up to 1280 × 720. It is suitable for being mounted on robots to capture depth information of external targets.

Robot control terminal: The hardware is a Lenovo Y7000P computer; the operating system is Windows 10 ×64; the CPU processor model is Intel (R) Core (TM) i7-10875H; the memory type is DDR4; the maximum memory capacity is 32 GB; and the GPU graphics card is RTX 2060. The control terminal is equipped with visual recognition and positioning algorithms, motion control strategy algorithms for dual AUBO-i5 robotic arms, serial communication control programs for flexible gripper picking hands and suction cup picking hands, and parallel communication system control programs for dual robotic arms.

Robot tracked mobile chassis: Wheeled robot mobile platforms are not suitable for use on soft and bumpy ground in orchards, as their off-road mobility is weak. The advantages of tracked robot mobile platforms include small turning radius, less slip during movement, good obstacle-crossing ability, and strong adaptability to orchard working environments. Therefore, based on the complex terrain of modern apple orchards in China, tracked mobile platforms are determined as the power chassis for harvesting robots.

The prototype of the dual-arm picking robot and its main components are illustrated in Figure 20a.

The main integrated control cabinet of the dual-arm picking robot prototype built, which includes key components such as the picking hand, right robotic arm control cabinet, SCB-18IAS active controller, DC-DC stabilized power module, etc., is shown in Figure 20b and Figure 21.

3. Methodology of Robot Dual Arm Parallel Operation

3.1. Control Method of Robotic Arm and End Picking Hand

3.1.1. The Motion Control Method of Robotic Arm

The motion control of the robotic arm for harvesting robots mainly involves three aspects: firstly, the effective connection and real-time information communication between the computer and the AUBO-i5 robotic arm control end; secondly, there is a motion control method for robotic arms based on Python programming; finally, the operation control method of the end effector of the robotic arm.

The network setting unit of the AUBO-i5 control terminal can enable third-party interfaces to perform control-oriented network settings. In this setting interface, the specified network card name and its IP address, subnet mask, default gateway, and other parameters can be configured. The network IP address of the third-party control port needs to be in the same network segment as the IP address of the AUBO-i5 local machine. In the network debugging interface of the AUBO-i5 teaching pendant, the ping command can be used to check whether the robotic arm is pinging with external devices. The ifconfig command can be used to check the IP address, subnet mask, and other information of the AUBO-i5 body. The server status can be used to check whether the robotic arm’s server port number is in a listening state. The connection between the computer and the robotic arm control end requires setting the IP address of the computer end and the IP address of the robotic arm control end in the same network segment (different IP addresses).

The motion control of a robotic arm based on Python programming first requires obtaining the real-time pose information of the target apple obtained by the robot vision system, that is, obtaining the position and pose information that the end effector of the robotic arm needs to reach. Furthermore, through the Python programming interface on the control end of the robotic arm, based on the inverse solution algorithm of the six-axis robotic arm motion, the programming realizes the inverse solution value of the robotic arm axis motion according to the target apple information, that is, the angle values of the six joints of the robotic arm that need to be rotated separately are calculated. Finally, based on the Python programming interface, control the position and posture of the robotic arm axis to the target point. Among them, controlling the axial movement of the robotic arm is based on the AUBO-i5 Python programming application programming interface (API).

3.1.2. Control Method of End Picking Manipulator

The component of the apple picking robot that directly interacts with the fruit is the picking end effector, also known as the end effector. In order to adopt a flexible apple picking hand with stable grip function, based on the morphological characteristics of apples, this study determined to use two types of flexible picking hands for the picking robot: a flexible gripper picking hand and a suction cup picking hand. In the future, comparative experiments will be conducted on the picking performance of two types of pickers in orchards to determine the end effector manipulator that is more suitable for robots and can ensure the success rate of picking operations.

Motion Control Method of Flexible Claw Picking Hand: Fix the flexible gripper picking hand to the end of the robotic arm of the picking robot and connect the three gripping fingers of the gripper through pneumatic pipes and connect them to the pneumatic output port of the SCB-18IAS active controller. The gripping and opening motion of the flexible gripper is controlled by the SCB-18IAS active controller through a Python code program written on a PC. The PC is connected to the active controller through an RS232 serial communication cable. One end of the serial cable is connected to the USB interface of the PC, and the other end is connected to the CN2 electrical control port of the active controller. The physical image of the flexible gripper picking hand and the schematic diagram of the apple flexible gripper motion control method based on serial communication are shown in Figure 22 and Figure 23, respectively.

The control instructions for the grasping and opening actions of the flexible gripper based on the SCB-18IAS active controller are shown in

Table 1. Grasp/open control command based on SCB-18IAS active controller.

Serial Communication Command	Claw Grip		Claw Open
Function code	Send [0]	0xFA	Send [0]	0xFB
Gear selection	Send [1]	0x09	Send [1]	0x09
Supplement code	Send [2]	0xFF	Send [2]	0xFF
Supplement code	Send [3]	0xFF	Send [3]	0xFF

.

The motion control method of suction cup picking hand: Fix the suction cup picking hand to the end of the mechanical arm of the picking robot and connect the flexible suction nozzles of the four metal suction cups installed on the picking hand to the suction ports of the FA0520E two-position three-way solenoid valve through pneumatic pipelines. The suction outlet end of the solenoid valve is connected to the inlet end of the air pump. Among them, the solenoid valve is connected to the servo control port of the L293D motor drive board through electronic switch lines, and the positive and negative power lines of the four air pumps are, respectively, connected to the DC motor power input and output ports of the L293D motor drive board. The power supply for the air pump is provided by a portable battery module externally connected to the L293D motor drive board.

The battery is stepped down through the LM2596 DC-DC adjustable voltage reduction power module to provide the motor with the appropriate operating voltage. The suction and release actions of the suction cup are controlled by the Arduino UNO R3 development board embedded above the L293D motor drive board. The USB interface of the UNO R3 development board is connected to the PC through a serial communication cable. The Python code program written in the PC can control the Arduino UNO R3 board based on serial port instructions to complete the suction and release actions of the suction cup.

The schematic diagram and physical image of the mechanical dimensions of the end effector manipulator for air suction picking are shown in Figure 24.

Before controlling the suction cup based on serial communication, it is necessary to write C program code and burn the code into the UNO R3 development board using the Arduino programming development environment (IDE). The C code specifically burned to the UNO R3 development board includes the control program for negative pressure suction and release of the picking suction cup hand pump motor (including coordinated control of four solenoid valves).

The composition diagram of the suction and release action control system of the suction cup type picking hand and the schematic diagram of the system hardware composition are shown in Figure 25.

The working principle diagram of the vacuum pump, suction cup, and solenoid valve of the suction cup picking hand to achieve the suction and release of apples is shown in Figure 26.

3.2. Design of Dual-Arm Parallel Communication Mechanism

3.2.1. Establishment of Parallel Motion Communication System for Dual Robotic Arms

In order to achieve the coherent picking operation of the dual-arm robot, it is necessary to independently and parallelly control the two robotic arms to achieve synchronous parallel operation, thereby improving the efficiency of apple picking. The parallel control of two AUBO-i5 robotic arms is based on different IP addresses of the robotic arms and sending motion control signals to them separately.

The specific communication hardware implementation is based on the dual robotic arm parallel motion communication module, and the signal transmission equipment is the WLF22-1105 100 Mbps five-port Ethernet switch. A switch is a network device used for forwarding electrical (optical) signals, which can provide an exclusive electrical signal path for any two network nodes accessing the switch. Switches can perform data transmission between multiple port pairs at the same time, and each port can be regarded as an independent physical network segment. The network devices connected to it enjoy all the bandwidth independently, without competing with other devices for use. The WLF22-1105 switch provides 5 adaptive ports, and the hardware is designed based on a highly integrated structure, using store-and-forward technology and dynamic memory allocation to ensure effective allocation to each port. Based on traffic control, it ensures that data packets are not lost during the transmission and reception of nodes. The parameters of the 100 Mbps Ethernet switch are shown in

Table 2. Key technical parameters of Ethernet switch.

Switch Technical Specifications	Specific Parameters
model	WLF22-1105
Port transmission rate	5100 Mbps automatic flip RJ45 ports
Functional Features	One-click VLAN port isolation, 250 m long-distance transmission, loop detection
Standard	IEEE 802.3, IEEE 802.3u, IEEE 802.3x, IEEE 802.3az, IEEE 802.3ab
Power supply mode	DC, Micro USB

.

When port A sends data to port D, port B can simultaneously send data to port C, and both transmissions enjoy the full bandwidth of the network and have their own virtual connections. The hardware of the dual robotic arm parallel motion communication module is shown in Figure 27, where Line 1 connects to the Ethernet interface of the computer end, Line 2 connects to the Ethernet interface in the control cabinet of the right robotic arm of the picking robot, and Line 3 connects to the Ethernet interface in the control cabinet of the left robotic arm of the picking robot.

Two control terminals are opened on the computer side and connected to the Ethernet control interface IP of the two robotic arms, respectively, for matching. The parallel motion control of the dual robotic arms is achieved by sending motion control instructions to the IP addresses of the two robotic arms from different terminals opened on the computer side simultaneously.

3.2.2. Establishment of Communication Mechanism Between Visual System and Dual Robot Arm Control System

In order to transmit the three-dimensional spatial position values of the fruit to be picked to the robotic arm motion control system and then drive the robotic arm axis to the apple target position, it is necessary to establish communication between the picking robot vision system and the dual robotic arm motion control system separately.

If the running code of the visual inspection system is directly embedded into the motion control code of the robotic arm, there will be conflicts in calling the running visual system due to the parallel nature of the dual robotic arm motion. Therefore, an independent terminal needs to be opened for the operation of the visual system. Run the apple target recognition and positioning program in an independently opened terminal, based on the D435 depth camera for recognition and positioning, continuously import video streams for apple target recognition and positioning, and based on the picking operation partition of the dual robotic arm, refresh the spatial coordinates of the apples assigned to the left arm for picking in real time and write them to the specified “Left arm Apple Picking Coordinate Information.txt” folder, and refresh the spatial coordinates of the apples assigned to the right arm for picking in real time and write them to the specified “Right arm Apple Picking Coordinate Information.txt” folder.

The way to store the spatial coordinates of the apple target to be picked output by the visual system is to write the (x, y, z) information of the apple target output in the current frame relative to the base coordinate system of the robotic arm in each line.

After activating the motion control terminal of the dual robotic arm, the left and right arms will automatically read the spatial position information of the apple target to be picked from the “Left arm picking apple coordinate information.txt” and “Right arm picking apple coordinate information.txt”, respectively, and then execute the apple target coherent picking program for fruit picking operation.

Based on the above design, a communication mechanism between the robot vision system and the dual robotic arm control system was established through independent programming. Due to the fact that the visual recognition positioning system, left robotic arm control end, and right robotic arm control end are controlled based on three independent terminals opened by the computer end, the robotic arm completes the picking operation of the current fruit and needs to pick the next fruit in real time. It automatically obtains the spatial position information of the apple target from the TXT document stored in the visual system. The left/right robotic arm also obtains the spatial coordinate information of the apple to be picked from the corresponding TXT document position. The visual system and the motion control systems of the left and right arms do not interfere with each other and can run in parallel, thus forming a modular coupling mode between the dual-arm motion control system and the visual system that can run in parallel. The communication mechanism schematic of the visual system dual robotic arm control system is shown in Figure 28.

On the other hand, during the process of apple picking, once an apple is picked by the robot, the positions of other uncollected apples in the surrounding area may undergo slight changes due to changes in the force on the branches. Therefore, in this study, the recognition and localization of apple targets are carried out in real time by importing video streams. Therefore, the position information of the apple target to be picked is dynamically updated. The data obtained by the robotic arm in the TXT file before preparing to pick the apple are the real-time positioning information of the fruit to be picked, which can ensure the accuracy of the spatial positioning of the apple.

3.3. Design and Motion Trajectory Analysis of Dual-Arm Parallel Coherent Picking Strategy

In the X-axis direction of the camera coordinate system, the maximum prediction error of the apple recognition and positioning model is 4 mm, with an average error of 2.15 mm; in the Y-axis direction of the camera coordinate system, the maximum prediction error is 4 mm, with an average error of 2.15 mm; in the Z-axis direction of the camera coordinate system, the maximum prediction error is 6 mm, with an average error of 2.5 mm; the maximum root mean square error of the apple recognition and positioning model is 4.32 mm, with an average error of 2.45 mm. This value is relatively small and can meet the positioning accuracy requirements of the three-dimensional spatial position of apple picking robots in actual orchard applications.

The coherent picking process of the dual-arm picking robot is as follows: Firstly, the robot vision system recognizes and locates the apple target and stores information. Then, the left and right robotic arms actively obtain the spatial coordinates of the assigned fruit to be picked, and the inverse kinematics of the robotic arm is used to solve the model to obtain 6 joint angles. The suction cup robotic arm’s air pump is started (or the flexible gripper is opened), and the robotic arm’s axis is controlled to move to the position of the apple to be picked. The suction cup hand automatically completes the suction operation (or the flexible gripper grasping operation) of the apple, and then the robotic arm’s axis moves to the separation point of the picked fruit. Finally, the fruit release operation is performed to place the apple in the collection box of the orchard transportation robot. After that, the robotic arm moves to the picking preparation posture to prepare for the next fruit target picking work. The above series of dual-arm continuous picking operations can be summarized as a key operational step of the dual robotic arm “apple recognition and positioning—dual robotic arm movement—robotic arm adsorption and picking—robotic arm movement and fruit release” continuous picking strategy, which includes the following core steps:

① Launch the apple target recognition and localization program;

② Activate the left and right robotic arm motion control terminals;

③ The left and right robotic arms actively obtain the spatial position of the apple to be picked and perform inverse kinematics calculation;

④ Move the dual robotic arms to the picking preparation posture;

⑤ The dual robotic arms move to the picking approach point posture;

⑥ Suction cup picking hand air pump is turned on (or flexible gripper performs opening operation);

⑦ The dual robotic arms move to the position of the fruit to be picked, and the suction cup hand adsorbs the apple (or the flexible gripper performs the grasping operation);

⑧ The dual robotic arms move to the fruit separation point posture, and the fruit picking operation is completed;

⑨ The dual robotic arms move to the fruit release position, and the suction cup-type picking hand solenoid valve is activated (or the flexible gripper performs an opening operation). The fruit falls off into the flexible fruit release pipeline of the transport robot, and the fruit picking operation is completed;

⑩ Starting from step ③, the next fruit picking operation will be executed in a loop.

The continuous picking process of the dual-arm picking robot is shown in Figure 29.

Among them, the picking preparation posture needs to be preset, mainly to apply necessary constraints to the motion path of the robotic arm picking apples, in order to avoid collisions with tree branches and other obstacles during the process of the robotic arm moving directly from the fruit placement posture axis to the apple position. At the same time, it also avoids the mechanical arm from hitting the picking robot chassis equipped with the robotic arm due to the low position of the second joint when picking fruits at lower positions, which may cause damage to the robotic arm and failure of the picking operation.

When the robotic arm is in the picking approach point posture, its three-dimensional coordinates at the end are the same as the x and y coordinates of the apple to be picked, and the z coordinate is subtracted by a fixed value. The reasons for designing the picking approach point are as follows: firstly, the robotic arm receives the three-dimensional coordinate information of the fruit to be picked. After installing the picking hand, directly executing the motion planning of the robotic arm based on the inverse solution model to pick apples will cause collision between the robotic arm and the apple, resulting in picking failure; secondly, the orchard environment is complex. If no picking approach point is set, the joints of the robotic arm are highly likely to collide with tree branches or other fruits (not currently being picked) during the process of moving to the apple target.

The setting of the fruit release point posture is based on the fruit release positions of the transport robot corresponding to the left and right robotic arms. The setting of this posture needs to ensure that the picked fruits smoothly fall into the apple conveying pipeline after the electromagnetic valve of the air suction picking hand is opened (the suction cup of the picking hand is depressurized).

Based on the V-REP robotic arm motion simulation software, Python programming language is used to control the motion of the robotic arm. The proposed dual robotic arm coherent picking strategy is subjected to kinematic simulation. The AUBO-i5 model is used to visualize and analyze the motion trajectory of the robotic arm and output motion data parameters. The simulation trajectory of the coherent picking strategy of the robotic arm (indicated by the blue curve in Figure 30) and the schematic diagram of different key stages of the operation are shown in Figure 30. It can be seen that the motion trajectory is relatively smooth and the robotic arm has no movement phenomenon.

During the continuous picking simulation motion of the robotic arm, the spatial position changes of the end effector and the six joint angle changes of the robotic arm were monitored. The real-time change curves of the x, y, z coordinates and six joint angles of the end effector are shown in Figure 31 and Figure 32, respectively. Based on a comprehensive analysis of the three-dimensional spatial position and joint angle variation curves at the end, the position points where the robotic arm reaches each key link pose through continuous picking are marked in Figure 32. Through the variation curves of the end points and six joint angles, it can be visually seen that the spatial position and joint angle changes at the end during the operation of the robotic arm are relatively smooth without sudden changes, further verifying the feasibility of the proposed continuous picking strategy.

4. Apple Orchard Picking Experimental Results

4.1. Robot Picking Performance Test Based on Different End Effectors

Based on different end effectors, a robot picking experiment was conducted in an apple orchard from September to November 2022. After the robot arrived at the apple picking area, the visual terminal of the dual-arm picking robot and the control terminals of two mechanical arms were activated, totaling three terminals. The picking robot then began to perform the apple picking operation. The left arm end of the robot was equipped with a suction cup-type mechanical arm, and the right arm end was equipped with a flexible gripper picking hand. The experimental device and some examples of the experimental process are shown in Figure 33. Based on the dual-arm picking robots equipped with suction cup-type robotic arms and flexible gripper picking arms, 90 apples were picked from trees each, and the success rate of picking was calculated and the reasons for picking failure were analyzed.

The results of orchard picking experiments on apples using two types of end effectors are shown in

Table 3. Analysis and statistics of robot picking results based on different end effectors.

	Picking Hand Type	Flexible Claw Picking Hand		Suction Cup Picking Hand
Picking Result Statistics		Quantity	Proportion/%	Quantity	Proportion/%
Number of experimental fruits		90	100	90	100
Number of successful harvests		72	80.0	77	85.6
Number of failed suction/grip attempts		7	7.8	8	8.9
Number of failed fruit–branch separation		8	8.9	3	3.3
Number of other failures		3	3.3	2	2.2

. According to the table, the overall picking success rate of the suction cup robotic arm is 85.6%, which is higher than that of the flexible gripper picking hand (80%).

For flexible gripper picking hands, the failure of picking mainly occurs in the grasping stage, fruit–branch separation stage, and other stages, accounting for 7.8%, 8.9%, and 3.3%, respectively, with the highest proportion of failure in fruit–branch separation.

The main reason for the failure of the grasping process is that the fingers of the flexible gripper collide with the branches connected to the apple during the process of approaching and grasping it, resulting in displacement of the apple. Therefore, the flexible gripper cannot wrap and grasp the fruit, leading to the failure of picking. The main reason for the failure of the separation process is that the fruit is in close proximity or in contact with its growing branches, causing the flexible gripper to simultaneously wrap the connected branches when grasping the fruit, ultimately leading to the failure of fruit separation. The failure of other links is mainly caused by the following two reasons: the spatial position of the apple target is a singular point, and the inverse kinematics solution of the robotic arm fails to reach it; the environment inside the tree crown of the picking area is complex, and the mechanical arm collides with thicker branches, causing the mechanical arm to stop and start urgently.

For suction cup pickers, the failure of picking mainly occurs in the suction stage, fruit–branch separation stage, and other stages, accounting for 8.9%, 3.3%, and 2.2%, respectively, with the highest proportion of failure in the suction stage.

The reasons for the failure of the suction process mainly include the following situations: firstly, the growth posture of the fruit is bottom facing outward, and the physical depression structure at the bottom of the apple causes the suction cup hand to contact the apple, and the suction nozzle cannot fully adhere to the surface of the fruit, thus unable to guarantee the vacuum suction state of the suction cup, directly leading to suction failure and picking failure. Secondly, some fruits grow at the end of the twigs. When the suction cup picking hand moves to the fruit position to suck it up, the apple will produce a small displacement, causing the suction cup to be unable to attach the apple, resulting in picking failure. Thirdly, the surface of individual apples is covered by leaves, which prevents the four suction nozzles from fully adsorbing the apples, resulting in a decrease in overall adsorption force and leading to harvesting failure. Fourthly, a very small number of fruits have abnormal appearance, which causes some suction nozzles to not fully adhere to the surface of the apple, resulting in harvesting failure.

The main reason for the failure of the fruit–branch separation process is that some fruits grow at the end of the fine branches. When the suction cup picking hand holds the fruit, during the movement of the robotic arm to the fruit separation point, the direction of the force separating the fruit changes to the same direction as the branches associated with the apple, causing the fruit separation force to shift from mainly shear force to the longitudinal tensile force that needs to overcome the same direction as the fruit stem and branches. Therefore, a larger separation force is required, ultimately leading to fruit separation failure. The main reasons for the failure of other processes are the same as when using a flexible gripper robotic arm.

4.2. Efficiency Test of Parallel Harvesting with Dual Robotic Arms

Based on the AP-7200 electronic stopwatch timer (accuracy: 0.001 s, Guangzhou Aopi Sports Goods Co., Ltd., Guangzhou, China), during the continuous fruit picking experiment of the robot, the time from the movement of the robotic arm to the picking preparation posture, the picking approach point posture, the target position of the apple to be picked, the fruit separation point, and the fruit release point posture were recorded by reading seconds to comprehensively analyze the picking efficiency of the dual-arm robot. An example of different picking operation stages for robots is shown in Figure 34.

The vision system of the robot is embedded with a visual algorithm for apple target recognition for harvesting robots. Before executing the harvesting operation, the robot’s vision terminal is turned on to perform real-time recognition and positioning of apple targets. Due to the modular coupling mode between the dual-arm motion control system and the vision system that can run in parallel, the recognition and positioning time (in milliseconds) can be ignored in the total time consumption of single-arm single-fruit harvesting.

A timing analysis was conducted on the continuous picking process of the robotic arm of the dual-arm robot. The time taken for the robotic arm to reach the picking preparation posture, the picking approach point posture, the target position of the apple to be picked, the fruit separation point, and the fruit release point posture were counted. The average time consumption of each step is shown in

Table 4. Average time consumption of each operation link of single apple picking utilizing double-arm robot.

Arriving at Different Stages of Picking Operations	Picking Preparation Posture	Picking Approach Point Posture	Target Location for Picking Apples	Fruit Separation Point	Fruit Release Point	Total Time for Single-Arm Picking
Average time/s	2.1	1.2	0.8	2.5	0.9	7.5

.

As shown in the table above, it takes an average of 7.5 s to complete the picking operation of a single fruit by sequentially reaching five different robotic arm postures during the key stages of “robot arm movement—robot arm adsorption and picking—robotic arm fruit release”. Among them, the longest time required from the robot arm reaching the target position of the apple to be picked to the fruit separation point is about 2.5 s, which includes the process of the picking hand adsorbing the apple. On the other hand, the robotic arm takes a relatively long time, about 2.1 s, to reach the picking preparation posture from the fruit release point posture, which is due to the process of releasing the fruit by the suction cup picking hand. In addition, there is a significant difference between the posture of the robotic arm at the fruit release point and the preparation posture for picking, so increasing the motion range of each joint also requires a certain amount of time. The remaining steps: from the picking preparation posture to the picking approach point posture, from the picking approach point posture to the position of the fruit to be picked, and from the fruit separation point posture to the fruit release point posture, take relatively less time, which are 1.2 s, 0.8 s, and 0.9 s, respectively.

5. Discussion

In order to achieve dual robotic arm apple picking and improve the efficiency of robot operations, based on the analysis of three spatial layout schemes for dual robotic arms, a dual-arm spatial layout design for picking and collision avoidance was proposed. The strategy of apple picking without missing out was analyzed, and a prototype of a dual-arm picking robot was built. On this basis, control systems for flexible gripper picking hands and suction cup picking hands were built, and control program code was written to study their control methods. Then, the coherent picking strategy, parallel motion communication system, and communication mechanism between the vision system and the dual-arm control system of the dual-arm apple picking robot were studied. To achieve independent parallel control of the dual arm to achieve synchronous parallel operation of the dual arm and improve the apple picking efficiency of the robot, a dual-arm parallel motion communication system was established based on an Ethernet switch. Then, a communication mechanism between a robot vision system and a dual robotic arm control system was proposed and autonomously programmed. Furthermore, a dual-arm coherent picking strategy was proposed, and the motion trajectory and pose of the robotic arm based on this coherent picking strategy were monitored in real time. Motion simulation analysis was conducted to verify its feasibility.

Finally, performance tests of dual-arm harvesting robots were conducted in different dimensions in modern Red Fuji apple orchards. The results of the robot harvesting performance tests based on different end effectors showed that the overall harvesting success rate of the suction cup robotic arm was 85.6% for picking apples from 90 trees, which was higher than that of the flexible gripper harvesting hand (80%). The results of the dual-arm robot picking efficiency test show that the average time required for a single machine arm to pick a single fruit completely (including the robotic arm reaching the picking preparation posture, the picking approach point posture, the target position of the apple to be picked, the fruit separation point, and the fruit release point posture, totaling 5 processes) is 7.5 s, and the overall picking efficiency of the robot is 3.75 s per fruit.

Our field evaluations confirmed two key operational constraints within the tested modern orchard environment: Weather Dependency: The current prototype operates optimally under stable, dry conditions. Performance in rain, strong winds, or extreme humidity was not validated. Occlusion Challenges: While our vision system achieved high detection success on partially obscured targets, leaf occlusions directly contributed to suction failures (incomplete seal due to foliage). Crucially, no fruit damage occurred during successful harvests, confirming the safety-centric end-effector design.

Future research work mainly includes the following: (1) optimization of end effectors: improving suction cup materials, structures, or adsorption control strategies (such as variable adsorption force); continuously optimizing the adaptability and non-destructive gripping reliability of flexible grippers for apples of different sizes and shapes; (2) further improving work efficiency and intelligence level: further optimizing the picking strategy, researching dynamic task allocation strategy based on scene perception on the basis of existing coherent picking strategies, achieving more efficient autonomous collaboration between both arms, and further shortening the average time of single fruit picking; (3) high-speed and high-precision control: optimizing the trajectory planning and servo control algorithms of the robotic arm, improving the motion speed while ensuring positioning accuracy, and exploring lightweight robotic arm design or higher performance joint drive schemes.

From the perspective of single fruit picking time of robots, the fastest one is the air suction apple picking robot produced by Abundant Robotics [25], which only takes 1.5 s to pick a single fruit. However, on the other hand, the application scenario of this robot is for V-shaped fruit tree wall planting mode apple trees abroad, while the widely planted spindle-shaped apple trees in China have complex tree crown space (including obstruction of apples by branches). Therefore, the effect of applying it to spindle-shaped apple trees in modern apple orchards in China for automated fruit picking needs further verification. In addition, the single fruit picking time of the picking robots developed by the University of Florida [18] and Washington State University [23,24] in 2010, 2017, and 2019 were 7.1 s, 6.1 s, and 7.3 s, respectively, all higher than the single fruit picking speed of the robot developed in this paper. However, the robot developed in this paper can achieve dual-arm parallel operation. From the perspective of overall operating efficiency, the equivalent single fruit picking time of the dual-arm robot developed in this paper is about 3.75 s (7.5 s/2).

From the perspective of overall picking efficiency, the robot developed in this article has the function of parallel operation with dual robotic arms, resulting in higher picking efficiency compared to single-arm robots developed by other units, with an average picking efficiency of 3.75 s per fruit. However, compared with the air suction apple picking robot produced by Abundant Robotics [25], which can pick one fruit in 1.5 s, there is still room for improvement in picking efficiency. Further analysis shows that due to the fact that the company produces an air suction robot with a Cartesian coordinate system, its picking actuator is integrated with “picking and harvesting”. After separating the apple and fruit stem, the fruit can be directly sucked into the pipeline by the robot and flow into the collection device, thus shortening the time consumption of the picking process for individual fruits. Therefore, optimizing and improving the development of an air suction apple picking robot with a lightweight rectangular coordinate system configuration suitable for the spindle-shaped apple trees widely planted in China is an important research direction in the later stage.

Future research will also focus on building an intelligent apple harvesting system tailored to the characteristics of spindle-shaped orchards and comprehensively improving system performance and adaptability through multi-dimensional collaborative innovation. At the core technology level, the focus is on breaking through the bottleneck of environmental adaptability and intelligent control of end effectors, developing a new generation of biomimetic adsorption and flexible clamping mechanisms, and achieving dynamic force control based on fruit characteristics. Simultaneously, by deeply integrating multimodal perception technology, we aim to overcome the challenges of target recognition and localization under conditions of deep occlusion by branches and leaves and explore the active intervention mechanism of robotic arms to create a harvesting workspace. At the system architecture level, we will develop a dynamic task allocation strategy based on real-time environment understanding, break through the limitations of traditional sequential tasks, and achieve asymmetric collaboration and autonomous obstacle avoidance. Simultaneously, we will advance the research and development of lightweight high-speed harvesting platforms, explore integrated “picking–collection” conveying systems, optimize single fruit processing processes, and shorten harvesting cycles. At the level of work capability building, we will strengthen the adaptability to all working conditions, develop anti-interference perception systems to cope with complex conditions such as rain, fog, and changes in lighting, design protective ontology structures, and develop environment-adaptive control algorithms to compensate for external disturbances such as wind disturbance. We will form a reusable technology ecosystem, improve system reliability through orchard verification, and extend technology migration to intelligent harvesting scenarios for similar crops such as citrus and pear, promoting the transformation and upgrading of the fruit industry towards high efficiency, intelligence, and unmanned direction and providing technical support for addressing agricultural labor shortages and industrial upgrading needs.