Designing and Testing a Picking and Selecting Integrated Remote-Operation-Type Dragon-Fruit-Picking Device

Abstract

In order to effectively solve the problems of the complex growth state of dragon fruit and how the picking process is mostly manual, this study designed a picking and selecting integrated remote-operation-type dragon-fruit-picking device. Based on SOLIDWORKS 2020 software for the three-dimensional digital design and overall assembly of key components, the structure and working theory of the machine are introduced. By improving the high-recognition-rate dragon fruit target detection algorithm based on YOLOv5, better recognition and locating effects were achieved for targets with a small size and high density, as well as those in bright-light scenes. Serial communication, information acquisition, and the precise control of each picking action were realized by building the software and hardware platforms of the picking device control system. By analyzing the working principle of the mechanical system and the mechanics of the machine picking process, the critical factors affecting the net picking rate and damage rate of the dragon-fruit-picking device were confirmed. Based on the force and parameter analysis of the test results, it was confirmed that the machine had an optimal picking influence when the flexible claw closing speed was 0.029 m/s, the electric cylinder extending speed was 0.085 m/s, and the mechanical arm moving speed was 0.15 m/s. The net picking rate of the device reached 90.5%, and the damage rate reached 2.9%. The picking device can complete the picking of a single dragon fruit, as well as a plurality of fruits grown at a growing point, and integrates the integration of picking fruits, removing bad fruits, and sorting fruits, which can improve the efficiency of dragon fruit harvesting and replace manual work.

1. Introduction

Dragon fruit, also referred to as jade dragon fruit or dragon pearl fruit [1,2,3], has rich nutritional value [4,5]. Dragon fruit is rich in anthocyanins, vitamin E, and other antioxidant substances as well as bioactive compounds [6,7] and has been shown to have a variety of positive health effects, such as promoting human metabolism and antidiabetic, anti-inflammatory, anticancer, and antimicrobial effects [8,9,10]. With the rapid development of China’s economy and the continuous improvement of the living standards of its people, people’s demand for dragon fruit and other fruits is also increasing. With the continuous expansion of the dragon fruit market, China’s dragon fruit planting area is increasing [11]; by 2021, China’s dragon fruit planting area exceeded 66,700 hm², making it the biggest area in the world [12]. China is not only one of the largest dragon fruit producers, but also one of the largest dragon fruit consumers [13]. This greatly promotes the development of the domestic dragon fruit industry, as dragon fruit picking is an important part of the dragon fruit industry, as well as being an essential link [14]; however, because dragon fruit roots are very hard and the fruit growth stages are diverse, it is difficult to program a market fruit-picking machine to distinguish between the different fruit stages of dragon fruit picking. Therefore, it is difficult to realize the mechanized harvesting of dragon fruit [15], and, at present, dragon fruit is usually picked manually, which is labor-intensive and has low picking efficiency, restricting the development of the dragon fruit industry. In the main production areas of dragon fruit, owing to the large area and wide range of dragon fruit planting, manual picking cannot satisfy the requirements of efficient picking, so mechanized picking is an inevitable choice [16]. Dragon fruit belongs to the cactus family of plants [17], so the branches are hard and thorny in appearance; thus, picking not only requires special tools, but may also cause injury to workers. Therefore, the study of an automatic dragon-fruit-picking robot is of practical value.

No research reports on dragon-fruit-picking robots have been found abroad; however, developed countries have been studying fruit picking for a long time, and they have achieved many results by combining advanced technology, unique picking forms, and a highly developed level of intelligence. D.O. Khort et al. [18] developed a strawberry collection device with a computer vision system that can identify the position according to the size and ripeness of the fruit coordinates and pick the fruit using an automatic manipulator with a computer vision system. This collection device combines a robotic arm and visual recognition system with the ability to work synergistically. The visual recognition system and robotic arm work in conjunction to improve fruit-picking accuracy and picking efficiency. Arman Arefi et al. [19] developed a new segmentation algorithm for guiding a robotic arm to use a machine vision system to pick ripe tomatoes that achieves the fast detection and localization of fruits by the picking machine and improves the detection ability of the algorithm in complex scenes. Nguyen Pham Thuc Anh et al. [20] developed a robotic system for autonomous pineapple harvesting with a customized and special end effector as well as an image-based harvesting control unit, which is based on a convolutional neural network to automatically position and accurately harvest pineapples, recognized by the camera. Actual harvesting tests concluded that the machine worked well and had a high harvesting success rate. Henry Williams et al. [21] developed a kiwifruit-harvesting robot, which consisted of four robotic arms, as well as a machine vision system, which greatly improved the harvesting efficiency, and was capable of harvesting more than half of the kiwifruits in the three test orchards, greatly reducing the demand for manpower (i.e., reducing the number of workers) during the peak kiwifruit harvesting period. Johan Baeten et al. [22] developed an automated fruit-picking mechanism for an apple-picking robot, which used a flexible end effector to reduce the damage of the picking manipulator to the apples as well as a binocular recognition system for the visual recognition system, and the time taken for apple picking was 10 s for each apple through orchard tests, proving the feasibility and functionality of the autonomous fruit-picking machine (AFPM). Takeshi Yoshida et al. [23] proposed a method for automatic fruit harvesting using a fruit-harvesting robot equipped with a robotic arm to pick most of the fruits from a V-shaped trellis. The machine uses two arms for fruit picking, using sensors and computer vision to detect the position of the fruit, which improves the picking accuracy of the machine and proves the feasibility of this method through experiments. The machine can be used, with good results, for picking single dragon fruits; however, it cannot be used for picking multiple dragon fruits growing at one growing point. Although there is no research on dragon-fruit-picking robots, a variety of fruit-picking robots exist that can help lay the foundation for their design.

China has not yet finished products in mass production domestically, but still focuses on prototype design [24]. Zhang Yueyue et al. [25] designed a dragon-fruit-picking robot end effector to achieve rapid shear picking with the dragon fruit leaf edge as the positioning reference and, at the same time, to achieve the goal of shortening the length of the incision to be comparable with manual shear under the guarantee of picking quality. It is simple, lightweight, and well-designed, which can be good for single-dragon-fruit picking, but not for multiple dragon fruits growing at one growing point. Zhu Lixue et al. [26] proposed a dragon fruit image segmentation and attitude assessment method based on an improved U-Net to achieve a three-dimensional attitude assessment of dragon fruits, providing technical support for the automated and accurate picking of intelligent dragon-fruit-picking robots, but it is not suitable in the following cases: in a picking environment with a small fruit size, high density, and complex environment, the visual recognition effect will be reduced. Jialiang Zhou et al. [27] proposed a dragon fruit detection method based on RDE-YOLOv7 to identify and locate dragon fruits more accurately, which theoretically supports the development of dragon-fruit-picking robots, which have a very high recognition accuracy for a single fruit, but its visual recognition effect decreases in picking environments with a high fruit density and complex environment. Bin Zhang et al. [28] suggested the integration of a lightweight network, specifically an enhanced version of YOLOv5s, to achieve the consistent detection of dragon fruits across diverse orchard conditions and weather patterns, and the improved model has good robustness, which offers a good solution for dragon-fruit-picking robotics, providing a theoretical foundation and technical support. Ziyang Zhou et al. [29] proposed a control and actuation method for soft robots that exploits the nonlinear mechanical properties of compliant curve beams and carefully designed mechatronic control circuits to create a central pattern generator, and this work is a significant step forward for soft robots towards fully autonomous, electronics-free, SMA-based, on-board-actuation, and control-of-motion robots. This research points to the design and control of a flexible end effector for dragon-fruit-picking devices and future directions for improvement.

In response to the above, in order to effectively solve the problems of the long time and low efficiency of the manual picking of dragon fruit, a large labor force occupied by repetitive manual work, and the substantial time consumed in selecting and removing bad fruits after the completion of picking, in addition to promoting the development of the dragon-fruit-picking industry and improving the mechanization of dragon fruit picking, this study designs a picking and selecting integrated remote-operation-type dragon-fruit-picking device, which can realize the picking of single fruits and multiple fruits grown at one growing point, can realize accurate identification and positioning under a dragon-fruit-picking environment of small size, high density, and complexity, and can sort fruits according to their weights as well as remove damaged or over-ripe fruits according to the characteristics of the fruits in order to satisfy the market demand. At the same time, it can also effectively relieve repetitive manual labor intensity, reduce the workload on employees, enhance the efficiency of dragon fruit picking, reduce the time for selecting and removing bad fruits after dragon fruit picking, and promote the development of the semiautomatic field of dragon-fruit-harvesting machinery. The paper is laid out in a general-partitioned structure, detailing the mechanical structure and control system of the device.

2. Materials and Methods

2.1. Overall Structure and Working Principle of the Machine

2.1.1. Overall Structure of the Picking Device

The whole picking and selecting integrated remote-operation-type dragon-fruit-picking device adopts both electric and pneumatic drive to provide power for each part of the structure, which integrates the functions of a shifting blade, clamping, picking, removing, selecting, travelling, and speed regulation. The picking and selecting integrated remote-operation-type dragon-fruit-picking device employs the format of single-row picking, which includes a picking and sorting system, bad fruit removal system, visual identity system, aluminum profile frames, control system, and inverted trapezoidal caterpillar track travel system. The main structure of the system is shown in Figure 1.

The picking and selection of the integrated remote-operation-type dragon-fruit-picking device have the following three advantages: the integration of picking and sorting can save the time of fruit selection after the completion of fruit picking; the use of a flexible claw gripping dragon fruits can prevent damage to the skin of the fruits during picking, and reduce the rate of fruit loss; and, fitted with a blade-shifting device, can adapt to a variety of complex fruit growing environments and improve the picking rate.

2.1.2. Working Principle and Main Technical Parameters

The picking and selecting integrated remote-operation-type dragon-fruit-picking device adopts the design scheme of remote operation, branch swinging and clamping, picking and removing, and sorting. During the orchard operation, the operator controls the dragon-fruit-picking device to reach the designated working position through remote control, which recognizes the dragon fruit image through the visual recognition system and completes the location of the dragon fruit to ensure the accuracy of the picking work. According to the image information and location information of the dragon fruit, the picking device uses the end of the flexible claw and picking scissors to complete single-dragon-fruit and multiple-dragon-fruit picking. In the picking process, the fruits are covered with a blade-shifting device to clip out the branches to complete the picking of the covered fruit, so that the picking work is carried out normally. When there are damaged and over-ripe fruits in the picking process, the bad fruit removal system allows the suction cup to reach the designated sucking position and complete the elimination of bad fruits through the mutual cooperation of the cylinder and the slipway. The fruit picked through the funnel into the sorting device according to the weight of the fruit will be divided into large fruit and small fruit. The two kinds of fruit will go onto the tracks for transporting fruit into different collection boxes and complete the dragon fruit picking.

The key technical specifications of the entire machine are presented in

Table 1. The main technical parameters of the dragon-fruit-picking device.

Technical Parameters	Numerical Value
Sizes (L × W × H)/(mm × mm × mm)	1480 × 820 × 1720
Quality of the machine/(kg)	74.8
Stepper motor for lifting device torque/(N m)	2.8
End Flex Jaw motor torque/(N m)	1.6
Twin rod cylinder actuator speed/(m/s)	0.15
Sorting device conveyor belt line speed/(m/s)	0–0.2
Work efficiency/(pieces/hour)	499

.

The machine design schematic is shown in Figure 2.

2.2. Design of Mechanical Structures

2.2.1. Picking and Sorting System

The picking system mainly includes a picking device and a sorting device. The picking device can pick either a single dragon fruit growing alone or a dragon fruit growing together with multiple fruits. The growth state of dragon fruits is shown in Figure 3; the shape of the fruit is ellipsoid, the average long diameter of the fruit is 87.5 mm, the average transverse diameter of the fruit is 66.7 mm, and the average mass is 294.2 g. Through practical tests, it can be seen that the maximum force of compression of dragon fruit should not exceed 100.8 N, and the shearing force of the scissors inserted into the leaf flesh is 20 N, in order to make the picking device better adapted to the indeterminate growth characteristics of dragon fruit. In the process of picking individually grown dragon fruits, the picking position is determined by the movement of the lifting platform composed of the module and the round-trip movement of the electric cylinder, and the picking angle is adjusted by the rotation of the steering gear connected to the flexible claw. The flexible claw can completely wrap the fruit inside and cut the root of the fruit through the curved blade on the outside of the flexible claw to complete the picking. In the process of picking dragon fruits that grow together, the lifting platform is composed of modules and the movement of the mechanical arm drives the end of the scissors to reach the roots of the fruits and cut off the dragon fruits that grow together. In the picking process, if fruit is blocked, the blocked branch is clipped out by the blade-shifting device to facilitate the picking of shaded fruits. The sorting device through the front of the funnel to take over the fruit was completed, and the weight of the fruit was measured indirectly through the pressure sensor at the end of the funnel. At this time, the corresponding cylinder promotes the baffle plate and the other side of the baffle plate to form the corresponding channel for transporting fruits, which enter the collection box through the channel to complete the sorting of fruits. Reinforcement was installed underneath the central load-bearing plate to prevent the deformation of the middle load-bearing plate due to excessive weighing, and to prevent the free fall of the load caused by the reverse movement of the lead screw module a braking device was installed under the screw module [30]. The three-dimensional structure of the picking and sorting system is shown in Figure 4. In the entire picking system, the component that requires the highest stiffness is the connector between the end flexible claw and the steering engine, and a lack of stiffness of the connector will lead to the sagging of the end flexible claw, which cannot guarantee picking accuracy; therefore, the connector was analyzed using ANSYS Workbench 2023R1 software, and the static stress, strain, and displacement of the connector are analyzed and presented in Figure 5.

The connector was constructed from 45-gauge steel and possessed a yield stress of 530 MPa. After the static stress analysis of the connector in ANSYS Workbench 2023R1 software, it was found that the maximum stress of the part appeared at the connection position between the connector and steering gear, and the stress of the part was 46.4228 MPa. After the static displacement analysis of the connector in ANSYS Workbench 2023R1 software, it was found that the maximum displacement of the part appeared at the edge of the connector, and the static displacement of the part was 4.72208 × 10⁻³ mm. After performing static strain analysis in ANSYS Workbench 2023R1 software, the maximum static strain appeared at the connection position between the connector and the steering gear, and the maximum static strain of the connector was 2.46735 × 10⁻⁴. A reasonable demonstration of the static stress analysis, static displacement, and static strain analysis indicates that the part meets the necessary requirements for use under operating conditions.

2.2.2. Bad Fruit Removal System

The bad fruit removal system [31,32] is located on top of the aluminum profile frame and is carried out by the OpenMV (Yunjiang science and technology company in China Hangzhou) visual recognition module, through the movement of the double cylinder and screw module in the direction of the X,Y,Z-axis, such that the organ-type suction cups can achieve accurate positioning in space, and through the rotation of the steering engine the sucking position is changed, then completing the culling of damaged or over-ripe fruit. The three-dimensional diagram of the bad fruit removal system is shown in Figure 6a.

At present, most suckers on the market are single air cavity suckers, and the adsorption effect for irregular objects is not ideal and is easily affected by vibration and other factors; double air cavity suckers have greater suction force and higher stability. Therefore, in order to make suckers adapt better to the absorption of different sizes of dragon fruit and further improve the stability and diversity of suckers, the vacuum sucker is designed as a two-stage air cavity structure, the surface of the sucker cavity is covered with a film, and, during vacuuming, the air pressure, P_i1, in the sucker cavity is extracted into P_i2. The pressure difference between P and P_m1 pressurizes the film to cover the inner wall of the suction cups. The deformation of the film expands the enclosed space, S₀, between the object and sucker, so that the initial air pressure, P_m1, equal to the ambient atmospheric pressure in S₀ is simultaneously reduced to P_m2. The dragon fruit was adsorbed using the air pressure difference generated between P_m2 and atmospheric pressure, P₀.

For the sucker, the temperature of the ideal gas remains constant and the amount of substance of the ideal gas remains constant. Thus, an increase in the volume of the ideal gas leads to a decrease in the gas pressure, and the sucker cavity is a circular truncated cone, which simplifies the deformation process of S₀ to an increase in the volume of the circular truncated cone:

$$\mathrm{\Delta }V=\frac{1}{3}\pi \left[3D^2-3D\mathrm{\Delta }{h}_s\cot \left(\theta \right)+{\left(\mathrm{\Delta }{h}_s\cot \left(\theta \right)\right)}^2\right]\mathrm{\Delta }{h}_s$$

(1)

where D is the diameter of the outer circle of the suction cup, θ is the angle of inclination of the inner wall, Δh_s is the change in height of S_0, and V₁ is the initial volume of S₀. Based on the analysis above, the equation for the adsorption force of the suction cup is as follows:

$$F=\mathrm{\Delta }pS=\frac{\mathrm{\Delta }V}{\mathrm{\Delta }V+V_1}{p}_{m1}S$$

(2)

where S is the contact area between the sucker and object, and it can be seen from the formula that the adsorption force of the sucker is related to the volume change and contact area. The larger the volume change, the larger the contact area, and the stronger the adsorption capacity; therefore, this cavity design can make the vacuum suction cup better adapted to the dragon-fruit-removing process, and the schematic diagram of the sucker structure is shown in Figure 6b.

2.2.3. Inverted Trapezoidal Caterpillar Track Travel System

The road surface of the dragon fruit orchard has poor flatness, high water content, and soft soil that is easy to collapse. The mobile platform should have strong off-road performance, and the inverted trapezoidal tracked chassis can provide higher stability and off-road capability compared to the traditional tracked chassis; it has more significant adaptability and load-bearing capacity. Therefore, the picking and selecting integrated remote-operation-type dragon-fruit-picking device adopts an inverted trapezoidal tracked travelling mechanism, with two parallel screw sliding platforms mounted on top of the chassis. Thus, when the tracked travelling mechanism can no longer move forward, the position above the sliding platform can be further adjusted via the sliding platform, further improving the feasibility of the picking device. According to the “Crawler transport machinery” for motor components and structure selection, the chassis motor model 110BM-48-1000 DC brushless motor (DMKE company in China Guangzhou) has rated torque of 6 N m, a supporting gear reducer selection RV50 worm gear reducer (Hengming intelligent equipment company in China Shenzhen), and a reduction ratio of 1:50. It also has walking system selection of gear rubber tracks, track pitch P = 0.065 m, track width b = 0.18 m, track thickness of 0.02 m, and B = 0.75 m between the left- and right-side tracks [33]. The three-dimensional diagram of the inverted trapezoidal caterpillar track travel system is shown in Figure 7.

2.3. Design of the Control System

2.3.1. Visual Identity System

The visual recognition system [34] is located in the uppermost part of the picking and selecting integrated remote-operation-type dragon-fruit-picking device, which adopts the eye-to-hand calibration method; the hardware system adopts the XTOM-MATRIX 3D (Xintuo 3D technology company in China Shenzhen) scanning system based on the principle of binocular stereoscopic vision [35] and blue light projection aberration multifrequency phase-shift technology to achieve non-contact object 3D data in detailed, accurate, and fast acquisition. Moreover, to achieve the localization of dragon fruit using a dragon-fruit-picking device, a high-recognition-rate dragon fruit target detection algorithm based on YOLOv5 is improved by adding fewer labeled category images to the targeted data expansion, finding the optimal parameters adapted to this self-constructed dataset by using hyperparametric evolution, applying an image weighting strategy for the imbalance in the categories of the dataset, to increase the recognition accuracy of dragon fruit, and using a label smoothing strategy to prevent overfitting of the network and enhance the generalization ability of the network.

To reduce the influence of the category imbalance problem during the training process, an image weighting strategy is applied to the category imbalance of the dataset, which calculates the weight of the image acquisition during the training process. If the image weight is larger, the probability of the image being sampled is larger. While traversing the image, it was calculated according to the recaptured index. With the use of image weights, the images that were poorly trained in the previous round are given increased weights in the next round of training, so that the network can make full use of the data in the dataset. Because manual labeling inevitably leads to mislabeling and omission, the use of a label smoothing strategy is a good solution to this problem.

First, we train according to the parameters recommended by YOLOv5, and then add or subtract on this basis to train again; the optimal parameter is finally determined to be 0.05. In order to optimize the loss function IOU, we introduce an IOU loss function with different characteristics and adopt five IOU loss functions (DIOU [36], GIOU [37], EIOU [38], SIOU [39], and Alpha-EIOU [40]) which were trained in combination with the YOLOv5s model, and the results were compared to select the most suitable loss function.

Changes in various indicators during training are shown in Figure 8.

According to Figure 8, it can be concluded that the difference between the indicators of mAP_0.5, mAP_0.5:0.95, P, and R using different IOUs is not very large; the maximum value does not differ by more than 1%. However, under the three indexes of classification loss, verification loss, and target loss of the training set and verification set, the use of EIOU can effectively reduce the loss value, which indicates that the use of EIOU can better locate the target and complete the task.

After adding the above optimization method, the network is trained again and tested using the weights obtained after the completion of training; a comparison graph of the metrics between the YOLOv5 network with the optimization method added and the original YOLOv5 network during the training process is shown in Figure 9.

The EIOU curve (blue curve) is the network training process curve after the addition of the improvement measures, and the other curve is the original YOLOv5 network training process curve. According to Figure 9, after adding the optimization measures, mAP_0.5:0.95 has been improved considerably, and the simulation precision and recall improved. According to Figure 10, after the improvement, the performance of the model has been improved in all aspects compared with the pre-improvement period. Among them, the mAP_5:0.95 of all categories in the model increased by 0.12 and 0.205, respectively. For mature dragon fruit, mAP_0.5:0.95 increased by 0.032 and 0.092, respectively; the mAP_0.5:0.95 of immature dragon fruit increased by 0.191 and 0.297, respectively; and the mAP_0.5:0.95 of the model for flowers increased by 0.134 and 0.255, respectively. From Figure 11, it can be concluded that the optimized network model can achieve superior recognition and localization results for targets in small, dense, and bright-light scenes. A comparison of the visual recognition effect on the test set is shown in Figure 10. The visual recognition effect diagram of some scenes is shown in Figure 11.

2.3.2. Dragon Fruit Weight Information Access

The sorting system is used to achieve the weighing and weight sorting of dragon fruits. The STM32F4 microcontroller is selected to be the controller; after the target dragon fruit is in contact with the pressure sensor, the pressure of the fruit makes the resistance of the resistance strain gauges change and causes a change in the voltage, so as to output a signal. The signal is transmitted to the HX711 A/D converter module and converted into a digital signal; the signal is then converted into an accurate weight value by the microcontroller program. The pressure sensor weighs the dragon fruit, and the weight information is transmitted to the microcontroller through an HX711 A/D converter module. Processed by the internal program of the microcontroller, the signal was converted into an accurate weight value. The conversion process is as follows.

The output voltage, U, of the sensor at full scale is as follows:

$$U=U_I\cdot k$$

(3)

where the following is the case:

• U_I—excitation voltage.
• k—sensitivity, k = 1 mV/V.

If the range of the pressure sensor is set to 5 kg and 5 V is used for power supply, then the output voltage, U, is 5 (mV), which is equivalent to the sensor under 5 kg gravity, the output voltage U is 5 mV, according to the circuitry of the load cell module, and the actual output voltage U₂ = 4.6 mV. When the pressure sensor is operating, the output voltage, U, is as follows:

$$U=\frac{4.6X}{R}$$

(4)

where the following is the case:

• X—fruit weight, kg.
• R—sensor range, kg.

Substituting the sensor range R = 5 kg into Equation (4), we obtain the output voltage U = 0.92X (mV). The HX711 A/D converter module can amplify the sensor output voltage 128 times, at which point the output voltage, U, is 128 × 0.92X = 117.8X (mV). Set the measured AD value to Y; convert the output voltage, U, into a 24-bit digital signal; and then obtain the following:

$$Y=\frac{U\cdot 2^{24}}{4.6}$$

(5)

Substituting U = 117.8X in Equation (5), we obtain Y = 429,496.7X. Hence, we obtain the following:

$$\mathrm{X}\approx \frac{Y}{429.5}\left(g\right)$$

(6)

Equation (6) shows the relationship between the measured AD value and the fruit weight. According to this formula, the corresponding microcontroller program can be written, which increases the processing of the oscillation signal in the program, and the acquisition of dragon fruit weight information can be completed by eliminating the oscillating signals.

2.3.3. Control System

The system adopts an STM32F4 microcontroller (Mir technologies company in China Shenzhen) as the core chip of the control system for picking and selecting an integrated remote-operation-type dragon-fruit-picking device. The hardware of the control system includes a steering engine for regulating the rotation of the vacuum suction device (model: TD-8120MG, with a blocking torque of 20 kg cm, Feite model company in China Shenzhen); a ball screw stepping motor module in the horizontal drive module for controlling the left and right movements of the sucker positioning mechanism (model: 57HS22, with an effective travel of 600 mm and a torque of 2.3 N m, Process equipment manufacturing company in China Nanjing); a ball screw stepper motor module that controls the up and down movements of the lifting platform (model: 57BYG250H, effective stroke of 500 mm, torque of 2.8 N m, Process equipment manufacturing company in China Nanjing); an electric cylinder, used to control the movement of the end flexible claw expansion and contraction (speed of 100 mm/s, effective stroke of 250 mm); a servomotor, used to control the flexible finger opening and closing (model: 60M-R6430A5-E, torque of 1.6 N m, Huistong motor company in China Jiangsu); a DC gear motor for controlling the sorting device (model: 5D120-GU-2, Jinmaozhan micro motor company in China Shenzhen); a solenoid valve for controlling the working status of the cylinder and sucker (model: 4V210-08, 2-position 5-way, Fidelion precision manufacturing company in China Zhejiang); a relay for controlling the solenoid valve open/close (model: JQC-3FF-S-Z, Mikolay electronics company in China Shanghai); a DC24V switching power supply; a pressure sensor (model: FSR402, external diameter: 18.3 mm, Mack sensors company in China Shanxi); a laser diffuse reflection counter-reflective photoelectric sensor (model: LS18-DS80C1, sensing distance: 80 cm, Jingwei Laser Equipment company in China Liaocheng); and a stepper motor driver (model: PFDE-DM542, incorporating three generations of 32-bit DSP processors, Process equipment manufacturing company in China Nanjing).

The software part of the control system was programmed in the C language using Keil5 software (5.25 version), and the modification as well as debugging of the target detection algorithm was carried out in the Python language on PyCharm 2023. The pressure sensor, photoelectric sensor, and visual recognition system constitute the fruit information acquisition system, and the DC power supplies power for the control system. Before the work of the control system, the components are first initialized to bring the control system into a ready state; the visual recognition system then begins to identify and locate the fruit, the digital image data collected are transmitted to the microcontroller through the USB interface, and the microcontroller analyzes and processes the obtained information. When the fruit is judged to be a single fruit, the microcontroller controls the end of the flexible claw picking; when it is judged that the fruit is multifruit, the microcontroller controls the mechanical scissors picking; and when it is judged that the fruit is blocked, the microcontroller then controls the blade-shifting device to clip out the branches and fruit picking. During the process of picking, the OpenMV (Yunjiang science and technology company in China Hangzhou) module monitors the status of the fruit in real time and transmits the signal to the microcontroller through the USB interface; when it is judged to be over-ripe or damaged, the microcontroller controls the bad fruit removal system to remove the fruits. After the picking is completed, the picking system puts the fruits into the funnel; the fruits enter the sorting device through the funnel, the pressure sensor weighs the fruits, the weight signal is converted and transmitted to the microcontroller, and the voltage signal is converted into accurate weight data by the microcontroller program. Meanwhile, they will be sent to the upper computer through the USART serial port; the weight data are displayed in real time, and the microcontroller controls the corresponding cylinder to push the baffle plate to form different transmission channels according to different weights. The fruits enter the different collection boxes through the channels, and after the fruits enter the collection boxes the microcontroller controls the electrical components to reset and complete the process of one operation. The machine is driven by a combination of electric and pneumatic components, and the various control hardware aspects of the picking and selecting integrated remote-operation-type dragon-fruit-picking device are connected by a combination of an interrupt control and a timer control. The feedback signals from the laser diffuse reflective photoelectric sensors, pressure sensors, and visual recognition system were used as signs with which to control the work of various electric and pneumatic components. The precise control of the slider position of the stepper motor ball screw module, the speed of the stepper motor, the speed of the servomotor, the opening and closing of relays and solenoid valves, the opening and closing of the end flexible jaws, the expansion and contraction of the electric cylinders, the removal of damaged and over-ripe fruits, and the picking and sorting of fruits are based on the position of the fruit and the state of fruit growth. The system uses a visual recognition system to judge the position of fruit growth and growth state and provides feedback to the microcontroller to regulate each mechanism to carry out the corresponding action [41]. The control system controls the motion state of the electric device by controlling the rotation of different motors through electrical signals, in order to achieve the function of each system. In this paper, the control of the motors uses PWM signals, and the control of the motors is completed through the PWM signals generated by the STM32. The mechanical control architecture is shown in Figure 12.

2.4. Dynamic Analysis of a Picking and Selecting Integrated Remote-Operation-Type Dragon-Fruit-Picking Device

2.4.1. Mechanical Analysis of the End Effector of the Blade-Shifting Device

The end effector should have an appropriate clamping force during the working process to ensure that the dragon fruit branches are stably clamped without causing damage to the branches. The magnitude of the clamping force is related to the specific structure of the end effector and the diameter of the branch. By analyzing the force on the end effector, the relationship between the driving force of the end effector, the clamping force, and the mass of the branch can be established. To ensure that the end effector can complete the clamping work with the appropriate clamping force during the operation, researchers conducted force analyses to ascertain the direction and magnitude of the forces acting on the end effector. Subsequently, force diagrams were plotted, and force balance equations were formulated, as depicted in Figure 13.

$$F=\frac{F_P}{2}$$

(7)

$$F_N{L}_1=\frac{F}{\cos \alpha }\times \frac{L_2}{\cos \alpha },F_N\ge K_1{K}_2{K}_{3}G$$

(8)

where the following is the case:

• F_P—the driving force of the end effector, provided by the steering engine, acts on the pin.
• F—distribution of the driving force on the gripping fingers.
• F_N—clamping force on the leaf ridge, N.
• α—the angle between the direction of the groove of the finger and the line between the two rotary pivot points.
• L₁—distance between the rotary pivot point and center of symmetry of the overall structure.
• L₂—distance from the rotary pivot point to the center of the branch.
• K₁—safety factor, take 2.0.
• K₂—working condition factor K₂ = 0.98.
• K₃—azimuth coefficient.
• G—gravity of the branch.

In order to reduce the weight of the end effector, the material of all parts of the end effector is 7075-T6, which can meet the required fatigue strength of the place where the slide groove contacts the pin during the operation. In order to prevent the end effector damaging the branch, the rubber is affixed to the inner and outer walls of the clamp finger as a buffer medium. The power of the end effector is supplied by the steering engine, the model of the steering engine is HTS-35H (Feite model company in China Shenzhen), and the maximum torque is 343 N. In the case of a certain clamping force, the size of the driving force depends on the α angle; if the α angle is too large, this will lead to the end effector size being too large, which is not conducive to the normal operation of the end effector, so α = 45° is selected. It is known through the test that the maximum compression critical resistance of the dragon fruit is 100 N, and that its minimum clamping force is 15 N. Of the maximum compression critical resistance, 30% is taken as the safety pressure, F_N = 30 N, and, according to Equations (7) and (8), it can be concluded that the driving force, F_P, is 70.5 N. From this, it can be seen that the torque provided by the steering engine can meet the requirements of the normal use of the end actuator. Among them, safety factor K₁ = carrying capacity/load = 100/30 = 3.3, and working condition factor K₂ = actual steering engine output power/rated steering engine output power = 24.5/25 = 0.98.

2.4.2. Mechanical Analysis of the End Effector of a Scissor Device

The end effector should have an appropriate shear force during the working process to ensure that the end effector can cut the tree diameter at the connection with the dragon fruit. By analyzing the force diagrams of the scissors and connecting rod, the relationship model between the driving force and shear force can be established, force diagrams can be plotted, and force balance equations are established, as shown in Figure 14.

$${\displaystyle \sum F_x=F_{Dx}}+F_{Cx}+F_{Bx}=0$$

(9)

$${\displaystyle \sum F_y=F_{Dy}+F_{Cy}+F_{By}}=0$$

(10)

$${\displaystyle \sum M_C=F_{Bx}\cdot c+}{F}_{By}\cdot b-F_{Dx}\cdot c-F_{Dy}\cdot d+F_{Dx}\cdot \left(c-e\right)=0$$

(11)

$${\displaystyle \sum M_B=F_{Cx}\cdot c+F_{Cy}\cdot b-F_{Dy}\cdot \left(b+d\right)}-F_{Dx}\cdot e+F_{Dy}\cdot \left(b+d\right)+F_{Dx}\cdot e=0$$

(12)

$$F_{Dx}=F\sin \beta ,F_{Dy}=F\cos \beta$$

(13)

where the following is the case:

• F—shear force.
• F_Ax, F_Ay, F_Bx, F_By, F_Cx, F_Cy, F_Dx, F_Dy—force on the scissor device in the x, y direction at A, B, C, and D, respectively.
• β—half of the angle between the two scissors when the fruit root is subjected to maximum shear force.
• b—transverse distance between pins B, C when the fruit root is subjected to the maximum shear force.
• c—longitudinal distance between pins B, C when the fruit root is subjected to the maximum shear force.
• d—lateral distance between the fruit root and pin C when the fruit root is subjected to maximum shear force.
• e—longitudinal distance between the fruit root and pin B at the time of maximum shear on the fruit root.

The association of Equations (9)–(13) yields the following:

$$F_{Bx}=-\left[\frac{\left(1+2c+e\right)\cdot F\cdot \sin \beta +2\left(b+d\right)\cdot F\cdot \cos \beta }{1+c}\right]$$

(14)

Based on the analysis of the force relationship in Figure 14, we can obtain the following:

$$F_{Ax}=-F_{Bx}$$

(15)

Because the steering engine generates a driving force acting on the slider bar in the direction of the horizontal to the left, the direction of the force synthesized by F_Ax and F_Ay is horizontally to the left. To prevent the scissors from opening and closing too much, the maximum angle between F_Ax and the horizontal direction is set to 40 °; therefore, the driving force can be calculated according to F_Ax. To minimize the cost of the end effector and improve the convenience of use, the steering engine model HTS-35H (Feite model company in China Shenzhen) was chosen as the power unit with which to simplify the control system while ensuring the efficiency of the picking, and it can be seen from the test that the shear force required by the scissors to cut a fleshy stem with mesophyll is known to be 20 N. From the above formula, the driving force required by the scissor device can be calculated as 160 N when the shear force is 20 N. The maximum torque produced by the steering engine was 343 N. The maximum torque generated by the steering engine is 343 N, which indicates that the torque provided by the steering engine can meet the requirements of normal end effector use.

3. Results

3.1. Experimental Materials, Conditions, and Equipment

In order to complete the final experiment of the machine, the test conditions and test steps are based on the national standard NY/T 3911-2021 [42] “Dragon Fruit Harvesting, Storage and Transportation Technical Specification”, to test whether the various systems of the picking and selecting integrated remote-operation-type dragon-fruit-picking device can work properly. In December 2023, a picking test was carried out on the campus of Liaocheng University to test the picking of prepared dragon fruit branches. The length of the experimental branch was 0.9~1.2 m, and the installation height of the experimental branches was 1.3~1.7 m. The power source for the system was a German Komax 220 v small oil-free silent air compressor, rated at 1490 W and capable of achieving a maximum pressure of 0.8 Mpa. Additionally, a 220 V–2500 W outdoor mobile power supply was chosen to supply power to the air pump and electrical components within the machine. The selected prototype is the picking and selecting integrated remote-operation-type dragon-fruit-picking device studied in this study, as shown in Figure 15.

At the onset of the design stage, the mechanical structure of each component of the picking machine was modeled, simulated, and analyzed using SOLIDWORKS software 2020. Additionally, a simulation was conducted for the dragon-fruit-picking device that encompassed picking and selecting functionalities. By integrating the data from preliminary investigations and design calculations, any deficiencies or irrationalities within the design were rectified and optimized throughout the trial process. Subsequently, drawings for processing and assembly were utilized to produce a prototype for trial use.

3.2. Assessment of Indicators

To visually demonstrate the picking effect and quality of the picking and selecting integrated remote-operation-type dragon-fruit-picking device, the net picking rate and the damage rate were selected as evaluation indicators. Each index is the average value of multiple trials and is defined as follows:

$$N=\frac{T_1}{T}$$

(16)

In this equation, N denotes the net harvesting rate throughout the trial, %.

T₁ is the actual number of fruits picked by the device.

T is the number of fruits to be picked throughout the trial.

$$Z=\frac{T_2}{T_1}$$

(17)

In this equation, Z denotes the rate of fruit damage throughout the trial, %.

T₂ is the total number of damaged fruits after picking is completed.

3.3. Analysis of Orthogonal Experiments

To better optimize the parameters and elucidate the interactions among the experimental factors, the ranges of various factors such as flexible claw closing speed, A, electric cylinder extending speed, B, and robotic arm movement speed, C, were determined through meticulous calculations and analyses. Subsequently, a three-factor and three-level quadratic regression orthogonal experiment was conducted, utilizing the net picking rate, N, and damage rate, Z, as the response values. The corresponding code for each experimental factor is listed in

Table 2. Encoding of experimental factors.

Coded Value	Factor
	Flexible Claw Closing Speed, A/(m/s)	Electric Cylinder Extending Speed, B/(m/s)	Speed of Movement of the Robotic Arm, C/(m/s)
−1	0.02	0.05	0.1
0	0.05	0.15	0.2
1	0.1	0.25	0.3

.

The data were analyzed and processed using Design Expert software (22 version); the three-factor and three-level experiments were designed in accordance with Box–Behnken’s experimental principle, as shown in Table 2.

Utilizing Design Expert software (22 version), multivariate linear regression and quadratic fitting were conducted to determine the net harvesting rate, N, and damage rate, Z. Based on the data presented in

Table 3. Encoding of experimental factors.

Product Key (Software)	Factor			Performance Indicator
	A (m/s)	B (m/s)	C (m/s)	N/%	Z/%
1	0.05	0.15	0.2	92.7	2.8
2	0.02	0.05	0.2	88.9	2.3
3	0.05	0.25	0.3	91.8	3.9
4	0.02	0.15	0.1	92.8	2.6
5	0.05	0.15	0.2	92.6	2.7
6	0.05	0.05	0.3	93.0	2.8
7	0.1	0.15	0.1	94.0	4.0
8	0.05	0.25	0.1	91.0	3.5
9	0.05	0.15	0.2	92.6	2.7
10	0.1	0.05	0.2	93.8	3.7
11	0.02	0.25	0.2	86.7	2.9
12	0.05	0.15	0.2	92.6	2.7
13	0.1	0.25	0.2	96.4	4.6
14	0.1	0.15	0.3	97.9	4.4
15	0.02	0.15	0.3	84.2	2.6
16	0.05	0.15	0.2	92.6	2.7
17	0.05	0.05	0.1	89.7	1.9

, quadratic regression equations for both N and Z were derived.

$$N=93.39-43.2A+16.3B-23.41C+240AB+312.5AC-31.25BC-64A^2-101B^2-5.875C^2$$

(18)

$$Z=3.02875-30.7A+1.425B-1.475C+15AB+10AC-6.25BC+206A^2+14B^2+4.125C^2$$

(19)

The variance analysis of the regression equation simulation was carried out using Design Expert software (22 version) to obtain

Table 4. Analysis of variance regression equation of the net picking rate of dragon fruit.

Source	Sum of Squares	df	Mean Square	F-Value	p-Value
Model	160.11	9	17.79	12.82	0.0014
A—Flexible claw Closing speed	108.78	1	108.78	78.42	<0.0001
B—Electric cylinder pushing speed	0.0313	1	0.0313	0.0225	0.0089
C—Speed of movement of the robotic arm	0.0450	1	0.0450	0.0324	0.8622
AB	5.76	1	5.76	4.15	0.0081
AC	39.06	1	39.06	28.16	0.0011
BC	1.56	1	1.56	1.13	0.3238
A2	0.1078	1	0.1078	0.0777	0.7885
B2	4.30	1	4.30	3.10	0.1219
C2	0.2325	1	0.2325	0.1676	0.6945

and

Table 5. Analysis of variance regression equations for dragon fruit damage rate.

Source	Sum of Squares	df	Mean Square	F-Value	p-Value
Model	9.06	9	1.01	24.25	0.0002
A—Flexible claw closing speed	4.96	1	4.96	119.55	<0.0001
B—Actuator extension speed	2.21	1	2.21	53.13	0.0002
C—Speed of movement of the robotic arm	0.3613	1	0.3613	8.70	0.0214
AB	0.0225	1	0.0225	0.5422	0.0486
AC	0.0400	1	0.0400	0.9639	0.0359
BC	0.0625	1	0.0625	1.51	0.2594
A2	1.12	1	1.12	26.91	0.0013
B2	0.0825	1	0.0825	1.99	0.2013
C2	0.1146	1	0.1146	2.76	0.1405

; from Table 4, it can be seen that the net picking rate of the picking device was p < 0.05, suggesting that the regression model is statistically significant. Variance analysis demonstrated that the speed of flexible claw closure (p < 0.0001) had a statistically significant impact on the net picking rate, and that the speed of the electric cylinder extension (p < 0.05) exerted a notable influence on the net picking rate. Variance analysis demonstrated that the influence of each factor on the net picking rate, ranked in descending order of significance, was as follows: the speed of flexible claw closure, and the speed of electric cylinder extension. There was no significant effect of the mechanical arm movement speed.

As is evident from Table 5, the quadratic regression model of the damage rate of the dragon-fruit-picking device is p < 0.05, indicating that the regression model was significant. Variance analysis demonstrated that the flexible claw closing speed (p < 0.0001) has a very significant effect on the damage rate of dragon fruit and that the electric cylinder extending speed and the mechanical arm movement speed (p < 0.05) had a significant effect on the damage rate of dragon fruit. Analysis of variance showed that the effects of the factors on the damage rate in descending order were as follows: flexible claw closing speed, electric cylinder extending speed, and mechanical arm movement speed. The significant effect of the interaction term proves that there is an interaction between the flexible claw closing speed and the electric cylinder extending speed on the damage rate; there is also an interaction between the flexible claw closing speed and the mechanical arm movement speed on the damage rate.

3.4. Response Surface Analysis

Design Expert software (22 version) was used to optimize the solution, and the optimal solution is obtained; when the flexible claw closure speed was 0.029 m/s, the electric cylinder extension speed was 0.085 m/s and the robotic arm movement speed was 0.15 m/s. At this time, the net picking rate was 91.7% and the damage rate was 2.1%. Using the value obtained at this time as the standard and as the variable value under the interaction in the response surface analysis, we can derive the response surfaces influenced by each interaction, as demonstrated in Figure 16a–f.

From the response surface analysis depicted in the aforementioned figure, it is evident that the variation rule of the factors affecting the response surface is basically consistent with the results of the model and regression equation analysis of variance; A—flexible claw closing speed, B—electric cylinder extending speed, and the interaction term between AB and AC have a significant effect on the net picking rate and damage rate. The movement speed of the robotic arm had no significant effect on the net picking rate, but had a significant effect on the damage rate. Using the optimization function of Design Expert (22 version), an optimization analysis was carried out to constrain the levels of each test factor; the maximum net picking rate and the minimum damage rate of the dragon fruit picker were taken as the optimization indexes, and the optimal parameter combinations were obtained as follows: in the case of a flexible claw closure speed of 0.069 m/s², a push rod extension speed of 0.085 m/s², and a robotic arm movement speed of 0.100 m/s², the net picking rate of the dragon fruit picker was 91.7%, and the damage rate was 2.1%.

4. Discussion

Based on the model design, simulation experiment, and trial production of key components, an integrated remote-operation-type dragon-fruit-picking device was designed. Compared with the manual picking of dragon fruit, according to the calculation of 12 h of work per day, manual picking can only pick 1500 dragon fruits per day, but this picking device can pick 7000 fruits per day. The cost of the manual picking of dragon fruits is approximately CNY 1500/mu, but the cost of machine operation is approximately CNY 200/mu. In other words, the efficiency of this picking device is 4.67 times that of manual picking, and the cost saving is approximately CNY 1300/mu. This picking device only needs one person to complete the picking operation of the machine, saving labor while reducing costs. The picking device integrates picking, sorting, and removing fruit, greatly reducing the labor force and labor time, freeing up the hands of fruit farmers. The picking device can complete both the picking of a single fruit and the picking of a plurality of fruits grown at a growing point, so that the device can better complete the picking of dragon fruits with complex growth. The entire control system of the device selects the STM32F4 microcontroller as the main control module, the software part of the control system is programmed in the C language using Keil5 software (5.25 version), and the modification and debugging of the target detection algorithm is carried out in the Python language on PyCharm, through the establishment and debugging of the hardware and software of the control system. The sensing ability of the picker to the external environment was enhanced, and the working efficiency of the mechanism was improved.

Compared with the domestic DELIXI fruit-picking mechanism, the dragon fruit damage rate of the DELIXI fruit-picking mechanism was 6.0%, the net picking rate was 80.7%, the dragon fruit damage rate of the machine was 2.9%, and the net picking rate was 90.5%. Moreover, the DELIXI fruit-picking mechanism is only able to complete the picking of single fruits; it is not able to pick more than one fruit growing at one growing point. Under identical picking conditions, this machine can guarantee the efficient harvesting of dragon fruit and satisfy the operational demands specific to dragon fruit picking. Vietnam, Thailand, and the Philippines are world-renowned dragon fruit producers, and Vietnam alone accounts for more than 50 per cent of global dragon fruit production; therefore, the dragon-fruit-picking device mentioned in this paper has promising applications in countries such as Vietnam, Thailand, and the Philippines. At present, the picking device still suffers from the disadvantage of a limited picking area, and the shortcomings of the device will be addressed in the future by optimizing the mechanical structure and transmission system.

5. Conclusions

This article addresses the issues of the manual picking of dragon fruits being time-consuming and inefficient, requiring substantial labor due to repetitive physical work and the lengthy process of selecting and removing spoiled fruits after harvesting. A picking and selecting integrated remote-operation-type dragon-fruit-picking device is designed and manufactured.

• The picking device integrates picking, sorting, and removing fruits, which greatly reduces the labor force, liberates the hands of fruit farmers, and has broad application prospects.
• The YOLOv5-based high-recognition-rate dragon fruit target detection algorithm is improved by using the image weighting strategy for the category imbalance of the dataset, hyperparameter evolutionary optimization, label smoothing strategy, loss function optimization, etc. The adaptability of the visual recognition system to a complex picking environment was investigated, and the results showed that the improved device can achieve better recognition and localization results for targets with a small fruit size, high density, and bright-light scenes.
• The HX711 A/D converter module completes the process of converting the voltage signal output from the pressure sensor into weight data, and designs a procedure with which to reduce the interference of shock and vibration with the weight signal to realize the real-time display of the weight of the fruits and their sorting.
• Through the design of the tracked mobile platform, the picking device achieves excellent obstacle-crossing ability and good shock-absorbing capability. By eliminating vibration interference, the picking device can better perform localization.
• By optimizing the suction pad structure of the bad fruit removal system into a double air cavity design, the bad fruit removal system is better adapted to the process of dragon fruit picking.
• The feasibility of the end effector of the plucking device and shearing device is investigated through mechanical analysis, and the results show that the structure of both can satisfy the requirements for use.

Eventually, through several orthogonal experiments and data processing, the picking effect was found to be optimal when the closing speed of the flexible claw was 0.029 m/s, the extension speed of the electric cylinder was 0.085 m/s, and the movement speed of the robotic arm was 0.15 m/s. The net picking rate of the device can reach 90.5%, and the damage rate can be up to 2.9%, satisfying the requirements of the dragon-fruit-picking operation.