Automatic Fruit Harvesting Device Based on Visual Feedback Control

Abstract

With aging populations, and people′s demand for high-quality or high-unit-price fruits and vegetables, the corresponding development of automatic fruit harvesting has attracted significant attention. According to the required operating functions, based on the fruit planting environment and harvesting requirements, this study designed a harvesting mechanism to independently drive a gripper and scissor for individual tasks, which corresponded to forward or reverse rotation using a single motor. The study utilized a robotic arm in combination with the harvesting mechanism, supported by a single machine vision component, to recognize fruits by deep-learning neural networks based on a YOLOv3-tiny algorithm. The study completed the coordinate positioning of the fruit, using a two-dimensional visual sensing method (TVSM), which was used to achieve image depth measurement. Finally, impedance control, based on visual feedback from YOLOv3-tiny and the TVSM, was used to grip the fruits according to their size and rigidity, so as to avoid the fruits being gripped by excessive force; therefore, the apple harvesting task was completed with a 3.6 N contact force for an apple with a weight of 235 g and a diameter of 80 mm. During the cutting process, the contact point of the metal scissors of the motor-driven mechanism provided a shear force of 9.9 N, which was significantly smaller than the simulation result of 94 N using ADAMS and MATLAB software, even though the scissors were slightly blunt after many cuts. This study established an automatic fruit harvesting device based on visual feedback control, which can provide an automatic and convenient fruit harvest by reducing harvesting manpower.

1. Introduction

Current agricultural harvesting processes have been significantly aided by the development of industrial automation for specific planted crops. The development of automated mechanical harvesting in the United States is particularly advanced: it is common for crops such as rice, potatoes, carrots, and lettuce to be harvested in batches, using large mechanical vehicles; however, fruit harvesting cannot be performed in the same manner. The harvesting of most fruit trees is still manpower-dependent or indirectly assisted by stairs, transport trolleys, and picking machines. The aforementioned crops, that can be harvested in large quantities, are mostly in structured environments, and the planting positions between a single crop and other crops can be controlled at equal intervals, which is done so that mechanical harvesting can create a simple and excellent design for this structure; by contrast, the harvesting of fruit species is limited by the vertical distribution of fruit tree growth and the random uncertainty of the horizontal growth position at the same vertical height, which makes the planting method dense and dependent on the high flexibility of manpower to complete the harvesting work. Owing to the increasingly obvious problems of lack of labor and an aging population in the agriculture industry [1], many researchers are engaged in R&D and improvement in related fields, to try to automate harvesting, by the full use of robots. Abundant Robotics [2] uses a human-driven traction machine to drag an automated harvesting robot. The robot uses a parallel robot as an extension of its mechanical arm, and a vacuum-absorbing harvesting mechanism, to harvest in a structured picking environment. Although the vacuum suction method is feasible, it requires extremely good design arrangements—such as vibration reduction measures—for material collection, and it must be equipped with compressors, transfer pipelines, and control equipment. FFRobotics [3] is designed using a telescopic mechanical rod, which is matched with a three-claw gripper to grip apples. Baeten et al. [4] have developed fruit gripping machines—mainly for gripping apples—which combine industrial robotic arms, personal computers, photographic lenses, and vacuum suction to pick up a single apple, recover it, and move the carrier. Zhao et al. [5] have also harvested apples, have moved a car body with crawler tracks, and have used visual algorithms, grippers, electric scissors, and a design composed of collision, position, and pressure sensors. Given that consumers expect higher-quality fruit and vegetable products, mechanical damage during mechanical gripping is a key factor in reducing their value. The mechanical damage mentioned by Knee et al. [6] shows that from the commencement of apple harvesting, the manual handling process still relies on human hands in most cases, and that apples are easily damaged by uneven manual forces when gripping is not easy: this is especially serious when only two or three fingers are used to pull the fruit, so the grower often asks the harvester to grip the fruit with the whole hand, and to use the twisting method, so as to concentrate on the fruit stem, to pull it off. In addition, Li et al. [7] used a glove that could sense the contact force of human fingers, to compare four fruit-gripping modes: their experimental results showed that manual gripping, with one finger pushing the stem and two fingers holding the apple and applying a pendulum motion, can cause minimal stress and bruising of the apple. However, it is difficult to implement a low-cost and low-degree-of-freedom robotic arm, because the gripping mechanism imitates the gripping action of the human hand; moreover, this approach would force the entire automated harvesting system to be implemented with a high-order multi-degree-of-freedom industrial robotic arm, thereby increasing its speed costs, and reducing its commercial viability. In addition, Liu et al. [8] proposed a compliant gripper for gripping fruit: this gripper can be driven by a linear actuator, and adheres to an object by changing the shape and size of the gripping object. The compliant gripper is a good solution for fragile objects: however, its disadvantage is that the shape and structure design requirements are relatively high, and there are specific deformable material requirements. In recent years, there have been several studies on the use of visual recognition for positioning control [9,10,11,12,13], in addition to several research studies proposing the use of visual recognition for fruits [14,15,16,17]. Reference [17] recognizes the ripeness and stem position of strawberries, so as to provide image-positioning control information for robotic automated strawberry harvesting. Based on the above literature, to avoid fruit damage, the design of the gripping force control of the harvesting mechanism must be considered. To increase the automatic visual identification of fruit species for the positioning control of the gripping mechanism, the harvesting mechanism must be integrated with a multi-dimensional actuator, a gripping mechanism, and a visual lens: the overall production complexity, weight, and cost are high, which makes practical application challenging.

This study utilized a single motor to drive and grip the fruit species and cut the fruit stem, and a charge-coupled device (CCD), based on the image recognition of the You Only Look Once (YOLO) algorithm [18,19,20,21,22] in automatic fruit harvesting devices. At the beginning of the harvesting process, the two-dimensional visual sensing method (TVSM) [11], combined with the YOLO algorithm, could be used to measure fruit position coordinates, to achieve visual feedback. Because only one motor was used to complete the gripping and cutting of the fruit stem, and only one CCD was used to achieve fruit recognition and distance measurement, this mechanism was very light, which was advantageous for the stable gripping of apples in an automated manner with minimal force requirements. In addition, when the size of the fruit was different from its rigidity, the gripper required different forces to grip the fruit, to avoid the fruit being pinched by excessive force; therefore, this study utilized impedance control [23,24,25], which could adjust the dynamic relationship between the displacement and contact force of the system. The device was divided into four parts: control and image recognition computing elements, machine vision elements, the main body of the developed robotic arm, and the harvesting mechanism. In the mechanism design method, the computer-aided design software SolidWorks, and the computer-aided engineering software ADAMS, were used to design, analyze, and simulate the mechanical arm and harvesting mechanism. Based on the image information provided by the machine vision components, the coordinates were measured by the algorithm, so that the robotic arm could complete the fruit harvesting automatically, based on visual feedback control, so as to replace manual harvesting behavior and improve harvesting efficiency.

2. Design of the Harvesting Device

This study used a single motor to drive the gripping of fruit species and subsequently cut the fruit stem. A pair of ratchets were driven by the clockwise and counterclockwise rotation of the motor, to drive components A and J, respectively. The gripper and scissors were driven as shown in Figure 1. In this study, the motor rotated clockwise to drive the ratchet, slider crank mechanism, and gears, so as to sequentially drive components A to I, and thus control the gripper, as shown in Figure 2. Component C drove the slider crank mechanism (component D), so that components F and I could be controlled to rotate the back-and-forth θ angles, so as to complete the grip and release control of the θ angle. The motor rotated clockwise from 0° to 180° to grip the fruit, and from 180° to 360° to loosen the fruit.

Figure 3 shows the gripping forces of the gripper. The fruit is assumed to be a sphere with a radius $r$; $L$ is the distance between the contact point of the fruit and the rotation axis of component F; $F$ is the force of the contact point of the fruit; $\phi$ is the angle between the contact point of the fruit and the vertical line; θ is the angle of rotation of the gripper; $d_x$ is half the distance between the rotation axes of components F and I; $d_y$ is the vertical distance between the fruit and rotation axis of components F or I. The relationship equations are represented as:

$$L=\sqrt{{{(r-d}_x)}^2{+(r+d}_y{)}^2},$$

(1)

$${\phi =\mathit{sin}}^{-1}(\frac{{r-d}_x}{L}),$$

(2)

$$F=\frac{\mathrm{K}\phi }{L},$$

(3)

$$F_n=F\times \mathit{cos}(\phi ),$$

(4)

$$F_t=F\times \mathit{sin}(\phi ),$$

(5)

where $\mathrm{K}$ is the torsion spring constant, $F_n$ is the normal force when the apple produces friction, and $F_t$ is the thrust perpendicular to the direction of the normal force.

The load of the gripper is the composite result of the thrust $F_t$ and the weight of the apple $W$. Considering that the maximum load of the gripper in any attitude is when the thrust and weight directions coincide, the balance relation equation of one gripper is

$$F_n\times \mu \ge F_t+\frac{Wg}{2},$$

(6)

where $\mu$ is the friction coefficient of rubber, and $g$ is the gravitational acceleration. According to the literature [26], substituting Equations (3)–(5) into Equation (6) obtains

$$\mathrm{K}=\frac{{Ed}^4}{{10.19DN}_a}\ge \frac{\mathit{LWg}}{2\phi (\mathit{cos}(\phi )\times \mu -\mathit{sin}(\phi ))}.$$

(7)

where $E$ is Young′s coefficient of the material, $d$ is the metal wire diameter, $D$ is the average diameter of the torsion spring, and $N_a$ is the effective number of turns. This study completed the design of the torsion spring using Equation (7).

According to the literature [27], Washington apples have the largest share of the market in the United States (64%); in addition, they constitute nearly 30% of the apple export volume every year. In consideration of apple packing, Washington apples grade their products into apples with diameters ranging from 58.7 mm to 92.5 mm [28]; therefore, this study designed the dimensions of the harvesting device to have harvested fruit diameters ranging from 55 mm to 110 mm, as shown in Figure 4. Figure 4 shows the cut control in the upper left, top, and upper right views, using a pair of scissors. Figure 1 shows that the ratchet (component J) was driven counterclockwise by a motor, which, in turn, drove component N. Subsequently, a pair of four-bar linkages, with components L and M as fixed fulcrums, were used to complete the cutting of the fruit stem. The motor’s rotation counterclockwise from 0° to 180° was the process of cutting the fruit stem, and its rotation from 180° to 360° was the process of releasing the scissors. To ensure that the harvesting mechanism had more degrees of freedom, this study integrated a robotic arm with X-and Z-axis motions, and a 0–180° rotation around the Z-axis, as shown in Figure 5. A CCD was installed in the middle of the gripper, to complete fruit recognition, and thus to complete the track approach and the fruit harvesting.

3. Visual Feedback Control

3.1. Visual Position

To harvest fruits automatically, this study used the YOLO algorithm for fruit recognition, which facilitated the automatic positioning and tracking of the harvesting device. YOLOv2 [18] refers to the design of an anchor to improve performance, and YOLOv3 [19] uses multi-size anchor points and a residual network to further improve performance. The anchor points calculated by YOLOv3 additionally scale the input image 1/16 and 1/32 times, before calculating the anchor point size. This method effectively increases recognition performance, and the anchor point is calculated using the k-means clustering algorithm [29]. After confirming the number of clusters to be classified in advance, the following equation is satisfied:

$$\mathit{arg}\underset{so}{\mathit{min}}{\sum }_{i=1}^k{\sum }_{x\in s{o}_i}{\left|\left|x-{\mu }_i\right|\right|}^2,$$

(8)

where $x$ is the vector of a point in space, ${\mu }_i$ is the vector average of all points in space, $k$ is the number of classifications, and so is the cluster. During the training process, the defined loss function was gradually optimized in the process of training the neural network model using a backpropagation algorithm. The loss function $L$ predicted the location, range, and category of the fruits; therefore, the loss function was represented as:

$$L(x,y,w,h,c)={\lambda }_c+l_w+l_C+{\lambda }_n+l_p,$$

(9)

where ${\lambda }_c={\lambda }_{coord}{\sum }_{i=0}^{S^2}{\sum }_{j=0}^B{l}_{ij}^{obj}\left[{(x_i-{\widehat{x}}_i)}^2+{(y_i-{\widehat{y}}_i)}^2\right]$, $l_w={\sum }_{i=0}^{S^2}{\sum }_{j=0}^B{l}_{ij}^{obj}\left[{(\sqrt{w_i}-\sqrt{{\widehat{w}}_i})}^2+{(\sqrt{h_i}-\sqrt{{\widehat{h}}_i})}^2\right]$, $l_C={\sum }_{i=0}^{S^2}{\sum }_{j=0}^B{l}_{ij}^{obj}{(C_i-{\widehat{C}}_i)}^2$, ${\lambda }_n={\lambda }_{noobj}{\sum }_{i=0}^{S^2}{\sum }_{j=0}^B{l}_{ij}^{noobj}{(C_i-{\widehat{C}}_i)}^2$, and $l_p={\sum }_{i=0}^{S^2}{l}_i^{obj}\sum _{(C\in classes)}{(p_i(C)-{\widehat{p}}_i(C))}^2. x_i$ and $y_i$ were the predicted coordinates, ${\widehat{x}}_i$ and ${\widehat{y}}_i$ were the actual coordinates, $w_i$ and $h_i$ were the predicted width and height, ${\widehat{w}}_i$ and ${\widehat{h}}_i$ were the actual width and height, $C_i$ was the predicted target type, and ${\widehat{C}}_i$ was the actual target type, $p_i(C)$ was the confidence level of the predicted target, ${\widehat{p}}_i(C)$ was the confidence level of the actual target, ${\lambda }_{coord}$ and ${\lambda }_{noobj}$ were the loss weights, $S$ was the grid, $B$ was the bounding box, $obj$ was the cluster with the target, and $noobj$ was the cluster without the target. According to the literature [18,19,20,21,22], YOLOv3-tiny is a model based on a simplified version of YOLOv3, and has good execution speed and recognition efficiency; therefore, this study used YOLOv3-tiny to recognize the fruits.

As the harvesting device was harvesting fruit, the weight of the harvesting device itself could not be too heavy; therefore, this study utilized a single motor to perform gripping and cutting actions, so as to reduce the weight of the harvesting device using a CCD. TVSM [11] was combined with the YOLOv3-tiny algorithm to visually position the fruit coordinates. In Figure 6, assuming that the width of the target object remains unchanged after the CCD moves $\alpha$ toward the target fruit, the distance $x$ relationship between the CCD and the target fruit is

$$\frac{(x+\alpha )}{n}\mathit{tan}\psi =\frac{x}{m}\mathit{tan}\psi ,$$

(10)

where $n$ and $m$ represent the width pixels of the field of view (FOV) divided by the width pixels of the target before and after, respectively, the CCD moves $\alpha$. $\psi$ is the angle of the FOV. After arranging Equation (10), the distance between the CCD and the target can be obtained as

$$x=\frac{m\alpha }{(n-m)}.$$

(11)

3.2. Cut and Grip control of Harvesting Device

After YOLOv3-tiny recognized the fruit, this study used the robotic arm, with the X-, Z-, and Z-axis rotation motors and TVSM, to continuously track and approach the fruit, and to measure the diameter of the fruit, so as to calculate $\phi$, thus achieving a contact position between the fruit and the harvesting device, where the fruit was gripped and its stem was cut. In this study, a single motor was used to drive the gripper clockwise, so as to rotate the angle of θ to grip the fruit, and counterclockwise, so as to rotate the angle of θ to drive the scissors to cut the fruit stem; therefore, gripping and cutting involved the same mechanism, and the two control actions were completed independently. For cutting control of the harvesting device, this study’s controls scissors were used to cut fruit stems by counterclockwise rotation of the motor. The speed of the motor was inversely proportional to the torque, according to the fixed power of the motor; therefore, in this study, the cutting force control of the cutting fruit stem was achieved by controlling the motor speed. In this study, the input signal to the motor (motor angle from 0° to 180°), ranged from slow cutting frequency (0.25 Hz) to fast frequency (1 Hz); through the output motor speed measurement and instrumental variable method [30], the parameters of the system were identified, to obtain the transfer function of the motor speed to the angle error, as follows:

$$G(s)=\frac{sY(s)}{E(s)}=\frac{417.4}{s+25.11}.$$

(12)

$E(s)$ is the angle error and $Y(s)$ is the output angle obtained by the angle sensor. Figure 7a shows a block diagram of the cut control of the harvesting device, where P is the power of the driving motor. This study used ADAMS and MATLAB software to simulate the cutting force analysis of the fruit stems. According to the motor (Dynamixel/MX-106) used in this study, a maximum torque of 5.7 Nm was provided at 180 deg/s, as the driving condition to cut an apple with a stem thickness of 2 mm and a diameter of 80 mm. Based on the final simulation results, the cutting force was approximately 94 N. The literature [31] discusses the cutting force requirement for cutting off mango fruit stems. The highest cutting force requirement of the fruit stem was 8.64 N, which shows that this design is sufficient to provide a cutting force.

This study used the clockwise rotational force of the motor to control the stable grip of the harvesting device. Because a gripper with torsion spring was used in this study, when the size of the fruit was different from its rigidity, the gripper used different forces to grip the fruit, so as to avoid the fruit being gripped by an excessive force; therefore, this study utilized impedance control [23,24,25], which can adjust the dynamic relationship between the displacement and contact force of the system. This study assumed that the gripping dynamics of the harvesting device were a linear second-order system, to control the displacement and contact force of the system by the behavior of the mass-spring-damping system; the target impedance relationship was obtained as follows:

$$M_d({\ddot{\mathsf{\theta }}}_d-{\ddot{\mathsf{\theta }}}_r)+D_d({\dot{\mathsf{\theta }}}_d-{\dot{\mathsf{\theta }}}_r)+K_d({\mathsf{\theta }}_d-{\mathsf{\theta }}_r)=(F_r-F)L=e_{F}L,$$

(13)

where $M_d$, $D_d$, and $K_d$ were the expected mass, damping, and stiffness coefficients for the target gripping dynamic impedance, ${\mathsf{\theta }}_r$ was the reference angle of the rotation of the gripper, $F_r$ was the reference force of the gripper, F was the force of the gripper to grip the fruit, $e_F$ was the force error, and ${\mathsf{\theta }}_d$ denoted the expected angle of the gripper, generated by the target impedance. The motor drove the gripper to follow the expected position through the mechanism, and when the gripping angle θ of the gripper was greater than the contact angle ${\mathsf{\theta }}_c$(${\mathsf{\theta }}_c={\mathsf{\theta }}_{max}-\phi$), the gripper began to contact the fruit, to compress the torsion spring, so as to generate force. The target impedance relationship between force and angle was:

$$\frac{{\mathsf{\theta }}_d-{\mathsf{\theta }}_r}{(F_r-F)L}=\frac{1}{M_d{s}^2+D_{d}s+K_d}.$$

(14)

The force of the gripper to grip the fruit was the torsional force generated by the compression of the gripper spring under moment arm $L$ (Figure 3) at the contact point of the fruit; therefore, according to Equation (3), the force of the gripper for gripping the fruit was:

$$F(\mathsf{\theta })=\{\begin{array}{ll}\hfill 0,& (\mathsf{\theta }-{\mathsf{\theta }}_c)<0\\ \hfill \frac{\mathrm{K}(\mathsf{\theta }-{\mathsf{\theta }}_c)}{L},& (\mathsf{\theta }-{\mathsf{\theta }}_c)\ge 0\end{array}.$$

(15)

Assuming $e_{\mathsf{\theta }}={\mathsf{\theta }}_d-\mathsf{\theta }$, then:

$$\begin{array}{ll}\hfill e_F=F_r-F=F_r-\frac{\mathrm{K}(\mathsf{\theta }-{\mathsf{\theta }}_c)}{L}& =F_r+\frac{{\mathrm{K}\mathsf{\theta }}_c}{L}-\frac{\mathrm{K}}{L}({\mathsf{\theta }}_d-e_{\mathsf{\theta }})\\ & =F_r+\frac{{\mathrm{K}\mathsf{\theta }}_c}{L}+\frac{\mathrm{K}{e}_{\mathsf{\theta }}}{L}-\frac{\mathrm{K}}{L}({\mathsf{\theta }}_r+\frac{e_{F}L}{M_d{s}^2+D_{d}s+K_d}).\end{array}$$

(16)

Rearranging Equation (16), we get:

$$\left[M_d{s}^2+D_{d}s+(K_d+\mathrm{K})\right]e_F=\left[M_d{s}^2+D_{d}s+K_d\right](F_r+\frac{\mathrm{K}{\mathsf{\theta }}_c}{L}+\frac{\mathrm{K}{e}_{\mathsf{\theta }}}{L}-\frac{\mathrm{K}{\mathsf{\theta }}_r}{L}).$$

(17)

From Equation (17), the steady-state error $e_F^{ss}$ of the gripping force was obtained as follows:

$$e_F^{ss}=K_s(\frac{F_r}{\mathrm{K}}+\frac{{\mathsf{\theta }}_c}{L}+\frac{e_{\mathsf{\theta }}}{L}-\frac{{\mathsf{\theta }}_r}{L}),$$

(18)

where $K_s=\frac{K_d\mathrm{K}}{(K_d+\mathrm{K})}$ was the coefficient of the equivalent stiffness of the system, and $K_d$ was the stiffness of the fruit. In the design of this study, $K_d\gg \mathrm{K}$, so the equivalent stiffness coefficient is rewritten as:

$$K_s=\frac{K_d\mathrm{K}}{(K_d+\mathrm{K})}\approx \mathrm{K}.$$

(19)

Because the contact angle $\widehat{{\mathsf{\theta }}_c}$, the diameter $2r$ of the fruit, and $\widehat{\phi }$ were calculated by the recognition of fruit based on YOLOv3-tiny and image processing, to obtain $\widehat{{\mathsf{\theta }}_c}={\mathsf{\theta }}_{max}-\widehat{\phi }$, the contact angle ${\mathsf{\theta }}_c$ contained the error $∆{\mathsf{\theta }}_c$ and was expressed as

$${\mathsf{\theta }}_c=\widehat{{\mathsf{\theta }}_c}+∆{\mathsf{\theta }}_c.$$

(20)

According to Equation (15), ${\mathsf{\theta }}_r$ was the reference angle of the gripper rotation, and was expressed as

$${\mathsf{\theta }}_r={\mathsf{\theta }}_c+\frac{F_{r}L}{\mathrm{K}}.$$

(21)

Because $e_{\mathsf{\theta }}\cong 0$, Equation (18) was rewritten as

$$e_F^{ss}=\frac{K_d\mathrm{K}∆{\mathsf{\theta }}_c}{(K_d+\mathrm{K})L}.$$

(22)

The results indicated that $∆{\mathsf{\theta }}_c$ changed the steady-state error of the gripping force. Figure 7b shows a block diagram of the harvesting device grip control.

To simulate the clamping analysis of the impedance control of fruits (apples with a diameter of 80 mm and a weight of 235 g), ADAMS and MATLAB software were used in this study. According to Equations (2), (3), and (7), the torsion spring provided a force F of 5.5 N for this apple ($\mathrm{r}$ = 40 mm, ${\mathrm{d}}_{\mathrm{x}}=28 \mathrm{mm}$, $\mathrm{L}=$ 70 mm, $\phi = {10}^{°}$, $\mathrm{K}=38.75\mathrm{mNm}/\mathrm{deg}$ based on $\mathrm{E}=195\mathrm{GPa}$, $\mathrm{d}=3.2\mathrm{mm}$, $\mathrm{D}=21.8\mathrm{mm}$, and ${\mathrm{N}}_{\mathrm{a}}=6.6 rev$). According to Equations (3) and (7), and its actual required minimum force, $F_r$ is 3.6 N ($\mathrm{W}=235\mathrm{g}$, $\mathrm{g}=9.8 \mathrm{m}/{\mathrm{s}}^2$, $\phi = {10}^{°}$, $\mu =0.5$ (according to [32], the friction coefficient between apple and rubber ranges from 0.462 to 0.941)); therefore, the compliant gripping mechanism alone ensured that the apple had enough force to be gripped, but it was not the optimal amount of force, and could cause grip damage. Consequently, impedance control was utilized, such that the gripper did not have to be completely closed when gripping, but was adjusted to the optimal angle of rotation of the gripper under a state of force control. According to the mechanism design of this study, if ${\mathsf{\theta }}_{max}$ was ${60}^{°}$, then ${\mathsf{\theta }}_c$ was ${50}^{°}$ (apples with a diameter of 80 mm and a weight of 235 g). According to Equation (19), this study adopted K = 38.75 mNm/deg, assuming that $K_d$ varied from 0 to 500 mNm/deg, to calculate $K_s$ as shown in Figure 8.

In Figure 8, $K_s$ begins to converge when $K_s$ is greater than 250 mNm/deg; therefore, this study assumed $M_d=1 {\mathrm{kgm}}^2/\mathrm{deg}$ and $D_d=0 {\mathrm{kgm}}^2/\mathrm{sdeg}$; then, using impedance control with $K_d$ = 200, 250, 300, and 350 mNm/deg, the grip force F was obtained, as shown in Figure 9. The convergent and stable performances of the grip force with $K_d$ = 300 and 350 mNm/deg, are shown in Figure 9. In addition, as this study assumed $D_d=0 {\mathrm{kgm}}^2/\mathrm{sdeg}$ and $K_d=300 \mathrm{mNm}/\mathrm{deg}$, with $M_d=1, 0.5, 0.05, \mathrm{and} 0.01 {\mathrm{kgm}}^2/\mathrm{deg}$, the force F of the gripper to hold the fruit was obtained, as shown in Figure 10. The convergent and stable performances of grip force with $M_d=0.05 \mathrm{and} 0.01 {\mathrm{kgm}}^2/\mathrm{deg}$ are represented in Figure 10.

As this study assumed $M_d=0.05 {\mathrm{kgm}}^2/\mathrm{deg}$ and $K_d=300 \mathrm{mNm}/\mathrm{deg}$, with $D_d=0, 0.1, \mathrm{and} 1 {\mathrm{kgm}}^2/\mathrm{sdeg}$, the force F of the gripper, to hold the fruit, was obtained, as shown in Figure 11. According to Figure 11, the performance of the grip force with a $D_d$ of 1 was the best among the three simulation results. Finally, the simulation results of the grip control of the harvesting device, with and without impedance control, based on $M_d=0.05 {\mathrm{kgm}}^2/\mathrm{deg}$, $D_d=1 {\mathrm{kgm}}^2/\mathrm{sdeg}$, and $K_d=300 \mathrm{mNm}/\mathrm{deg}$, are shown in Figure 12. As shown in Figure 12, the gripper contacted the apple at 0.8 s. The gripping forces and angles were 36 N and 56.5°, respectively, with impedance control, and 55 N and 60°, respectively, without impedance control. Accordingly, a gripping force of 9 N more than that of impedance control can easily pinch apples.

4. Experiment and Discussion

The first step in automatic fruit harvesting is to recognize the fruit in the image space, and to follow up with the fruit closest to the harvesting device, for subsequent tracking and harvesting. After completing the harvesting of the first fruit, the above-mentioned steps are repeated, to search for the nearest fruit, and to finish harvesting. When all the fruits on the tree have been harvested, the harvesting operation is complete. This study used the YOLOv3-tiny algorithm to recognize fruits. For convenience of verification, we used apple harvesting as the object. In this study, a motor-rotating platform was used: an apple was placed on it, and pictures of the apple were continuously taken under various angles of rotation, to establish the model for the YOLOv3-tiny algorithm. This study used four kinds of apples: Braeburn, Crimson Snow, Pink Lady, and Red. A total of 2512 apple images from various angles and 2512 blank images were obtained in this study, totaling 5024 images. Among them, 3768 images were used for modeling, and 1256 images were used to test the accuracy of the YOLOv3-tiny algorithm. Finally, after 1500 training iterations, the mean average precision (mAP) reached 100%, to complete the accurate YOLOv3-tiny algorithm modeling.

To verify the apple harvesting experiment, a field was constructed, as shown in Figure 13. This study used a lithium battery of 3300 mAh to drive four motors (MX-106), a motor controller (OpenCM9.04-C), a V2 CCD (Camera Module V2), and a Raspberry Pi 4 Model B. The Raspberry Pi 4 Model B controlled the motor clockwise, to rotate the angle of θ to drive the gripper to grip the fruit, and counterclockwise to rotate the angle of θ to drive the scissors to cut the fruit stem by OpenCM9.04-C, and measured the images of the fruits by Camera Module V2. As shown in Figure 13, a bracket hung an apple (80 mm in diameter, 235 g in weight) at a distance of 200 mm from the automatic fruit harvesting device. First, V2 CCD used the YOLOv3-tiny algorithm to move the harvesting device through the X-axis, Z-axis, and Z-axis rotation motors, to search for and recognize the apple, as shown in Figure 13b,c. To measure the diameter of the apple, this study utilized the image process of a hue saturation value (HSV)-mask filter, to obtain the profile of the apple. Accordingly, the diameter of the apple was calculated for the profile of the apple, by curve-fitting of the circular curve equation, as shown in Figure 13d. Finally, φ was calculated from the diameter of the apple and the geometric dimensions of Figure 3, and ${\mathsf{\theta }}_C$ was obtained by subtracting $\phi$ from ${\mathsf{\theta }}_{max}$. After searching for the apple and correcting the alignment of the apple in the direction of the X-axis, by driving the X-axis, Z-axis, and Z-axis rotation motors, the height of the apple was also aligned with the height of the gripper. The X-axis was forward, and the distance between the apple and gripper was sensed by the TVSM. The X-axis motor was then driven along the X-axis direction, to approach the apple and sensing distance. When the gripper touched the apple, it began to drive the motor, to rotate the angle of θ, so as to grip the apple. When θ was greater than ${\mathsf{\theta }}_c$, the Raspberry Pi 4 Model B began to use impedance control to control the gripping force F, for gripping the apple according to its size and rigidity, so as to avoid excessive force during the gripping process. Finally, after a firm grip was achieved, the motor was triggered, to rotate the angle of -θ, so as to cut off the stems and complete the apple harvesting action. A flowchart of the apple harvest process is shown in Figure 14. To record the harvesting process in detail, this study took an image of the harvesting process, with the top-view CCD and V2 CCD, during the harvesting process, as shown in Figure 15.

The time results of the gripping force, and the angle in the process of gripping the apple, are shown in Figure 16. In the figure, when the gripper did not touch the apple, the gripping force was 0. Until 0.58 s, the gripping angle was 48.7°, then the apple began to be stressed. At 0.72 s, the gripping force was at a maximum of 3.55 N, and the gripping angle was 57.4°. At 0.9 s, the gripping force and the angle began to converge to 3.54 N and 55.5°, respectively. It can be seen from the experimental results that the difference between the maximum gripping force and final gripping force was 0.01 N. Compared with the simulation results of the gripping force, the experimental results based on impedance control had a smaller overshot. In addition, the difference in the final gripping angle between the simulation and the experiment was 1°. This result indicates that the impedance control algorithm used in this study was successfully applied to an actual fruit gripping application.

The time-resulting graph of the torque, and the angle of the motor in the process of shearing the stem, is shown in Figure 17. When the motor was switched on, there was a starting torque. The scissors were then driven at a constant speed, started touching the fruit stem at 0.6 s, and reached the maximum torque at 0.9 s, to cut the fruit stem. Subsequently, the motor ran at the same speed, and stopped operation when the angle reached 180°. The maximum motor torque when cutting the fruit stem was 0.6 Nm, and the cutting force of the fruit stem—calculated from the geometric dimensions of Figure 4 to the contact point of the fruit stem—was 9.9 N. This result was significantly lower than the theoretical upper limit of the shear force of 94 N, which was simulated by ADAMS, and provided a maximum torque of 5.7 Nm to drive the motor at a speed of 180 deg/s. Accordingly, the harvesting device used in this study was sufficient to complete the pruning of fruit stems in such apples, even though the scissors were slightly blunt after many cuts. Although the verification field of this study was in a laboratory construction field, according to the experimental results, the harvesting device would sufficiently complete fruit harvesting in an orchard, in daytime and in slight wind; moreover, the robotic arm based on the harvesting mechanism was modified into a rainproof wearable device, which could harvest fruits from the high parts of the tree.

5. Conclusions

This study designed a harvesting mechanism to independently drive a gripper and scissor for their individual tasks, which corresponded to forward or reverse rotation by a single motor. Consequently, this study utilized a robotic arm, combined with the harvesting mechanism, and supported by a single machine vision component, to successfully recognize apples using the YOLOv3-tiny algorithm. Furthermore, this study completed the coordinate positioning of the fruit, using TVSM, which was used to achieve an image depth measurement. Finally, this study used impedance control, based on the visual feedback of the YOLOv3-tiny algorithm and TVSM, to grip the apple according to its size and rigidity, so as to avoid excessive force during the gripping process. The gripping process was experimentally verified by observing the gripping of an apple. The apple harvesting task was completed with a 3.6 N contact force for an apple with a weight of 235 g, based on a diameter of 80 mm. During the cutting process, the contact point of the metal scissors of the motor-driven mechanism provided a shear force of 9.9 N, which was significantly smaller than the simulation result of 94 N obtained using ADAMS and MATLAB software, even though the scissors were slightly blunt after many cuts. Although the verification field of this study was in a laboratory construction field, according to the experimental results, the harvesting device would sufficiently complete fruit harvesting in an orchard in daytime and in a slight wind. The robotic arm, based on the harvesting mechanism, was modified into a rainproof wearable device, which could harvest fruits from high parts of the tree. The automatic harvesting technology established in this study can reduce harvesting manpower, and increase the speed and convenience of harvesting.