1. Introduction
Cost and availability of human labor are two major concerns for sweet cherry growers. As cherries are characterized by clusters of small fruit in random spatial locations and orientations in tree canopies, hand picking is highly labor intensive. Though the labor-saving technology for sweet cherry harvesting is a critical need for the industry [
1], no commercially adoptable sweet cherry harvesters have been available to the growers so far. The advancement in mechanical sweet cherry harvesting would help the cherry industry to be more competitive and sustainable in the long term [
2].
Over the past several decades, many studies have been conducted to develop mechanical and automated solutions for harvesting tree fruit crops [
3,
4,
5,
6]. One of the widely used harvesting technique is using vibrational energy, which has been shown to be an effective way of harvesting various types of fruit crops, especially berries and cherries with smaller fruit growing in clusters [
7,
8,
9]. In this harvesting method, fruit removal is accomplished by delivering vibrational energy at appropriate frequency using trunk shakers, limb shakers, or canopy shakers [
10,
11,
12,
13], which generally detaches fruit at its abscission zone [
7].
Peterson et al. (1999) [
14] proposed a fully automated bulk apple harvester with an imaging system to guide a robotic arm. Automatic image segmentation was attempted using a color camera. It was reported that detection of apples was successful but the detection of branches was considerably difficult. Therefore, identifying shaking positions on the branches was completed manually by clicking at the desired point on the tree images. Such manual operation would provide the two-dimensional co-ordinates (
x,
y) of the shaking location and the depth was determined physically by allowing the actuator to move until a limit switch was pressed when it came in contact with the branch. The development of a robust image processing system that can detect branches and a method to automatically locate shaking positions in the tree branches is the desirable next step towards fully automated tree fruit harvesting with a mechanical shaker.
Amatya et al. (2016) [
15] developed a method of detecting cherry tree branches often covered with leaves and fruit using morphological properties of visible branch segments and reported a detection accuracy of 89% in vertical planar architecture. Amatya and Karkee (2016) [
16] further improved the branch detection method by using location of cherry clusters as a clue to detect heavily or completely occluded branches in high foliage density orchards trained in Upright Fruiting Offshoots (UFO) architecture. The improved method reported a branch detection accuracy of 93% in a UFO orchard architecture with a Y-trellis training system. These studies showed promise for a machine vision system to automate shake-and-catch cherry harvesting systems. The next step in automated shake-and-catch harvesting is to automatically determine and locate the shaking positions in the detected tree branches for shaking off cherries.
There have been numerous studies in improving mechanical harvesting technology for sweet cherries to increase harvesting efficiency and quality of harvested fruit while keeping the harvest-induced damage to the tree at a minimum level. Some studies have indicated that higher fruit removal efficiency can be obtained by prolonged shaking at multiple shaking positions [
17,
18]. Zhou el al. (2013) [
18] showed that fruit removal efficiency is affected by shaking frequency and duration, and reported 81% fruit removal with four intermittent shaking of 5 s each at a frequency of 18 Hz. Other studies have indicated that excitation position on the tree branches also influences the transfer of energy along the branch and consequently affects the fruit removal efficiency [
19,
20]. Zhou et al. (2014) [
20] tested fruit removal efficiency at different excitation positions with a hand-held shaker by dividing Y-trellis cherry trees into four excitation zones. The results indicated that maximum fruit removal was achieved for lowest shaking position followed by the highest shaking position. It was also reported that up to 97% fruit removal efficiency could be achieved if shaking was performed at both top and bottom excitation zones.
Peterson and Wolford (2001) [
9] developed a mechanical cherry harvester with a branch impacting mechanism and catching conveyor system, which transported harvested cherries to a collection bin. They reported a potential for harvesting up to 85–92% of sweet cherries but also resulted in damage to the tree and fruit with multiple impacts during harvesting. Another major drawback was the difficulty in seeing the branches and subsequent difficulty in positioning the impacting mechanism on tree branches. Chen et al. (2012) [
7] also reported that the limited visibility for the operator was challenging for accurately aiming the shaking mechanism to targeted branch locations. Larbi et al. (2015) [
21] modified a cherry harvester by replacing the impacting mechanism with a continuous shaking mechanism in order to reduce the damage caused by the impact. In addition, the shaking mechanism was operated using a remote controller to improve the harvester’s operability. With a remote controller in hand, the operator had more flexibility to move around to get a proper view and target the shaking mechanism accurately. With this added flexibility, the efficiency of hitting a target branch with an impactor was 93% and the average time required for such maneuvering was 19.9 s per position. The results indicated that the positioning of the shaking mechanism in the target branches was still problematic because the presence of a catching conveyor under the target branch would limit the operator from going too close to have a better view. On the other hand, the operator’s skill level affects the harvest rate and positioning time. Larbi and Karkee (2014) [
22] showed that there was an operator variability of 8.3% in shaker positioning time and up to 16.1% in fruit removal rate.
The difficulty in positioning the shaking mechanism on target branches is due to the limited visibility of the branches. A machine vision system can be more effective in detecting tree branches in dense canopies and position the shakers in target locations [
23]. This automated method will not only eliminate the visibility issues but also reduce positioning time taken by operators. The detection of tree branches has been carried out in previous studies with satisfactory accuracy. However for positioning of shaking mechanisms on proper shaking locations, the shaking positions need to be identified in detected branches. Therefore, this research has focused on developing a method of automatically selecting shaking positions in tree branches for effective cherry harvesting. The shaking positions are selected considering the amount of cherries available for harvest and their relative location in the canopy.
4. Conclusions
For automated sweet cherry harvesting with the shake-and-catch system, the machine is required to make decisions on the number and location of shaking positions and estimate their 3D location in the canopy. This research focused on developing a method for determining shaking positions in cherry tree branches detected by a machine vision system. The localization of shaking positions on tree branches included; (i) determination of shaking position in each branch to harvest cherries; and (ii) estimation of distance to shaking position from the camera location by mapping depth information onto the color images. The root mean square error (RMSE) on estimating distance to shaking positions was found to be 6.4 cm. It was also observed that the distance estimation was more accurate (RMSE of 4.8 cm compared to reference measurements from laser distance measure) when the shaking position was selected over the visible branch regions compared to the positions in occluded regions of the canopy where distance was estimated using a linear interpolation method.
The mechanical shaking of tree branches at the shaking positions determined by the algorithm was carried out in Y-trellis and vertical trellis canopies. The first shaking event removed the largest amount of fruit from tree branches regardless of the tree architecture. The maximum fruit removal achieved with shaking at multiple positions was 92.9% for the Y-trellis system, which required as many as five shaking positions per branch in some cases. For the vertical trellis system, the maximum fruit removal efficiency was 86.6%, which took up to four shaking positions per branch. However, it was found that three shaking positions per branch would be enough for harvesting most of the cherries that could be removed by branch shaking in most of the cases. The first shaking event in the vertical trellis removed more fruit (47.4%) compared to that in Y-trellis system (29.1%). The results indicated that relatively larger diameter (of the vertical system) might play a role in increasing the effectiveness of energy transfer along the branch and therefore more efficient fruit removal. Overall, fruit removal in the vertical trellis system was lower than in the Y-trellis system because of undetected branches and cherries. Maintaining a good two-dimensional fruiting wall structure and spacing between branches may help to improve branch and cherry detection accuracy and the overall harvest efficiency.