Sensing and Automation in Pruning of Tree Fruit Crops : A Review 1

1Department of Agricultural and Biological Engineering, Pennsylvania State University 3 2Penn State Fruit Research and Extension Center 4 3Department of Plant Science, Pennsylvania State University 5 Corresponding author: Long He, luh378@psu.edu 6 7 Abstract: Pruning is one of the most important tree fruit production activities, which is highly 8 dependent on human labor. Skilled labor is in short supply, and the increasing cost of labor is 9 becoming a big issue for the tree fruit industry. Growers are motivated to seek mechanical or 10 robotic solutions for reducing the amount of hand labor required for pruning. This paper reviews 11 the research and development of sensing and automated systems for branch pruning for tree fruit 12 production. Horticultural advancements, pruning strategies, 3D structure reconstruction of tree 13 branches, as well as practice mechanisms or robotics are some of the developments that need to 14 be addressed for an effective tree branch pruning system. Our study summarizes the potential 15 opportunities for automatic pruning with machine-friendly modern tree architectures, previous 16 studies on sensor development, and efforts to develop and deploy mechanical/robotic systems for 17 automated branch pruning. We also describe two examples of qualified pruning strategies that 18 could potentially simplify the automated pruning decision and pruning end-effector design. 19 Finally, the limitations of current pruning technologies and other challenges for automated 20 branch pruning are described, and possible solutions are discussed. 21


Introduction
The tree fruit industry is an important component in the U.S. agricultural sector, accounting for 26% ($11 billion) of all specialty crop production (USDA NASS, 2015).Presently, the majority of tree fruit crop production systems are highly dependent on seasonal human labor.Many critical activities are not only labor intensive, but are also highly time sensitive.This intense labor demand creates a significant risk of growers not having a sufficient supply of labor to conduct seasonal tasks for tree fruit production (Fennimore and Doohan, 2008;Calvin and Martin, 2010).Going forward, it is critical to minimize dependence on labor for the long-term sustainability of this industry (Gonzalez-Barrera, 2015).
In recent decades, automation technologies, especially the use of autonomous tractors has created enormous gains in efficiency for agriculture in general (Noguchi et al., 2002;Kise et al., 2005;Zhang and Pierce, 2016).However, for the specialty crops including tree fruit crops, the application of automation and precision has lagged behind due to the complexity of field operations and inconsistency of crop systems (Karkee and Zhang, 2012).Studies have shown great potential for using mechanical or robotic system for trees with compatible canopy training, including autonomous assist platform (Lesser et al., 2008;Hamner et al., 2011), and mechanical/robotic harvesting (Gongal et al., 2015;De Kleine and Karkee, 2015;Davidson et al., 2016).New mechanization-friendly orchard architectures with better machine accessibility to either fruits or branches is presenting the promising opportunities for automation in tree fruit production.Dormant pruning of fruit trees refers to removing unproductive parts of trees, and is essential to maintain overall tree health, control plant size, and increase fruit quality and marketable yield.
Presently, dormant pruning is still accomplished by field crews either using manual loppers or powered shears.Pruning is the second largest labor expense for tree fruit field production behind harvesting, accounting for 20% or more of total pre-harvest production cost (Gallardo et al., 2011;Hansen, 2011).Due to the high cost and declining availability of skilled labor, alternative solutions for pruning fruit trees are becoming essential.Non-selective mechanical pruning and precise robotic pruning are solutions proposed to address these issues.
Mechanical pruning mainly referring to hedging is a non-selective process with a high rate of throughput, which will usually require follow-up hand pruning (Sansavini, 1976).Hedging has been tried in the past as a supplement or replacement to selective hand pruning.Ferree and Lakso (1979) evaluated hedging for dormant pruning of vigorous semi-dwarf apple trees, and reported negative consequences of low within-canopy light levels and poor fruit color.These conditions resulted from a proliferation of new shoots arising from the numerous non-selective heading cuts made by the hedger.Summer hedging with dormant selective hand pruning, conversely, was shown to be beneficial in creating higher light levels in the lower canopy (Ferree, 1984).
Robotic pruning, which is a selective pruning with accurate cuts, typically cuts the branch by an end-effector consisting of a cutting blade and anvil that uses a scissors motion (Lehnert, 2012).
Before cutting, the challenge is to detect the targeted branches, and identify the pre-determined cutting point.The location, orientation, and dimension of the branch are the critical information for conducting accurate pruning.For cutting itself, it also requires a highly specific degree of accuracy with respect to the placement of the end effector, the jaws of which must be maneuvered into a position over the branch and perpendicular to branch orientation.This level of specificity in the spatial placement of the end effector results in a complex set of maneuvers and slows the pruning process, resulting in low efficiency.In the following sections, we will discuss the horticultural advancement for automated pruning, the involved core technologies, the current development of automated pruning, and the issues and challenges that remain in the procedure.

Horticultural Advancement
Tree architecture is critical to the success of adopting automation in orchard production systems.
Previously, studies were conducted on free-standing trees on semi-dwarf rootstocks, established at moderate planting density and trained as central leader trees, with numerous scaffolds and a complex branching hierarchy.Modern intensive orchards have a smaller canopy with less branching hierarchy, and are grown at close spacing on size controlling rootstocks that restrict tree vigor, resulting in a smaller simpler canopy.Trellising reduces the variability in canopy shape and position.This serves to make the operation of machinery simpler and less fatiguing to the machine operator and should facilitate more predictable and repeatable results.
Detection and accessibility to branches and fruits is a key factor for automating labor-intensive orchard tasks.Establishment of intensive orchards systems at close spacing and using sizecontrolling rootstocks and training systems is a global trend.Modern intensive orchard systems could provide easier detection and access to both tree canopy and fruits, resulting in higher potential of applying mechanical and robotic technologies (Dininny, 2017).A 'Robot Ready' concept was proposed recently by Washington State University scientists to train and manage tree orchards for robotic harvesting (DuPont and Lewis, 2018).Horticultural advancement along with the attempt of conducting automated activities in tree fruit production has been blooming recently.Varieties of intensive modern tree architecture systems have been developed and tested for production and labor efficiency.Figure 1 shows two examples: a V-trellis fruiting wall system with horizontal branches, and the tall spindle tree system.For intensive fruit systems, especially those trained to 2D planar fruiting wall, the narrow canopy becomes much simpler and easier to access with machines, which has brought great benefit to tree fruit growers in many aspects.Previous studies have documented the effect of intensive tree architectures in terms of light interception and distribution (Willaume et al., 2004;Zhang et al., 2015), the influence on yield and fruit quality (Robinson et al., 1991;Hampson et al., 2002;Whiting et al., 2005), earlier production and higher returns (Balmer and Blanke, 2001), and compatibility to mechanical solutions such as blossom thinning machines or harvest aids (Lyons et al., 2017;Zhang et al., 2016).Additionally, mechanical or robotic harvesting is also becoming more promising on intensive trees (De Kleine et al., 2015;He et al., 2017a;Silwal et al., 2017).He et al. (2017a) developed a shake and catch harvesting system for trellis trained V-trellis apple trees.Their results indicated that this tree system provides an opportunity to shake only targeted fruiting limbs and catch the fruit just under those limbs, increasing the potential to keep fruit quality at a desirable level for the fresh market.In Silwal et al. (2017), the fruit detection and picking rate could be reach to 100% and 85% for robotic apple picking if working with horizontal branch fruiting wall system.Fewer studies have been focused on automated pruning for the intensive architectural trees.With the simplified tree architecture, tree branches could be detected and identified much easier, thus to apply simple rules for robotic pruning (Karkee et al., 2014).

Pruning Strategies for Automated Pruning
Pruning strategies for fruit trees were mainly determined by certain rules that manage canopy size and shape to improve the light distribution, with the primary goal of improved fruit quality.
Another goal for pruning is to remove a certain amount of flower buds to manage crop density (Robinson, et al., 2014).Schupp (2014) proposed a severity level pruning strategy for tall spindle apple trees to provide guidelines for determining the cutting threshold for robotic pruning.In robotic pruning, cuts must be precisely defined so the computer can transfer accurate information to the pruner.Researchers have studied on developing simple and quantified pruning rules to increase the feasibility of using robotic and automated pruning (Karkee et al., 2014).Two examples of quantified pruning rules are given here, which could be potentially used for robotic pruning systems.
Case 1: Severity pruning levels for tall spindle apple trees (Schupp et al., 2017) Dr. Schupp and his team have been working on creating an effective pruning strategy for tall spindle apple trees based on severity levels.This study was a component of USDA NIFA project Automation of Dormant Pruning of Specialty Crops.The initial phase of the project established four pruning rules; the pruning strategies for the pruning task.A pruning severity index, namely limb-to-trunk ratio (LTR), was calculated from the sum of the cross-sectional area of all branches on a tree at 2.5 cm from their union to the central leader divided by the trunk crosssectional area at 30 cm above the graft union (Figure 2).In the LTR index, a lower value means less limb area relative to trunk area, which represents more severe pruning.Six severity levels ranging from LTR 0.5 to LTR 1.75 have been applied by successively removing the largest branches from the apple trees.The LTR provides a measurable way to define and create different levels of pruning severity and achieve consistent outcomes.This allows a greater degree of accuracy and precision to dormant pruning of tall spindle apple trees.The use of the LTR to establish the level of pruning severity provides a simple and consistent rule for using of autonomous pruning systems.

Figure 2.
Pruning the apple trees with the proposed severity pruning levels (Schupp et al., 2017) The rules and pruning strategy generated in this study are very easily implemented, removing only those branches with diameters greater than the setting level.Once the pruning severity level is determined, the maximum allowable branch diameter could be calculated easily based on the calculation.Those with diameters greater than the threshold will be cut from the branch base or about one inch away the trunk depending on the necessary of new branch growth.Therefore, the first critical information needed for automated pruning is to measure the diameter of the trunk as well as the diameter for each individual branch, then the cross-sections and LTR are calculated, and pruning decision are made.This method is suitable for intensively planted trees with minimal branching complexity and no permanent branches, such as the tall spindle or the super spindle.To apply robotic pruning, it will require machine vision to locate branches and map a pruning path.
Case 2: Pruning based on twig length and length/diameter ratio (Zhang et al., 2017) While the primary goal of this study is to investigate the effect of mechanical harvesting with different pruning treatments, it is still a good model for developing automated pruning, since the proposed rules are simple and measureable.Four different treatments were applied, with 10, 15, 20 cm twig length and variable pruning with diameter to length ratio setting to 0.06 based on our previous study (He et al., 2017b).Figure 3 shows an example of cutting a twig to the length of 15 cm.The pruning rules proposed in this study are mainly for the trellis trained trees with horizontal permanent branches, the removed part is from the twigs growing from the permanent horizontal branches.Three different pruning treatments with different twig lengths were compared, and results showed that shorter length twigs had higher fruit removal efficiency, while it required more cuts since there were more twigs needed to be cut if the targeted twig length is shorter.
Furthermore, by adopting the twig diameter into consideration, an index was proposed based on the ratio of twig diameter and length.With determined index, the twigs with larger diameter could retain longer.The results indicated that the treatment of using an index determined pruning treatment achieved very promising fruit removal as well as fruit yield.The pruning strategy created in this study gained the guidance for the autonomous pruning by providing the specific rules to cut the branches.To apply the created pruning strategy for automated pruning, identifying the twig length as well as the diameter would be essential.

Machine Vision Sensing for Branch Detection and Identification
Normally, the first step of automated pruning is to find tree branch and target the cutting location in the branch.A proper sensing technique is essential to identify the branch in the fruit trees, and then select unwanted branches and cutting points based on the desired pruning rules to conduct selective pruning.Machine vision, is a system combined with sensors and algorithms to obtain information of the target objects.Machine vision sensing has been used in agricultural application for several decades for various operations (Chen et al., 2002;Davies, 2009;McCarthy et al., 2010;Radcliffe et al., 2018).There are many different sensors have been used in machine vision system for detection of agricultural objects, e.g., cameras and Lidar sensor.
Table 1 listed some past studies using different sensors and techniques for tree/branch detection and identification.Among those applications, identifying branches and pruning points for robotic pruning is one.Lidar based machine vision systems have been mainly used for biomass mapping or individual tree detection, especially for the forest application (Brandtberg et al., 2003;Edson and Wing, 2011;Van Aardt et al., 2008).Recently, Li et al. (2017) proposed an adaptive extracting method of tree skeleton based on the point cloud data with a terrestrial laser scanner, and obtained consistent tree structure.A Lidar sensor also has been tried for the branch length and diameter identification (Bucksch and Fleck, 2011).In their study, a 3D canopy structure of trees was modeled using Lidar sensor and a reconstruction algorithm, the results indicated that the correlation could be up to 0.78 and 0.99 for branch length and branch diameter respectively.
Mapping the pruning wood for grape vines in the vineyards was another application of laser scanner (Tagarakis et al., 2013).Furthermore, in Medeiros et al. (2017), a laser sensor was used to collect observation of fruit tress aiming for automatic dormant pruning, the results showed that the system is able to identify the primary branches with an average accuracy of 98% and estimate their diameters with an average error of 0.6 cm.Even the current system is too slow for large-scale practice, the study shows the proposed approach may serve as a fundamental building block of robotic pruners in the near future.
Camera based machine vision system is widely studied and/or applied in the agriculture production in the past several decades for many applications.Among those, fruit detection for tree fruit robotic harvesting was the one attracted most attention (Pla et al., 1993;Jimenez et al., 2000;Hannan et al., 2010;Silwal et al., 2016).There are also a few studies on the branch detection to determine shaking positions using 3D sensing for mechanical massive harvesting (Amatya et al., 2017;Zhang et al., 2017).Amatya et al. (2017) used RGB and 3D cameras to detect and reconstruct the branches that could be used for determining the shaking points for the   pruning points in grapevines (Naugle et al., 1989;McFarlane et al., 1997;Gao and Lu, 2006;Corbett-Davies et al., 2012).Furthermore, a stereo vision system based 3D machine vision system was used and cutting points on the branches were determined with remaining certain length of branches by segmenting the branches and measuring the length of branches (Hosseini and Jafari, 2017).A computer vision system builds a three dimensional (3D) model of the vines, an artificial intelligence (AI) system decides which canes to prune, and a six degree of -freedom robot arm makes the required cuts (Botterill et al., 2016).
Some progress has been made in the development of robotics technology for pruning more complex canopies such as apple trees.Dr. Karkee and his team from Washington State University proposed a machine vision system with 3D camera to detect and identify tree branches for pruning for tall spindle apple trees (Karkee et al., 2014;Karkee and Adhikari, 2015).The studies developed an algorithm with two simple rules to determine the pruning points, i.e., branch length and inter-branch spacing.The results showed that the algorithm removed 85% of long branches, and 69% of overlapping branches.Researchers from Purdue University have been working on tree modeling using different vision sensing for automatic pruning, such as Kinect 2 (Elfiky et al., 2015); RGBD (Akbar et al., 2016a), depth image (Akbar et al., 2016b), and time-of-flight data (Chattopahdyay et at., 2016).Tabb and her collaborator focused on developing a 3D reconstruction of fruit trees (Figure 5) for automatic pruning with identifying the branch parameters such as length, diameter, angle etc. (Tabb, 2009;Tabb and Mederios, 2017;Tabb and Mederios, 2018).Through the studies above on the tree branch pruning, the accuracy of branch detection and identification, the efficiency of branch reconstruction as well as the cost of the system would be critical for the success of the robotic tree branch pruning system.

Mechanical Pruning System
Among those commercialized mechanical systems for tree fruit crop production, mechanical pruning system is one of them.Here, mechanical pruning mainly refers to hedging.There are a few types of pruning machine available on the market designed for tree fruit crops, such as disctype cutter and teeth-type cutter (Figure 6).Depending on the requirement, mechanical pruning could be performed with topping the canopy parallel to the ground, and/or hedging on both sides of canopy (Dias et al., 2014).Hedging is a non-selective mass pruning systems in which a cutting tool was run over rows in orchards keeping pre-determined distance from the trees.With this approach, everything beyond certain distance from canopy center and/or above certain height was removed (Gautz et al., 2002).Hedging pruning has gained extensive application for grape vine pruning (Morris et al., 1975;Bate and Morris, 2009;Poni et al., 2016).For tree fruit crops, non-selective mechanical pruning has been investigated for different crops, such as sweet cherry (Guimond et al., 1998), citrus (Marti and Gonzalez, 2010), and olive (Albarracin et al., 2017).While, those non-selective pruning systems are limited in their ability to ensure the quality of pruning (Carbonneau, 1979;Jensen et al., 1980).Non-selective and excessive pruning can result in excessive growth of shoots, which may lead to reduced fruit quality and yield (Moore and Gough, 2007).On the other hand, inadequate pruning will result in a tree populated with unproductive woods (Carbonneau, 1979;Jensen, 1980).Therefore, mechanical pruning for fruit trees was mainly used for summer pruning with removing some exterior shoots to increase the light interception to the fruits (Ferree and Rhodus, 1993).With intensive tree architecture, hedging technology would be beneficial to those trees due to the less possibility of branch regrowth with simple tree structure.
Furthermore, automated hedging pruning with precise canopy size control could be considered by estimating the canopy size/shape using sensors such as Lidar.

Robotic Pruning System
Robotic pruning is a selective pruning operation, which aims to mechanically prune the tree branches at the same quality and level as human hands.Pruning for tree fruit crops is highly labor intensive, but no work specific to automated pruning has been carried out in the past due to a few challenges.First challenge is the complex environments of tree canopy/structure, second challenge is moving robotic parts quickly, efficiently and delicately.Compare to fruit trees, grape vines are relatively in more uniform architecture, which gained certain amount of studies and field trials.Sevilla (1985) conducted research on a robotic grapevine pruning manipulator with modeling and simulation.Ochs and Gunkel (1993) worked on a machine vision system for grapevine pruner.Similarly, Lee et al. (1994) reported work in the electro-hydraulic control of a vine pruning robot.Kondo et al. (1993 and1994) developed a manipulator and vision system for multi-purpose vineyard robot.Especially, there were two serial robots developed and tested for grape vine pruning (Figure 7), one is from vision robotics Inc. (Koselka, T., 2012), and the other one is from Botterill et al. (2016).However, all of these robotic systems focused on grapevines, which have relative uniform and organized canopy architecture.pruner (Koselka, 2012); Right) A router mill-end cutter (Botterill et al., 2016).
Even there is no actual machine developed for robotic pruning for tree fruit crops, a little progress has been made in the development of robotics technology for pruning more complex canopies such as apple and cherry trees.As we discussed earlier, most of the studies for robotic fruit tree pruning focused on developing machine vision system for the branch identification and reconstruction.While, a few studies focused on the robotic arm for simulation of pruning task  (Korayem et al., 2014;Megalingam et al., 2016).Furthermore, as plenty of robotic systems have been developed for pruning grape vines as well as picking fruits, it would be possible to develop an effective robotic pruner for tree fruit crops when the tree architecture is getting more uniform.
The challenges and solutions were discussed in the following section.

Discussion: Challenges and Solutions
As we discussed earlier, tree structures in modern orchard are getting much simpler by adopting the intensive system.While even with these trees, the pruning task is still relative complex due to the natural of biological system.For robotic pruning, the cuts on branches require high precision with a cutting end-effector applied at the right locations and perpendicular to branch orientation.
A successful robotic pruning system would be considered as accurate, robust, fast, or even inexpensive system.Therefore, the critical points for success of robotic pruning for fruit trees are the accuracy of branch identification/reconstruction, the spatial requirement of pruning endeffector, and the efficiency of pruning operation (time for branch identification and the time for maneuvering the end-effector).
To apply robotic pruning, firstly, the tree branch and cutting location need to be accurately identified.Majority of studies on automated pruning focused on the tree branch identification and reconstruction using machine vison system as we discussed earlier.And some more studies have been reported on developing algorithms to improve the accuracy of the branch reconstruction (Krissian et al., 2000;Chuang et al., 2000;Duan et al., 2004).While, most of these studies focused on the tree skeleton from the 3D images, which typically could get the location and the length of the tree branches.While, it is hard to get other information, such as the diameter and angle of the branches.One of recent study from Tabb and Medeiros showed the capability to detect and automatically measure the branching structure, branch diameters, branch lengths, and branch angles.Those information are required for tasks such as robotic pruning of trees as well as structural phenotyping.While at this stage, it takes about 8 minutes to finish one tree reconstruction, which is too long for practical pruning process (Tabb and Medeiros, 2017).
Not only branch identification task, but also the accessibility of the robotic manipulator and endeffector is challenging due to complexity and variability of agricultural environment, as well as the required speed of operation.The previous developed pruning robots were typically using serial robotic arm with a fix cutter (Figure 7), while this level of specificity in the spatial placement of the end effector results in a complex set of maneuvers and slows the pruning process.Meanwhile, the serial robot arm with an end-effector requires large space for the cutter to engage with the branches.Although it is not for pruning directly, effort has been made to simplify the maneuvers and improve the efficiency of robotic operations in harvesting.Two robotic fruit picking robots have been developed and tested, one is from FFRbotics (Gesher HaEts 12, Israel) and the other one is from Abundant Robotics (California, USA).These robotic arms are in parallel type, which limited the spatial requirement of the picking end-effector.The position of the end-effector could be adjusted at the base of the overall robotic arm, and then the picking end-effector could reach the fruit directly or by extending the rod.Similar robotic arms could be considered for developing the pruning system.While, there is one thing needs to consider, that normally no specific orientation was required for the end-effector to engage fruits.
For robotic pruning, the end-effector (cutter) needs not only to reach the right location, but also to be placed perpendicularly to the branch.To be always perpendicular to the branch, and well as using the parallel type robotic arm, the end-effector should be with adjustable orientation (He et al., 2018, unpublished document).With this kind of end-effector, the cutter itself could be rotated with very small spatial need.Moreover, the cutter could be made of saw blade with no specific orientation constraints.
At last, the economics of the robotic pruning system also needs to be considered.The robotic pruning machine may be too expensive with little or no gain in pruning efficiency compared to human pruners on the self-steering motorized platforms and simple trees like the Tall Spindle.
While, by considering the labor shortage issue as well as putting effort on building low cost robotic pruning system with off-the-shelf components, the benefit of developing a robotic pruning system would be obvious.Meanwhile, multiple robots could be employed to improve the working efficiency.The cost/benefit ratio of a robotic pruning machine will have to be analyzed after the machine is built.

Conclusion
In this study, automated pruning related technologies have been reviewed, from the horticultural advancement, machine vision sensing, pruning strategies, as well as mechanical and robotic pruning development.Through these comprehensive review and discussion, the following statement could be concluded.
1. Tree architecture is very critical for adopting automated orchard operations like pruning and harvesting.Intensive tree orchard with narrow tree canopy or even 2D planar fruiting wall would be suitable for fully autonomous pruning system in the future.
2. In order to develop robotic pruning, simple and quantified pruning rules are the essential of practical pruning strategies.3.Even plenty of studies have been focused on the tree branch identification and reconstruction, the accuracy and efficiency still require to be improved for practical pruning operation.
4. Robotic pruning technologies have been successfully investigated in some uniformed crops, such as grapevines.With the adopting the intensive tree architectures as well as the improvement of cutting end-effector, it is very promising to have a robotic pruning system for tree fruit crops.

Figure 1 .
Figure 1.Intensive modern fruit tree systems.Left) Horizontal branch fruiting wall V-Trellis

Figure 3 .
Figure 3. Pruning based on the length of the twigs and the ratio of length and diameter of twig mechanical sweet cherry harvesting (Figure7a); andZhang et al. (2017)  using Microsoft Kinect sensor and CNN deep learning algorithm to detect and model the horizontal branches in the Vtrellis fruiting wall apple trees for mechanical harvesting purpose (Figure7b).

Figure 4 .
Figure 4. Left) Branch detection for sweet cherry trees in Y-Trellis; Right) Branch detection for

Figure 5 .
Figure 5.An example of tree reconstruction with 3D machine vision.Left) RGB image of the

Figure 7 .
Figure 7. Robotic pruning systems.Left) Vision robotics-Robotic arm with attached customized

Table 1 .
Sensors and techniques used for branch detection and identification in different applications