Sensing and Automation in Pruning of Apple Trees: A Review

: Pruning is one of the most important tree fruit production activities, which is highly dependent on human labor. Skilled labor is in short supply, and the increasing cost of labor is becoming a big issue for the tree fruit industry. Meanwhile, worker safety is another issue in the manual pruning. Growers are motivated to seek mechanical or robotic solutions for reducing the amount of hand labor required for pruning. Identifying tree branches/canopies with sensors as well as automated operating pruning activity are the important components in the automated pruning system. This paper reviews the research and development of sensing and automated systems for branch pruning in apple production. Tree training systems, pruning strategies, 3D structure reconstruction of tree branches, and practice mechanisms or robotics are some of the developments that need to be addressed for an effective tree branch pruning system. Our study summarizes the potential opportunities for automatic pruning with machine-friendly modern tree architectures, previous studies on sensor development, and efforts to develop and deploy mechanical/robotic systems for automated branch pruning. We also describe two examples of qualiﬁed pruning strategies that could potentially simplify the automated pruning decision and pruning end-effector design. Finally, the limitations of current pruning technologies and other challenges for automated branch pruning are described, and possible solutions are discussed.


Introduction
The tree fruit industry is an important component in the U.S. agricultural sector, accounting for 26% ($11 billion) of all specialty crop production.Among all crops, apple is one of the most valuable agriculture commodities in the United State [1].Presently, the majority of tree fruit crop production systems are highly dependent on seasonal human labor.Many critical activities are not only labor intensive, but are also highly time sensitive.This intense labor demand creates a significant risk of growers not having a sufficient supply of labor to conduct seasonal tasks for tree fruit production [2,3].Going forward, it is critical to minimize dependence on labor for the long-term sustainability of this industry [4].
In recent decades, automation technologies, especially the auto guidance for field tractors has been investigated widely [5][6][7].The auto guidance technologies were adopted by large industry companies, such as John Deere, to develop autonomous tractors.However, for the specialty crops including tree fruit crops, the application of automation and precision has lagged behind due to the complexity Agronomy 2018, 8, 211 2 of 18 of field operations and inconsistency of crop systems [8].Studies have shown great potential for using mechanical or robotic system for trees with compatible canopy training, including autonomous assist platform [9,10], and mechanical/robotic harvesting [11][12][13].Machine vision is one of the core technologies in the automation of tree fruit production.A machine vision system is used to detect and localize the fruit for harvesting, and branch detection and reconstruction for robotic tree training and branch pruning [14].
Proper training through correct pruning is important for a healthy, strong fruit tree [15].There are many different tree training systems for apple trees (see Table 1 for the detail).Each training system has its own advantages and disadvantages.Many modern tree fruit orchards are planted high-density using dwarfing rootstocks and training systems designed for maximum sunlight interception, higher fruit yields and quality, and easier worker access [16].New mechanization-friendly orchard architectures with better machine accessibility to either fruits or branches is presenting the promising opportunities for automation in apple production.To achieve targeted tree systems as well as obtain satisfied fruit production, pruning is required in the dormant season and/or summer.Of course, different pruning strategies would be applied to regarding training system for fulfilling the target tree systems.
Dormant pruning of fruit trees refers to removing unproductive parts of trees, and is essential to maintain overall tree health, control plant size, and increase fruit quality and marketable yield.Presently, dormant pruning is still accomplished by field crews either using manual loppers or powered shears, which is a very labor-intensive operation.Pruning is the second largest labor expense for tree fruit field production behind harvesting, accounting for 20% or more of total production cost in the full production years (the fixed investment breakdown for each year, such as land, trellis, irrigation setup and others, is not included) [17,18].Figure 1 shows an example of Gala apple orchard cost distribution in full production years (reproduced from [17]).Meanwhile, worker safety is another concern in the manual pruning, including falls, lacerations, slips, head/eye injuries, strain to shoulder/back, and repetitive motion injuries [19].Due to the high cost and declining availability of skilled labor, and the safety issues in the manual pruning, alternative solutions for pruning fruit trees are becoming essential.Automated pruning, including non-selective mechanical pruning and precise robotic pruning, was proposed to address these issues.
Agronomy 2018, 8, 211 2 of 18 potential for using mechanical or robotic system for trees with compatible canopy training, including autonomous assist platform [9,10], and mechanical/robotic harvesting [11][12][13].Machine vision is one of the core technologies in the automation of tree fruit production.A machine vision system is used to detect and localize the fruit for harvesting, and branch detection and reconstruction for robotic tree training and branch pruning [14].
Proper training through correct pruning is important for a healthy, strong fruit tree [15].There are many different tree training systems for apple trees (see Table 1 for the detail).Each training system has its own advantages and disadvantages.Many modern tree fruit orchards are planted high-density using dwarfing rootstocks and training systems designed for maximum sunlight interception, higher fruit yields and quality, and easier worker access [16].New mechanizationfriendly orchard architectures with better machine accessibility to either fruits or branches is presenting the promising opportunities for automation in apple production.To achieve targeted tree systems as well as obtain satisfied fruit production, pruning is required in the dormant season and/or summer.Of course, different pruning strategies would be applied to regarding training system for fulfilling the target tree systems.Dormant pruning of fruit trees refers to removing unproductive parts of trees, and is essential to maintain overall tree health, control plant size, and increase fruit quality and marketable yield.Presently, dormant pruning is still accomplished by field crews either using manual loppers or powered shears, which is a very labor-intensive operation.Pruning is the second largest labor expense for tree fruit field production behind harvesting, accounting for 20% or more of total production cost in the full production years (the fixed investment breakdown for each year, such as land, trellis, irrigation setup and others, is not included) [17,18].Figure 1 shows an example of Gala apple orchard cost distribution in full production years (reproduced from [17]).Meanwhile, worker safety is another concern in the manual pruning, including falls, lacerations, slips, head/eye injuries, strain to shoulder/back, and repetitive motion injuries [19].Due to the high cost and declining availability of skilled labor, and the safety issues in the manual pruning, alternative solutions for pruning fruit trees are becoming essential.Automated pruning, including non-selective mechanical pruning and precise robotic pruning, was proposed to address these issues.An example of production cost breakdown in percentage for each category for a Gala apple orchard (40 Acres).The fixed cost, such as land, trellis, irrigation setup, etc., is not included [17].
Mechanical pruning, mainly referring to hedging, is a non-selective process with a high rate of throughput, which will usually require follow-up hand pruning [20].Hedging has been tried in the past as a supplement or replacement to selective hand pruning.Ferree and Lakso [21] evaluated hedging for dormant pruning of vigorous semi-dwarf apple trees, and reported negative consequences of low within-canopy light levels and poor fruit color.These conditions resulted from a proliferation of new shoots arising from the numerous non-selective heading cuts made by the hedger.Summer hedging with dormant selective hand pruning, conversely, was shown to be beneficial in creating higher light levels in the lower canopy [22].
Robotic pruning, which is a selective pruning with accurate cuts, typically cuts the branch by an end-effector consisting of a cutting blade and anvil that uses a scissors motion [23].Before cutting, Figure 1.An example of production cost breakdown in percentage for each category for a Gala apple orchard (40 Acres).The fixed cost, such as land, trellis, irrigation setup, etc., is not included [17].
Mechanical pruning, mainly referring to hedging, is a non-selective process with a high rate of throughput, which will usually require follow-up hand pruning [20].Hedging has been tried in the past as a supplement or replacement to selective hand pruning.Ferree and Lakso [21] evaluated hedging for dormant pruning of vigorous semi-dwarf apple trees, and reported negative consequences of low within-canopy light levels and poor fruit color.These conditions resulted from a proliferation of new shoots arising from the numerous non-selective heading cuts made by the hedger.Summer hedging with dormant selective hand pruning, conversely, was shown to be beneficial in creating higher light levels in the lower canopy [22].Robotic pruning, which is a selective pruning with accurate cuts, typically cuts the branch by an end-effector consisting of a cutting blade and anvil that uses a scissors motion [23].Before cutting, the challenge is to detect the targeted branches, and identify the pre-determined cutting point.The location, orientation, and dimension of the branch are the critical information for conducting accurate pruning.For cutting itself, it also requires a highly specific degree of accuracy with respect to the placement of the end effector, the jaws of which must be maneuvered into a position over the branch and perpendicular to branch orientation.This level of specificity in the spatial placement of the end effector results in a complex set of maneuvers and slows the pruning process, resulting in low efficiency.
In the following sections, we discuss the tree training systems and strategies for automated pruning, the machine vision sensing technologies for tree branch identification, the current development of automated pruning systems, and the issues and challenges that remain in the procedure.

Tree Training Systems
Tree training systems are critical to the success of adopting automation in orchard production.Previously, studies were conducted on free-standing trees on semi-dwarf rootstocks, established at moderate planting density and trained as central leader trees, with numerous scaffolds and a complex branching hierarchy.Modern intensive orchards have a smaller canopy with less branching hierarchy, and are grown at close spacing on size controlling rootstocks that restrict tree vigor, resulting in a smaller simpler canopy.Trellising reduces the variability in canopy shape and position.This serves to make the operation of machinery simpler and less fatiguing to the machine operator and should facilitate more predictable and repeatable results.Table 1 shows the typical tree training systems of apple trees and their specifications.
Detection and accessibility to branches and fruits is a key factor for automating labor-intensive orchard tasks.Establishment of intensive orchards systems at close spacing and using size-controlling rootstocks and training systems is a global trend.Modern intensive orchard systems could provide easier detection and access to both tree canopy and fruits, resulting in higher potential of applying Agronomy 2018, 8, 211 4 of 18 mechanical and robotic technologies [32].A "Robot Ready" concept was proposed recently to train and manage tree orchards for robotic harvesting [33].Horticultural advancement along with the attempt of conducting automated activities in tree fruit production has been blooming recently.Various intensive modern tree architecture systems have been developed and tested for production and labor efficiency.Figure 2 shows two examples: a V-trellis fruiting wall system with horizontal branches, and the tall spindle tree system.
This serves to make the operation of machinery simpler and less fatiguing to the machine operator and should facilitate more predictable and repeatable results.Table 1 shows the typical tree training systems of apple trees and their specifications.
Detection and accessibility to branches and fruits is a key factor for automating labor-intensive orchard tasks.Establishment of intensive orchards systems at close spacing and using size-controlling rootstocks and training systems is a global trend.Modern intensive orchard systems could provide easier detection and access to both tree canopy and fruits, resulting in higher potential of applying mechanical and robotic technologies [32].A "Robot Ready" concept was proposed recently to train and manage tree orchards for robotic harvesting [33].Horticultural advancement along with the attempt of conducting automated activities in tree fruit production has been blooming recently.Various intensive modern tree architecture systems have been developed and tested for production and labor efficiency.Figure 2 shows two examples: a V-trellis fruiting wall system with horizontal branches, and the tall spindle tree system.
(A) (B) For intensive fruit systems, especially those trained to 2D planar fruiting wall, the narrow canopy becomes much simpler and easier to access with machines, which has brought great benefit to tree fruit growers in many aspects.Previous studies have documented the effect of intensive tree architectures in terms of light interception and distribution [34,35], the influence on yield and fruit quality [36][37][38], earlier production and higher returns [39], and compatibility to mechanical solutions such as blossom thinning machines or harvest aids [40,41].Additionally, mechanical or robotic harvesting is also becoming more promising on intensive trees [12,42,43].He et al. [42] developed a shake and catch harvesting system for trellis trained V-trellis apple trees.Their results indicated that this tree system provides an opportunity to shake only targeted fruiting limbs and catch the fruit just For intensive fruit systems, especially those trained to 2D planar fruiting wall, the narrow canopy becomes much simpler and easier to access with machines, which has brought great benefit to tree fruit growers in many aspects.Previous studies have documented the effect of intensive tree architectures in terms of light interception and distribution [34,35], the influence on yield and fruit quality [36][37][38], earlier production and higher returns [39], and compatibility to mechanical solutions such as blossom thinning machines or harvest aids [40,41].Additionally, mechanical or robotic harvesting is also becoming more promising on intensive trees [12,42,43].He et al. [42] developed a shake and catch harvesting system for trellis trained V-trellis apple trees.Their results indicated that this tree system provides an opportunity to shake only targeted fruiting limbs and catch the fruit just under those limbs, increasing the potential to keep fruit quality at a desirable level for the fresh market.In [43], the fruit detection and picking rate could reach 100% and 85% for robotic apple picking if working with horizontal branch fruiting wall system.Fewer studies have focused on automated pruning for the intensive architectural trees.With a simplified tree architecture, tree branches could be detected and identified much more easily, by applying simple rules for robotic pruning [44].

Pruning Strategies for Automated Pruning
Pruning strategies for fruit trees were mainly determined by certain rules that manage canopy size and shape to improve the light distribution, with the primary goal of improved fruit quality.Another goal for pruning is to remove a certain amount of flower buds to manage crop density [45].Schupp [46] proposed a severity level pruning strategy for tall spindle apple trees to provide guidelines for determining the cutting threshold for robotic pruning.In robotic pruning, cuts must be precisely defined so the computer can transfer accurate information to the pruner.Researchers have studied on developing simple and quantified pruning rules to increase the feasibility of using robotic and automated pruning [44].Table 2 summarizes some of the pruning strategies for apple trees with different tree training systems.Tall spindle -Certain branch length and spacing -Measure the length of each branch -Determine the density of the branches [44] Super/tall Spindle -Branch simplification pruning: removing lateral secondary branches -Identify the diameter and length of each secondary branches [45] Vertical/V-trellis fruiting wall -Keep secondary branches in certain length -Identify the diameter and length of each secondary branches [48] Trellis training central leader -Hedging and topping -Identify tree structure -Determine the cutting location [49] Slender spindle -Remove tree top -Remove unwanted shoot -Identify tree structure -Determine the cutting location [50] Hedgerow -Hedging and topping -Identify tree structure -Determine the cutting location [20] In Table 2, for automated pruning, the technique difficulties were mainly on localizing the pruning locations through branch detection and reconstruction.Meanwhile, the accessibility of pruning machine/robot is another challenge, namely the difficulty of cutting end-effector to reach the cutting point.For hedging pruning, the technical critical would be the location identification for precise cutting.To provide more detail information about pruning strategies, two examples of quantified pruning rules are given here, which could be potentially used for robotic pruning systems.
Case 1: Severity pruning levels for tall spindle apple trees [47].Dr. Schupp and his team have been working on creating an effective pruning strategy for tall spindle apple trees based on severity levels.This study was a component of USDA NIFA project Automation of Dormant Pruning of Specialty Crops.The initial phase of the project established four pruning rules, i.e., the pruning strategies for the pruning task.A pruning severity index, namely limb-to-trunk ratio (LTR), was calculated from the sum of the cross-sectional area of all branches on a tree at 2.5 cm from their union to the central leader divided by the trunk cross-sectional area at 30 cm above the graft union (Figure 3).In the LTR index, a lower value means less limb area relative to trunk area, which represents more severe pruning.Six severity levels ranging from LTR 0.5 to LTR 1.75 have been applied by successively removing the largest branches from the apple trees.The LTR provides a measurable way to define and create different levels of pruning severity and achieve consistent outcomes.This allows a greater degree of accuracy and precision to dormant pruning of tall spindle apple trees.The use of the LTR to establish the level of pruning severity provides a simple and consistent rule for using of autonomous pruning systems.
Agronomy 2018, 8, 211 6 of 18 trunk area, which represents more severe pruning.Six severity levels ranging from LTR 0.5 to LTR 1.75 have been applied by successively removing the largest branches from the apple trees.The LTR provides a measurable way to define and create different levels of pruning severity and achieve consistent outcomes.This allows a greater degree of accuracy and precision to dormant pruning of tall spindle apple trees.The use of the LTR to establish the level of pruning severity provides a simple and consistent rule for using of autonomous pruning systems.The rules and pruning strategy generated in this study are very easily implemented, removing only those branches with diameters greater than the setting level.Once the pruning severity level is determined, the maximum allowable branch diameter could be calculated easily based on the calculation.Those with diameters greater than the threshold will be cut from the branch base or about one inch away the trunk depending on the necessary of new branch growth.Therefore, the first critical information needed for automated pruning is to measure the diameter of the trunk as well as the diameter for each individual branch, then the cross-sections and LTR are calculated, and pruning decision are made.This method is suitable for intensively planted trees with minimal branching complexity and no permanent branches, such as the tall spindle or the super spindle.To apply robotic pruning, it will require machine vision to locate branches and map a pruning path.
Case 2: Pruning based on twig length and length/diameter ratio [48].
While the primary goal of this study was to investigate the effect of mechanical harvesting with different pruning treatments, it is still a good model for developing automated pruning, since the proposed rules are simple and measurable.Four different treatments were applied, namely 10 cm, 15 cm, and variable twig length pruning, as well as a control set of 20 cm twig length pruning.In variable length pruning, the diameter to length ratio was set to 0.06 based on our previous study [51].Figure 4 shows an example of cutting a twig to the length of 15 cm (reproduced from [48]).The rules and pruning strategy generated in this study are very easily implemented, removing only those branches with diameters greater than the setting level.Once the pruning severity level is determined, the maximum allowable branch diameter could be calculated easily based on the calculation.Those with diameters greater than the threshold will be cut from the branch base or about one inch away the trunk depending on the necessary of new branch growth.Therefore, the first critical information needed for automated pruning is to measure the diameter of the trunk as well as the diameter for each individual branch, then the cross-sections and LTR are calculated, and pruning decision are made.This method is suitable for intensively planted trees with minimal branching complexity and no permanent branches, such as the tall spindle or the super spindle.To apply robotic pruning, it will require machine vision to locate branches and map a pruning path.
Case 2: Pruning based on twig length and length/diameter ratio [48].
While the primary goal of this study was to investigate the effect of mechanical harvesting with different pruning treatments, it is still a good model for developing automated pruning, since the proposed rules are simple and measurable.Four different treatments were applied, namely 10 cm, 15 cm, and variable twig length pruning, as well as a control set of 20 cm twig length pruning.In variable length pruning, the diameter to length ratio was set to 0.06 based on our previous study [51].Figure 4 shows an example of cutting a twig to the length of 15 cm (reproduced from [48]).The pruning rules proposed in this study are mainly for the trellis trained trees with horizontal permanent branches, the removed part is from the twigs growing from the permanent horizontal branches.Three different pruning treatments with different twig lengths were compared, and results showed that shorter length twigs had higher fruit removal efficiency, while it required more cuts since there were more twigs needed to be cut if the targeted twig length is shorter.Furthermore, by adopting the twig diameter into consideration, an index was proposed based on the ratio of twig diameter and length.With determined index, the twigs with larger diameter could retain longer.The results indicated that the treatment of using an index determined pruning treatment achieved very promising fruit removal as well as fruit yield.The pruning strategy created in this study gained the guidance for the autonomous pruning by providing the specific rules to cut the branches.To apply the created pruning strategy for automated pruning, identifying the twig length as well as the diameter would be essential.

Tree/Branch Detection/Reconstruction using Machine Vision System
Normally, the first step of automated pruning is to find tree branch and target the cutting location in the branch.A proper sensing technique is essential to detect the branches in the fruit trees, and then select unwanted branches and cutting points based on the desired pruning rules to conduct selective pruning.With the detection of the tree branches, the branches and the location of each branch could be identified.To further obtain branch detail information, such as diameter, orientation, as well as overall tree structure, 3D reconstruction is essential.Machine vision is a system combined with sensors and algorithms to obtain information of the target objects.Machine vision sensing has been used in agricultural application for several decades for various operations [52][53][54][55].A review summarized the sensors and technologies of 3D imaging systems for agriculture applications [56].Many different sensors have been used in machine vision system for detection of agricultural objects, e.g., cameras and Lidar sensor.Table 3 lists some past studies using different sensors and techniques for tree/branch detection and 3D structure reconstruction.Among those applications, identifying branches and pruning points for robotic pruning is one.The pruning rules proposed in this study are mainly for the trellis trained trees with horizontal permanent branches, the removed part is from the twigs growing from the permanent horizontal branches.Three different pruning treatments with different twig lengths were compared, and results showed that shorter length twigs had higher fruit removal efficiency, while it required more cuts since there were more twigs needed to be cut if the targeted twig length is shorter.Furthermore, by adopting the twig diameter into consideration, an index was proposed based on the ratio of twig diameter and length.With determined index, the twigs with larger diameter could retain longer.The results indicated that the treatment of using an index determined pruning treatment achieved very promising fruit removal as well as fruit yield.The pruning strategy created in this study gained the guidance for the autonomous pruning by providing the specific rules to cut the branches.To apply the created pruning strategy for automated pruning, identifying the twig length as well as the diameter would be essential.

Tree/Branch Detection/Reconstruction using Machine Vision System
Normally, the first step of automated pruning is to find tree branch and target the cutting location in the branch.A proper sensing technique is essential to detect the branches in the fruit trees, and then select unwanted branches and cutting points based on the desired pruning rules to conduct selective pruning.With the detection of the tree branches, the branches and the location of each branch could be identified.To further obtain branch detail information, such as diameter, orientation, as well as overall tree structure, 3D reconstruction is essential.Machine vision is a system combined with sensors and algorithms to obtain information of the target objects.Machine vision sensing has been used in agricultural application for several decades for various operations [52][53][54][55].A review summarized the sensors and technologies of 3D imaging systems for agriculture applications [56].Many different sensors have been used in machine vision system for detection of agricultural objects, e.g., cameras and Lidar sensor.Table 3 lists some past studies using different sensors and techniques for tree/branch detection and 3D structure reconstruction.Among those applications, identifying branches and pruning points for robotic pruning is one.

Lidar Based Machine Vision System
Lidar based machine vision systems have been mainly used for biomass mapping or individual tree detection, with plenty of them for the forest application [57][58][59].Especially, Brandtberg et al. [57] developed a 3D tree reconstruction method by fitting cylinders into a point cloud derived from a terrestrial laser-scanned tree.Using the developed method, the coefficient of determination was 0.965 and showed high potential of using for forest inventories.Furthermore, they developed an open source tool named Simple Tree to provide an efficient optimization approach for tree reconstruction [73].Recently, Li et al. [74] proposed an adaptive extracting method of tree skeleton based on the point cloud data with a terrestrial laser scanner, and obtained consistent tree structure.A Lidar sensor also has been tried for the branch length and diameter identification [75].In their study, a 3D canopy structure of trees was modeled using Lidar sensor and a reconstruction algorithm, the results indicated that the correlation could be up to 0.78 and 0.99 for branch length and branch diameter, respectively.Mapping the pruning wood for grape vines in the vineyards was another application of laser scanner [62].Furthermore, in [66], a laser sensor was used to collect observation of fruit trees aiming for automatic dormant pruning, the results showed that the system is able to identify the primary branches with an average accuracy of 98% and estimate their diameters with an average error of 0.6 cm.Even though the current system is too slow for large-scale practice, the study showed the potential of using the proposed approach to develop robotic pruning systems in the near future.

Camera Based Machine Vision System
Camera based machine vision system has been widely studied and/or applied in agriculture production in the past several decades for many applications.Among those, fruit detection for tree fruit robotic harvesting attracted most attention [76][77][78][79].There are also a few studies on the branch Agronomy 2018, 8, 211 9 of 18 detection to determine shaking positions using 3D sensing for mechanical massive harvesting [60,61].Amatya et al. [60] used RGB and 3D cameras to detect and reconstruct the branches that could be used for determining the shaking points for the mechanical sweet cherry harvesting (Figure 5A); and Zhang et al. [61] used a Microsoft Kinect v2 (ToF camera, Microsoft, Seattle, DC, USA) and a convolutional neural network (CNN) deep learning algorithm to detect and model the horizontal branches in the V-trellis fruiting wall apple trees for mechanical harvesting purpose (Figure 5B).Some other applications of 3D tree reconstruction with machine vision system include a vision sensor (stereo camera) based 3D tree reconstruction for automated blossom thinning [80], 3D models developed with source images to reconstruct the overall tree shape [81], and a single range image used to construct a 3D model of the trunk and branches of a real tree [82].
error of 0.6 cm.Even though the current system is too slow for large-scale practice, the study showed the potential of using the proposed approach to develop robotic pruning systems in the near future.

Camera Based Machine Vision System
Camera based machine vision system has been widely studied and/or applied in agriculture production in the past several decades for many applications.Among those, fruit detection for tree fruit robotic harvesting attracted most attention [76][77][78][79].There are also a few studies on the branch detection to determine shaking positions using 3D sensing for mechanical massive harvesting [60,61].Amatya et al. [60] used RGB and 3D cameras to detect and reconstruct the branches that could be used for determining the shaking points for the mechanical sweet cherry harvesting (Figure 5A); and Zhang et al. [61] used a Microsoft Kinect v2 (ToF camera, Microsoft, Seattle, DC, USA) and a convolutional neural network (CNN) deep learning algorithm to detect and model the horizontal branches in the V-trellis fruiting wall apple trees for mechanical harvesting purpose (Figure 5B).Some other applications of 3D tree reconstruction with machine vision system include a vision sensor (stereo camera) based 3D tree reconstruction for automated blossom thinning [80], 3D models developed with source images to reconstruct the overall tree shape [81], and a single range image used to construct a 3D model of the trunk and branches of a real tree [82].

Branch Detection/Reconstruction for Automated Pruning
Most effort for machine vision towards branch detection for robotic pruning was on grape vines due to its more uniform and organized canopy architecture.For example, some researchers used a single 2D camera and image processing techniques for identification of pruning points in grapevines [64,67,83,84].Furthermore, a stereo vision system based 3D machine vision system was used and cutting points on the branches were determined with remaining certain length of branches by segmenting the branches and measuring the length of branches [85].A computer vision system builds a three-dimensional (3D) model of the vines, an artificial intelligence (AI) system decides which canes to prune, and a six degrees of freedom robot arm makes the required cuts [61].
Some progress has been made in the development of robotics technology for pruning more complex canopies such as apple trees.A machine vision system was proposed and evaluated with 3D camera to detect and identify tree branches for pruning for tall spindle apple trees [44,86].The studies developed an algorithm with two simple rules to determine the pruning points, i.e., branch length and inter-branch spacing.The results showed that the algorithm removed 85% of long branches, and 69% of overlapping branches.A few other studies have also been conducted on tree modeling using different vision sensing for automatic pruning, such as Kinect 2 [68], RGBD [71], depth image [72], and time-of-flight data [67].Tabb and her collaborator focused on developing a 3D reconstruction of fruit trees (Figure 6, reproduced from [69]) for automatic pruning with identifying the branch parameters such as length, diameter, angle, etc. [69,70,87].Based on the studies above on the tree branch pruning, the accuracy of branch detection and identification, the efficiency of branch

Branch Detection/Reconstruction for Automated Pruning
Most effort for machine vision towards branch detection for robotic pruning was on grape vines due to its more uniform and organized canopy architecture.For example, some researchers used a single 2D camera and image processing techniques for identification of pruning points in grapevines [64,67,83,84].Furthermore, a stereo vision system based 3D machine vision system was used and cutting points on the branches were determined with remaining certain length of branches by segmenting the branches and measuring the length of branches [85].A computer vision system builds a three-dimensional (3D) model of the vines, an artificial intelligence (AI) system decides which canes to prune, and a six degrees of freedom robot arm makes the required cuts [61].
Some progress has been made in the development of robotics technology for pruning more complex canopies such as apple trees.A machine vision system was proposed and evaluated with 3D camera to detect and identify tree branches for pruning for tall spindle apple trees [44,86].The studies developed an algorithm with two simple rules to determine the pruning points, i.e., branch length and inter-branch spacing.The results showed that the algorithm removed 85% of long branches, and 69% of overlapping branches.A few other studies have also been conducted on tree modeling using different vision sensing for automatic pruning, such as Kinect 2 [68], RGBD [71], depth image [72], and time-of-flight data [67].Tabb and her collaborator focused on developing a 3D reconstruction of fruit trees (Figure 6, reproduced from [69]) for automatic pruning with identifying the branch parameters such as length, diameter, angle, etc. [69,70,87].Based on the studies above on the tree branch pruning, the accuracy of branch detection and identification, the efficiency of branch reconstruction as well as the cost of the system would be critical for the success of the robotic tree branch pruning system.reconstruction as well as the cost of the system would be critical for the success of the robotic tree branch pruning system.

Automated Pruning System
As introduced above, manual pruning is a labor intensive and costly operation in the tree fruit production.Mechanical pruning and robotic pruning are two alternative methods to replace the manual pruning.By integrating with autonomous platform, these two methods could accomplish automated pruning task.
Among those commercialized mechanical systems for tree fruit crop production, mechanical pruning system is one of them.Here, mechanical pruning mainly refers to hedging.There are a few types of pruning machine available on the market designed for tree fruit crops, such as disc-type cutter and teeth-type cutter [88].Depending on the requirement, mechanical pruning could be performed with topping the canopy parallel to the ground, and/or hedging on both sides of canopy [89].Figure 7 shows a typical mechanical pruning system with saw-tooth cutter.Hedging is a non-selective mass pruning system in which a cutting tool is run over rows in orchards keeping pre-determined distance from the trees.With this approach, everything beyond certain distance from canopy center and/or above certain height is removed [90].Hedging pruning has gained extensive application for grape vine pruning [91][92][93].The application of mechanical pruning for vine grapes showed significant cost saving while maintained yield and fruit quality [94].For tree fruit crops, non-selective mechanical pruning has been investigated for different crops, such as sweet cherry [95], citrus [88], and olive [96].Mechanical pruning of citrus obtained increased yields and reduced pruning cost [97].Mechanical pruning showed certain level of promise for citrus and olive.For example, medium-low vigor olive trees can be successfully controlled by mechanical

Automated Pruning System
As introduced above, manual pruning is a labor intensive and costly operation in the tree fruit production.Mechanical pruning and robotic pruning are two alternative methods to replace the manual pruning.By integrating with autonomous platform, these two methods could accomplish automated pruning task.
Among those commercialized mechanical systems for tree fruit crop production, mechanical pruning system is one of them.Here, mechanical pruning mainly refers to hedging.There are a few types of pruning machine available on the market designed for tree fruit crops, such as disc-type cutter and teeth-type cutter [88].Depending on the requirement, mechanical pruning could be performed with topping the canopy parallel to the ground, and/or hedging on both sides of canopy [89].Figure 7 shows a typical mechanical pruning system with saw-tooth cutter.
Agronomy 2018, 8, 211 10 of 18 reconstruction as well as the cost of the system would be critical for the success of the robotic tree branch pruning system.

Automated Pruning System
As introduced above, manual pruning is a labor intensive and costly operation in the tree fruit production.Mechanical pruning and robotic pruning are two alternative methods to replace the manual pruning.By integrating with autonomous platform, these two methods could accomplish automated pruning task.
Among those commercialized mechanical systems for tree fruit crop production, mechanical pruning system is one of them.Here, mechanical pruning mainly refers to hedging.There are a few types of pruning machine available on the market designed for tree fruit crops, such as disc-type cutter and teeth-type cutter [88].Depending on the requirement, mechanical pruning could be performed with topping the canopy parallel to the ground, and/or hedging on both sides of canopy [89].Figure 7 shows a typical mechanical pruning system with saw-tooth cutter.Hedging is a non-selective mass pruning system in which a cutting tool is run over rows in orchards keeping pre-determined distance from the trees.With this approach, everything beyond certain distance from canopy center and/or above certain height is removed [90].Hedging pruning has gained extensive application for grape vine pruning [91][92][93].The application of mechanical pruning for vine grapes showed significant cost saving while maintained yield and fruit quality [94].For tree fruit crops, non-selective mechanical pruning has been investigated for different crops, such as sweet cherry [95], citrus [88], and olive [96].Mechanical pruning of citrus obtained increased yields and reduced pruning cost [97].Mechanical pruning showed certain level of promise for citrus and olive.For example, medium-low vigor olive trees can be successfully controlled by mechanical Hedging is a non-selective mass pruning system in which a cutting tool is run over rows in orchards keeping pre-determined distance from the trees.With this approach, everything beyond certain distance from canopy center and/or above certain height is removed [90].Hedging pruning has gained extensive application for grape vine pruning [91][92][93].The application of mechanical pruning for vine grapes showed significant cost saving while maintained yield and fruit quality [94].For tree fruit crops, non-selective mechanical pruning has been investigated for different crops, such as sweet cherry [95], citrus [88], and olive [96].Mechanical pruning of citrus obtained increased yields and reduced pruning cost [97].Mechanical pruning showed certain level of promise for citrus and olive.For example, medium-low vigor olive trees can be successfully controlled by mechanical pruning without compromising their yield [98]. Non-selective pruning systems are limited in their ability to ensure the quality of pruning for some other fruit trees, such as sweet cherry, peach and apple [95], which could result in excessive growth of shoots, leading to reduced fruit quality and yield [99].On the other hand, inadequate pruning will result in a tree populated with unproductive woods [95,100].Therefore, mechanical pruning for fruit trees is mainly used for summer pruning with removing some exterior shoots to increase the light interception to the fruits [49].Dwarf and semi-dwarf trees trained in thin hedgerows can be mechanically pruned more easily than single large trees [20,49].With intensive tree architectures in apple trees, hedging technology would be beneficial to those trees due to the less possibility of branch regrowth with simple tree structure.Furthermore, automated hedging pruning with precise canopy size control could be considered by estimating the canopy size/shape using sensors such as Lidar.
Robotic pruning is a selective pruning operation, which aims to mechanically prune the tree branches at the same quality and level as human hands.Pruning for tree fruit crops is highly labor intensive, but no work specific to automated pruning has been carried out in the past due to a few challenges.The first challenge is the complex environments of tree canopy/structure.The second challenge is moving robotic parts quickly, efficiently and delicately.Compared to fruit trees, grape vines are relatively more uniform, which benefited from certain amount of studies and field trials.
Sevilla [101] conducted research on a robotic grapevine pruning manipulator with modeling and simulation.Ochs et al. [102] worked on a machine vision system for grapevine pruner.Similarly, Lee et al. [103] reported work in the electro-hydraulic control of a vine pruning robot.Also a manipulator and vision system was developed for multi-purpose vineyard robot [104,105].Especially, there were two serial robots developed and tested for grape vine pruning; one is from vision robotics Inc. [106], and the other one is from [63].However, these robotic systems focused on grapevines, which have relatively uniform and organized canopy architecture.
Software framework is another important component for automated systems.Some open source software has been widely used, including PCL (Point cloud library), Open CV (Open Source Computer Vision Library), and ROS (Robot Operating System).Holz et al. [107] illustrated a few examples that used PCL for 3D point cloud processing, including RGB-D point clouds by consumer color and depth camera, high-resolution laser scans from 3D scanners, and low resolution sparse point clouds by a custom lightweight 3D scanner [107].Within PCL, an open source implementation of the kinectFusion algorithm is included, and can be used to reconstruct a 3D scene using the Kinect sensor like a handheld laser scanner [108].Open CV has been widely used for the fruit detection for robotic harvesting, such as an apple detection algorithm for robotic harvesting using an RGB-D camera [109]; Open CV was used to test the haze-removal effects in apple harvest robot vision system [110]; and ROS was adopted for a navigation system in a robotic bin handling robot [111].A set of robotic operating system nodes was created to the sensing and motion control of the robotic harvesting system [112].In a robotic pruning system, a path planner used RRT-Connect implementation from the Open Motion Planning Library in Robot Operating System (ROS), and a Radial basis function SVM (RBF-SVM) was implemented to label each pixel as foreground, back ground, or wire, based on its color and the color of its neighborhood [63].
Even though no machine has been developed for robotic pruning for tree fruit crops, some progress has been made in the development of robotics technology for pruning more complex canopies such as apple and cherry trees.As discussed above, most studies on robotic fruit tree pruning focus on developing machine vision system for the branch identification and reconstruction, while few studies focus on the robotic arm for simulation of pruning task [113,114].Furthermore, as plenty of robotic systems have been developed for pruning grape vines as well as picking fruits, it would be possible to develop an effective robotic pruner for tree fruit crops when the tree architecture is getting more uniform.The challenges and solutions are discussed in the following section.

Discussion: Challenges and Solutions
As discussed above, tree structures in modern orchard are getting much simpler by adopting the intensive system.Even with these trees, the pruning task is still relatively complex due to the natural biological system.For robotic pruning, the cuts on branches require high precision with a cutting end-effector applied at the right locations and perpendicular to branch orientation.A successful robotic pruning system needs to be c accurate, robust, fast, and even inexpensive.Therefore, the critical points for success of robotic pruning for fruit trees are the accuracy of branch identification/reconstruction, the spatial requirement of pruning end-effector, and the efficiency of pruning operation (time for branch identification and the time for maneuvering the end-effector).
To apply robotic pruning, firstly, the tree branch and cutting location need to be accurately identified.Most studies on automated pruning focus on the tree branch identification and reconstruction using machine vison system, as discussed above.Some studies have been reported on developing algorithms to improve the accuracy of the branch reconstruction [115][116][117].Most of these studies focus on the tree skeleton from the 3D images, which typically could get the location and the length of the tree branches.It is hard to get other information, such as the diameter and angle of branches.One recent study from Tabb and Medeiros showed the ability to detect and automatically measure the branching structure, branch diameters, branch lengths, and branch angles.That information is required for tasks such as robotic pruning of trees as well as structural phenotyping.At this stage, it takes about 8 min to finish one tree reconstruction, which is too long for practical pruning process [69].The time for finishing a pruning is determined by both the speed of sensing system and the mechanical/robotic executive system.Both speeds could be potentially improved in the future with the advancement of related technologies.It would be more practical to use stop-and-go since the pruning task for each tree varies, and the time for finish each individual tree also varies.A platform is needed for the automated pruning system, and the platform used in the mechanical harvesting of apples could be an option [48].With the sensing system mounted on the front of the platform, and the overall pruning system could still be considered as real-time system if the pruning operation could be conducted right after the platform reaches the tree.
Not only branch identification task, but also the accessibility of the robotic manipulator and end-effector is challenging due to complexity and variability of agricultural environment, as well as the required speed of operation.Previously developed pruning robots typically use serial robotic arm with a fix cutter, while this level of specificity in the spatial placement of the end effector results in a complex set of maneuvers and slows the pruning process.Meanwhile, the serial robot arm with an end-effector requires large space for the cutter to engage with the branches.Although it is not for pruning directly, effort has been made to simplify the maneuvers and improve the efficiency of robotic operations in harvesting.Two robotic fruit picking robots have been developed and tested, one is from FFRbotics (Gesher HaEts 12, Benei Dror, Israel) and the other one is from Abundant Robotics (Abundant Robotics, Hayward, CA, USA).These robotic arms are parallel type, which limits the spatial requirement of the picking end-effector.The position of the end-effector could be adjusted at the base of the overall robotic arm, and then the picking end-effector could reach the fruit directly or by extending the rod.Similar robotic arms could be considered for developing the pruning system.There is one thing needs to consider that normally no specific orientation was required for the end-effector to engage fruits.For robotic pruning, the end-effector (cutter) needs not only to reach the right location, but also to be placed perpendicularly to the branch.To be always perpendicular to the branch, as well as using the parallel type robotic arm, the end-effector should be with adjustable orientation [118].With this kind of end-effector, the cutter itself could be rotated with very small spatial need.Moreover, the cutter could be made of saw blade with no specific orientation constraints.
Finally, the economics of the robotic pruning system also needs to be considered.Typically, the cost for a robotic system is high.With the use of off-the-shelf robotic arms (such as Robolink, Igus, Cologne, Germany) and low cost sensing system (such as Kinect v2, Microsoft), the overall cost of the robotic system could be dramatically decreased.Therefore, with the consideration of labor shortage issue as well as putting effort on building low cost robotic pruning system with off-the-shelf components, the benefit of developing a robotic pruning system would be obvious.Meanwhile, multiple robots could be employed to improve the working efficiency.The cost/benefit ratio of a robotic pruning machine will have to be analyzed after the machine is built.

Conclusions
In this study, automated pruning related technologies have been reviewed, from the tree training systems, machine vision sensing, pruning strategies, as well as mechanical and robotic pruning development.Through this comprehensive review and discussion, the following statement could be concluded.

1.
Tree architecture is very critical for adopting automated orchard operations such as pruning and harvesting.Intensive tree orchard with narrow tree canopy or even 2D planar fruiting wall would be suitable for fully autonomous pruning system in the future.2.
To develop robotic pruning, simple and quantified pruning rules are the essential of practical pruning strategies.

3.
Many studies have focused on the tree branch identification and reconstruction, however the accuracy and efficiency still needs to be improved for practical pruning operation.4.
Robotic pruning technologies have been successfully investigated in some uniformed crops, such as grapevines.With the adoption of intensive tree architecture as well as the improvement of cutting end-effector, it is very promising to have a robotic pruning system for tree fruit crops.

Figure 1 .
Figure 1.An example of production cost breakdown in percentage for each category for a Gala apple orchard (40 Acres).The fixed cost, such as land, trellis, irrigation setup, etc., is not included[17].

Figure 2 .
Figure 2. Intensive modern fruit tree systems: (A) horizontal branch fruiting wall V-Trellis system in Washington; and (B) tall spindle tree system in Mid-Atlantic fruit region.

Figure 2 .
Figure 2. Intensive modern fruit tree systems: (A) horizontal branch fruiting wall V-Trellis system in Washington; and (B) tall spindle tree system in Mid-Atlantic fruit region.

Figure 3 .
Figure 3. Pruning the apple trees with the proposed severity pruning levels.

Figure 3 .
Figure 3. Pruning the apple trees with the proposed severity pruning levels.

Figure 4 .
Figure 4. Pruning based on the length of the twigs and the ratio of length and diameter of twig (from [48], used with permission).

Figure 4 .
Figure 4. Pruning based on the length of the twigs and the ratio of length and diameter of twig (from [48], used with permission).

Figure 5 .
Figure 5. (A) Branch detection for sweet cherry trees in Y-Trellis (reproduced from [60]); and (B) branch detection for apple trees in V-trellis fruiting wall (from [61], used with permission).

Figure 5 .
Figure 5. (A) Branch detection for sweet cherry trees in Y-Trellis (reproduced from [60]); and (B) branch detection for apple trees in V-trellis fruiting wall (from [61], used with permission).

Figure 6 .
Figure 6.An example of tree reconstruction with 3D machine vision: (A) RGB image of the test tree; and (B) reconstructed tree [69].

Figure 7 .
Figure 7. Mechanical pruning systems for apple trees with saw-tooth cutter driven by a tractor.

Figure 6 .
Figure 6.An example of tree reconstruction with 3D machine vision: (A) RGB image of the test tree; and (B) reconstructed tree [69].

Figure 6 .
Figure 6.An example of tree reconstruction with 3D machine vision: (A) RGB image of the test tree; and (B) reconstructed tree [69].

Figure 7 .
Figure 7. Mechanical pruning systems for apple trees with saw-tooth cutter driven by a tractor.

Figure 7 .
Figure 7. Mechanical pruning systems for apple trees with saw-tooth cutter driven by a tractor.

Table 1 .
Typical tree training systems and specifications for apple trees.

Table 2 .
Pruning strategies and potential difficulties for automated pruning.

Table 3 .
Sensors and techniques used for tree/branch detection and reconstruction in different applications.