Tree Branch Skeleton Extraction from Drone-Based Photogrammetric Point Cloud

Abstract

Calculating the complex 3D traits of trees such as branch structure using drones/unmanned aerial vehicles (UAVs) with onboard RGB cameras is challenging because extracting branch skeletons from such image-generated sparse point clouds remains difficult. This paper proposes a skeleton extraction algorithm for the sparse point cloud generated by UAV RGB images with photogrammetry. We conducted a comparison experiment by flying a UAV from two altitudes (50 m and 20 m) above a university orchard with several fruit tree species and developed three metrics, namely the F1-score of bifurcation point (FBP), the F1-score of end point (FEP), and the Hausdorff distance (HD) to evaluate the performance of the proposed algorithm. The results show that the average values of FBP, FEP, and HD for the point cloud of fruit tree branches collected at 50 m altitude were 64.15%, 69.94%, and 0.0699, respectively, and those at 20 m were 83.24%, 84.66%, and 0.0474, respectively. This paper provides a branch skeleton extraction method for low-cost 3D digital management of orchards, which can effectively extract the main skeleton from the sparse fruit tree branch point cloud, can assist in analyzing the growth state of different types of fruit trees, and has certain practical application value in the management of orchards.

1. Introduction

Point cloud data are commonly used to describe the 3D structure of objects and have important applications in 3D reconstruction, engineering measurement, and morphological analysis, among other applications. With the widespread use of agricultural automation technology, the demands for point cloud analysis for high-throughput orchard phenotyping and growth modeling, such as plant shape, canopy structure, organ morphology, and stress response, are gradually increasing. Skeletonization is one of the essential steps in point cloud analysis in which a three-dimensional skeleton of a tree point cloud is obtained. This step makes it easier to calculate tree height, branch length, and angle, which comprise useful information for precise fruit tree growth management.

In the existing research on tree skeleton extraction, algorithms usually consist of two stages: firstly, skeleton points from the tree point cloud are extracted, and then skeleton topology connections are formed from the skeleton points. The skeleton points extraction is the fundamental step of the whole process. Current research approaches to skeleton point extraction can be broadly divided into three categories:

Graph structure-based [1,2,3,4]: a graph structure is a discrete structure consisting of vertices and edges connecting the vertices. The skeleton of a tree-like point cloud can be regarded as a combination of vertices and edges in a graph structure. The point cloud is converted into a graph structure, the shortest distance from the root node to each point in the point cloud is calculated to determine the skeleton points, and adjacent skeleton points are connected to form the branch skeleton. This method designs a skeleton point extraction algorithm for point clouds with a tree-like structure, which conforms to the structure of tree branches and has a good skeleton extraction effect. However, this method needs to calculate the shortest path on the finely constructed graph structure, which causes high computational costs.

Generic point cloud skeleton extraction algorithms-based [5,6,7,8,9,10,11,12,13,14]: generic point cloud skeleton extraction algorithms mainly include generalized rotationally symmetric axis (ROSA), L1-median and Laplace-based contraction (LBC). Many researchers have used this class of algorithms for tree branch skeleton extraction. Since these algorithms have no specific requirements on the structure of point cloud objects, they are adapted to a wider range of point cloud skeleton extraction tasks and can be easily used for skeleton extraction of point clouds of many types of tree branches. However, such methods were not designed for tree point clouds, such algorithms may extract structures such as annular or detached skeletons from point clouds, but these structures do not exist in natural tree branches, which may not guarantee the accuracy of tree skeleton extraction.

Other point cloud skeleton extraction algorithms [15,16]: other tree skeleton extraction methods are based on nearest neighbor shrinkage or clustering to extract the skeleton, and the skeleton points are obtained by the result of nearest neighbor shrinkage or clustering and connected to form the dendritic skeleton, and this kind of methods get better skeleton extraction results on the point cloud model with regular shape. However, these methods require high quality of point clouds, and the algorithms are sensitive to parameters, which need to be finely set to obtain better skeleton extraction results.

In general, point cloud acquisition for branch skeleton extraction uses laser scanning, multi-view reconstruction, and virtual model construction. Tree point clouds acquired using terrestrial laser scanning (TLS) or scanners have high accuracy [1,2,3,4,7,8,9,10,11,12,13,14], but this approach is expensive for large scale data acquisition for scenarios such as orchards and is not suitable for actual orchard production. Although the use of unmanned aerial vehicle (UAV) laser scanning (ULS) to acquire point cloud has the advantages of large acquisition range and high speed, the acquisition process is affected by airframe shaking, which makes it difficult to align into higher quality point cloud, and if the same quality of point cloud as TLS is to be acquired, the cost of the acquisition equipment will increase significantly. Although the point cloud acquisition using photogrammetry [15,16] reduces the cost of laser scanning, the spatial information of tree objects needs to be strictly calibrated when acquiring tree point cloud, and the process requires a relatively stable data acquisition environment, which limits its application in complex environmental scenarios such as orchards. The construction of point cloud from virtual 3D models [14,17,18] is not affected by the actual acquisition environment, but the virtual point cloud often does not represent the real 3D spatial structure of a fruit tree. In recent years, with the development of UAV aerial photography technology, the use of UAV equipped with RGB cameras to generate point clouds has been widely used, which combines the advantages of ULS while significantly reducing the equipment cost [19,20,21]. However, due to the limitation of the resolution of RGB cameras, the point clouds collected by this method are sparse and irregularly distributed in space, which makes fruit tree skeleton extraction challenging. Therefore, an extraction algorithm for sparse RGB point cloud is needed to complete the fruit tree skeleton extraction task.

Meanwhile, reasonable evaluation metrics are important for verifying the performance of the tree point cloud skeleton extraction algorithms. In fruit tree growth management, parameters such as tree height, branch length, and branch angle need to be accurately calculated, and the fruit tree branch skeleton is an important basis for calculating these parameters. Therefore, setting reasonable metrics to evaluate the performance of the skeleton extraction algorithm is of great significance for the extension of the tree skeleton extraction algorithm for use in scenarios such as orchards. In the evaluation work on tree point cloud skeleton extraction, Bucksch et al. [4] graded the branches of tree branches by length, counted the number of extracted tree branch skeletons and real branch skeletons under different length gradations, and evaluated the effectiveness of the skeleton extraction algorithm by calculating the correlation coefficient R² between the two. This approach evaluates the performance of skeleton extraction only in terms of quantity, ignoring the spatial information of branches. Li et al. [14] manually measured the real parameters of tree branches and calculated the F1 value (F1-Score), correlation coefficient (R²), and root mean square error (RMSE) between the extracted tree branch skeleton parameters and the real measured parameters to evaluate the branch skeleton structure parameters for accuracy. Fu et al. [18] proposed a quantitative evaluation of the skeleton extraction effect, where the point cloud was generated from a virtual branch skeleton and the evaluation of the skeleton extraction effect relied on the pre-existing virtual tree skeleton. However, the tree branch skeleton is a combination of a series of spatial curves, and drawing the curves in 3D space by hand is often inaccurate and subjective, which affects the reliability of the evaluation results. Therefore, this evaluation method is not applicable to the skeleton extraction evaluation task of a real tree point cloud. We observed that there is no unified standard for the evaluation of skeleton extraction algorithms at this stage, and there is a need for evaluation metrics to verify the performance of actual tree point cloud skeleton extraction algorithms.

In summary, in this paper, we propose a branch skeleton extraction algorithm for UAV and photogrammetry-generated sparse RGB fruit tree point cloud and suggest evaluation metrics for tree branch point cloud to verify the performance of the algorithm proposed in this paper. The overall aims of the research were as follows.

(1)

To address the problem of low accuracy of existing algorithms in skeleton extraction for sparse point cloud data. In this paper, a spatial density-based regional point cloud aggregation algorithm is designed to aggregate sparse tree point cloud before skeleton point extraction, and the aggregated point cloud can describe the 3D skeleton morphology of branches, which can effectively improve the accuracy of subsequent skeleton point extraction.

(2)

To address the problem that the generic point cloud skeleton extraction algorithm is prone to broken branches and self-loops when the skeleton topology is connected, which leads to unrealistic tree point cloud skeleton extraction results. In this paper, we propose a skeleton topology connection method with spherical shrinkage from the outside to the root node, which can better adapt to the bifurcated tree structure, effectively avoiding the appearance of non-tree branch structure, and can effectively extract the initial skeleton of tree branches.

(3)

To objectively evaluate the skeleton extraction algorithm and thus verify the performance of the skeleton extraction algorithm on real tree branch point cloud. In this paper, based on the consideration of the characteristics of tree branch structure morphology, the metrics for evaluating the accuracy of skeleton morphology: FBP (F1-score of bifurcation point), FEP (F1-score of end point) and the metric for evaluating the accuracy of skeleton topology: HD (Hausdorff distance) were designed, which can reasonably evaluate the skeleton extraction performance of the algorithm.

(4)

To release an easy-to-use software application that helps the community to test the proposed algorithm and use it for other related applications.

2. Materials and Methods

2.1. Data Acquisition

The data set used in this experiment were acquired by an onboard RGB camera (1 inch CMOS, lens FOV 84 degrees, focus length 8.8 mm, Max image size 5472 × 3648 pixels) mounted on a low-cost industrial level UAV (DJI Phantom 4 RTK, DJI, Shenzhen, China). The flight altitude was set to 50 m and 20 m, and both the front and side overlap in between photos were set to 80%. Synthesis of RGB point cloud was performed using DJI Terra. All the flights were conducted at an orchard of the Institute for Sustainable Agro-ecosystem Services (ISAS), University of Tokyo Orchard (Tokyo, Japan) (35°44′16.0″ N, 139°32′20.9″ E) on the following dates: 13 January 2022, 14 February 2022, and 4 March 2022 (as shown in Figure 1a). There are multiple types of fruit trees growing in the orchard. In this paper, peach trees, persimmon trees, chestnut trees, and plum trees were selected as experimental objects, and five fruit trees were randomly selected from each type of fruit tree to tag and conduct comparative experiments (as shown in the white dashed box in Figure 1b). More details of the orchard are provided in

Table 1. Orchard point cloud containing tree species and number of individuals.

Tree Type	Number of Individuals
Apple	17
Cherry	2
Chestnut	86
Citrus	25
Kiwifruit	15
Loquat	5
Peach	51
Persimmon	95
Plum	46

.

2.2. General Architecture of the Algorithm

This section describes the general architecture of the algorithm in this paper, and Figure 2 shows the structure of the algorithm. The algorithm consists of the following components.

• Point cloud pre-processing module: this module first denoises the original branch point cloud data, and then aggregates the sparse branch point clouds using the regional point cloud aggregation method proposed in this study, which can form a dense point cloud that roughly describes the morphology of the branch skeleton and prepares for the subsequent skeleton point extraction.
• Skeleton point extraction module: this module is based on the octree algorithm to spatially divide the clustered point cloud and extract the branch skeleton points in the divided subspace.
• Skeleton construction module: this module uses the spherical shrinkage based skeleton topology connection method proposed in this paper to form the thick skeleton of the branch.
• Branch skeleton morphology optimization module: this module fine-tunes the positioning of key points in the coarse skeleton first, and then smooths the branch coarse skeleton to output the final fine skeleton.

2.3. Point Cloud Pre-Processing Module

To better extract the skeleton points from the original fruit tree branch point cloud, this section first denoises the original branch point cloud and then aggregates the denoised branch point cloud using a regional point cloud aggregation algorithm.

(1) Point cloud denoising processing

In this study, we used the density-based spatial clustering of applications with noise (DBSCAN) algorithm [22] to remove the anomalous points in the original point cloud. The clustering approach is used to remove not only the anomalous outliers generated during the synthesis of point cloud data but also the anomalous point cloud patches (classes of reduced samples within clusters) generated during the reconstruction of point cloud data.

(2) Branch point cloud density enhancement

The branch skeleton is a series of curve combinations describing the centers of fruit tree branches, and the skeleton points are the spatial points constituting these curves [23]. These spatial points are the centers of branch point cloud in a certain segment. Determining the centers of fruit tree branch point cloud in a certain segment of space is the focus of skeleton point extraction. Since the branch point clouds generated by the RGB camera are sparse and not uniform, extracting the skeleton points directly from the denoised point cloud will produce the problem of inaccurate positioning.

If the tree point cloud branches can be aggregated to the centerline region of the branch skeleton (as shown in Figure 3a) in some way to form a higher density point cloud that can describe the 3D curve shape of the skeleton, and then the skeleton points are extracted from it, this can effectively improve the accuracy of skeleton point extraction. Therefore, an algorithm for regional point cloud aggregation is proposed in this paper.

(1) Let the denoised tree branch point cloud be $C_s=\{x_1,x_2\dots x_i\dots x_n | x\in R^3\}$, and set the regional aggregation radius $r$, as shown in Equation (1):

$$r=\frac{{\sum }_i^{n}dist\left(x_i, k_i\right)}{n}, i\in \left[1, n\right]$$

(1)

where $k_i$ is the $k$ nearest neighbor point of $x_i$ ($k$ is determined according to the empirical value of the preparatory experiment and adjusted according to different tree species).

(2) Traverse each point $x_i$ in the point cloud, set a search region of radius $r$ around $x_i$, set the set of points contained in this search region as $O_i=\{x_1,x_2\dots x_j\dots x_m | x\in R^3\}$, take the shape center $c_i$ of the point set $O_i$, as shown in Equation (2):

$$c_i=\frac{{\sum }_j^m{x}_j}{m}, j\in \left[1, m\right]$$

(2)

As the target location for $x_i$ to move, the new point cloud set $C_a=\{c_1,c_2\dots c_j\dots c_n | c\in R^3\}$ obtained after the point cloud traversal is completed, and Figure 3b shows the process of point cloud aggregation in the region.

(3) Keeping the search radius $r$ constant, $n$ iterations of step (2) are carried out to finally obtain the aggregated branch point cloud $C_t$.

Figure 4 shows the process of branch point cloud aggregation after n iterations $\left(n=8\right)$. With increasing n, the distribution of the fruit tree branch point cloud gradually aggregates toward the center line of the skeleton, and since the number of points of the branch point cloud does not change in the process, the density of the branch point cloud gradually increases in the process of aggregation. After n iterations of iterative aggregation, the spatial distribution of branch point cloud is dense and concentrated near the center line of the branch skeleton, which can already show the three-dimensional morphology of the fruit tree branch skeleton. Extracting the skeleton points subsequently on this basis can effectively improve the accuracy of skeleton point positioning.

2.4. Skeleton Point Extraction Module

Skeleton point extraction involves finding spatial points that can describe the branch skeleton from the branch trunk point cloud. Since the spatial distribution of the aggregated fruit tree branch trunk point cloud can already describe the three-dimensional form of the branch skeleton, the skeleton points can be considered as the concentrated expression of the fruit tree branch trunk point cloud after aggregation in a subdivision space. Therefore, this paper proposes a skeleton point extraction method based on subdivision space point cloud shape center calculation, which first divides the aggregated fruit tree branch trunk point cloud spatially, and then calculates the shape center of the point cloud in the subdivision space as the skeleton point (as the previous preparation for the construction of 2.5 section branch trunk skeleton). The specific implementation steps are as follows.

(1) The octree based spatial partitioning

The octree algorithm [24], as one of the commonly used data structures for 3D spatial partitioning, can accelerate spatial queries and efficiently manage 3D space. In this paper, we first use the spatial octree structure to subspace the aggregated fruit tree branch trunk point cloud, as shown in Figure 5.

$S_i$: smallest subspace divided using octree, $O_i$: geometric center of subspace, $c_{ij}$: branch point cloud location in subspace, $k_i$: skeleton points in subspace.

(2) Calculation of skeleton points in subspace

The skeleton point of the branch can be regarded as the center of the point cloud geometry of the branch in a certain region. Considering that the center point ${\mathrm{O}}_i$ of the subspace may have a large offset compared with the distribution of the point cloud in the space, it could not represent the distribution of the point cloud in the subspace. Therefore, to better reflect the representativeness of the skeleton point to the point cloud in the subspace, this paper adopts the shape center of the point cloud in the subspace as the skeleton point ${\mathrm{k}}_i$, which is defined as shown in Equation (3):

$$k_i=\frac{{\sum }_j^n{c}_{ij}}{n},j\in \left[1,n\right]$$

(3)

where $k_i$ denotes the skeleton points calculated in each subspace, $n$ is the number of point clouds contained in each subspace, and $c_{ij}$ is the location of the branch point cloud in each subspace.

Calculating the point cloud shape centers as skeleton points in the subspace of the octree subdivision can ensure that the spatial distribution of its skeleton points can better reflect the fruit tree branch trunk morphology, which is ready for the skeleton construction in Section 2.5. Figure 6a shows the effect of spatial partitioning of the aggregated whole branch trunk point cloud using the octree algorithm; Figure 6b shows the extracted fruit tree branch trunk skeleton points, and the extracted skeleton point distribution can still better reflect the fruit tree skeleton morphology.

2.5. Skeleton Building Module

After extracting the skeleton points of fruit tree branches, the skeleton points need to be topologically connected to form the tree branch skeleton. In previous work, the skeleton of the tree point cloud was usually obtained based on the nearest neighbor or bottom-up topology connection [25], and the number of sub-branches at the bifurcation was not well determined in the case of uneven distribution of skeleton points due to the need to set a reasonable number of nearest neighbor skeleton points, which affects the correctness of the extracted skeleton topology. Since the point cloud data in this paper are obtained from the reconstruction of RGB images collected by an overhead UAV, the point cloud data are sparse and contain less details of branches, which mainly reflect the main branch structure of the fruit trees (e.g., main branches, submain branches, etc.). Therefore, to quickly construct the skeleton topology of the main branches of fruit trees, we designed a spherical shrinkage skeleton topology connection method from the outside to the root node, which can effectively adapt to the bifurcated structure of fruit tree branches, and can extract the key points (end points, bifurcation points) and skeleton segments (end point-bifurcation point, bifurcation point-bifurcation point) of the skeleton while obtaining the main skeleton of fruit tree branches.

Figure 7 shows the spherical contraction topology connection process of the local branch skeleton, and the specific process of the algorithm is as follows.

(1)

Firstly, starting from the farthest skeleton point $p_1$, make a ball with the radius of this point and the root node, connect this point with the nearest skeleton point $p_3$ which is located inside the sphere shell. Following this, select the next far point $p_2$ as the starting point, make a ball with the radius of this point and the root node, select the nearest skeleton point $p_5$ which is located inside the sphere shell to connect, and iterate the above process to traverse all skeleton points.

(2)

After all skeleton points are traversed, the $div$ of each skeleton point is calculated ($div$: the number of connections between a skeleton point and other skeleton points around it. In this paper, we define $div=1$ for end points, $div=2$ for ordinary branch skeleton points, and $div>=3$ for bifurcation points) to extract the end points and bifurcation points in the skeleton points, and calculate each skeleton segment of the branch according to the connection status between the skeleton points.

(3)

Due to the possible redundancy of the extracted skeleton points, there may be shorter burr branches (skeleton segments containing fewer skeleton points) in the topologically connected skeleton. Set the threshold value $l$. If the skeleton segment contains skeleton points greater than $l$, the skeleton segment is retained, otherwise the skeleton segment is removed.

After the skeleton topology connection, the algorithm obtains the endpoints, bifurcation points, and the set of skeleton segments of the fruit tree branches along with the whole skeleton. Figure 8 shows the coarse skeleton of a tree formed after the spherical shrinkage topology connection, where red dots indicate the fruit tree branch skeleton end points, green dots indicate the fruit tree branch skeleton bifurcation points, and different color connection segments in the figure represent different segments of the fruit tree branch skeleton.

Red dots indicate fruit tree branch skeleton endpoints, green dots indicate fruit tree branch skeleton bifurcation points, and the connected segments of different colors in the figure represent different segments of the fruit tree branch skeleton.

2.6. Skeleton Morphology Optimization Module

In 3D space, the real tree branch skeleton can be considered as a combination of multiple smooth spatial curves, and the two endpoints of the curves are always at the endpoints and bifurcation points of the branch skeleton. Therefore, the positions of the skeleton endpoints and bifurcation points need to be kept from moving during the smoothing process of the skeleton. However, the algorithms commonly used for skeleton smoothing (e.g., polynomial smoothing, B spline fitting smoothing, etc.) are not adapted to the prerequisite of fixed skeleton curve endpoints and bifurcation points. Therefore, to solve this problem, the spatial Bezier curve [26] is chosen in this paper to perform spatial curve smoothing on the extracted skeleton. To ensure the correctness of the optimized skeleton morphology, the key points in the thick skeleton of the branches were first fine-tuned in this paper before the pre-processing of the skeleton smoothing using Bezier curves (see Appendix A: branch skeleton key point adjustment strategy).

In the extracted fruit tree branch skeleton, let a skeleton segment be $BH=\left[P_0, P_1, P_2, \dots ,P_n\right]$, ($n$ is the number of skeleton points contained in the skeleton segment), where $P_0$ is the end point (or bifurcation point) of the skeleton segment, $P_n$ is the bifurcation point of the skeleton segment, and $P_0$ and $P_n$ are used as the starting point and end points, and $P_1, P_2, \dots ,P_{n-1}$ as the control points of the Bezier curve, and the smoothing of this skeleton segment using an $m\left(m=n-1\right)$ order spatial Bezier curve, the smoothed curve $B\left(t\right)$ of this skeleton segment can be obtained, and the definition of $B\left(t\right)$ is shown in Equation (4):

$$B\left(t\right)={\displaystyle \sum }_{i=0}^m{C}_m^i{\left(1-t\right)}^{m-i}{t}^i{P}_i ,t\in \left[0, 1\right]$$

(4)

In the equation:

$$C_m^i=\frac{m!}{i!\ast \left(m-i\right)!}$$

As shown in Figure 9b, after the smoothing process of the spatial Bezier curve, the segments of the skeleton are smoother and more natural, which can better reflect the three-dimensional spatial shape of the real fruit tree branches.

3. Results

In this section, the proposed tree point cloud skeleton extraction algorithm is evaluated. First, evaluation metrics are designed to verify the effectiveness of the tree point cloud skeleton extraction algorithm, and based on the evaluation metrics, the proposed algorithm in this paper is compared with the L₁-medial and the Laplacian based contraction (LBC) algorithm [6,11] for experiments. Section 3.1 introduces the metrics proposed in this paper to evaluate the effectiveness of skeleton extraction, F1-score of bifurcation Point (FBP), F1-score of end Point (FEP), and Hausdorff distance (HD). In Section 3.2, the environment of the comparison experiments and the specific comparison experimental data and results are presented in this paper.

3.1. Evaluation Metrics

We propose FBP and FEP to evaluate the accuracy of skeleton topological connection and HD to evaluate the accuracy of skeleton morphology. Note that the generated branch point cloud and its extracted skeleton of different tree species are normalized in the spatial range prior to evaluation by Equation (5):

$$P_{\mathrm{normalization}}=\left(\frac{x-x_{min}}{x_{max}-x_{min}}, \frac{y-y_{min}}{y_{max}-y_{min}}, \frac{z-z_{min}}{z_{max}-z_{min}}\right)$$

(5)

where $\left[x_{min},x_{max}, y_{min},y_{max},z_{min},z_{max}\right]$ indicates the spatial range of the real fruit tree point cloud.

The original fruit tree branch point cloud and the extracted branch skeleton are transformed into the normalized 3D space, and the evaluation of skeleton extraction in this paper is carried out in the normalized 3D space.

3.1.1. Skeleton Topology Accuracy Metrics FBP and FEP

Branching angles are usually obtained by calculating the angles between branch bifurcation points and endpoints, while the accuracy of bifurcation point and endpoint positioning affects the topological accuracy of the fruit tree branch skeleton. Therefore, a reasonable evaluation of the accuracy of bifurcation point and endpoint positioning is an important part of the skeleton evaluation work. Since it is simpler to obtain the bifurcation points and endpoints of branches from real branch point cloud and is not easily influenced by subjective judgment, the spatial coordinates of these points in real branch point cloud obtained by manual labeling are used as real values to evaluate the positioning effect of the points of the skeleton extracted by the algorithm.

In this study, FBP is used to evaluate the extraction effect of bifurcation points of the skeleton, and the definition of FBP is shown in Equation (6):

$$FBP=\frac{2\ast PBP\ast RBP}{PBP+RBP}$$

(6)

In the equation:

$$PBP\left(Precision of Bifurcation Point\right)=\frac{BP}{BN}$$

$$RBP\left(Recall of Bifurcation Point\right)=\frac{BP}{BT}$$

where $BP$ is the number of accurately located skeleton bifurcation points extracted by the algorithm (set the Euclidean distance $d$ between the skeleton bifurcation points extracted by the algorithm and their nearest true skeleton bifurcation points, and if $d$ is less than the set threshold $t$, the skeleton bifurcation points are considered to be accurately located), $BN$ is the total number of skeleton bifurcation points extracted by the algorithm, and $BT$ is the total number of true branch skeleton bifurcation points.

Similarly, FEP is set in this paper to evaluate the extraction effect of skeleton end points. FEP is defined as shown in Equation (7):

$$FEP=\frac{2\ast PEP\ast REP}{PEP+REP}$$

(7)

In the equation:

$$PEP\left(Precision of End Point\right)=\frac{EP}{EN}$$

$$REP\left(Recall of End Point\right)=\frac{EP}{ET}$$

where $EP$ is the number of accurately located skeleton endpoints extracted by the algorithm (set the Euclidean distance $d$ between the skeleton endpoints extracted by the algorithm and their nearest real skeleton endpoints, and if $d$ is less than the set threshold $t$, the skeleton endpoints are considered to be accurately located), $EN$ is the total number of skeleton endpoints extracted by the algorithm, and $ET$ is the total number of real branch skeleton endpoints.

3.1.2. Skeleton Morphological Accuracy Metric HD

To evaluate the accuracy of the skeleton morphology extracted by the algorithm, it is necessary to evaluate the degree of fit between the extracted skeleton and the real fruit tree. However, it is difficult and subjective to annotate the 3D skeleton from the real fruit tree branch point cloud, which will affect the accuracy of the skeleton morphology evaluation results. Therefore, in this paper, we generated a point cloud around the extracted skeleton and measured the morphological accuracy of the extracted skeleton by comparing the similarity between the generated fruit tree branch point cloud and the real fruit tree branch point cloud. The HD is a measure to describe the degree of similarity between two point sets [27], and the smaller the HD, the greater is the degree of similarity between the two point sets; therefore, it can be used to measure the degree of similarity between two point clouds.

To evaluate the morphological accuracy of the fruit tree branch skeleton, we set the real branch point cloud after denoising as $C_s$, generated the point cloud $C_g$ around the fruit tree branch skeleton by random sampling, and calculated the average of HD between $C_s$ and $C_g$ for $n$ times ($n=10$ in this paper) as the final HD between $C_s$ and $C_g$, which can reduce the error caused by a single calculation. The HD between $C_s$ and $C_g$: H($C_s$,$C_g$) is calculated as shown in Equation (8):

$$H\left(C_s,C_g\right)=\frac{{\sum }_i^n\max \left[h\left(C_s,C_g\right),h\left(C_g,C_s\right)\right]}{n}$$

(8)

In the equation:

$$h\left(C_s,C_g\right)=\underset{P_i\in C_s}{\max }\underset{P_j\in C_g}{\min }\Vert P_i-P_j\Vert$$

$$h\left(C_g,C_s\right)=\underset{P_j\in C_g}{\max }\underset{P_i\in C_s}{\min }\Vert P_j-P_i\Vert$$

Figure 10 shows the generated point cloud $C_g$ around the extracted skeleton and the overlap effect with the real point cloud $C_s$.

3.2. Comparison Experiments

In this section, based on the evaluation metrics proposed in Section 3.1, the algorithm proposed in this paper is compared with the L₁-medial and the LBC. Section 3.2.1 introduces the comparison experiment environment; Section 3.2.2 introduces the comparison experiment data and results, and the analysis and discussion of the results.

3.2.1. Experimental Environment

The algorithms used for the experiments in this paper are performed on the same computer, which is configured with an AMD Ryzen 7 4800U processor with 1.80 GHz and 16 GB running memory, and the experimental environment is a Windows 10 operating system. This algorithm was run using python 3.7, and the GUI was developed using C++ and the Qt framework.

3.2.2. Comparison Experiments

Currently, the L₁-medial and the LBC algorithm, as the most commonly used algorithms in point cloud skeleton extraction work, have been largely applied in fruit tree skeleton extraction tasks [7,8,9,10,11,12,13,14]. Further, the existing tree point cloud skeleton extraction work is also usually carried out based on these two algorithms. In this section, on the point cloud of fruit tree branches generated by UAV images at 50 m and 20 m, the point cloud skeleton extraction algorithm of fruit tree branches proposed in this paper is compared with L₁-medial and LBC, and the specific comparison experiments and experimental results are shown in Appendix B Exhibit 2-1.

(1) 50 m altitude comparison experiment

In this study, we first conducted comparative experiments on the point cloud of fruit tree branches collected by UAV at a 50 m altitude to qualitatively evaluate the skeleton extraction effect of fruit tree branches, and quantitatively evaluate the three algorithms in terms of skeleton connection accuracy and skeleton morphology accuracy by combining three metrics, FEP, FBP, and HD.

Figure 11 shows the visualization of the fruit tree branch skeleton extracted by the three algorithms on the point cloud of fruit trees collected at a 50 m altitude (one tree of each fruit tree is selected for display).

From the results of the skeleton extraction in the figure, although the L1-medial and the LBC extracted the skeleton of the fruit tree branches, the skeleton has different degrees of defects (as shown by the orange dashed circles in Figure 11. Using the L1-medial algorithm, when processing the sparser RGB point cloud, the skeleton of some branches could not be connected well. Due to the irregular distribution of the point cloud generated by the RGB images, the LBC extracts a ring-like structure in the branch skeleton of fruit trees, which does not exist in real trees, and affects the realism of the algorithm skeleton extraction effect. In contrast, the proposed algorithm extracts a more complete branch skeleton and correct topological connections, and the final extracted branch skeleton has a smoother and more natural shape due to the smoothing process of the branch skeleton in the algorithm.

From the average values of the metrics in

Table 2. The average values and variation of metrics for the skeleton extraction results of the three algorithms on the branch point clouds of the four types of fruit trees generated from images collected at a 50 m and 20 m altitude (t = 0.03).

		50 m			20 m			Variation
Tree_Species_Number	Metric	L1-Medial	LBC	Proposed	L1-Medial	LBC	Proposed	L1-Medial	LBC	Proposed
Average_Peach	FBP	70.76%	57.60%	83.60%	60.26%	70.41%	86.30%	−10.50%	13.81%	2.71%
	FEP	73.63%	75.02%	83.64%	64.50%	65.41%	83.35%	−9.13%	−9.61%	−0.29%
	HD	0.0647	0.0574	0.0433	0.0762	0.0698	0.0546	0.0115	0.0124	0.0113
Average_Persimmon	FBP	58.04%	58.13%	74.76%	68.80%	70.82%	84.77%	10.76%	13.69%	10.01%
	FEP	61.60%	67.36%	85.45%	70.71%	61.49%	90.04%	9.11%	−5.87%	4.59%
	HD	0.0878	0.063	0.0582	0.0996	0.0688	0.0527	0.0118	0.0058	−0.0055
Average_Chestnuts	FBP	33.74%	30.35%	45.92%	59.05%	56.01%	79.93%	25.31%	26.66%	34.01%
	FEP	42.48%	33.40%	48.54%	55.51%	44.86%	83.64%	13.03%	11.46%	35.10%
	HD	0.1094	0.1032	0.0909	0.0708	0.06	0.0393	−0.0386	−0.0432	−0.0516
Average_Plum	FBP	40.72%	32.43%	52.31%	65.38%	63.26%	81.98%	24.66%	31.83%	29.67%
	FEP	60.20%	36.41%	62.12%	77.85%	52.79%	81.59%	17.65%	16.38%	19.46%
	HD	0.1184	0.1068	0.0871	0.097	0.0736	0.0432	−0.0215	−0.0332	−0.0439
Average_Total	FBP	50.82%	44.63%	64.15%	63.37%	61.13%	83.24%	12.56%	21.50%	19.10%
	FEP	59.48%	53.05%	69.94%	67.14%	56.14%	84.66%	7.67%	3.09%	14.72%
	HD	0.0951	0.0826	0.0699	0.0859	0.0681	0.0474	−0.0092	−0.0146	−0.0224

, the point cloud skeleton extraction algorithm proposed in this paper has higher FBP and FEP than the other two algorithms, reaching 64.15% and 69.94% for the four fruit trees, respectively. Meanwhile, the HD of the extracted skeleton by the algorithm was 0.0699, which is also smaller than those of the other two compared algorithms. Furthermore, from Tables in the Appendix section, the algorithm proposed in this paper performed the best among 47 metrics out of the total 60 metrics for 20 fruit trees. The experimental results show that the algorithm is more accurate in locating the bifurcation points and end points of the skeleton on the point cloud of fruit tree branches collected at a 50 m altitude, and the point cloud generated from the skeleton extracted by the algorithm has higher similarity with the real fruit tree branch point cloud. This suggests that the skeleton of fruit tree branches extracted by the proposed algorithm has better skeleton topology and morphological accuracy, and better reflects the real morphology of fruit tree branches.

To verify the stability of the skeleton extraction algorithm under different acquisition times, acquisition light conditions, and other environments, we collected point clouds of fruit tree branches at a 50 m altitude on 13 January, 24 February, and 4 March 2022, and conducted comparative experiments, and the specific experimental results are shown in the Appendix section following the paper (Appendix C Exhibit 3-1, Exhibit 3-2, Exhibit 3-3). Appendix C Exhibit 3-4 in the Appendix of the paper show the standard deviations of the different algorithms for each metric of the fruit trees in the three comparative experiments mentioned above. Based on the information presented in Appendix C Exhibit 3-4, we can see that the algorithm of this paper has a smaller standard deviation in the experimental metrics compared with the L₁-medial and the LBC. This indicates that the data fluctuation of the three 50 m altitude comparison experiments of this paper’s algorithm is lower. Therefore, the algorithm proposed in this paper can maintain a good skeleton extraction effect under different fruit tree branch point cloud acquisition environments.

In summary, combining the effect of fruit tree skeleton extraction and experimental evaluation metrics, it can be seen that in the comparison experiments of RGB point cloud generated from images collected at a 50 m altitude, compared with the other two algorithms, the skeleton of fruit tree branches extracted by the algorithm in this paper has a more realistic 3D structure and has a better performance of evaluation metrics on most of the fruit trees. Further, the proposed algorithm can more effectively and stably extract the point cloud of fruit tree branches from the point cloud of fruit tree branches, and ensure realism of the skeleton. However, we also observed that the experimental metrics of all three algorithms on chestnut and plum trees are poor compared to the results on peach and persimmon trees. To verify the effectiveness of our algorithm on the point cloud of chestnut and plum trees with higher quality of point cloud, we also reduced the altitude of image acquisition to 20 m to generate a higher quality RGB point cloud.

(2) 20 m altitude comparison experiment

Figure 12 shows the visualization of the fruit tree branch skeleton extracted by the three algorithms on the point cloud of fruit trees collected at 20 m altitude (one tree of each fruit tree is selected for display).

The black points are the original fruit tree branch point cloud, the blue curve is the skeleton extracted by the algorithm, the green points are the bifurcation points of the fruit tree branch skeleton, and the red points are the end points of the fruit tree branch skeleton

As shown in the figure, compared with the point cloud of fruit tree branches collected at 50 m, the point cloud collected by the UAV at a flight altitude of 20 m is of higher quality, presenting a larger number of fruit tree branches and a more complex branching structure of fruit trees. The effect of skeleton extraction shows that although the skeletons extracted by the L₁-medial and LBC can show the morphology of fruit tree branches, there are still different degrees of broken branches and self-loops (as shown by the orange dashed circles in Figure 12); especially in the peach and persimmon trees with more complex branch structures, the two algorithms show more skeleton topology connection errors, which affect the authenticity of the extracted fruit tree branch skeleton. Compared with the two comparative algorithms, the proposed algorithm extracted more complete branch skeletons and accurate topological connections in terms of the point cloud of fruit tree images captured at an altitude of 20 m with more complex structures, although the proposed algorithm also showed local skeleton linkage errors (as shown in the gray dashed circles in Figure 12).

Based on information provided in Table 2, in the comparison experiments on four fruit trees at 20 m altitude, the average value of FBP for the proposed algorithm is 83.24%, which is 19.87% higher than that of L₁-medial and 17.11% higher than that of LBC; the average value of FEP is 84.66%, which is 17.52% higher than that of L₁-medial and 28.52% higher than that of LBC; the average value of HD is 0.0474, which is 0.0385 lower than that of L₁-medial algorithm and 0.0207 lower than that of LBC, which shows that the proposed model achieved the best values in all three metrics. Based on the information presented in Appendix C Exhibit 3-5 in the Appendix text, the algorithm proposed in this paper performs best in 52 metrics among 60 metrics for 20 fruit trees. It can be seen that in the point cloud of fruit tree branches collected at a 20 m altitude, the skeleton of fruit tree branches extracted by the algorithm proposed in this paper can still maintain better topology and morphological accuracy compared with the L₁-medial and LBC. In other words, the skeleton extracted by the proposed algorithm is more realistic in the point cloud of fruit tree branches collected at a 20 m altitude.

(3) Analysis of experimental results at different altitudes

Figure 13 shows the average values of the experiment metrics on all fruit tree branch point clouds generated from images collected at both altitudes, and Table 2 shows the amount of variation in the average values of the metrics for the three algorithms on the four types of fruit trees at both altitudes.

As shown in Figure 13, with a decrease in the fruit tree point cloud collection altitude, the average values of FBP and FEP of the three skeleton extraction algorithms on the four fruit trees showed an increasing trend, and the average values of HD showed a decreasing trend, which showed that the quality of the collected fruit tree branch point cloud was improved with a reduction in the data collection altitude.

From the column of the amount of variation in Table 2, chestnut and plum trees whose metrics performed poorly in the comparison experiment at a 50 m altitude, showed significant improvement in the skeleton extraction results in the comparison experiment with images taken at a 20 m altitude. For chestnut and plum trees with sparse branches, the improvement of fruit tree branch point cloud quality can effectively improve the accuracy of the algorithm in extracting skeleton topology and morphology. The proposed algorithm improved the average values of FBP, FEP, and HD metrics by 34.01%, 35.10%, and 0.0516, respectively, for the five chestnut trees, and improved the average value by 29.67%, 19.46%, and 0.0439, respectively, for the five plum trees. This indicates that the proposed algorithm can improve the skeleton extraction effect to a greater extent with improvement in the quality of fruit tree branch point cloud.

We also noted that the experimental metrics of peach and persimmon trees in the comparison experiment at a 20 m altitude did not improve significantly with the improvement of point cloud quality compared with those of chestnut and plum trees, but deteriorated in some metrics (e.g., the average values of FEP and HD experimental metrics of peach trees). The reason for this is that although the fruit tree branch point clouds collected by lowering the UAV flight altitude to 20 m are of higher quality, the fruit tree branch point cloud of peach and persimmon trees also present a more complex structure, and the fine branches of the fruit tree are more distinct (as shown in Figure 14). These fine branches can easily be ignored in the process of skeleton extraction, which makes the fruit tree branch skeleton extraction more difficult.

From Table 2, although the average value of FEP decreased by 0.29% and the average value of HD increased by 0.0113 for peach trees for the proposed algorithm, the metrics deteriorated less compared to that in the L₁-medial and LBC algorithms. The L₁-medial algorithm reduced the FBP metric of peach tree by 10.50%, the LBC reduced the FEP metric of persimmon tree by 5.87%, and both algorithms increased the HD metric of persimmon tree by 0.0118 and 0.0058, respectively, which shows that the proposed algorithm can still maintain a better performance in the complex fruit tree branch point cloud skeleton extraction task compared with the two other algorithms.

4. Discussion and Conclusions

In this paper, a branch skeleton extraction algorithm is proposed for the sparse RGB fruit tree point cloud collected by UAV, which can effectively extract the skeleton from the sparse fruit tree branch point cloud, approximate the 3D structure of the original fruit tree, and more realistically reflect the branch morphology of the fruit tree. The algorithm proposed is based on point clouds generated from images captured by UAV RGB cameras. Compared with the method of extracting skeleton points using point cloud data collected by devices such as LiDAR [7,8,12,13,14], the algorithm proposed can effectively reduce the cost of point cloud data collection while maintaining the accuracy of skeleton point extraction, such as evaluate the growth of orchard trees by analyzing the tree morphometry using UAVs [28,29].

Combining the comparison experiments at 50 m and 20 m altitudes, it can be found that the algorithm proposed in this paper has better performance in terms of FBP, FEP and HD metrics compared with L₁-medial and LBC. At the same time, the results of comparison experiments at different altitudes show that for fruit trees with sparse branch structure (e.g., chestnut and plum trees), the reduction in UAV flight altitude can improve the quality of the collected fruit tree branch point cloud and improve the skeleton extraction effect of the algorithm. However, for the more complex branch structure (e.g., peach and persimmon trees), with a reduction in flight altitude, the UAV can collect a more complex fruit tree branch point cloud structure, which brings challenges to the skeleton extraction of the algorithm. Therefore, when a more complex fruit tree branch structure is dealt with, whether the skeleton can be effectively extracted from the branch point cloud is an important basis to test the adaptability of the fruit tree point cloud skeleton extraction algorithm. Compared with the L₁-medial and LBC, the algorithm proposed in this paper can still maintain good experimental results on more complex fruit tree branch point cloud, indicating that the algorithm proposed in this paper can also adapt to the extraction task of the skeleton of more complex fruit tree branch point cloud (Figure 15).

As a conclusion, after the experiments and discussions in this paper, the proposed method can effectively extract the skeleton from the point cloud of fruit tree branches measured by UAV photogrammetry. It is undeniable that extracting the skeleton from a complex tree branch point cloud is still a challenging task. Unlike conventional objects with clear 3D structures, tree branches have open surfaces and complex branches, and the collected point cloud often fails to clearly represent the tree branch morphology, causing greater difficulties in the subsequent branch skeleton extraction. Therefore, the algorithm proposed in this paper still has areas for improvement:

First, based on the experimental results, it can be found that the algorithm in this paper shows advantages in fruit tree branch point cloud skeleton extraction. However, there are still some problems, such as poor performance in FEP and FBP metrics of some fruit trees in the comparison experiments. The reason for this is that the algorithm in this paper aggregates the fruit tree branch point cloud, and the skeleton point extraction and subsequent skeleton topology connection are performed on the aggregated point cloud, and the deviation of skeleton key point positioning occurs. In future work, optimizing the positioning of skeleton end points by using the search method with direction and the positioning of skeleton bifurcation points by using the local density judgment can be considered.

Second, at this stage, the parameters need to be set manually during skeleton extraction. For example, the number of points and iterations of regional point cloud aggregation will affect the point cloud aggregation and the recurrence depth of spatial octree, and subspace point threshold will affect the extracted skeleton points, among other such issues. The parameters during skeleton extraction need to be adjusted according to the type and height of fruit trees and other attributes, which may require some experience. It is possible to conduct several experiments on branch point clouds of different tree species to acquire empirical values to apply to the needs of different types of fruit trees, and also to try to project reasonable parameter thresholds based on the projected area and space size of fruit trees in the vertical direction.

In the actual work of fruit tree morphological analysis, the extracted fruit tree skeleton can be used to calculate parameters such as branch length and branch angle (e.g., the angle between main and secondary branches, and other parameters, as shown in Figure 16) to achieve the task of automated fruit tree morphological analysis. On the basis of the extracted fruit tree branch skeleton, phenotypic information of fruit tree canopy can be calculated, which can be used in practical applications for tasks such as orchard canopy cover, biomass density and carbon sequestration estimation [30,31,32]. In addition, the real 3D tree branch skeleton is the basis for modeling scenes of orchards and woodlands, and the algorithm proposed in this paper can extract a realistic fruit tree branch skeleton, and the reconstructed tree model on this basis can better restore the real 3D morphology of fruit trees. Therefore, the virtual 3D model of fruit trees can be formed based on the extracted skeleton by regularized programming and mapping in future work (as shown in Figure 17) to meet the demand for 3D visualization display of orchards and other scenes [33,34,35,36].