Filtering Stems and Branches from Terrestrial Laser Scanning Point Clouds Using Deep 3-D Fully Convolutional Networks

Abstract

Terrestrial laser scanning (TLS) can produce precise and detailed point clouds of forest environment, thus enabling quantitative structure modeling (QSM) for accurate tree morphology and wood volume allocation. Applying QSM to plot-scale wood delineation is highly dependent on wood visibility from forest scans. A common problem is to filter wood point from noisy leafy points in the crowns and understory. This study proposed a deep 3-D fully convolution network (FCN) to filter both stem and branch points from complex plot scans. To train the 3-D FCN, reference stem and branch points were delineated semi-automatically for 14 sampled areas and three common species. Among seven testing areas, agreements between reference and model prediction, measured by intersection over union (IoU) and overall accuracy (OA), were 0.89 (stem IoU), 0.54 (branch IoU), 0.79 (mean IoU), and 0.94 (OA). Wood filtering results were further incorporated to a plot-scale QSM to extract individual tree forms, isolated wood, and understory wood from three plot scans with visual assessment. The wood filtering experiment provides evidence that deep learning is a powerful tool in 3-D point cloud processing and parsing.

1. Introduction

Modern wood management is a balance of economic activities and environmental stewardships. It is a dynamic system integrating chains of wood production, marketing, conservation, energy, biodiversity, carbon cycling, and climate consequences. It is therefore important to inventory storage and change of wood resources in a timely and comprehensive manner. A typical indicator of wood resources is wood biomass. Estimating wood biomass directly from harvesting or destruction is infeasible given a forest’s massive extent. In practice, regional or national wood biomass is estimated from plot-level surveys of inventory attributes (e.g., diameter and height) extrapolated with remote sensing imagery [1,2]. Plot-level surveys can provide accurate inventory attributes, but most are gross dimension attributes (e.g., the diameter-at-breast-height (DBH) and tree height). Regional or national wood biomass is usually modeled in simple allometric equations using these gross dimension attributes. Using such simple allometric equations, relative root mean squared error (RMSE) of biomass is around 30% [3,4] and absolute error has 50% variation with different tree forms [5]. Large errors are found in branch biomass estimation [6]. With the advent of three-dimensional (3-D) point clouds from Terrestrial laser scanning (TLS), detailed stem and plot-level structural attributes become extractable [7]. Kankare et al. [6] proposes an allometric equation with 20 TLS features for Scots pine and Norway spruce and reported a total relative biomass RMSE of 11.90%. Further efforts, particularly quantitative structure modeling (QSM), present lower relative biomass RMSE (<10%) by modeling biomass simply as a product of wood volume, basic density, and biomass expansion factor (BEF), with volume extracted automatically from TLS point clouds [8,9].

Compared to allometric approaches, the success of QSM depends on whether stems and branches can be delineated. Apparently, QSM is challenged by leafy crown and understory where wood points are hardly separable. A compromise is to limit analyses to deciduous trees in leaf-off condition [8,10,11], or simply ignore the leaf issue because leaf points can be partially filtered out in wood reconstruction or branch skeletonizing models [12,13,1415,16]. The side effect of ignoring leaf points in QSM is overestimation of branch volume [12]. In contrast, many studies recognize the importance of independent filtering algorithm prior to QSM. Several wood filtering algorithms have been proposed, which can be categorized into seeking high laser intensity [9,17,18,19], seeking high point density [9,20,21] or comparing dual-wavelength difference [22,23]. Nevertheless, all existing filtering algorithms rely on ‘hard’ thresholds in filter configuration. Their filtering performance are susceptible to varying scanning scenarios, for example, species and sensor variation, point density attenuation, surrounding shadowing, wind sway, or environmental noise. A robust and versatile wood filtering algorithm is demanded in terms of dataset imperfection.

The complexity of scanning scenarios is a constraint to developing successful wood filtering algorithms. Fortunately, artificial intelligence, especially the revolutionary deep learning network, is tailored to handle complex recognition and filtering problems [24]. In general, deep learning networks comprise multiple layers with millions of parameters that explain complex data structures and achieve high intelligence. Initial parameter values need to be trained in a gradual optimization process where gradients between the predicted and the desired are backpropagated and minimized. The configuration of layers, however, has various forms and depends on application type. A close application to our wood filtering case is image classification, which is also termed image semantic segmentation, under the context of computer vision and machine learning. Correspondingly, the most popular deep learning network is the Fully Convolutional Network (FCN). As the name implies, FCN typically has hierarchical convolutional layers that extract abstract image features in hyperspace, then fully-connected layers that define wall-to-wall mappings between the abstract features and classes, and finally transposed convolution layers that interpolate class predictions to fine resolution. The convolutional layers are also termed Convolutional Neural Network (CNN) layers. Such a layer configuration of FCN has had significant success in practice for the capability of automatic feature selection and high-degree nonlinearity. Different FCN realizations and variations are reviewed in Garcia-Garcia et al. [25].

Applying FCN to 3-D wood filtering has several practical difficulties. First, training an FCN structure with 3-D point clouds is time and storage challenging. So far, the convolution step of FCN from most deep learning programs does not support a discrete point format, and 3-D point clouds need to be voxelized first. However, inputting the FCN structure with all voxels in fine resolution easily leads to computation overflow. This tradeoff between computation load and 3-D information abundance calls for additional transformation of 3-D point data or FCN structure. Existing transformation methods involve projecting 3-D points to multi-view or multi-feature 2-D images [26,27,28], subdividing point clouds into blocks [29,30], or enabling sparse computation of CNN layers [31,32]. Second, a gradient vanishing problem arises. Most voxels are empty voxels because point clouds are sparse. Non-empty voxel values have little effect on network computing, and their influence decreases or even disappears when passed on to deeper layers. Similarly, backpropagation of gradients to shallower layers hardly affects non-empty voxels. This phenomenon is a symptom of gradient vanishing [33], which significantly slow the training process or even lead to training failure. The gradient vanishing problem in 3-D has no sound solution yet, albeit compromises can be replacing CNN with a multi-layer perceptron structure that weakens spatial relationships [34,35]. Third, it is difficult to have either ground truth or manual reference of wood points. The training process of FCN requires a considerable number of samples with correct labels. Yet it is laborious to either obtain ground truth by repetitively scanning a plot with and without leaves or create a manual reference by picking and labeling individual points from dense and noisy scans at branch level.

A 3-D FCN structure was customized in this study, aiming to filter stems and branches from TLS point clouds in a robust and flexible manner. TLS scans were collected with varying scanning and geometric conditions from three common species. An automatic region-growing method was developed in aid of manual reference generation. The filtering results from 3-D FCN were finally input to a QSM model to test application possibility for plot-level wood delineation. From this experimental study, we hope to unveil the potential of applying deep learning methods in complicated 3-D processing tasks, particularly in parsing forest plot scans, in pursuit of automatic and intelligent wood resource management.

2. Data and Sampling

In 2015 and 2016 summers, we collected plot scans for three widespread species in Canada: sugar maple (Acer saccharum) from Vivian Forest (Ontario), trembling aspen (Populus tremuloides) from West Castle (Alberta), and lodgepole pine (Pinus contorta) from Cypress Hills (Saskatchewan). Maple, aspen, and pine plots have an approximate stem density of 400, 200, and 700 ha⁻¹, and average tree height of 17, 12, and 20 m, respectively. All sites were scanned with an Optech ILRIS LR (1064 nm) from plot center and three-to-five corners. The mean point spacing was between 10–20 mm and range accuracy was ±2 mm at a distance of 20 m from the scanner. Laser intensity values ranged from 0–255. For the sampling and evaluating purpose, all our scans were not trimmed in size.

From corner scans, we manually clipped typical areas with at least one tree to be our sample areas. Our training sample included six maple areas, six aspen areas, and two pine areas, whereas our testing sample includes three maple areas, two aspen areas, and two pine areas. We intentionally selected sample with varying tree numbers and scanning conditions, with a quantitative summary in

Table 1. Quantitative characteristics of each training and testing sample. DBH min-max means minimum and maximum DBH extracted from sample point clouds at a height of 1.3 m. A DBH with 0 value means unmeasurable. Height means vertical distance between lowest and highest points. Point spacing means median nearest neighbor distance. Quality ranking is described in Table 2.

Dataset (Size)	Sample ID	Tree Count	DBH Min-Max (cm)	Height (m)	Point Spacing (cm)	Quality Ranking
Training sample (14)	maple 1	2	8.3–29.1	17.9	0.9	2
	maple 2	3	8.3–12.2	11.7	0.6	7
	maple 3	4	8.7–37.0	19.0	1.1	3
	maple 4	3	0–41.5	20.8	2.0	4
	maple 5	2	0–15.3	17.6	3.1	4
	maple 6	1	0–0	20.3	2.2	6
	aspen 1	4	0–18.8	9.5	3.2	6
	aspen 2	1	0–0	6.7	4.7	10
	aspen 3	2	0–27.1	10.8	0.9	1
	aspen 4	6	9.3–22.9	11.3	1.7	3
	aspen 5	7	0–0	7.5	7.9	9
	aspen 6	3	10.1–19.4	8.8	2.7	4
	pine 1	4	11.6–29.7	19.9	1.0	1
	pine 2	1	0–0	18.3	1.0	5
Testing sample (7)	maple 7	4	3.9–19.4	13.8	2.1	8
	maple 8	2	12.0–30.6	15.7	0.6	2
	maple 9	2	0–0	21.2	1.8	3
	aspen 7	1	16.4–16.4	11.5	0.6	2
	aspen 8	1	30.2–30.2	13.8	0.5	1
	pine 3	1	39.8–39.8	17.9	2.4	5
	pine 4	4	0.0–0.0	18.4	0.5	3

. We also created a sample quality ranking between ‘1’ and ‘10’, explained in

Table 2. Criteria for sample quality ranking.

Quality Ranking	Rationale
1	explicit branch and stem forms, very low occlusion degree
2	moderate branch and stem details, full stem form, very low occlusion degree
3	full stem form, low occlusion degree, few branch details
4	fragmentary stem form, moderate occlusion degree
5	partial stem form, low occlusion degree
6	partial stem form, moderate occlusion degree
7	partial stem form, high occlusion degree
8	partial stem form, sparse branch or stem points, high occlusion degree
9	indiscernible stem points
10	indiscernible stem points, surrounded with noisy points

, indicating how difficult it is to infer inventory details from human interpretation. Point sparsity and occlusion are important reference to our subjective quality ranking, which can be reflected by the high Pearson’s correlation (r) of 0.72 between quality ranking and point spacing, and by the low p value of 0.0003. It is noteworthy that a quality ranking of ‘1’ does not mean perfect quality. All our samples encounter a certain degree of point occlusion, especially in upper crown. Such scan quality imperfection and inconsistency are notorious characteristics of natural forest scans and are where the challenge is. Great quality variation can help develop a robust or universal filtering model to maximize plot data usage.

All sample areas were labeled with three classes: ‘stem’, ‘branch’, and ‘other’ components. The labeling step was semi-automatic, to be described in Section 3.1. For completeness, stems and branches of understory saplings (height < 4 m) were also labeled in the sample areas. Finally, center scans and corner scans were manually co-registered in preparation for plot-level filtering visualization and tree reconstruction.

3. Methodology

Our experimental design of training, evaluating, and applying a 3-D FCN filtering model in wood reconstruction is presented in Figure 1. The process of generating training and testing reference is described in Section 3.1, specification of our 3-D FCN in Section 3.2, and schemes of plot-level QSM to retrieve tree structure and volume in Section 3.3.

3.1. Labeling Reference Points

Per-point labeling of subtle branches, especially in crown and understory areas, can be difficult. This study relied on an automatic segmentation algorithm to help locate tube-shaped branches and filter out noise [36]. Generally, the algorithm started from selecting a few core points from point clouds, which iteratively incorporated neighboring points and eventually grew to be tube-shaped regions. Major requirements of the region-growing routine were to constrain growth direction and tube width changes. As a result, most points were segmented into tube-shaped regions, while other remaining points especially from leaves and ground had bulky shapes. The tube-shaped regions were extracted based on their three-dimensional features from principal components analysis (PCA) [37]: any regions with linearity >0.4 and sphericity <0.3. The extracted regions were then exported to CloudCompare Open Source Software for further editing. Specifically, the tube-shaped regions were manually trimmed and refined to represent clean wood segments. Wood segments, with thick long shape, near-vertical direction, tight connection to base part, or high laser intensity, were preferably recognized as stems. The remaining wood segments were thus intended to be branches. We labeled stems as class 2, branches as class 3, and other points as class 1.

The above-stated region-growing algorithm was an iterative process, meaning that later region-growing steps were affected by errors in earlier steps. It was therefore critical to ensure correctness of initial region-growing steps. For example, locating initial region-growing points should avoid areas with ambiguous shapes, such as joints between branches. Our experiments suggested that region-growing initialized from branch tip points would yield most correct shapes and cause less conflict with following region-growing steps. We then proposed a simple method of locating branch tips, to improve the initialization step in Xi et al. [36]. First, for operational convenience, all points were converted to voxels at a resolution of 5 × 5 × 5 cm³ (xyz). A metric termed mean curvature was used to describe the spatial local sharpness (Figure 2). Specifically, for each voxel, we calculated its unit vectors directing to all neighboring voxels. The average of those unit vectors was a vector (red arrow in Figure 2), the direction and amplitude of which represented surface normal and sharpness degree, respectively. This amplitude value was indeed a discrete expression of surface mean curvature [38]. Using surface mean curvature, branch tips were coarsely located where a voxel’s mean curvature exceeded 0.75. These voxels representing branch tips were regarded as initial region-growing points. Each branch tip was searched in turn to grow initial regions, until no tips were available. Then any arbitrary point was selected to continue region-growing until no points were selectable. Both region-growing and initialization algorithms were programmed in MATLAB.

3.2. Configuring 3-D FCN

Our 3-D FCN model and evaluation steps were programmed in Tensorflow (GPU edition). We adopted the point cloud subdivision approach [29,30] and limited the model input to be a block of 128 × 128 × 128 voxels, with a voxel size of 5 × 5 × 5 cm³. Larger block size and smaller voxel size were desired but could cause computing overflow problems. We cut raw point clouds evenly into blocks and then shuffled blocks randomly before the training process. Each voxel was assigned with three attributes, derived from summary statistics of all points within that voxel. The first attribute was binary, with 1 meaning point occupancy and 0 otherwise. The second attribute was average of laser intensity values, whereas the third attribute was average of point heights (z value). The final FCN input was a 128 × 128 × 128 × 3 block. In addition, the class label of a voxel was calculated as the dominant label from all points within the voxel. If a voxel was empty, its class was set to be 0. Hence, the total class number was four.

Our FCN structure (Figure 3) was customized based on the FCN-VGG-16 structure described in Long et al. [39]. In addition to expanding 2-D layers to a 3-D version, we changed the fusion strategy between transposed convolution (TC) and low-level pooling, shown by the plus sign in Figure 3. The initial method was to fuse 1st TC results with 4th pooling results, 2nd TC with 3rd pooling, and so on. Our strategy was fusing 1st TC directly with 3rd pooling results, and 2nd TC with 2nd pooling. The omission of fusing 4th pooling results was intended to reduce blurring degree on final prediction. The final prediction output a 128 × 128 × 128 × 4 block, with last dimension denoting the probability of each class. The final class label was predicted as the maximum value in the last dimension of the probability block.

We trained our model from scratch, because existing 2-D pretrained models were hardly adaptable to our scenes and data structure. Our convolution filter size was 3 × 3 × 3, and dropout ratio was 0.5. The weights of convolution and fully-connected layers were initialized using default values in Tensorflow, and the weights of transposed convolution layer were based on trilinear interpolation. To train our 3-D FCN model, errors (or disagreement) between probability block and reference, namely loss function, were measured using a mean softmax cross entropy metric [40]. Nevertheless, directly minimizing this loss function could encounter the gradient vanishing issue. Our solution was to calculate loss function only based on non-empty voxels in the probability block. We used an Adam optimizer [41] with an initial step of 10⁻⁴ to iteratively minimize the loss function. It was observed that loss function from only non-empty voxels could always descend to a minimum level, albeit the convergence process was slow and variable.

Prediction accuracy (or filtering accuracy) was defined as an agreement between predicted and referenced labels for this study. Three accuracy metrics were adopted: Overall Accuracy (OA), Intersection over Union (IoU) per class, and its mean value (mIoU) [25]. The OA divided the total number of matched labels between prediction and reference by the total number of points. It represented an overall matching degree. The IoU was class-specific, which divided the total number of matched labels by the total number of labels for one class. The mIoU was an average of IoUs among all classes. Comparing to OA, mIoU weighed per-class accuracy equally, and therefore less influenced by the accuracy of dominant classes. The metrics of OA, IoU, or mIoU all ranged from 0–1, with higher value indicating greater agreement.

3.3. Reconstructing 3-D Tree Geometries

We designed a follow-up wood reconstruction model (or QSM) at plot-level to showcase a potential application value of our 3-D FCN. The results of FCN filtering were labelled voxels. We extracted stem points from stem voxels, and branch points from branch voxels. The non-wood points were discarded, as well as the height and intensity attributes from wood points. Only 3-D coordinates of wood points were reserved for the reconstruction model. The wood reconstruction modeling (or QSM) was a top-to-bottom and bottom-to-top process: decomposing the plot to individual trees, extracting nodes from branch or stem points, connecting nodes along separate branches or stems, and finally connecting branches to stems. Success of all reconstruction steps relied significantly on clean filtering of stems and branches from the 3-D FCN. All wood points were assumed to be tube-shaped.

At the plot level, we used a simple region-growing algorithm to decompose plots into individual trees. Region growing process started from an arbitrary point, kept incorporating its buffer points within 0.1 m, and terminated when no buffer points were found. Results were isolated components of branches and stems. We applied a simple rule to merge isolated stem components into complete individual stems. Any two stem components were identified as being from the same stem if one stem component was completely above the other, and the upper components fell within five times stem radius of the lower component. Then, with individual stems identified, branch components were assigned to their nearest stem, together to form an individual tree. We limited the nearest distance between branch components and stem to be no greater than 2 m, otherwise the branch components would be identified as isolated branches. We also identified any branches not higher than 4 m from the ground as understory wood. Those branches were not assigned to their nearest stem. Consequently, a plot of wood points was decomposed into individual trees, isolated branches, and understory wood.

At the individual-tree level, stem and branch points were segmented respectively using the same region-growing algorithm as described in Section 3.1, except for filtering tubes based on linearity and sphericity criteria. The resulting grown regions were evenly sliced along their main axis at 1 cm intervals, to form a sequence of nodes representing the tube. The main axis direction was estimated from PCA [36], and each node location was at the center of the slice. At each node location, we applied a circle-fitting algorithm for its neighboring points [42]. The search radius of neighboring points was set to be three times the tube width retrieved from the region-growing process. The fitted circle center became the new node location, the circle radius became node radius, and the circle plane normal becomes the node direction.

At the individual-stem or branch level, nodes were connected based on a weighted similarity score (Equation (1)) utilizing distance $D$, angle $\theta$, and radius $R$. Figure 4 has intuitive illustration of these geometric terms, with $D$ meaning distance between two nodes, $\theta$ included angle between node direction and node connection, and $\mathsf{\Delta }R$ node radius difference. Any large $\theta$, $\mathsf{\Delta }R$ or $D$ can lead to a low similarity score. The weight of each term was customizable, and this study empirically set $w_1=0.4,w_2=0.4,w_3=0.1$ and $w_4=0.1$ to approximate consistent and smooth branch curves. The process of node connection started with an arbitrary node, searched its neighboring nodes within 0.1 m, connected nodes with highest similarity scores, and repeated search and connection, until no node is available. Nodes with $\theta \ge 90°$ were discarded to avoid turn-around of connection lines. We then reversed the direction of the starting node and performed the same connection process, so that both sides of starting nodes could be searched and connected. The entire connection process produced individual stems and branches.

$${\mathrm{Score}}_{2,1}={\left(w_1\left|\tan {\theta }_1\right|+w_2\left|\tan {\theta }_2\right|+w_{3}D+w_4\left|\mathsf{\Delta }R\right|\right)}^{-1},where{\theta }_1,{\theta }_2<90°$$

(1)

At the individual-tree level, each branch, only represented by its starting and ending nodes, was connected to the stem or another branch based on the similarity score shown in Equation (2). Comparing to Equation (1), no penalty was required for radius difference or direction difference between target and connection line. The connection process was iterative. First, branches nearest to the stem were connected to a stem node with highest similarity score. This branch-to-stem connection shaped a general 3-D tree form. Then, remaining branches nearest to the tree form were connected to it based on a similarity score, and so forth, until no branches were available. The resulting tree form was further refined by combining similar branches (Figure 5). Any two branches with similar branch directions (<50° difference) would join at their shared node, if the joint branch were to be longer than both branches.

$${\mathrm{Score}}_{2,1}={\left(w_2\left|\tan {\theta }_2\right|+w_{3}D\right)}^{-1},where{\theta }_2<90°$$

(2)

Returning to the plot level, in addition to individual trees, isolated branches and understory wood were simply reconstructed following the steps of node connection. We did not apply branch-to-branch connection to avoid an excessive number of plausible connection lines. Wood volumes of individual trees, isolated branches and understory were all computed. All the QSM parameter settings were the same for different plots.

4. Results and Discussion

4.1. Reference Creation

Results of branch tip extraction from ‘aspen 3’ are shown as white dots in Figure 6a. Initialized with these branch tips, our region-growing algorithm can segment point clouds mostly into tube shapes. Those segments are assigned random colours in Figure 6b to visualize differences. After segment-wise PCA filtering, manual selection and minor trimming, the resultant wood reference is displayed in Figure 6c. The stem and branch points after splitting are shown in green and red, respectively, whereas other points are in blue (Figure 6d). Two trees are covered in this sample area, with only one tree exposing its main stem. Both trees contain many ghost points, particularly near the crown area, due to the difficulty of interpretation and trimming. These are typical quality issues in preparing reference data which hinder deep learning algorithms from detecting branches correctly. Yet eliminating reference errors simply from human interference is limited and costly. It is anticipated in the future that a benchmarking reference dataset could be developed for an algorithm optimizing purpose where scene complexity and data quality could be finely controlled.

4.2. 3-D FCN Filtering and Evaluation

Figure 7a,b tracks accuracy changes during the training process for a training sample (aspen) and a testing sample (maple). A total of 60,000 training iterations are presented, corresponding to a running time of 30.19 h using a computer with Intel^® Core™ i7-6700K 8 × 4.00 GHz, 32 GB RAM and NVIDIA GeForce GTX 1070. As iteration increases, both training sample and testing sample show an increasing trend for mIoU and OA, with only one or two abrupt slumps, indicating our 3-D FCN is a global optimization process. Note that an IoU or OA of 1.0 means perfect prediction. After 30,000 iterations, mIoU and OA of training sample converges to 0.98, indicating the strong fitting ability of the 3-D FCN. On the other hand, mIoU and OA of testing sample levels off to a limit of 0.76 and 0.96, respectively, after 10,000 iterations. This result indicates that the generalizability of FCN would saturate and that the later learning phase after 10,000 iterations may only focus on classifying marginal features.

The testing sample has a high OA limit but relatively lower mIoU limit. This result indicates that the majority of wood points are filtered accurately, but some classes may be predicted incorrectly. Class-wise IoU changes for the training and testing samples are thus examined in Figure 7c,d. An increasing trend is shown for all classes of both training and testing samples. Apparently, branch points, due to geometric complexity, have slower and lower convergence values than stem and other points. Therefore, the FCN model can identify simple classes in the early phase, but learning difficult classes takes much longer and may not achieve the same high accuracy. The testing sample has a low branch IoU convergence of 0.4, which also explains the low mIoU in Figure 7a. The branch IoU issue can be caused by imperfect branch reference and complex branch structure, which should be mitigated by adding training samples, or refining the deep learning model with higher spatial resolution and stronger feature extraction ability.

Filtered wood points and reference can be visually contrasted using examples in Figure 8a–c. The main wood points are recognized and most difference occur in crown branches in Figure 8a,c, understory stems in Figure 8a and ghost points in Figure 8b,c. The classified wood appears slightly fatter than reference wood partly due to the setting of 5 cm voxel resolution.

The final accuracies of all training and testing samples are provided in

Table 3. 3-D FCN filtering accuracies (IoU, mIoU, and OA) among training and testing samples.

Dataset (size)	Sample ID	IoU (Stem)	IoU (Branch)	IoU (Other)	mIoU	OA
Training sample (14)	maple 1	0.974	0.501	0.950	0.808	0.960
	maple 2	0.278	0.361	0.973	0.537	0.968
	maple 3	0.979	0.764	0.955	0.899	0.973
	maple 4	0.943	0.835	0.935	0.904	0.956
	maple 5	0.860	0.489	0.995	0.781	0.975
	maple 6	0.980	0.933	0.996	0.970	0.996
	aspen 1	0.972	0.562	0.952	0.829	0.964
	aspen 2	0.970	0.988	0.998	0.985	0.998
	aspen 3	0.929	0.563	0.935	0.809	0.942
	aspen 4	0.921	0.728	0.950	0.866	0.953
	aspen 5	-	1.000	1.000	1.000	1.000
	aspen 6	0.933	0.863	0.968	0.921	0.967
	pine 1	0.984	0.792	0.942	0.906	0.975
	pine 2	0.940	0.742	0.907	0.863	0.933
Testing sample (7)	maple 7	0.779	0.658	0.985	0.808	0.981
	maple 8	0.936	0.308	0.954	0.733	0.955
	maple 9	0.704	0.323	0.888	0.638	0.873
	aspen 7	0.963	0.629	0.958	0.850	0.963
	aspen 8	0.943	0.774	0.945	0.887	0.961
	pine 3	0.980	0.782	0.930	0.898	0.955
	pine 4	0.920	0.303	0.843	0.689	0.889

. An average of stem IoU, branch IoU, ‘other’ IoU, mIoU, and OA is 0.90, 0.72, 0.96, 0.86, and 0.97 for the training sample, respectively, and 0.89, 0.54, 0.93, 0.79, and 0.94 for the testing sample, respectively. The mIoU shows no significant difference (p = 0.15) between training and testing samples based on a two-sample t test, implying that deep learning model is generalizable. The mIoU difference between the three species is also insignificant (p = 0.16) based on an ANOVA test using all samples. This result presents a possibility of having a universal wood filtering solution without need for exhaustive sampling of species.

Not all IoU results from Table 3 are satisfactory. Many factors could contribute to the low IoU situation, e.g. geometrical complexity and details. Results from a left-tailed t test show that branch IoU is significantly lower than stem IoU (p < 0.001), probably due to complex forms of branches. More details, denoted by a smaller point spacing, would lead to a lower mIoU. This statement can be indicated by a high Pearson’s r of 0.51 between point spacing from Table 1 and mIoU from Table 3. Similarly, worse quality ranking (Table 1) tends to produce higher branch IoU (r = 0.39) and lower stem IoU (r = −0.21). As quality ranking descends, stem form becomes fragmentary or blank at a rising risk of misclassification, whereas branches become simplified and conducive to classification. However, a clear understanding of IoU variation requires systematic investigation of model configuration, reference uncertainty and sample sufficiency, which is beyond the scope of this study.

Attribution of input voxels plays a vital role in our 3-D FCN filtering accuracy. As

Table 4. Effect of voxel attribution on 3-D FCN filtering accuracies (per-class IoU, mIoU, and OA) at the 25,000th iteration.

Sample ID	Voxel Attribute Combination	IoU (Stem)	IoU (Branch)	IoU (Other)	mIoU	OA
aspen 6	occupancy	0.922	0.630	0.878	0.810	0.911
	occupancy + intensity	0.924	0.648	0.881	0.817	0.914
	occupancy + intensity + height	0.958	0.854	0.938	0.909	0.958
maple 8	occupancy	0.789	0.284	0.951	0.674	0.923
	occupancy + intensity	0.847	0.355	0.960	0.721	0.942
	occupancy + intensity + height	0.911	0.445	0.960	0.772	0.960

shows, with only point occupancy, the 3-D FCN model can still achieve an impressive accuracy, for example, a stem IoU of 0.922 for the ‘aspen 6’. It is clear that local spatial pattern is mostly relied by the 3-D FCN filter. Adding the intensity attribute shows a slight improvement over accuracy. By contrast, introducing the height attribute provides leads to a greater accuracy increase, because branch is height-dependent.

Traditionally, wood points (branch + stem) can be filtered efficiently based on simple metrics such as intensity threshold. The accuracy of applying an intensity threshold filter to our sample is examined in Figure 9. We represent wood filtering accuracy with the ‘other’ IoU metric. For each sample, ‘other’ IoU is plotted against a range of intensity thresholds, and the highest ‘other’ IoU is marked with a red triangle. It is obvious that no universal threshold exists that can promise the highest wood filtering accuracy for all the samples. The upper boundary of ‘other’ IoU from intensity filter also vary drastically between 0.5 and 0.9. In comparison, ‘other’ IoU from our FCN filter, denoted by the blue starred line, is drawn in Figure 9. An overwhelming advantage of using FCN filter is presented, considering the accuracy level and stability among all the samples.

After the training process, the 3-D FCN model can be directly applied to filter stem and branch points from entire plots, based on a moving block of 128 × 128 × 128 voxels. Our plot scan has an average file size of about 1 GB. Filtering an aspen plot of about 500 trees requires 3.12 hrs, a maple plot of 1000 trees 1.95 hrs, and a pine plot of 2000 trees 2.23 hrs.

4.3. Wood Reconstruction

Individual wood reconstruction results are illustrated in Figure 10 with ‘aspen 7’ provided as an example. Branch and stem curves are reconstructed separately from FCN-filtered branch and stem points, and branch curves are finally connected to the stem curve. Shown in Figure 10, our node connection can overcome a certain degree of stem occlusion in the crown area, and the slice-based circle fitting algorithm is tolerant to some branch noise in the top of the canopy. The extracted stem volume is 0.45 m³, and the branch volume 0.22 m³. Replacing FCN prediction with reference labels and using the same QSM settings produces a stem volume of 0.38 m³, and a branch volume of 0.23 m³. Based on a wood density of 0.35 g cm⁻³ [43], the resulting stem biomass and branch biomass from FCN-QSM are 159 kg and 77 kg, respectively, and from reference-QSM are 133 kg and 80 kg, respectively. The ground truth of biomass is not available in this study. A referable biomass can be allometric biomass [44] based on inventory information of species, sites, DBH, and height. The allometric stem biomass (including bark) is 149 kg and allometric branch biomass 26 kg. In this case, the branch biomass from QSM is probably overestimated. The causes can be noise and ghost points (as indicated in Figure 10) occurring during scanning collection, reference preparation and voxelization. However, considering all 21 sample areas, the mean branch biomass from FCN-QSM is only 15% of allometric branch biomass, due to high degree of branch occlusion in most sample areas. For branch biomass, the relative root-mean-square-error (RMSE%) and r² between FCN-QSM and allometric approach are 130% and 0.05, respectively, comparing to the RMSE% of 19.6% and r² of 0.95 for stem biomass.

An example of plot-level wood reconstruction results is visualized in Figure 10. It is the maple plot, with FCN-filtered components (stem, branch and other) in different colours in Figure 11a. Its wood reconstructed stem, branch, isolated, and understory are also assigned with different colours in Figure 11b. The plot scan comprises five corner scans and one center scan, stretching over 100 m distance from plot center. In our maple plot, the farthest reconstructed tree is 148 m away from the plot center. However, a problem is also apparent in that reconstructing farther woods is strongly affected by a lower signal-noise ratio. Availability of both stem and branch points descends rapidly with the distance from the plot center. The reconstructed maple plot has 80% of stem volumes within a distance of 44 m, and 80% of branch volumes within 26 m. The individual tree branch-to-stem volume ratio also descends, from 0.15 within 10 m, to 0.13 within 25 m, to 0.05 within the entire plot. To mitigate the weak wood signal problem, we consider 25 m as an effective plot size for evaluating plot-level wood reconstruction. The reconstructed wood volume of maple plot has a composition of 92.3% stem, 6.5% branch, 0.3% isolated, and 0.9% understory. Correspondingly, the reconstructed aspen plot has 81.2% stem, 17.6% branch, 0.05% isolated, and 1.2% understory, and the pine plot has 91.5% stem, 8.4% branch, 0.03% isolated, and 0.1% understory. The average branch-to-stem ratio of individual tree with branch detected is 0.15 for maple, 0.24 for aspen, and 0.13 for pine. According to Ung and colleagues’ [45] biomass database, the average branch-to-stem ratio and its standard deviation should be 0.27 ± 0.24 for sugar maple, 0.14 ± 0.14 for trembling aspen, and 0.15 ± 0.12 for lodgepole pine. Comparing to maple and aspen in leaf-on conditions, lodgepole pine has better visibility of wood points, leading to less bias of branch-to-stem ratio. A future study should involve rigid evaluation and calibration of occlusion effect on QSM’s volume and branch-to-stem ratio calculation.

Finally, it should be clarified that reconstruction results were not validated, due to lack of ground truth measurement of branch curve and volume. The parameters of reconstruction model are also not optimized, for the same reason of lacking accurate reference. Yet one function of this QSM model can be exposing detailed wood components, and reflecting any issues of integrating filtering and QSM for natural forest scans. The delineated tree curves can also provide high-level geometric features, in terms of raw point metrics, to help develop more accurate allometric models of wood volume or biomass.

5. Conclusions

Delineating wood components in complex natural forest environments has attained wide research interest, represented by QSM development, due to its importance to modern wood management in pursuit of correct and complete biomass, carbon budget, and tree physiology. At the plot level, the wood amount in dense and noisy areas such as crown and understory are however, uncharted or gross in common QSM models. Recent QSMs incorporate simple wood filtering methods aimed to reduce noise. In contrast, this study introduces the use of a deep learning model FCN in 3-D space to filter both stem and branch points from complex forest scans, with an average testing accuracy of 0.89 (stem IoU), 0.54 (branch IoU), 0.79 (mIoU), and 0.94 (OA) over three plots. We further explored the potential application of wood filtering in constructing a plot-level QSM and extracting the wood volume component. From visual inspection, wood filtering generally produces tube shapes beneficial to QSM modeling.

Some challenges remain, however. Creating reference data, although assisted with a semi-automation algorithm, inevitably contains interpretation errors and further induces low branch filtering accuracy. Besides, point cloud is a discrete form of 3D data, but continuous voxel form is the only supported choice in deep learning tools. As a result, coarse voxel resolution was fixed to tradeoff deep learning accuracy with sufficient computation efficiency. Last but not least, wood filtering cannot address the most severe problem of wood delineation, which is occlusion. Improving TLS scan coverage and QSM modeling, although sophisticated, are still the most reliable solutions to overcome occlusion.

Deep learning modeling offers significant potential to intellectualize processing and analysis in various data-driven applications. This study only intends to probe the feasibility of filtering 3-D wood points of interest, without activating diverse options of refinement. For example, the instability issue of branch accuracy can be mitigated by larger sample sizes, more classes, and broader species. Filtering low-quality scans, particularly from needle-leaf species (e.g., pine and spruce), remains a challenge. Tentative refinement can be introducing more complex network structures such as ResNet [46], Inception [47,48], and CRF [49], sparse point network such as PointNet [34,35], instance segmentation network such as R-CNN [50,51]. However, refinement and surgery of the all-purpose deep networks are not optimal solutions to many practical 3-D problems. It is strongly hoped that deep network mechanisms can be rationalized systematically in the future, with regard to how data structure, network architecture, sample property, and problem dimension are interrelated and delimited, before optimal practice is achieved.