Accurate Geo-Referencing of Trees with No or Inaccurate Terrestrial Location Devices

Abstract

Accurate and precise location of trees from data acquired under-the-canopy is challenging and time-consuming. However, current forestry practices would benefit tremendously from the knowledge of tree coordinates, particularly when the information used to position them is acquired with inexpensive sensors. Therefore, the objective of our study is to geo-reference trees using point clouds created from the images acquired below canopy. We developed a procedure that uses the coordinates of the trees seen from above canopy to position the same trees seen below canopy. To geo-reference the trees from above canopy we captured images with an unmanned aerial vehicle. We reconstructed the trunk with photogrammetric point clouds built with a structure–from–motion procedure from images recorded in a circular pattern at multiple locations throughout the stand. We matched the trees segmented from below canopy with the trees extracted from above canopy using a non-rigid point-matching algorithm. To ensure accuracy, we reduced the number of matching trees by dividing the trees segmented from above using a grid with 50 m cells. Our procedure was implemented on a 7.1 ha Douglas-fir stand from Oregon USA. The proposed procedure is relatively fast, as approximately 600 trees were mapped in approximately 1 min. The procedure is sensitive to the point density, directly impacting tree location, as differences larger than 2 m between the coordinates of the tree top and the bottom part of the stem could lead to matching errors larger than 1 m. Furthermore, the larger the number of trees to be matched the higher the accuracy is, which could allow for misalignment errors larger than 2 m between the locations of the trees segmented from above and below.

1. Introduction

Many forest management decisions are based on attributes measured under the canopy, such as diameter at breast height (dbh) or height to the base of the live crown [1,2]. Acquisition of the below canopy data is slow and not inexpensive, which focused the efforts of many forest management organizations on the usage of information from airborne or spaceborne sensors that are cost-efficient for a large area. Attempts have been directed toward inferring under the canopy attributes from above canopy attributes such as crown diameter [3]. However, limited success was achieved because of the quality of data (e.g., low spatial resolution or small point density for lidar) or the lack of algorithms to extract the relevant information from the remotely sensed data.

The technological developments in material sciences, specifically sensors, information technology, and harvesting equipment allows fast and accurate estimation of the attributes relevant to forest activities while moving under the canopy, particularly dbh, taper, and total height. Whereas procedures for precise, accurate, and fast estimates of dbh and total height are available for a reduced set of trees (i.e., plots or samples), difficulties are encountered when estimates of all trees in a stand are needed. Stand level information is particularly needed for intermediate cuts, when the decision is taken considering all the trees not only a portion of them. Although total height of each tree in a stand can be relatively easy estimated from point clouds, lidar or photogrammetric, there are at least two attributes that are difficult to estimate accurately and precisely for all trees: dbh and location. Many algorithms have been developed to estimate dbh from above canopy data (e.g., point clouds or images) with significant success [4,5,6,7]. However, while expected accuracy is in many instances achieved, the precision of estimating dbh from above canopy is limited, which restricts its operational usage. The effort placed in estimation of tree location mirror the one for tree size, but the success was clearly less impressive, as the accuracy is still measured in meters [8].

Point clouds play a significant role in forest operations, particularly in road design and road maintenance [9,10]. Point clouds generated without an active sensor, such as lidar, or stereopsis are sometimes labeled photogrammetric point clouds [11] or phodar [12]. Photogrammetric point clouds (PPC) made inroads in harvesting, but they were employed similarly with aerial laser scanning, as the applications were based on the nadir view [13]. However, the true potential of point clouds, particularly PPC, rests in their usage under the canopy, where the most relevant forest attributes can be estimated with precision and accuracy. For example, the choice of tree to remove in an early thinning may depend upon its diameter at the top of the first merchantable log length, among other factors.

The usage of PPC in current forest activities occurring under the canopy is hindered by two issues: inability to accurately position individual trees and absence of real-time algorithms that estimates various tree attributes, such as diameter along stem, taper, or sweep. Tree position is critical in matching below-the-canopy tree measurements with above-the-canopy measurements, as accurate attributes (e.g., total height) are crucial in the decision process. Whereas significant progress has been made in estimation of tree attributes [14,15,16] from PPC, little advancement occurred in estimation of tree position from PPC.

Location of the trees while navigating under the canopy has had even more limited success, as most of the operational approaches relies on GPS technology, even though simultaneous location and mapping (SLAM) or stereopsis methods made significant progress [8]. The main difficulty in locating trees under the canopy with GPS is the lack of accuracy in a feasible amount of time, due to multipath errors. The lack of GPS accuracy in real-time disconnects the information acquired under the canopy from the information acquired above the canopy. A significant improvement can be noted if an ad-hoc positioning system is employed, but with a substantial increase in costs and field measurements time. Therefore, the objective of the present study is to develop a fast, accurate, and relatively inexpensive procedure for geo-referencing trees with no or low accuracy GPS units. We will show that even coarse GPS estimates may speed the geo-referencing procedure, such that real-time navigation would be possible.

2. Methods

To position individual trees in the absence of accurate location information we will use tree coordinates estimated from information acquired above the canopy, such as aerial laser scanning or images. Significant advancements occurred in the last decade in estimation of the Cartesian coordinates of trees from lidar, but as of now, there is no algorithm known to the authors that identifies with no or minute error for each tree in a stand. A similar lack of accuracy is encountered when multispectral images or a combination of lidar—multispectral images are used for tree segmentation. However, it is easy to identify the local maxima in a point cloud, which will likely contain the dominant and codominant trees. The local maxima will include, besides the trees, some of their branches. For our procedure, all dominant and codominant trees are needed, even though many commission errors will be present, sometimes called “false positive”. As we will see, the commission errors will not impact the procedure, but will increase the time to render a solution, as more computations are executed.

The proposed approach to geo-reference all the trees in a stand combines three types of algorithms: (1) acquisition and processing of images to render the PPC, (2) extraction of trees from the PPC, and (3) matching of the trees extracted from below canopy PPC with the possible trees extracted from above canopy (Figure 1). The proposed procedure relies on the location of the trees computed from the images captured above canopy. The subsequent sections detail each step of our procedure to geo-reference trees without GPS or with low accuracy GPS.

2.1. Study Area

The proposed procedure for geo-referencing trees was applied to a 7.1 ha plot located inside a 12.3 ha Douglas-fir (Pseudotsuga menziesii Mirb.) stand from the HJ Andrews Experimental Forest in western Oregon (Figure 2a). The soils on the site are from the Browder, hummocky-Cadenza complex, with more than 30 cm of silt loam. The B horizon, almost stone-free, is made of silty clay loam. The C horizon appears below 1.0 m depth. The soils are at least moderately well-drained, which lead to superior productivity and allows active forest management. According to Means and Helm [17], the site index for Douglas-fir at base age 50 is 34 m. The average dominant and codominant height is 35 m. Besides Douglas-fir, some western red cedar (Thuja plicata Donn) are present in the stand. The stand has a southern aspect, with slopes between 0° and 35°. There is limited understory, which allows for unobstructed view of all the trees at least 10 m away from any point within stand (Figure 2b). The clear sight within the stand allows the capture of the lower portion of the stems using red-green-blue (RGB) cameras.

2.2. Procedure

2.2.1. Tree Segmentation from the Above Canopy Acquired Point Clouds

To locate the trees, we use georeferenced point clouds. The point clouds can be developed from active or passive sensors. Passive sensors, particularly CCD (i.e., charge-coupled device) or CMOS (i.e., complementary metal-oxide semiconductor) that record RGB images, are attractive to forestry applications because of robustness and low-cost (e.g., at the time when the paper is written a Phantom 3 Standard by DJI, which includes the vehicle and the sensor, is approximately 500 USD). Therefore, we produced a point cloud from the RGB images acquired with the unmanned aerial system DJI Phantom 3 Professional, which is a quadcopter equipped with a 1/2.3″ CMOS sensor capable of recording 12.4 mega pixels pictures. The Phantom 3 Professional is equipped with a satellite positioning system that is GPS and GLONASS capable, with a manufacturer stated accuracy of 1.5 m horizontally and 0.5 m vertically. To ensure precise positioning of the PPC, three ground control points identified with 1 m × 1 m crosses were placed in the middle of the road bordering the stand. The position of the center of each cross was estimated with a Trimble Geo XH, which has a stated accuracy of 0.3 m. We execute the flight at an average above ground elevation of 100 m and using 80% forward-overlap and 60% side-overlap, to ensure proper 3D reconstruction of the stand. We created the point cloud using structure-from-motion [18,19,20,21], as implemented in Agisoft version 1.3.4 [22]. The Agisoft parameters used to process the images were based on the recommendations of Fang and Strimbu [15,23].

We infer stem location from tree crowns, which were identified using the point cloud created from the above canopy images. Because the point clouds are georeferenced, the tree crowns will also be georeferenced. Once the tree crowns were delineated, we consider that the stem is located at the tallest point within the crown. However, the coordinates of the tallest point within the crown does not necessarily accurately represent the actual coordinates of the lower portion of the stem, as trees often respond opportunistically to canopy openings to access light. Therefore, an adjustment is needed, as the highest point would likely not be located on top of the axis of the lower portion of the stem, the portion that is visible from the ground. To ensure that only the points describing a tree are used for the correction of stem location we assumed that the stem is located close to the centroid of the crown projected area. To represent the assumption, we considered that the stem cannot be further from the Cartesian coordinates of the tallest point within the crown by more than 1/3 the radius of a circle with an area equal to the crown projected area (similar to the centroid of a triangle):

$$distanc{e}_{stem}=\sqrt{Are{a}_{crown}/\pi }/3$$

(1)

Therefore, the stem position will be the highest point within a circle centered in the crown centroid and with the radius computed with Equation (1). The location of the stem, depends not only on the crown shape, which is responsible for the centroid position, but also on the point density, the higher the density, the closer the stem location would be to the location of the highest point inside the canopy. To avoid inclusion of parameters that are not necessarily directly related to the main study objective, we have created PPC with a density of at least 30 points/m². Such a high density, particularly for the PPC, which are focused on the outer part of the canopy, will ensure that the tallest return is close to the terminal bud. For actively managed forest the terminal bud is not horizontally offset more than 2 m from the stump [24], which will warrant the presence of the highest point and the stem within the same point cloud defined by the projected crown. To ensure the correspondence between the above and below canopy stems, only the low elevation points within the distance defined by Equation (1) are considered. We considered the points from the first quartile of heights as low-elevation points, which contain more stem points than the other quartiles. Furthermore, to reduce the chance of classifying terrain and small shrubs as trunk, only the points above 2 m from the ground are included in computations. The final (x, y) coordinates of the stem are the mean (x, y) coordinates of the points from the first quartile (i.e., low-elevation), weighted with the horizontal distance to the tallest point (an approximation of the top of the tree).

Several algorithms are available for individual tree crown segmentation (i.e., projected crown on the datum), none operating without error [4,5,25]. For this study, we used the oriented weighted graph algorithm of Strimbu and Strimbu [5], which currently seems to perform with minor errors given limited resources, and Dalponte and Coomes [6], which was proven to produce relatively accurate results. Because geo-referencing is executed by matching the trees segmented from above canopy with the trees from below canopy all the trees from the stands should be identified. Therefore, commission errors (i.e., false positive) are preferred over omission errors (i.e., false negative). Nevertheless, to ensure real-time computations, the commission errors should be kept minimal, otherwise, any local maxima could be considered a tree. The individual tree crown segmentation algorithms of Strimbu and Strimbu [5] and Dalponte and Coomes [6] perform with high accuracy if the point cloud is normalized, meaning the ground elevation is subtracted from each point. Several algorithms are available for ground normalization, almost all requiring point classification. However, the PPC created with Agisoft is not classified, therefore we have used an alternative algorithm, implemented in Quick Terrain Modeler [26], which is an iterative selection of minimum elevation points within a predefined grid.

We preferred the classification from Quick Terrain Modeler over other algorithms, as it is executed very fast (i.e., less than 1 s on a Dell Precision 7810 with an Intel Xeon E5-2630 v3 CPU and 32 Gb RAM) and supplies results comparable with more sophisticated implementations, such as TerraScan [27]. To ensure that not only speed but also point quality is present in the classification we compared the points classified with Quick Terrain Modeler with the classification performed by lastools [28] and PDAL [29]. The three classifications were similar in term of number of points and produced terrain models comparable with the US Geological Service models [30], which supported our choice for points classification. The tree segmentation algorithm based on oriented weighted graphs of Strimbu and Strimbu [5] as implemented in TrEx ver. 022 [31] and of Dalponte and Coomes [6], as implemented in LidR package of R ver. 3.5.1 [32]. The two segmentation algorithms produced similar results, but we have chosen the results supplied by Strimbu and Strimbu [5], as it outperformed the Dalponte and Coomes [6] algorithm when measured by omission, commission and overall accuracy error (i.e., omission error was 6% vs. 8%, commission error was 10% vs. 12%, and overall accuracy was 16% vs. 20%).

2.2.2. Generation of Point Clouds from below Canopy Images

Similarly to the above canopy point cloud, the trees seen from the ground will be 3D reconstructed from images. The only difference between the generation of point clouds consists in the view point: nadir for above canopy and perspective for below canopy. To capture a large section of the bole of the trees a wide-angle lens is preferred. In this study we have used two cameras: one with no GPS (i.e., Nikon D3200) and one with GPS (i.e., GoPro Hero 5). The Nikon camera was equipped with the Sigma EX 10 mm fisheye lens, to capture the entire tree in one image. The images were captured with a resolution of 24.06 mega pixels with no flash (Figure 3a), which resulted in high ISO sensitivity for some images (e.g., 400). To reconstruct the trees, we have captured at least 20 images at locations approximately 10 m apart, almost uniformly distributed inside the stand. A total of 20 locations were captured, as trees at the edge of the plot were constructed from the images captured from locations within the plot (approximately a 6 m buffer inside the plot). The images recorded at each location were approximately evenly distributed along a circle (i.e., ~20° between two adjacent images) with camera pointed away from location (Figure 3b). A similar procedure, and consequently point cloud, was obtained from the images captured with GoPro camera. We produced separate point clouds for each device by processing the images with Agisoft version 1.3.4 [22] using the parameters of Fang and Strimbu [15,23]. The major difference between the two-point clouds (i.e., from Nikon and from GoPro) being their positioning, one in relative coordinates (i.e., disconnected from reality-Nikon), and the other relatively positioned in the correct region (i.e., no accuracy—GoPro).

2.2.3. Identify the Relative Coordinates of the Trees from below Canopy Point Cloud

The PPC generated at each location under the canopy rendered at least 10 trees, but they were not accurately geo-referenced. To position them, even in relative coordinates, the trees must be identified. To extract the trees from the PPC we have implemented a two-step procedure: first, eliminate the points that most likely are not the stem, and second, locate the most likely position of the stem within the thinned point cloud. For terrestrial lidar scans, there are many procedures that identify the points that are not stem. The procedures used on lidar data have limited success on PPC because the point cloud contains more noise. However, lidar does not contains colors, which are available with the PPC. Therefore, we considered that points that are green (i.e., leaves) or blue (i.e., sky), are not stem, therefore, should be eliminated from stem identification. Images captured throughout the stand facing different directions record tones varying from location to location. Therefore, the color of the PPC created at each location should be normalized, to ensure consistency throughout the stand. Among the multiple algorithms [33,34] and equations available for normalization we have used the feature scaling [35], which does not produce unnatural appearances, an issue of complex algorithms, while homogenizing the hue throughout the stand:

$$normalize{d}_{hue}=\frac{hue value-\mathrm{in}\left(hue value\right)}{\max \left(hue value\right)-\min \left(hue value\right)}\times 255$$

(2)

After normalization of each wavelength, we differentiated the stem from non-stem points by implementing two constraints (Equations (3) and (4)), one operating in absolute values and one in relative terms. The restriction using absolute values (Equation (3)) identifies a non-stem point as one for which the normalized color is mostly blue or green (hue of blue and green > 127) while red have a reduced presence (hue of red ≤ 127). The constraint using relative terms (Equation (4)) identifies non-stem points as the ones for which both blue and green have larger hues than red.

$$\mathrm{if}\{\begin{array}{c}normalize{d}_{blue}>127\mathrm{or}normalize{d}_{green}>127\\ normalize{d}_{red}\le 127\end{array}\mathrm{then}\mathrm{non}-\mathrm{stem}$$

(3)

if normalized_blue > normalized_red or normalized_green > normalized_red then non-stem

(4)

To identify the location of the stems within the filtered PPC we developed a simple and fast procedure that is accurate enough, considering that the final identification of the trees will be executed by combining the above canopy information with the below canopy information. Our procedure relies on the assumption that trunks will be represented by more points than other forest features; therefore, a counting of points across an area should display the trunks as spikes. We represented a tree by the center of the stem; therefore, for each spike we computed its centroid. To identify each stem, we have used hierarchical clustering [36], with the measure of similarity being the average linkage (distance). We preferred the average distance over other measures of similarity because it is relatively robust to outliers and directly computes the location of each tree. The number of clusters, namely the number of trees, was chosen using the property that the proximity measure changes exponentially with the number of clusters [37]. We considered that a sudden change in proximity measure followed by a relatively flat line indicates that the significant number of clusters was achieved (similar to the screen test). We executed cluster analysis in SAS 9.4 [38], as it has a more intuitional interface than other packages.

2.2.4. Match the Inaccurate or Unreferenced Trees with the Georeferenced Trees

The final step in geo-referencing the trees segmented from under the canopy images consists in their matching with the trees segmented from above canopy. Among the various approaches that matches trees, we have chosen a point set registration algorithm [39,40,41,42], namely the nonrigid transformation of Ma et al. [43]. Several algorithms are commonly used for point set registration, the Iterative Closest Point of Besl and McKay [44], the coherence point drift algorithm of Myronenko and Song [45], the robust point-matching-preserving local neighborhood structures proposed by Zheng and Doermann [46], and the Kuhn–Munkres algorithm [47], which were proven to perform well in fingertip detection [48,49], phalanges [50], and in general on problems that can be represented by graphs [51,52]. The Kuhn–Munkres algorithm has a cubic complexity [47], which makes it attractive for solving combinatorial problems encountered in many forestry problems. However, the Kuhn–Munkres algorithm is sensitive to noise, which leads to a reduced accuracy in tree position. Because the trees located from above-canopy and below-canopy PPC are not accurately positioned, the Kuhn–Munkres algorithm is not the best suited for the geo-referencing problem at hand. Therefore, we use the non-rigid point-matching algorithm of Ma et al. [43], which can accommodate outliers and inaccuracy in data, more specifically small discrepancies between the coordinates of the trees segmented from above canopy and the trees segmented from below canopy. The lack of accuracy in tree position is present in both PPCs: from above canopy, as the tallest point within the tree crown is likely not located on the stem axis, and from below canopy, as the location of the tree is probably not the average of the x and y coordinates of the points filtered to represent the trunk. Furthermore, the non-rigid algorithm is designed to match sets with different number of elements [53], in our case trees, which is exactly the situation encountered in real situations.

The algorithm of Ma et al. [43] estimates the transformation f, of the tree located under the canopy to their corresponding trees seen from above canopy using a cost function, called L₂E, which minimizes the L₂ distance between the densities defined by the transformation f. The algorithm operates in two steps, first feature descriptors, such as shape context, of the tree location are established, and second, the transformation of the unreferenced trees to the georeferenced ones is computed using the robust estimator L₂E, a L₂ measure minimizing estimate. The L₂E was tested on linear regression and estimated the correct values for the slope even when half of data were outliers, whereas the maximum likelihood estimators supplied a reliable solution only for a limited number of outliers [43]. In our study, we have used the Matlab [54] implementation of the algorithm by Ma et al. [43], as it relies on fast and near-optimal implemented computational algorithms. The logical structure of the algorithm is the following [43]:

Step 1

Input the location of the trees, as seen from below, ${\left\{x_i\right\}}_{i=1}^n$, and as seen from above, ${\left\{y_j\right\}}_{j=1}^l$

Step 2

Compute feature descriptors for the location of the trees seen from above ${\left\{y_j\right\}}_{j=1}^l$

Step 3

Start a recursive procedure with a preset number of iterations

Step3-1 

Compute feature descriptors for the relative location of the trees seen from below ${\left\{x_i\right\}}_{i=1}^n$

Step3-2 

Estimate the initial correspondences based on the feature descriptors of two-point sets locating the trees, one from above and one from below

Step3-3 

Solve the transformation f, $f\left(x\right)={{\displaystyle \sum }}_{i=1}^m\mathsf{\Gamma }(x_i, {\tilde{x}}_i)c_i$, which warps the set of points representing the trees seen from below (the model point set) to the set of points representing the trees seen from above (the target point set). The Γ is a positive defined matrix Γ: R^d × R^d → R^d×d and ${\left\{{\tilde{x}}_j\right\}}_{j=1}^m$ is a random subset of points to have nonzero coefficients in the expansion of the solution of f. The solution of f is found using the following steps:

Step3-3a 

Input the annealing rate γ, and the parameters that control the smoothness of the transformation f, β and λ. Ma et al. [43] argued that the method is robust to parameter changes, therefore we have chosen the values recommended by Ma et al., namely γ = 0.5, β = 0.8 and λ = 0.1.

Step3-3b 

Construct the Gram matrix Γ and matrix U, where

${\mathsf{\Gamma }}_{ij}=\mathsf{\Gamma }\left({\tilde{x}}_i,{\tilde{x}}_j\right)=e^{-\beta {\tilde{x}}_i,{\tilde{x}}_j}$

$U_{ij}=\mathsf{\Gamma }\left(x_i,{\tilde{x}}_j\right)=e^{-\beta x_i,{\tilde{x}}_j}$

Step3-3c 

Initialize C and σ² where

σ² is the variance of the point correspondence, computed as y_i − f(x_i)

d is the dimension of the coefficient c_i from the C, the coefficient matrix m × d used in conjunction with the Gram matrix: C = (c₁, …, c_m)

Step3-3d 

Start deterministic annealing [55] aimed at minimization of the function

$L_{2}C\left(C,{\sigma }^2\right)=\frac{1}{2^d{\left(\pi \sigma \right)}^{d/2}}-\frac{2}{n}{{\displaystyle \sum }}_{i=1}^n\frac{1}{2^d{\left(\pi \sigma \right)}^{\frac{d}{2}}}{e}^{-\frac{y_i^T-U_{i}C}{2{\sigma }^2}}+\lambda \mathrm{tr}\left(C^T\mathsf{\Gamma }C\right)$

where C^T is the transpose of C, and tr is the trace of the matrix C^TΓC

using the quasi-Newton algorithm with C as the old value and the gradient $\frac{\partial L_{2}E\left(C,{\sigma }^2\right)}{\partial C}$,

Step3-3d-I: 

update C ← arg min_C L₂E(C, σ²)

Step3-3d-II: 

anneal σ² = γ σ²

Step3-3e 

Select the solution of $f\left(x\right)={{\displaystyle \sum }}_{i=1}^m\mathsf{\Gamma }\left(x_i,{\tilde{x}}_i\right)c_i$ obtained by minimizing L₂E

Step3-4 

update the location of the trees seen from below (model point set) ${\left\{x_i\right\}}_{i=1}^n\leftarrow {\left\{f(x_i)\right\}}_{i=1}^n$

Step 4

Stop when the preset number of iterations is reached

Step 5

The georeferenced trees seen from below have the location ${\left\{f(x_i)\right\}}_{i=1}^n$

2.2.5. Robustness of Tree Georeferencing to Missing and Erroneous Positioned Trees

We preferred the algorithm of Ma et al. [43] because it has two properties that makes it suitable for forestry applications: it accurately models the transformation required to align the point sets without sacrificing computational complexity (i.e., linear complexity in the number of correspondences), and it is robust to noise, outliers, and missing points. The final location of the trees is established by preserving the relative distances among the trees determined from the below-canopy PPC but rotated, translated, and expanded or contracted according to the geo-referenced trees from above-canopy PPC. For our research, the main issue is the inaccurate location of trees from both above and below canopy rather than on missing trees. Therefore, besides the results supplied by the field data, we tested the robustness of the algorithm to erroneous coordinates using a simulation that considered all trees as wrongly positioned.

The erroneous data is not the only issue impacting the performances of the Ma et al. [43] algorithm. Besides the intrinsic parameters required by the algorithm, such as annealing rate, γ, the width of the range of interaction between samples (i.e., neighborhood size), β, and the control of the strength of regularization, λ, the algorithm depends also on the number of outliers and the trees to be matched. For the case when trees were wrongly located we considered five mean errors ranging from 1 m to 5 m, in increments of 1 m. The error was allocated to individual trees positioned using the point cloud developed from the nadir images, as they are georeferenced. In absence of the contrary, we assumed that errors are uniformly distributed with mean {1, 2, 3, 4, 5} and variance 1. To assess the impact of the missing trees and the number of the matching trees, as mentioned by Ma et al. [43], we considered that only 70% or 80% of the trees identified from above are found in the unreferenced trees (i.e., below). Therefore, we created a 5 × 2 factorial experiment to assess the robustness of the geo-referencing procedure to positional errors and missing trees.

3. Results

3.1. Geo-Referencing Trees Using Real Data

Initial Geo-Referencing

We produced the PPC from the images collected above-canopy based on the study of Fang and Strimbu [15]. The options used for image alignment and point densifications were: half size images, 100,000 key points, 40,000 tie points, and the outliers identified assuming that no meaningfully small details are present (i.e., the aggressive depth filtering option). The final PPC has approximately 2,600,000 points, with an average point density of 37 points per square meter. The PPC geo-reference accuracy depends on the location of the ground control points [56], which in our study where not uniformly spread throughout the stand; therefore, errors are expected. Nevertheless, the magnitude of the geo-referencing error is likely to be significantly less than the tree-matching error, as non-rigid matching algorithms, including the one of Ma et al. [43], are not perfect.

To extract trees from the above-canopy PPC with TrEx we followed the recommendations of Strimbu and Strimbu [5]. The set of parameters used to extract trees without a significant number of false positives were: a 1 m cell for computing maximum elevation, a Gaussian smooth of 0.7 pixels, 32 height levels, and 4 adjacent cells. The graph created by TrEx had weights of 50% for level degree and 50% for top distance, whereas the trees were finally delineated using a 70% best sufficient parent and more than 10 pixels for each crown (Figure 4).

The total number of trees extracted by TrEx from the normalized PPC was 641 trees; out of which few are not necessarily actual trees, but branches mistakenly identified as trees. The initial location of each tree was selected to be the coordinates of the highest point within the crown. Once the trees were extracted, TrEx delineated their crown by tracing the pixels identified within the oriented weighted graph as belonging to neighboring trees. The final coordinates of the trees segmented from the above canopy PPC were determined using the lower quartile of the points above 2 m that were located inside the circle centered in the initial tree location and with radius computed with Equation (1). The final coordinates differ, on average, 0.45 m from the initial tree coordinates (i.e., coordinates of the highest point within the crown), as computed by TrEx. The differences range from 0.1 m to 1.7 m.

The parameters used to reconstruct the PPC from below-canopy images were based on Fang and Strimbu [23], but adjusted to produce fast results, less than 1 min per location. Therefore, to align the images we have used only ⅟16 × ⅟16 of each image (i.e., accuracy set to low), 40,000 key points, and 5000 tie points (Figure 5). Once the images were aligned, we densified the tie points by filtering the outliers assuming a lack of small meaningful details and ¼ of the linear information stored in each image. We integrate the PPCs generated at each location by first aligning them with the scale-invariant feature transform [57], and second by merging them.

In the absence of reference points, the PPC produced with structure-from-motion [20,58,59,60] software will likely not have a nadir view. In addition, the PPC based on unreferenced images will have the relative coordinate system determined by the software assumptions and may place the point cloud in an unrealistic position, as we see in one location (Figure 5a), where the trees are positioned almost horizontally. Considering that nadir view of the PPC is required for stem identification, the initial PPC has to be rotated such that a coordinate system that ensure a top view is obtained. To position the PPC obtained from Nikon 5200 images (Figure 5a) in nadir view we manually rotated the point cloud by visual inspection (Figure 5c). A rotation with error is allowed, as a relative nadir view is sufficient for computing the stem location from the below-canopy PPC. The final geo-referencing (i.e., accurate nadir) is executed in the last step of the procedure. Manual rotation is executed only for the first location where images will be collected, as the rest of the PPCs will be successively aligned, and consequently having a nadir view of the trees. In comparison with the Nikon camera, the PPC produced from the Go Pro camera does not need any rotation, the point cloud being correctly oriented. Even though the position of the PPC from the GoPro is not accurate, the benefit of having an inaccurate GPS helps to eliminate a step needed for geo-referencing.

Our assumption that colors will separate the trunks from the non-trunks was shown to be true (Figure 6a is a selection from Figure 5c). The delineation was not clear on the 3D image (Figure 6b) but was obvious on the grid created from the count of points (Figure 7). Considering that the PPC is unreferenced, there is the possibility that the grid will produce statistics that are too coarse or too fine. To reduce the risk of meaningless statistics, we have selected a grid with unit cell size, in relative units, as Agisoft Photoscan [22] distributes the points relative to the location of the cameras. Cluster analysis found 612 trees from under the canopy PPC, and for the locations from Figure 6 only 19 trees.

The point-matching of the 612 trees identified from below canopy with the 641 trees delineated from above canopy not only took a significant amount of time (i.e., more than 12 h) but also did not produce valid results, as the distance between some trees were more than 5 m. Therefore, we tried a different approach based on the observation of Ma et al. [43] that the number of trees under the canopy should not be large (in their simulations they used 15 points). Consequently, we divided the trees segmented from the PPC under the canopy in quadrants, each containing at most 20 trees. A simple-trial and error approach led to a grid with 30 m cell sizes. Matching the points from each quadrant into the 641 trees led to results even worse than before, as trees there spread throughout the 7.1 ha whereas, in reality, they are neighbors. To address this issue, we decided to reduce the size of the matching point sets by allocating the trees segmented from above the PPC into grid cells that would include at most 40 trees, twice as many as the number of trees from each cell with trees identified from below. This task is not difficult to implement when matching trees segmented from PPC developed from GoPro Hero 5 camera (i.e., low GPS precision), as each cell should be enlarged such that it would contain the cell from below and enlarged with the size of the manufacturer accuracy. Information about the accuracy of the GPS from the GoPro Hero 5 is not available from the manufacturer; therefore, we assumed that the accuracy is similar to other hobby GPS devices, such as Garmin or Magellan), which under the canopy is approximately 10 m. To ensure that all cells created from below PPC are georeferenced, we used a moving window with size 20 m (i.e., 10 m on each side) larger then under the canopy PPC grid, which led to a grid with 50 m cell sizes (Figure 8). However, this repeated process would take a significant amount of time, and at the end, the trees from all cells must be combined anyway. Therefore, we decided to select only five grid cells covering the trees identified from under the canopy PPC that are completely located inside the stand, four located close to the corners and one close to the centroid of the stand, namely 8, 11, 17, 22, and 32 (Figure 8). Furthermore, to reduce the computation time, we have selected from the trees segmented from the under the canopy PPC only the ones that are located inside the 30 m square within the five cells. The reduced number of cells used to geo-reference all the trees segmented from the below canopy PPC is of little significance, as the conversion from relative or imprecise coordinates to accurate coordinates is an affine transformation, namely a linear function. In theory, only one grid cell would have sufficed to georeference all the trees, but to ensure representability at the stand level and to correct for possible local distortions we have selected more cells evenly distributed through the stand. The original, imprecise, coordinates of the trees from the five cells, X, would be used to establish the translation vector, B, that converts to accurate coordinates, $\widehat{X}$: $\widehat{X}=B\times X$. The translation vector, B is obtained by simple algebraic manipulations of the coordinates of the matched trees: accurate, $\widehat{X}$ and inaccurate, X (Equation (5)):

$$B=\left(\widehat{X}\times X^T\right){\left(X\times X^T\right)}^{-1}$$

(5)

where ${\left(X\times X^T\right)}^{-1}$ is the inverse of matrix X × X^T.

The unmatched trees, Y, would have the coordinates corrected, $\widehat{Y}$ using B (Equation (6)):

$$\widehat{Y}=B\times Y$$

(6)

The usage of georeferenced images ensure that the grid cells used in the matching process contains some common trees. However, for the un-referenced PPC the location is relative; therefore, there is no overlapping of the grid created from trees segmented from above PPC with the grid created from the trees segmented from below PPC. To use the same procedure for unreferenced images that was employed for georeferenced images, we had to relatively overlap the two PPC, from above and below. To implement this task, first we matched the relative distance among the trees segment from above canopy PPC with the distance among the trees from segmented from below canopy PPC, and second, by rotating and translating the two set of tree locations (i.e., above and below) such that a relative overlap occurs. The first step ensures that the tree locations have similar magnitudes, as in the inaccurate georeferenced images, whereas the second step warrants that the grid cells created from above would include some of the trees segmented from below. Once the two sets of tree locations (i.e., above and below) are relatively aligned, we used the same process as for the relatively referenced trees to accurately locate the stems.

Irrespective the source of images to build the PPC (i.e., georeferenced images or not), we assessed the accuracy of procedure to georeference the trees by computing a matching error, defined as the mean distances between the actual and matched trees:

$$accuracy=\frac{\left({{\displaystyle \sum }}_{i=1}^g{{\displaystyle \sum }}_{j=1}^{n_g}{d}_{i,j:above,matched}\right)}{g\times n_g}$$

(7)

where g is the number of the grids used to match the trees, in this study is 5

n_g is the number of trees to be matched in grid cell g

d_{i,j:above,matched} is the distance between the location of the matched tree i obtained from the below-canopy PPC and the closest trees segmented from the above-canopy PPC, j.

The 2500 m² grid cells, created from the trees segmented from the above canopy PPC, contain at least one tree (i.e., cells 34 and 40) and more than 20 trees, when they are completely inside the stand (Figure 8). Our results suggest that the accuracy of tree matching depends on the number of trees segmented from the above canopy PPC found inside the 2500 m² grid cell and on the number of trees segmented from below canopy PPC found inside the 900 m² grid cell, with values between 0.98 m and 1.15 m (

Table 1. Accuracy of tree matching for the 2500 m² grid cells used for geo-referencing all the trees created from trees segmented using above canopy PPC.

Grid Cell	# Trees Inside 2500 m2 Cell	#Trees from GoPro Images Inside 900 m2 Cell	Accuracy [m]	#Trees from Nikon Images Inside 900 m2 Cell	Accuracy [m]
8	24	8	1.06	7	1.08
11	21	8	1.01	9	0.98
17	21	7	1.12	8	1.05
22	24	8	1.11	7	1.15
32	27	11	1.05	9	1.10
Accuracy	-	-	1.07	-	1.07

). We noticed a constant trend that the accuracy is inversely related to the number of trees segmented from the under the canopy PPC, irrespective of the number of trees identified from above canopy PPC. The accuracy of tree matching was virtually the same (i.e., 1.07 m) for the five plots, but larger for all the trees [i.e., 1.09 m for the trees segmented from georeferenced images (GoPro) and 1.11 m for the trees segmented from unreferenced images (Nikon)].

3.2. Accuracy of the Georeferencing Procedure Using Simulations

The factorial experiment designed to assess the impact of erroneous positioning of the trees, as well as of the missing trees, could not be implemented when the trees were georeferenced at once as a non-operational accuracy (>5 m) was present. Therefore, we have applied the same factorial experiment to the five grid-cells used for geo-referencing, as the selection of the initial positioning cells would define tree location accuracy. The results reveal that, irrespective the simulated overlap between the above and below canopy segmented trees (i.e., 0.7 and 0.8), for misalignment errors less than 2 m the trees will be geo-referenced with at most 1 m accuracy (Figure 9). Considering that usually mature trees are more than 1 m apart, a lack of alignment between the lower portion of the stem and the highest point likely to be achieved (i.e., <2 m) would lead to relatively accurate tree geo-referencing. However, the standard deviation of the tree geo-referencing depends on the overlap, as the more overlap, the more precise the tree location (Figure 9). In fact, for 70% overlap, the geo-referencing is operational only for misalignment errors less than 1 m (Figure 9a), whereas for 80% overlap, the misalignment error can be up to 2 m and the tree location is still with acceptable limits (Figure 9b).

The computation times for the five grid cells on a Dell Precision 7910 equipped with an Intel Xeon CPU E5-2630 [email protected] GHz is less than 10 s. The entire stand was geo-referenced in less than 1 min.

4. Discussion

To render trees using cameras under the canopy requires unobstructed visibility throughout the stand, particularly the lower portion of the stem. Good visibility of the stem ensures a fast and correct reconstruction of the trunk with structure-from-motion algorithms. The lack of understory vegetation is critical because it will not add noise around the stems, which helps extract the bole from the PPCs, irrespective the placement of the images, above or below canopy. The PPC created from images mirror reality and have all points colored. The presence of colors dramatically enhances processing of the PPC, permitting simple algorithms to supply faster and similar results to complex algorithms developed for colorless point clouds. If our color-based approach will still lead to a PPC with multiple non-stem points we will consider more sophisticated algorithms, but for mature Douglas-firs it seems that would not be needed. Visibility inside the stand is needed not only for 3D reconstruction of the stems but also for correct positioning of the trees segmented from above canopy PPC. Our results show that errors of more than 1 m in locating trees from above can lead to inaccurate results.

The non-rigid point-matching algorithm of Ma et al. [43] accommodates the lack of accuracy in positioning the trees from the above-canopy PPC and the below-canopy PPC, which makes it more attractive than the Kuhn–Munkres algorithm. However, its flexibility also possesses challenges, as the trees are not matched properly if multiple combinations are available. To address this issue, we reduced the size of the problem by overlaying a grid on the tree segmented from above PPC aiming at matching only a reduced set of trees. While the decrease in the number of combinations proved beneficial, as it solved the matching accuracy, it introduces an extra level of complexity. The addition of the grid does not ensure a complete solution, as it is possible that, in some situations, not all the trees identified from the below-canopy PPC would fall inside one grid cell. However, we demonstrated that our approach is operational and, perhaps more importantly, it is simple and easy to implement, given the current technology.

Our results show that irrespective the existence of a positioning device (such as GPS) trees would be accurately georeferenced. However, the lack of geotagged images requires extra steps, which increase the computation time and are prone to errors that could impact the accuracy of tree positioning. Our findings show that that absolute accuracy of trees location segmented from the below canopy PPC is not relevant, what is important is the relative position of the stems. It is the relative position that allows the usage of an area adjusted simultaneously to the trees delineated from the above-canopy PPC and below-canopy PPC, which ensures the matching accuracy. Additionally, the PPC that is spatially located eliminates its rotation, as the point cloud has a nadir view.

The proposed geo-referencing procedure of trees under the canopy with no or in-accurate GPS devices produces the expected results relatively fast. All the computations directly related to geo-referencing are executed in less than 1 min. The final location of the trees is sensitive to the misalignment between the tree top and lower bole, which for accurate positioning cannot be larger than 2 m. The requirement for aligning the top with the bottom of tree is necessary not only for overall accuracy but also because beyond 2 m, the standard deviation increases to levels that are not operational anymore. However, advances in sensor development would most likely reduce the difference between the stem and tree top coordinates. Furthermore, the inclusion of more trees segmented from the under the canopy PPC increases the accuracy of geo-referencing. Consequently, any improvement in locating the trees would reflect in a more accurate and less variable tree positioning.

We expect the georeferencing procedure that we developed to supply accurate results for deciduous species, especially if the images are acquired during the leaf-off season. The lack of leaves allows reconstruction of the trees with PPC containing sufficient stem points to ensure that the misalignment between the top of the tree and the lower part of the bole is less than 2 m. In fact, we expect that the difference between the two locations (i.e., tallest point and lower bole centroid) to be less than 1 m, as the globular shape of the crown would likely produce a maximum elevation point closer to the stem axis even when crown moves. It is possible that for the leaf-off season, the selection of the points from the lower quartile as basis to estimate the location of the trees from below canopy images could be replaced with the lower quintile, as more stem points are available.

5. Conclusions

Positioning trees on a geo-referenced map based on under-the-canopy measurements is challenging and time-consuming. The advancement in computer vision allows 3D reconstruction of the entire stands, but significant difficulties are encountered when they are placed on geo-referenced maps. Therefore, in this study we focused on geo-referencing trees segmented from under the canopy images that contains either no spatial coordinates or low accuracy GPS values. Furthermore, we aimed at developing a procedure for geo-referencing 3D point clouds representing the lower section of the trunk that is relatively inexpensive, fast, and accurate using only RBG images. Our procedure is based on using the coordinates of the trees seen from above canopy to position the same trees seen from below canopy. Multiple photogrammetric point clouds are developed using structure-from-motion implemented in Agisoft [22], one acquired with an unmanned aerial vehicle (i.e., DJI Phantom 3 Professional) that will be used for tree segmentation from above canopy, and multiple at different locations throughout the stand, which will serve to identify the absolute location of the trees as seen from below canopy. The location of the trees segmented from below canopy are matched with the location of the trees extracted from above canopy, in the sense that their coordinates will be affine-transformed based on the georeferenced trees. We segmented the crown of the trees from above using TrEx 022 [5] and estimated the coordinates of the trunks by using the lower quartile of the points 2 m above ground but within the horizontally projected crown. The location of the stems developed from under-the-canopy images were estimated with hierarchical cluster analysis [36] executed on the color filters points (i.e., green and blue points eliminated). We matched the trees identified from above canopy with the trees from below canopy with the non-rigid point-matching algorithm of Ma et al. [43]. We found that tree matching algorithm of Ma et al. [43] is too flexible, in the sense two sets of trees will almost always will be matched, given a preset accuracy. To ensure correct results, we reduced the number of the trees to be considered for matching by dividing the trees segmented from above using a quadratic grid of 50 m. The trees segmented from the below canopy PPC falling inside a 50 m cell buffer with the manufacturer GPS accuracy were matched with the trees from the 50 m grid-cell. For a misalignment of the trees less than 2 m the trees are mapped with an average accuracy of less than 1 m. The proposed procedure is relatively fast, as for a 7.1 ha stand, the entire matching process is approximately 1 min (on a Dell Precision 7910 equipped with a CPU Intel^® Xeon^® E5-2630 v3 and 32 GB RAM). The pair-matching procedure depends on the number of trees to be matched; the larger the number of trees, the larger the accuracy. A limitation of the proposed procedure is the need to rotate the point clouds generated from unreferenced images, such that a close to nadir view is obtained. However, this weakness is eliminated when geotagged images are used during the structure-from-motion reconstruction. The accuracy of positioning the images captured under the canopy is irrelevant, because the main purpose is to ensure the nadir view of the point clouds. We think that tree matching is possible to be executed in real-time if SLAM algorithms [61,62,63] are used for 3D tree rendering instead of structure-from-motion algorithms [57,64,65]. Furthermore, accurate geo-referencing of tree stems could help improve species identification when analysis is executed data fusing spectral information with locational data [66,67,68].