Method of 3D Voxel Prescription Map Construction in Digital Orchard Management Based on LiDAR-RTK Boarded on a UGV

Abstract

Precision application of pesticides based on tree canopy characteristics such as tree height is more environmentally friendly and healthier for humans. Offline prescription maps can be used to achieve precise pesticide application at low cost. To obtain a complete point cloud with detailed tree canopy information in orchards, a LiDAR-RTK fusion information acquisition system was developed on an all-terrain vehicle (ATV) with an autonomous driving system. The point cloud was transformed into a geographic coordinate system for registration, and the Random sample consensus (RANSAC) was used to segment it into ground and canopy. A 3D voxel prescription map with a unit size of 0.25 m was constructed from the tree canopy point cloud. The height of 20 trees was geometrically measured to evaluate the accuracy of the voxel prescription map. The results showed that the RMSE between tree height calculated from the LiDAR obtained point cloud and the actual measured tree height was 0.42 m, the relative RMSE (rRMSE) was 10.86%, and the mean of absolute percentage error (MAPE) was 8.16%. The developed LiDAR-RTK fusion acquisition system can generate 3D prescription maps that meet the requirements of precision pesticide application. The information acquisition system of developed LiDAR-RTK fusion could construct 3D prescription maps autonomously that match the application requirements in digital orchard management.

1. Introduction

In fruit production, the rational use of pesticides can reduce yield losses caused by pests and diseases. Wind-assisted application is often employed in orchards to improve the penetrability of pesticides [1,2]. During the wind transportation, only a part of the pesticide is intercepted by the canopy and deposited on the leaves, and the other part is lost to the ground and evaporated into the atmosphere.

During pesticide application, many parameters such as wind speed, sprayer type, droplet size, and spray volume can affect the distribution of pesticides in the canopy [3,4,5]. The wind speed and canopy characteristics can significantly influence the drift potential of small droplets [3,6,7,8]. In the gap between trees [4], droplets can be directly transported into the air by the wind due to the lack of canopy interception, resulting in significant drift losses [9]. Uniform spraying at the same application rate in canopies of different volume and leaf density, which is commonly used in pesticide application, ignores differences in the target canopy’s structure and makes pesticides underused or overused [10]. This may have exacerbated the problem of off-target loss and drift.

Therefore, an accurate three-dimensional model of the plant canopy can be constructed and applied to the application process to reduce the problem of pesticide drift. With the development of sensor technology, it has become possible to use industrial sensors to detect crop canopy for pesticide variable application. Currently, ultrasound sensor, LiDAR, and RGB camera are widely used to detect the canopy features. Ultrasound sensors could detect the volume of the plant canopy [11,12,13], and some specific models of ultrasound sensors which record the echo intensity can calculate the leaf density of the canopy [14,15]. The gradual reduction in the cost of Lidar equipment has led to the widespread use of Lidar for the precision pesticides application. Real-time decision making and execution of variable spraying based on accurate detection of tree structure by LiDAR has been successfully developed and tested in field experiments [16,17,18,19,20,21].

However, the real-time decoding of LiDAR data and the dynamic complexity of the canopy volume require high-performance computing equipment, which challenges the computational power of computers for the real-time acquisition and feature extraction of canopy point clouds. Meanwhile, real-time variable systems require sensors and an onboard high-performance computer on each sprayer, which means higher financial investment and cost management. It can reduce the cost of sensors and computers on each sprayer by sharing offline prescription maps. Through installation of RTK and a variable spray control system on each sprayer, while following a pre-generated prescription map with geographic information, multiple sprayers can execute variable spraying simultaneously by obtaining 3D canopy information in advance and generating prescription maps.

The main ways to obtain 3D information are terrestrial LiDAR or Unmanned Aerial Vehicle (UAV) photography [22,23,24,25,26,27]. Terrestrial LiDAR can capture rich details in the canopy and segment tree trunks, branches, and leaves, but mapping the structure of large orchards requires multiple scans at different locations that are stitched to create a complete point cloud of the orchard [19]. Plant height analysis using UAV photography is based on the structure from motion (SfM), which reconstructs the canopy surface using multi-view aerial imagery. However, because the images are captured from the air and the lower canopy is obscured by the upper canopy, there is usually a lack of details on the lower canopy. Compared to UAV photography and SfM technology, LiDAR has higher distance resolution and is unaffected by weather or wind. By fusing LiDAR with global navigation satellite systems (GNSS), information such as digital surface models (DSM) and canopy volume can be obtained [28,29,30]. However, DSM cannot be used directly for digital orchard management, especially variable rate pesticide application.

As shown in Figure 1, in this study regarding an acquisition system through integrating LiDAR and RTK in an all-terrain vehicle (ATV) which equipped an autonomous driving system, LiDAR-RTK fusion data were used to obtain canopy point clouds with geographic information, construct a digital orchard model, and generate a 3D occupancy voxel prescription map autonomously. The effectiveness of the algorithm was verified by evaluating the measurement error of tree height and the canopy details contained in the prescription maps of different voxel sizes. It could be a reference for low-cost multi-machine cooperative precision pesticide application. In addition, the voxel prescription map includes canopy dimensions and leaf density at different locations within the orchard, allowing for precise management actions such as estimating canopy biomass and pruning. Therefore, this study has the potential to provide valuable information for accurate orchard management. The main contributions of this study are as follows:

(1) A data acquisition system based on LiDAR-RTK fusion was proposed to achieve synchronized canopy point cloud acquisition and geographic information, which could serve as a reference for precise orchard management.

(2) The optimal prescription map was established under the voxel size of 0.1 m, 0.25 m, and 0.5 m.

(3) The accuracy of the voxel prescription map was evaluated by comparing the LiDAR-measured tree height to the true tree height.

2. Materials and Methods

The experimental orchard is located in Xiying Village, Pinggu District, Beijing (40.18°N, 116.97°E), planted with pears in 2019 date (Figure 2). The trees were pruned to a high spindle architecture with a row-to-row spacing of 4.5 m and a plant-to-plant spacing of 1.5 m, and the mean tree height was about 4 m. The seasonal growth stage was BBCH 91 (shoot growth completed; foliage still fully green) [31]. The weather was clear and windless during the experiment.

2.1. Design of the LiDAR-RTK Fusion Data Acquisition System

The hardware of the LiDAR-RTK fusion data acquisition system includes a LiDAR (RS-16, Suteng Innovation Technology Co., Ltd., Shenzhen, China), an Ethernet converter, a RTK rover with autonomous driving controller (MC5, Beijing UniStrong Science and Technology Co., Ltd., Beijing, China), an industrial computer (i7-8700T, 16G RAM, and a 1T SSD), and a battery power system (Figure 3). The Lidar was installed on an aluminum frame at 45 degrees and has 16 scan lasers, a field of vertical view of 30 degrees, a vertical resolution of 2°, a horizontal resolution of 0.1°, and a ranging accuracy of ±2 cm.

In this study, an autonomous ATV was utilized to traverse pre-determined trajectories and automatically collect digitized information from orchards. Signals from the autonomous driving controller were received by the electric steering wheel to provide precise direction control, while speed was regulated via a voltage signal.

Different installation angles of the LiDAR have a significant impact on the field of view. As shown in Figure 4a, when the LiDAR is installed vertically at 90 degrees, the scan line for downward direction is largely blocked by the ATV and only can detect the ground with weeds under the trees, which makes it difficult to split the ground. Moving the vertically mounted LiDAR forward some distance can prevent the obstruction of the ATV, but the irregular motion caused by the uneven ground may cause LiDAR to shake more. When the LIDAR is installed horizontally, its field of view is only 30°. This results in LiDAR being unable to detect the entire tree at close range, while the laser lines diverge and are widely spaced at long distance, reducing detection accuracy. Therefore, the LiDAR needs to be mounted as closely as possible to the fixation point to be able to detect the entire tree at close range.

Therefore, as shown in Figure 4b, the LiDAR was fixed at 45 degrees in this study. Given the slow forward speed and the LiDAR scan speed being set at 600 rpm, the motion distortion caused by the LiDAR rotation could be ignored.

The point cloud obtained by the LiDAR was referenced to the origin of the LiDAR. However, the prescription map used for variable application must include geographic information, the sprayer then makes control decisions based on the location, and the prescription map.

GNSS could provide the position of the antenna phase center in the geodetic coordinate system. RTK is a commonly used method to improve positioning accuracy. As shown in Figure 5, the RTK system consists of a base station and a rover station, which communicate with each other via 4G network for differential data transmission. The RTK rover station with directional function has two antennas, which could not only solve the real-time position of the main antenna but also obtain the angle between the vector from the main antenna to the slave antenna and the north direction, also known as the heading angle.

In this study, custom Python (3.8.10, Python Software Foundation) scripts were used to collect binary data from the LiDAR and RTK systems. The LiDAR point cloud data were transmitted via UDP protocol over the Ethernet in a binary format. The main data stream was sent through UDP datagrams on port 6699 by default. The RTK position statements were transmitted to the computer via a serial port at a baud rate of 115,200, and the data acquisition script stored the binary data from the LiDAR along with the position information in one file.

2.2. Voxel Prescription Map Generation

2.2.1. Establish the Coordinate System

The geographic coordinate system of the LiDAR-RTK fusion data acquisition system does not match the coordinate systems of the LiDAR and RTK in the initial state (Figure 6). The right-handed coordinate system was used throughout the decoding process. To reduce the additional computation and error caused by rotation of voxels, an appropriate global coordinate system was established to make the edges of the voxel parallel to the operation trajectory.

First, one point was selected as the global origin to create an East–North–Up (ENU) coordinate system, which was rotated around the z-axis until the y-axis was parallel to the tree row direction. The rotated coordinate system was used as the global coordinate system, and the rotation angle was the tree row angle. During the scanning, the ATV was driven in the middle of the tree rows, where its direction was parallel to the row of trees. Therefore, the distribution histogram can be created by analyzing the angles between the heading and the north in the travel trajectory, and the tree row angle was calculated based on its peak.

2.2.2. Point Cloud Pre-Processing

For a better stitching result, the point cloud was pre-processed in the LiDAR coordinate system. Due to the diffusion of LiDAR laser lines, fewer feature points could be detected at longer distances, while too-close noise points should also be excluded, and, finally, only points in the range of 0.2–20 m were retained.

Random sample consensus (RANSAC) is a planar segmentation method which fits randomly sampled points to a plane through multiple iterations. In this study, the plane fitting function in Open3D was directly called. The parameters were empirically set to 50 sampling points, 10,000 iterations; a distance threshold of 0.2 m. RANSAC was used to segment the ground in each frame of the LiDAR scanned point cloud. Canopy and ground points were marked differently, and only the canopy point cloud was used to construct the prescription map.

2.2.3. Point Cloud Registration

The Point cloud stitching involved multiple coordinate systems. To align all point clouds to the global coordinate system, the homogeneous coordinate was used to simplify the computations.

After decoding, the coordinate of the points was relative to the LiDAR coordinate origin. The homogeneous coordinate $P_{lidar}$ of point $P$ in the LiDAR coordinate system is ${\left[x,y,z,1\right]}^T$.

The point was rotated around the X-axis to fix the installed angle; the rotation matrix $R_x$ is Equation (1), where α is the installed angle of −45°.

$$R_x=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& cos\alpha & -sin\alpha & 0\\ 0& sin\alpha & cos\alpha & 0\\ 0& 0& 0& 1\end{array}\right]$$

(1)

Then the point was transformed to the RTK coordinate system; the transform matrix $T_{lidar}$ is as follows:

$$T_{lidar}=\left[\begin{array}{cccc}1& 0& 0& t_x\\ 0& 1& 0& t_y\\ 0& 0& 1& t_z\\ 0& 0& 0& 1\end{array}\right]$$

(2)

where point ${\left[t_x,t_y,t_z,1\right]}^T$ was the position of the LiDAR coordinate system origin in the RTK coordinate system.

The coordinate of point $P$ in the RTK coordinate system $P_{RTK}$ is as follows:

$$P_{RTK}=T_{lidar}{R}_x{P}_{lidar}$$

(3)

The angle which was used to transform point $P$ to the global coordinate could be calculated with Equation (4):

$$\mathsf{\gamma }={\theta }_{heading}+{\theta }_{tree}-{180}^{\circ }$$

(4)

where ${\theta }_{heading}$ is the heading angle obtained by RTK rover, ${\theta }_{tree}$ is the tree angle.

The rotation matrix $R_z$ is as follows:

$$R_z=\left[\begin{array}{cccc}cos\mathsf{\gamma }& -sin\mathsf{\gamma }& 0& 0\\ sin\mathsf{\gamma }& cos\mathsf{\gamma }& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$

(5)

The transform matrix ${\mathrm{T}}_{\mathrm{RTK}}$ which was used to move the point in the global coordinate system could be calculated using the RTK rover obtained position, so the point coordinate $P_{global}$ is as follows:

$$P_{global}=T_{RTK}{R}_z{P}_{RTK}$$

(6)

The point cloud in the region area was calculated by transforming all obtained points to the global coordinate system.

2.2.4. Generating Voxel Prescription Map

Digital orchard management decisions are difficult to make in real-time at a fine level based on point clouds, especially in a variable application. Therefore, a voxel prescription map must be created to extract details from the point clouds. The data were voxelized in the global coordinate system at the selected sizes, and the coordinates of the point cloud were converted to voxel indices. The number of detected points in each voxel was counted, and voxels with less than 5 points were considered as noise and ignored. Due to the acceleration and deceleration in the scan, the amount of accumulated points in the voxels varied depending on the scan time, so only the point cloud acquired under steady driving was used to generate the prescription map.

2.3. Accuracy Analysis

2.3.1. Ground Measurements

Tree height was defined as the distance between the tallest point of the tree and the ground. In the region area, Figure 7a, the position of the tree base were measured using a Tilt-featured RTK Receiver (E500, Beijing UniStrong Science and Technology Co., Ltd., Beijing, China), and the true height of the tree was measured using a tower ruler, Figure 7b, with a height accuracy of ±5 cm. The actual positions and true heights of 20 trees were measured.

The tree height measured by the point cloud was achieved by selecting points within 0.5 m from the root coordinates of the tree and calculating the difference between the maximum and minimum heights. The true row angles were obtained by measuring the coordinates of the trees in the six rows with a minimum of 25 tree positions measured per row and fitting them to a straight line.

2.3.2. Statistical Analysis

Root mean square error (RMSE), relative root mean square error (rRMSE), and mean absolute percentage error (MAPE) were used to evaluate the error between LiDAR measurements and true values. The equations are as follows:

$$RMSE=\sqrt{\frac{{\displaystyle \sum }{\left(y_i-y_i^{\prime }\right)}^2}{n}}$$

(7)

$$rRMSE=\frac{RMSE}{\overline{y}}*100\%$$

(8)

$$MAPE=\frac{100\%}{n}{\displaystyle \sum }\left|\frac{y_i^{\prime }-y_i}{y_i}\right|$$

(9)

where $y_i$ is true values, $y_i^{\prime }$ is predicted values, $n$ is the total number of samples, $\overline{y}$ is the mean of $n$ true values.

3. Results

3.1. Measurement of Tree Row Angles

Table 1. The angle of the tree rows fitted from the measured tree position.

Actual Tree Row Angle/°						Mean	Standard Deviation
1	2	3	4	5	6
12.90	12.71	12.60	12.49	13.03	12.77	12.75	0.198

shows the angles between the tree rows and the north direction, which represented the actual tree row angles. The actual mean tree row angle was 12.75°, while the estimated angle using the histogram (Figure 8) of heading angles was 12.50°, with an absolute error of only 0.25°.

3.2. Visualization of Point Cloud and Voxel Prescription Map

Point clouds scanned from two different trajectories were registered to check for any overlap of point clouds at the same feature locations. The rows of trees were scanned from both sides (Figure 9) and were shown in red and blue, respectively. Due to the slope of the laser lines and the gaps between the trees, the LiDAR was able to scan the trees in the behind rows.

The canopy details facing the driving path were well represented in the point cloud. The branches and details of the canopy on the side away from the driving path were not abundant due to shading of the branches and leaves inside the canopy. The point clouds of the two scans at the treetops basically overlapped, indicating that the point cloud accuracy based on LiDAR-RTK fusion could meet the application requirements.

Voxelization is a method to reduce the amount of data in point cloud processing, and the key parameter is voxel size. The space required to store a voxel prescription map increases exponentially as the voxel size decreases. Although too small a voxel size preserves more crown detail, it increases the memory space and traversal lookup overhead and also takes longer to generate the prescription map. Because of the diffusion of droplets, variable sprays are difficult to perform according to very small voxel intervals. The complexity of the algorithm needs to be balanced with the accuracy of the canopy resolution, so, in this study, the prescription maps were constructed with voxel sizes of 0.1 m, 0.25 m, and 0.5 m, respectively.

As shown in Figure 10b, the prescription map with a voxel side length of 0.1 m contained abundant details. Although the prescription map was generated by limiting the minimum number of points in each cell, there were still more isolated voxels, possibly noisy points and mis-segmented ground. The prescription map with a voxel side length of 0.25 m also contained sufficient information, and the gaps between trees and the height of the trees were still clearly visible, Figure 10c. However, when the voxel size rises to 0.5 m, inter-tree gaps and branches were already difficult to detect, Figure 10d,h. Variable rate spraying determines the number of nozzles to open based primarily on the height of the tree and turns off spraying in the gaps between trees to conserve pesticides and reduce drift pollution.

3.3. Accuracy of Tree Height Measurements

The true heights of 20 trees were measured, with a mean height of 3.86 m. The actual tree heights were mainly concentrated in the range of 3.0–4.5 m, and the measurement error was larger for trees with heights less than 3.5 m (Figure 11). This may be caused by adjacent taller branches growing over the lower trees, causing measurement errors. The RMSE between the laser-measured and actual tree heights was 0.42 m, with an rRMSE of 10.86% and an MAPE of 8.16%. This proved the accuracy of the stitching and reconstruction of the LiDAR point cloud.

4. Discussion

When the LiDAR-RTK fusion data acquisition system travels through orchards, the canopy was scanned multiple times from different angles. The point cloud obtained from directly stitching the point clouds based on RTK coordinate transformation was acceptable for the digital management, such as the pesticide applications. The outer canopy closer to the LiDAR was more likely to intercept and reflect more laser because the LiDAR lasers do not penetrate the leaves. Therefore, at least two directions were required to obtain sufficient canopy detail. The tests were conducted in a flat orchard, and the point cloud stitching assumed that the LIDAR had only yaw angle and no cross-roll or pitch motion. However, during the actual scanning process, the point clouds from multiple scans may not be perfectly aligned because the ATV was bumpy due to potholes or uneven ground (Figure 12). This can be improved in the future by using an IMU for attitude correction of roll and pitch angles.

Due to the uneven ground in mountain orchards, the LIDAR attitude is deflected [32], resulting in measurement errors that increase with detection distance (Figure 13). For continuous deflection in smooth motion, where both the ground and tree point clouds appear to be deflected, using the tree height minus ground height method will still give an accurate tree height. However, for shorter target trees, if a taller branch of an adjacent tree extended above the target tree, the additional branch was considered part of the target tree, resulting in a larger height error. More statistical indicators, such as quartiles, may be used to improve the accuracy.

Compared to real-time variable-rate spray systems based on ultrasound or LiDAR, a variable spray system based on prescription maps only requires precise positioning and spray control systems, without the need for expensive sensors on each machine. Remote measurement of canopy height and volume using UAV and SFM algorithms has been applied in various fields. Changes in drone flight parameters may result in the inability to complete reconstruction using the digital surface model, leading to an RMSE of tree height measurements of up to 2.7 m [23]. However, ground-based information collection equipment is still required to obtain detailed information below the canopy [33]. Ground-based methods can provide more details than UAV-based methods [28]. Real-time simultaneous localization and mapping (SLAM) based on LiDAR has been widely researched in urban scenes, achieving satisfactory mapping results. However, in an unstructured orchard, it may be difficult to extract feature points, and the matching may not converge [34,35]. Offline computing methods have no real-time requirements and can be used to extract more features and use more complex algorithms to achieve better registration.

Additionally, real-time variable spray decision-making requires powerful computers due to the complex image and LiDAR processing. The LiDAR-RTK fusion data acquisition system developed in this study obtained voxel canopy volume data with geographic information by driving though the orchard, which can be shared with multiple machines to improve application efficiency. The constructed voxel prescription map can be used to make on–off decisions, such as turning off spraying in areas without canopy cover to reduce pesticide pollution and drift. Using Voxel R-CNN for obstacle detection has the potential to implement semantic maps [36]. The voxel prescription map also includes point-count information, which can be used to determine leaf density [19,37] for pruning [38] and pesticide application for more precise spray and airflow control. The LiDAR-RTK fusion information acquisition system is installed on an electric ATV with autonomous driving system enabling automated acquisition of orchard canopy information.

5. Conclusions

A LiDAR-RTK fusion data acquisition system was developed, which is easy to implement, and deployed on an ATV with autonomous driving system which can be used for canopy point cloud scanning autonomously. This system was tested in a pear orchard to evaluate the accuracy of the voxel prescription map. The position and height of 20 trees were measured, and the absolute error between the estimated tree row angle based on the heading angle histogram and the actual angle was 0.25°. The RMSE of the tree height measured by the LiDAR point cloud and the actual tree height was 0.42 m, with an rRMSE of 10.86% and a MAPE of 8.16%. The prescription map with a voxel side length of 0.25 m contained sufficient canopy detail information and indirectly reflected leaf density in each voxel, which made it suitable for precision management of orchards, such as precise pesticide application and recommendation for pruning operations. This system provides a demonstration for unmanned digital information acquisition in orchards.