1. Introduction
Fruit tree canopies are an important basis for the intelligent management and mechanized operation of orchards. Fruit orchard canopies’ morphological characteristics can provide scientific guidance for the precise application and fertilization of orchards [
1], the evaluation of fruit tree yield and quality [
2], and the precise management of orchards [
3]. The precise detection of fruit tree canopies is at the core of realizing variables and precise spraying. By detecting the canopy information of trees (including canopy height, width, volume, etc.), a spray decision-making model matching the canopy characteristics of fruit trees is established, and the spray parameters are dynamically adjusted according to the feedback of the model, so as to achieve accurate and variable spraying [
4,
5,
6,
7].
In recent years, researchers have applied a series of advanced sensing technologies such as near-infrared photoelectric, ultrasonic, machine vision, and laser radar to the collection and processing of information related to canopy characteristics [
8,
9,
10]. The harsh environment of the orchard, accompanied by complex and changeable temperatures, sunlight, rainfall, fog, etc., restricts the detection accuracy and effect of the above sensors. As a new type of sensor, with the innovation and development of millimeter-wave technology, millimeter-wave radar shows excellent adaptability to environmental and meteorological conditions and anti-rain, fog, and light interference ability [
11,
12]. Benefiting from these excellent performance areas, millimeter-wave radar has been widely used in research on assisted driving [
13], vital sign detection [
14], human body recognition [
15], industrial detection [
16] and other fields, laying a foundation for millimeter-wave radar to operate in agricultural environments [
17].
Due to the excellent all-weather working advantages of millimeter-wave radar, Zhao et al. proposed a new millimeter-wave radar point cloud classification algorithm, using the newly proposed point cloud feature vector; based on the SVM classifier, the classification of people and vehicles is realized in bad weather (such as rain, fog, snow, etc.) [
18]. Henry et al. used millimeter-wave radar for precise cultivation management in vineyards to achieve yield assessment [
19]. Rapha et al. developed a pelican millimeter-wave radar, specifically for use in the field of agricultural robots, to achieve strong outdoor optical environment perception [
20]. Zhang et al. designed a multi-sensor fusion detection system based on millimeter-wave radar and camera to detect the distance and height information of the ridge. The results show that the average distance detection deviation based on the fusion algorithm is 0.11 m, the distance detection standard deviation is 6.93 cm, the average height detection deviation is 0.13 m, and the height detection standard deviation is 0.19 m, which meets the real-time and accuracy requirements of autonomous agricultural machinery [
21]. Wang et al. developed an orchard autonomous navigation system based on millimeter-wave radar environment perception and map construction for intelligent plant protection operations in orchards, and the unmanned vehicles can quickly identify obstacles and make emergency stops to meet the needs of orchard operations [
22].
In the research related to orchard canopy recognition and characteristics extraction, millimeter-wave radar has broken through the problems such as low accuracy of ultrasonic detection technology and large influence of machine vision by light. For example, Zhang et al. explored the detection results of target information feature information of millimeter-wave radar under spray conditions, which provided a basis for assessing the technical feasibility of target detection under spraying conditions [
23]. Yang et al. used millimeter-wave radar to detect the canopies of fruit trees in orchards; the point cloud of fruit tree canopy was processed using machine learning algorithms such as segmentation, clustering and reconstruction, and the canopy characteristic information, such as the plant height, crown width and volume of fruit trees, was effectively extracted, which provided important support for accurate pesticide spraying decisions [
24]. However, at present, millimeter-wave radar is still in its infancy in terms of the identification and characteristic extraction of the canopies of fruit tree orchards.
This study presents research on fruit orchard canopy recognition and the extraction of the canopy characteristics of fruit trees based on millimeter-wave radar. First, regarding fruit orchard canopy recognition, building on the traditional DBSCAN algorithm, an adaptive density clustering algorithm based on the variable-axis ellipsoid model (E-DBSCAN) is proposed, and the actual field scene is tested to verify the feasibility of the algorithm. For the extraction of the canopy characteristics of fruit trees, on the basis of fruit orchard canopy recognition, the coordinate method and RANSAC algorithm are used to extract plant height and crown width, respectively, and an adaptive Alpha_shape algorithm based on point cloud density was proposed to extract volume. Finally, the point cloud quality and recognition accuracy obtained using millimeter-wave radar under spray conditions are explored by imitating the spray conditions in the actual spraying scene.
2. Materials and Methods
2.1. Canopy Detection Platform
The canopy detection platform was composed of a detection system and a mobile platform. The detection system includes millimeter-wave radar (mini 60GHz, Xi’an Dabao Electronic Technology Co., Ltd. Xi’an, China), a rotary encoder (E6B2-CWZ1X, Omron, Kyoto, Japan), an IMU sensor (ATK-IMU901, Guangzhou Xingyi electronic technology Co., Ltd. Guangzhou, China), an STM32 microcontroller (STMicroelectronics, Crolles, France) and a PC terminal. Each sensor acquires the target canopy information and the detection platform motion information and transmits them to the STM32 microcontroller in real time. The STM32 microcontroller fuses the collected data into a three-dimensional point cloud. The PC terminal controls the detection system through different command codes, and the three-dimensional point cloud is transmitted to the PC terminal through serial communication and displayed in real time. The mobile platform is a self-made four-wheeled trolley, which gives the detection system linear motion, and it enables the full-coverage scanning of the fruit tree by the sensor. The mobile platform is driven by a brushless motor, and the running speed is controlled by the pulse-width modulation signal. The rotary encoder is installed at the wheel axle of the mobile platform to determine the changing position of the millimeter-wave radar during the scanning process and provide displacement data for the three-dimensional reconstruction of the tree canopy. Millimeter-wave radar is the core part of this canopy detection platform, and it is responsible for collecting three-dimensional coordinate information of the tree canopy in real time.
As the control module of the detection system, the STM32 microcontroller is mainly responsible for analyzing the data collected by each sensor and fusing the data into a three-dimensional point cloud in a custom coordinate system. The PC terminal reconstructs the collected three-dimensional point cloud and extracts the canopy structure information. The STM32 microcontroller communicates with the millimeter-wave radar, rotary encoder and IMU sensor through four UART serial ports. The millimeter-wave radar uses the development tools provided by TI to burn the appropriate point cloud code and obtain the target canopy information through two UART serial ports. The overall design diagram is shown in
Figure 1.
When fusing multi-sensor data into a custom coordinate system, using the pose information returned by the IMU, the point cloud coordinates are corrected according to the X-axis, Y-axis and Z-axis, so as to eliminate the deviation from the real-time position of the target canopy during the operation of the millimeter-wave radar. Taking the X-axis pose correction as an example, when the millimeter-wave radar rotates the θ angle around the X-axis, then the coordinate data returned by any point (x, y, z) in the new coordinate system are corrected to (x, ycosθ − zsinθ, ysinθ + zcosθ).
The point cloud data obtained via multi-sensor fusion need to be preprocessed to eliminate non-target point clouds. First, conditional filtering is applied to define the threshold value range of the three directions of x, y and z in the space, traversing the point cloud to eliminate non-target point clouds such as adjacent row interference, outliers and sensor malfunction data. Then, the RANSAC algorithm is used for plane fitting filtering, iteratively eliminating the ground point cloud. After conditional filtering and plane fitting filtering, the preprocessing operation is performed to obtain the point cloud data for clustering and extracting canopy information.
2.2. Adaptive Density Clustering Algorithm Based on the Variable-Axis Ellipsoid Model
2.2.1. DBSCAN Clustering Algorithm
The density-based spatial clustering of applications with noise (DBSCAN) algorithm is a classical clustering algorithm based on density. The parameter of the sample point neighborhood distance threshold Eps (Epsilon) and the threshold MinPts (minimum points) of the number of point clouds in the neighborhood of the Eps size are used to describe the compactness of the point cloud distribution. Different high-density areas are divided into point cloud clusters, and arbitrary shape clustering is achieved by eliminating noise points. It is defined as follows:
(1) The
Eps neighborhood refers to all point clouds contained in a sphere with any target point
p as the center of the sphere and Eps as the radius in the point cluster
D, that is:
where
D is the point cloud dataset,
Dist(
p,
q) is the distance between point
p and point
q, and
NEps(
p) is the set of all points in the point cloud dataset
D whose distance from point
p is smaller than
Eps.
(2) For any point
p∈
D, if |
NEps(
p)| ≥
MinPts, then it is the core point. When the point
q does not meet the core point determination condition but belongs to the core object
NEsp(
p), the point
p is considered as the boundary point. That is:
(3) In dataset D, all points except core points and boundary points are called noise points. If point b is in the Eps neighborhood of the core point a, then point b is said to be density-direct from point a. If there is a point cloud that conforms to p1, p2, …, pn∈D, for any pi∈D, if pi+1 is density-direct from pi, then pn is said to be density-reachable from p1. If there is a point c∈D, so that point a and point b are density-reachable from point c, then point a and point b are said to be density-connected.
The algorithm implementation is as follows:
(1) Input point cloud dataset D and set parameters Eps and MinPts.
(2) Randomly select the unlabeled point p from D.
(3) Determine all the density-reachable points with point p under Eps and MinPts conditions.
(4) If p is the core point, then create a new cluster and assign cluster labels to its inner points.
(5) If p is a boundary point and there is no density-reachable point, the algorithm turns to the next point in D and repeats the above steps until all points in D are processed.
It is worth mentioning that the algorithm needs to manually specify the parameters of Eps and MinPts before clustering, and the parameters directly determine the clustering effect. If the initial parameters are improperly selected, the DBSCAN algorithm is prone to over-segmenting or under-segmenting the point cloud with uneven density distribution, which affects the clustering accuracy.
2.2.2. E-DBSCAN Clustering Algorithm
Based on the characteristics of millimeter-wave radar parameters and the traditional DBSCAN clustering algorithm, a self-adaptive DBSCAN clustering algorithm based on variable-axis ellipsoid model (E-DBSCAN clustering algorithm) is proposed. The E-DBSCAN algorithm shows higher accuracy and adaptability in tree canopy point cloud clustering and segmentation, thus improving the clustering performance [
25]. The ellipsoid model is established for the points in the point cloud data set. The ellipsoid takes any point
A (
x1,
y1,
z1) as the origin,
hz as the major semiaxis,
hx as the middle semiaxis, and
hy as the short semiaxis. The horizontal angular resolution of the millimeter-wave radar is denoted as
θx, the vertical angular resolution as
θz, and the detection range resolution as
hy. The ellipsoid equation is expressed as:
where
hx and
hz are the semi-major axis and semi-middle axis of the ellipsoid, respectively,
hx = dtan
θx,
hz = dtan
θz,
d is the distance from point
A to the millimeter-wave radar, and
d = (
x12 +
y12 +
z12)
1/2.
In the clustering operation, after determining the ellipsoid model, it is necessary to determine the threshold of neighborhood density. For any target point P (x, y, z) in the target canopy, from the first detection of the target by the millimeter-wave radar to the last detection of the target, the forward driving distance of the sprayer is L, and the total number of detections of the target point during this period is the point number threshold T in the neighborhood of the ellipsoid model.
The algorithm implementation as follows:
(1) Input point cloud dataset D and randomly select a point xn∈D as the initial point.
(2) Construct the neighborhood point cloud set N∈(xn) of point xn in its ellipsoid model.
(3) If |N∈(xn)| ≥ T, then mark as the core point and start clustering to form a new point cloud cluster Cn.
(4) Complete the current point cloud clustering, select a new unvisited point and repeat the above steps.
(5) After traversing all points, all clusters C = (C1, C2, …, Cn) are output, and each cluster represents a canopy point cloud of the fruit tree target.
2.3. Canopy Characteristic Extraction Algorithm
Using the fruit orchard canopy point cloud data extracted using E-DBSCAN in
Section 2.2, the canopy point cloud is projected on the
XOY plane, and the RANSAC algorithm is used to fit the canopy point cloud to obtain the canopy width of the fruit tree. The plant height of the fruit tree is obtained by adding the installation height of the millimeter-wave radar and the coordinate value
z of the projection of the canopy point cloud on the
YOZ plane.
In this study, a point cloud density adaptive Alpha_shape algorithm was proposed, which improved the parameter of alpha based on the Alpha_shape algorithm. The core idea was to calculate the average distance of the k-nearest neighbors of each point, the distance was used as the reference value of alpha, and the dynamic alpha value was used in the three-dimensional reconstruction process. Until the point cloud is traversed, the rolling trace is the point cloud contour surface, and the boundary is extracted from the discrete point cloud to reconstruct the three-dimensional surface. After the three-dimensional reconstruction, the three-dimensional alpha shape is obtained. The plot3() function was used to draw the alpha shape in Matlab R2018a software, and the volume() function was used to obtain the volume of the alpha shape. Due to the uneven density of millimeter-wave radar point cloud, when the alpha value is selected too large, it can only extract the convex envelope structure of the point cloud, resulting in a large volume calculation value. When the alpha value is too small, the rolling circle will fall into the interior of the point cloud, resulting in a systematic error in the volume calculation value.
2.4. Fruit Orchard Canopy Recognition Experiment
The experiment site was located in the plum orchard of South China Agricultural University, Guangdong Province. It is a small arbor orchard managed by artificial planting with flat terrain and low weeds on the ground. On 19 December 2021, thirty-one plum trees were selected for fruit orchard canopy recognition research; the canopy was lush and spindle-shaped, with plant height between 2 and 3 m and plant spacing of 3.2 m. The real scene of the test area is shown in
Figure 2. The red arrow in
Figure 2b indicates the direction of the trajectory of the detection platform, and the operating speed is about 1 m/s. The detection system only carries out unilateral rapid scanning and relies on the penetration ability of millimeter-wave radar to quickly achieve the fruit orchard segmentation of canopy point clouds in a large range.
In order to evaluate the accuracy of fruit orchard canopy recognition, the precision rate
P, recall rate
R and
F1 score in statistics were introduced. The calculation formula is as follows:
where
TP,
FP and
FN represent the number of fruit trees correctly detected, wrongly identified, and undetected, respectively. The minimum value is 0. The higher the
F1 score is, the better the recognition result is.
F1 refers to the harmonic average of precision and recall. The maximum value is 1 and the minimum value is 0. The higher the
F1 is, the better the recognition result is.
2.5. Canopy Characteristics Extraction Experiment
In order to simulate the working environment of millimeter-wave radar in orchard spraying, the extraction of canopy characteristics under natural conditions and spraying conditions was carried out on 19 December 2021. The test objects were 7 artificial trees of different shapes, as shown in
Figure 3. The artificial trees were arranged in a straight line, the canopy spacing was 1 m, and the detection platform scanned the trees at a speed of 1 m/s. The millimeter-wave radar of the detection platform was 1.5 m away from the tree, and the detection range of the canopy was conical. The horizontal angle range was set at 120°, the vertical angle range was set at 50°, and the data refresh rate was 10 Hz. Under the spray condition, the vp110-001 nozzle (Spraying Systems Co., Wheaton, Illinois, USA) was selected, the spray volume was 0.39 L/min, and the atomization degree was 240 μm. The nozzle was placed under the millimeter-wave radar to ensure that the spray area covered the detection range of the millimeter-wave radar, so that it had to pass through the spray during scanning. To reduce the difference in the number of point clouds caused by the scanning time, the starting point was set to be 2 m from the artificial tree, and the end point was 2 m beyond the artificial tree to ensure the same scanning time for the canopy.
In order to verify the difference between the extracted values of the canopy characteristics under two conditions, the mean absolute error
MAE and the mean relative error
MRE of the extracted values of plant height, crown width, and volume were calculated to evaluate the accuracy of fruit orchard canopy characteristic extraction. The
MAE and
MRE of the extracted values were defined as follows:
where
EV is the parameter extraction value under normal conditions,
EV’ is the parameter extraction value under spray conditions, and
i is the tree number.
3. Results and Discussion
3.1. Fruit Orchard Canopy Recognition Results
The traditional DBSCAN algorithm and the improved E-DBSCAN algorithm were used to process the collected point clouds, and the fruit orchard canopy recognition results are shown in
Figure 4 and
Figure 5. When using the DBSCAN algorithm, it is necessary to find the appropriate global variables. First, the real average crown width of fruit trees is taken as the initial Eps, and the number of radar data refreshes during the scanning time was taken as the initial
MinPts. Then, the program is run, the global parameters are adjusted according to the calculation results, and the optimal solution of the global parameters is obtained. In
Figure 4, the clustering results of the DBSCAN algorithm under different parameters are presented.
Figure 5 shows the clustering results using the E-DBSCAN algorithm.
The performance of the traditional DBSCAN algorithm and the improved E-DBSCAN algorithm for fruit orchard canopy recognition is shown in
Table 1. When
Eps was 0.9 and
MinPts was 10, there was an under-segmentation phenomenon: a large number of point clouds were segmented into the same canopy, and fewer canopies were correctly identified, which directly leads to the lowest
F1 score. However, when
Eps was 0.4 and
MinPts was 40, there was an over-segmentation phenomenon; that is, the given global variable standard is high, and only a few point clouds are defined as core points, resulting in the loss of a large number of point clouds. We were unable to extract effective data from the identified fruit orchard canopy point clouds, but, due to the small number of core points, the computing speed was improved. When
Eps was 0.5 and
MinPts was 30, many fruit trees were correctly identified, and the canopy point cloud of the fruit trees was completely preserved, but there are still many fruit trees missing. This is already the best performance achieved by DBSCAN algorithm, but its F1 score was still lower than that of the improved E-DBSCAN algorithm. The recognition result of the E-DBSCAN algorithm shows the best performance: only fruit tree 13 showed an over-segmentation phenomenon, and fruit trees 27 and 31 could not be accurately recognized due to their low canopy density.
Furthermore, we conducted a comparative study with the classic K-means clustering algorithm. The K-means clustering algorithm had the shortest computation time, only 1.36 s, but its F1 score were all relatively low: lower than the DBSCAN algorithm and the adaptive E-DBSCAN algorithm when using the optimal global parameters. At the same time, it needs to combine the prior knowledge to set the K value and cannot remove the non-canopy point cloud in the point cloud, which will affect the extraction of canopy parameters.
In summary, due to its adaptability, after the algorithm proposed here integrates the ellipsoid model, although it increases a certain amount of operation time, it does not need to search for global parameters many times during use. Therefore, for real-time spray decision making in orchards, this algorithm in more efficient and saves more time. Meanwhile, among the above operation results, the E-DBSCAN algorithm has the highest F1 score of 96.7%, of which the precision rate was 93.5% and the recall rate was 95.1%.
3.2. Fruit Tree Canopy Characteristic Extraction Results
The original point cloud of the canopy was visualized in Matlab.
Figure 6 shows the projection of the canopy point cloud of artificial trees 1–7 on the
YOZ surface from left to right. It can be inferred from the figure that the overall number and density of canopy point clouds increase under the spray condition. There was no serious divergence in the canopy point cloud of the fruit trees, and the coverage area on the projection surface did not change significantly. However, there was a certain increase in noise between different canopies, such as the artificial trees No.1 and No.2, and No.4 and No.5.
The improved E-DBSCAN algorithm was used for fruit tree canopy recognition to determine whether the generated noise had an impact on the recognition. As shown in
Figure 7, the E-DBSCAN algorithm accurately recognized the canopy point cloud under spraying conditions and natural conditions, indicating that the noise generated under spraying conditions does not influence the recognition performance for fruit trees. After removing the noise points and the sparse point clouds around the canopy, the same artificial tree displayed some similarities in morphology. Under spraying conditions, due to the increase in the number and density of point clouds, it shows a fuller shape.
In order to further observe the influence of spraying on the reconstruction of canopy morphology, the Alpha_shape algorithm was used to conduct the three-dimensional reconstruction of the canopy point cloud of artificial tree no. 2, collected under spraying conditions and natural conditions. The canopy was fitted with three Alpha values of 0.5, 0.3 and 0.1. The performance was shown in
Figure 8.
The figure shows that, in the fitting of the three Alpha values, the canopy morphology under spraying conditions was always better than that under natural conditions. With the decrease in the Alpha value, the volume of the three-dimensional model becomes smaller and smaller, and the characteristics of the canopy surface become more obvious. When Alpha was 0.3, the model shape of the two condition was most similar and the error was lowest. When Alpha was 0.1, due to the number and density of point clouds, the three-dimensional model of the canopy under spraying conditions was more refined, while the three-dimensional model under natural conditions was broken and the canopy volume was lost to a significant degree.
To further explore the quality of point clouds under spraying conditions, the plant height, crown width and volume values of the artificial tree canopy under spraying condition and natural condition were extracted. The volume values under the two conditions were calculated when Alpha was 0.3, and the number of point clouds under the two conditions was calculated at the same time. As shown in
Figure 9, we determined that the number of point clouds under spraying conditions increased significantly. During the spraying process, the nozzle continued to release pressure, and the air flow caused the branches and leaves of the artificial tree to shake. Since the millimeter-wave radar is highly sensitive to moving objects, it is easier for the millimeter-wave radar to obtain the point clouds of the branches and leaves under this condition, resulting in an increase in the number of point clouds. The extracted values of plant height and crown width were the same under the two conditions, but there was a certain error, and the relative value of the extracted values of the two conditions does not follow any rules. For example, the extracted value of plant height under the spraying condition for artificial tree no. 3 was bigger than that under the natural condition, and the extracted value of artificial tree no. 5 was the opposite. The volume extraction value under spraying conditions was always larger than that under natural conditions. The main reasons for this can be summarized as follows: the volume value of the canopy increases due to the shaking of branches and leaves under spray conditions. Spray droplets cause point clouds to disperse in a small area. The point cloud density changes and, when Alpha is constant, low-density canopy point cloud fitting is more prone to canopy loss, resulting in lower volume calculation results.
In order to further verify the difference between the extracted values of canopy characteristics under the two conditions, the mean absolute error and mean relative error of the extracted values of plant height, crown width and volume were calculated, and the statistical results are shown in
Table 2. The average relative errors of plant height, crown width and volume under the two conditions are 2.3%, 2.1%, and 4.2%, respectively. The errors of plant height and crown width mainly come from random errors in different scanning processes, while the volume value is due to various factors that lead to higher calculation results under spray conditions. There is a slight difference between the volume extraction value and that under normal conditions, but, on the whole, the difference is small compared with that under natural conditions. Therefore, it can be concluded that the spraying conditions have no significant influence on the extraction of canopy characteristics using millimeter-wave radar, which can provide spray decision-making information for precision spraying.
4. Conclusions
In this study, fruit orchard canopy recognition and characteristic extraction based on millimeter-wave radar were investigated. Combining the working principle and the characteristics of the point cloud of millimeter-wave radar, based on the traditional DBSCAN algorithm, the E-DBSCAN clustering algorithm was proposed. After integrating the ellipsoid model, the algorithm has adaptability and does not need to search for global parameters many times during use. The field experiment results show that the F1 score of the proposed E-DBSCAN algorithm was the highest, reaching 96.7%, with a precision rate of 93.5% and a recall rate of 95.1%. The feasibility of the proposed algorithm was confirmed.
Based on the results of fruit orchard canopy recognition, a point cloud density adaptive Alpha_shape algorithm was proposed, and the accuracy of canopy characteristic extraction under natural and spraying conditions was explored. Based on the millimeter-wave radar canopy detection platform, a spray system was built to simulate the orchard spray environment to obtain the canopy point cloud. The experiment results showed that there was no significant difference in the extraction results of the plant height, crown width and volume of fruit tree canopy under natural conditions and spray conditions. The average relative errors were only 2.3%, 2.1%, and 4.2%, indicating that millimeter-wave radar can realize the recognition and characteristic extraction of fruit tree canopies under spray conditions, providing technical support and a decision-making basis for target spray operations.