Next Article in Journal
Building Dynamic Communities of Interest for Internet of Things in Smart Cities
Next Article in Special Issue
Mapping of Agricultural Subsurface Drainage Systems Using a Frequency-Domain Ground Penetrating Radar and Evaluating Its Performance Using a Single-Frequency Multi-Receiver Electromagnetic Induction Instrument
Previous Article in Journal
Intact Detection of Highly Occluded Immature Tomatoes on Plants Using Deep Learning Techniques
Previous Article in Special Issue
Monitoring Wheat Growth Using a Portable Three-Band Instrument for Crop Growth Monitoring and Diagnosis
Open AccessArticle

A Canopy Information Measurement Method for Modern Standardized Apple Orchards Based on UAV Multimodal Information

1
College of Engineering, Nanjing Agricultural University, Nanjing 210031, China
2
Jiangsu Province Engineering Lab for Modern Intelligent Facilities of Agriculture Technology & Equipment, Nanjing 210031, China
*
Author to whom correspondence should be addressed.
Sensors 2020, 20(10), 2985; https://doi.org/10.3390/s20102985
Received: 3 May 2020 / Revised: 22 May 2020 / Accepted: 22 May 2020 / Published: 25 May 2020
(This article belongs to the Special Issue Sensors in Agriculture 2020)

Abstract

To make canopy information measurements in modern standardized apple orchards, a method for canopy information measurements based on unmanned aerial vehicle (UAV) multimodal information is proposed. Using a modern standardized apple orchard as the study object, a visual imaging system on a quadrotor UAV was used to collect canopy images in the apple orchard, and three-dimensional (3D) point-cloud models and vegetation index images of the orchard were generated with Pix4Dmapper software. A row and column detection method based on grayscale projection in orchard index images (RCGP) is proposed. Morphological information measurements of fruit tree canopies based on 3D point-cloud models are established, and a yield prediction model for fruit trees based on the UAV multimodal information is derived. The results are as follows: (1) When the ground sampling distance (GSD) was 2.13–6.69 cm/px, the accuracy of row detection in the orchard using the RCGP method was 100.00%. (2) With RCGP, the average accuracy of column detection based on grayscale images of the normalized green (NG) index was 98.71–100.00%. The hand-measured values of H, SXOY, and V of the fruit tree canopy were compared with those obtained with the UAV. The results showed that the coefficient of determination R2 was the most significant, which was 0.94, 0.94, and 0.91, respectively, and the relative average deviation (RADavg) was minimal, which was 1.72%, 4.33%, and 7.90%, respectively, when the GSD was 2.13 cm/px. Yield prediction was modeled by the back-propagation artificial neural network prediction model using the color and textural characteristic values of fruit tree vegetation indices and the morphological characteristic values of point-cloud models. The R2 value between the predicted yield values and the measured values was 0.83–0.88, and the RAD value was 8.05–9.76%. These results show that the UAV-based canopy information measurement method in apple orchards proposed in this study can be applied to the remote evaluation of canopy 3D morphological information and can yield information about modern standardized orchards, thereby improving the level of orchard informatization. This method is thus valuable for the production management of modern standardized orchards.
Keywords: orchard; apple tree; canopy information; unmanned aerial vehicle (UAV); multispectral; multimodal orchard; apple tree; canopy information; unmanned aerial vehicle (UAV); multispectral; multimodal

1. Introduction

The canopy is the first part of the fruit tree exposed to light and the external environment and is the main place for photosynthesis and respiration. The shape, structure, volume, leaf area index, vegetation index, and other information of the fruit tree canopy not only reflect the growth status and yield potential of fruit trees [1] but also are important for decision-making about orchard canopy pruning [2,3], irrigation and fertilization [4], precision spraying [5], flowering management [6], and other key production steps [7].
Traditional fruit tree canopy information mainly relies on manual measurement methods based on previous experience. Based on field sampling data of orchard canopy information, survey results are extrapolated to the management scale and are used for reference [8]. Such traditional measurement and research methods have played an important role for quite a long time. With the development of intensive agriculture, traditional manual measurement methods have come to exhibit limitations in the rapid and accurate measurement of fruit tree canopy parameters. A low level of orchard informatization leads to problems such as low orchard production efficiency, low level of intelligent orchard management, and heavy pollution due to the large amounts of water, fertilizers, and pesticides/herbicides used. These shortcomings seriously restrict the development of precision production management in modern standardized orchards. It is urgently required to study canopy information measurement technologies in modern standardized orchards to lay a good foundation for the production management of modern standardized orchards.
With the development of sensor technology, the combination of unmanned aerial vehicles (UAVs) and visual imaging technology [9] allows the realization of nondestructive measurement of canopy information across large orchards, which can enable scientific and reliable decision-making for orchard production management. Hence, studying this technology will be key to the development of modern standardized orchards.
The main methods of measuring canopy information in large, modern standardized orchards are vehicle-mounted measurement systems [1,10] and UAV measurement systems [2,3,11]. A measurement system integrates ultrasound, a digital camera, a stereo camera, a time-of-flight (TOF) camera, a multispectral camera, a hyperspectral camera, two-dimensional (2D) light detection and ranging (LIDAR), three-dimensional (3D) LIDAR, high-resolution X-rays, and other technologies [7] to gather information about the orchard canopy 3D geometry, physiological indicators, and yield. Vehicle-mounted measurement systems have better measurement accuracy than UAV measurement systems, but they have lower measurement efficiency and a higher measurement cost than UAV measurement systems. Vehicle-mounted measurement systems are more suitable for the real-time orchard management practices, such as canopy target spraying [12], canopy pruning [2,3], and fruit positioning and picking [13], whereas the UAV measurement system is more suitable for information evaluation in orchard management, such as about canopy geometric morphology [14], biomass [15], diseases [16], water stress [17], vegetation indices [18], and yield [19,20].
At present, UAV visual imaging systems allow the establishment of multimodal information, such as orthophotographic maps, vegetation index images, digital surface models (DSMs), and 3D point-cloud models in large orchards, which have laid a good foundation for canopy information measurements in orchards. Scholars in this field have carried out a large body of research on two aspects: orchard canopy information extraction methods and orchard management. Studying UAV-based orchard canopy information extraction methods, Zarco-Tejada et al. [14] used orthorectification and DSMs of olive orchards to remotely evaluate the height of fruit trees. Koc-San et al. [11] used a UAV to collect citrus orchard images and proposed an automatic extraction method based on the circular Hough transform model for images and DSMs in citrus orchards. Comba et al. [21] used a UAV to collect 3D point-cloud models of orchards and proposed a method for 3D point-cloud model characteristic value extraction in hilly-terrain orchards. UAV measurement systems can also be used for orchard management, such as fruit tree canopy pruning, which can increase canopy light interception and canopy surface area and promote new growth of fruit trees, making fruits more readily pickable and possibly increasing the yield of the trees. To evaluate the effect of pruning on litchi orchards, Johansen et al. [3] used UAV to measure tree structure, such as canopy circumference, width, height, area, and plant projection coverage. Additionally, early diagnosis of orchard diseases and insect pests can improve the control effect of pests and diseases in orchards and can improve the quality and yield of fruits. To this end, through a UAV, Garcia-Ruiz et al. [16] collected multispectral canopy images in citrus orchards and established a diagnostic model of Huanglongbing (HLB) disease (citrus greening) in citrus orchards, which allowed them to classify fruit trees by HLB disease status. Through a UAV visual system, Park et al. [22] collected high-resolution thermal images of a peach orchard and established a diagnostic model for fruit tree water stress to guide the orchard irrigation management. As a UAV plant protection sprayer can improve the accuracy of target spraying and reduce the application amount, Cheng et al. [5] proposed an automatic segmentation method for canopy images in orchards and established an accurate spraying method based on canopy area. Meanwhile, UAV measurement technology has also been applied in a large body of research in field crops or plants, such as on cotton yield measurement [19], corn height measurement [23], barley biomass measurement [15], wheat height measurement [24], sorghum growth status measurement [25], sunflower yield measurement [26], field weed detection [27], and rapeseed yield measurement [20]. UAV measurement technology is highly efficient and nondestructive and is suitable for large-scale measurement of information about orchards and field crops. Compared with the ground measurement method [28], this technology is more suitable for applications in large, modern precision agricultural production management.
As modern standardized orchards must accommodate agricultural machinery and equipment, they need a large enough row spacing and plant spacing. A typical characteristic of modern standardized orchards is neat rows and columns, yet the presence of a few small trees or dead trees between fruit trees could affect the benefit of canopy segmentation. In this study, taking an apple orchard as the study object, a UAV visual imaging system was used to generate 3D point-cloud models and vegetation indices of the apple orchard. Automatic row and column detection methods were established based on vegetation index images, thereby achieving rapid segmentation. Based on the fruit tree point-cloud models, measurement methods for height, projection area, and projection volume, as well as yield prediction models, were established to achieve a remote evaluation of orchard canopy information. This paper provides a theoretical basis and technical reference for canopy information measurement in modern standardized orchards and will be helpful in orchard management.

2. Materials and Methods

2.1. UAV Measurement System for Canopy Information of Orchards

The UAV measurement system for modern standardized apple orchards mainly consisted of a Parrot Bluegrass UAV, Parrot Sequoia multispectral cameras, Pix4Dcapture flight control software, Pix4Dmapper postprocessing software, and a graphics workstation. The main technical parameters of the Parrot Bluegrass UAV were as follows: maximal take-off mass (MTOM) is 1.85 kg, battery life of 25 min, Parrot Skycontroller 2 with a control range of 2 km, and a vertical camera equipped with a built-in global positioning system (GPS), global navigation satellite system (GLONASS), altimeter, ultrasonic sensor, and optical flow vertical camera. The main technical parameters of the Parrot Sequoia multispectral camera are as follows: an RGB camera with the resolution of 16 megapixels (4608 × 3456 px); four global-shutter spectral-channel single-band cameras, including green (GRE, 550 ± 40 nm), red (RED, 660 ± 40 nm), red-edge (REG, 735 ± 10 nm), and near-infrared (NIR, 790 ± 40 nm), each with the resolution of 1.2 megapixels (1280 × 960 px) and the frame rate of 1 frame/s, it has four environmental light sensors, these sensors have the same band-pass filter as four single-band cameras, and pictures could be calibrated automatically according to the light conditions. The main functions of the Pix4Dcapture flight control software were as follows: UAV flight map selection, flight path planning, flight altitude, flight speed, horizontal coverage, vertical coverage, and other parameter settings, along with the display function of real-time status information parameters, such as the UAV battery, Parrot Skycontroller battery, memory, GPS, and operation processes. The main functions of the Pix4Dmapper postprocessing software were generating 3D point-cloud models, orthographic images, and DSM and vegetation index images based on the images collected by the UAV. The main technical parameters of the graphics workstation (HP Zhan 99 by Hewlett-Packard Corporation, Beijing, China) were as follows: a 6-core 12-thread Intel (R) Xeon (R) E-2176 M CPU @ 2.70 GHz, 32 GB of RAM, Intel (R) UHD Graphics P630 GPU and NVIDIA Quadro P600 4G graphics card. The operating system was Windows 10 Professional Workstation Edition, 64 bit. The software for calculating the 3D morphological information of the fruit tree canopy was Matlab 2017a (MathWorks Corporation, Natick, MA, USA).
The workflow of the UAV measurement system for canopy information in a modern standardized apple orchard is as follows: The first step was image acquisition. According to the selected orchard measurement area, Pix4Dcapture created a flight (including flight map boundary selection, flight path planning, flight altitude, flight speed, horizontal coverage, vertical coverage, and other parameter settings), the Parrot Bluegrass performed this flight, and the Parrot Sequoia multispectral camera collected multispectral images at each planned waypoint. The second step was the point-cloud output. Pix4Dmapper was used to read images collected by the flights, and according to the GPS information (longitude, latitude, altitude) of each waypoint, Pix4Dmapper carried out processing procedures, such as computing keypoints, computing matches, calibration, matching, and point-cloud densification, and finally output the PLY file of the point-cloud model of the orchard. The third step was the vegetation index output. According to GRE, RED, REG, and NIR orchard canopy orthophotos, Pix4Dmapper was used to generate various vegetation index images (such as normalized difference vegetation index (NDVI), normalized green (NG), green chlorophyll index (CIG), and renormalized DVI (RDVI)), and the orchard vegetation index images were output in JPG format (the type of vegetation index was set manually according to the actual situation). The fourth step was a row and column detection in the orchard canopy. The orchard point-cloud models and vegetation index images were read in Matlab, and the grayscale projection method based on 2D vegetation index images was used to realize the automatic row and column detection and segmentation of 2D/3D orchard canopy images. The point-cloud model of each fruit tree was saved in a 2D cell array. The fifth step was to calculate canopy information. According to the point-cloud model of each fruit tree, various morphological characteristic parameters were calculated, such as canopy height, canopy projection area, and circumscribed volume of the canopy. According to the vegetation index of each fruit tree, various color characteristic parameters were calculated, such as the first-, second-, and third-order color moments. Textural characteristic parameters were calculated, such as contrast, cross-correlation, energy, and inverse differential moment. The sixth step was fruit yield prediction and analysis. With the fruit tree morphological characteristic parameters, color characteristic parameters, and textural characteristic parameters as inputs, the back-propagation artificial neural network (BPANN) prediction model performed fruit yield prediction, and the UAV measurement accuracy of the morphology and yield information of the fruit trees was evaluated and analyzed.

2.2. UAV Measurement Tests for Gathering Orchard Canopy Information

2.2.1. Measurement of Multimodal Information about the Orchard Canopy

In this study, the apple orchard test site for canopy multimodal information measurement was in Yaojiazhuang, Guandao Town, Shandong Province, China (E 120.6352°, N 37.1716°). The measurement test time was from September 15, 2019 to September 25, 2019, when apple trees are at the mature stage. The variety in the apple orchard was Yanfu 3, and the tree age in the measured area was 8–9 years. The measured area was 140 m × 25 m (length × width), including five rows of fruit trees; the row spacing of the fruit trees was 5 m, and; the plant spacing was 4 m. There were few dead trees or young plants in the measured area of this orchard. Because there was plenty of sunlight between 10:00 a.m. and 2:00 p.m. on a sunny day, the impact of tree shadow on image processing was relatively small, so the UAV flight time was between 10:00 and 14:00. The parameters of the Parrot Bluegrass flights are shown in Table 1. The respective flight heights were specified as 22, 32, 43, 53, and 64 m, and the ground sampling distances (GSDs) were 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px. The flight speed was specified as Normal, with the averages of 2.6, 3.7, 4.9, 6.1, and 7.3 m/s. The expected flying time was 5 min 52 s, 2 min 50 s, 1 min 46 s, 1 min 32 s, and 1 min 20 s, respectively. The image front overlap was specified as 80%, and the image side overlap was 80%. The camera was a Parrot Sequoia, which contained a 16-megapixel RGB camera and four 1.2-megapixel spectral channel single-band cameras with a global shutter. The ground control adopted a Parrot Skycontroller 2 controller and Pix4Dcapture flight control software. During the study period, the sky was clear with very few clouds and a wind speed of less than 8 m/s, which met the requirements for remote sensing by UAV.

2.2.2. Preprocessing of Multimodal Information of Orchard Canopy

In this study, Pix4Dmapper software was used to generate 3D point-cloud models of the orchard and vegetation index orthographic images with GRE, RED, REG, and NIR orchard canopy images collected in the measured area. The main process included reading UAV measurement images, specifying coordinate systems, initial processing, and generation of point-clouds, textures, orthophotos, and indices. Pix4Dmapper parameter settings are shown in Table 2. Based on the structure-from-motion (SfM) principle [29], Pix4Dmapper carries out computing keypoints, computing matches, calibration, matching, and point-cloud densification on images, and it finally outputs the PLY file of the orchard point-cloud model, orthographic images, and various JPG files of vegetation index images.
Figure 1a shows an image of the 3D point-cloud model from the top view (XOY perspective) of the measured area of the apple orchard. Figure 1b shows the 3D point-cloud model from the side view (YOZ perspective) of the measured area in the apple orchard. Figure 1c–l, respectively, show the NG, normalized red (NR), NDVI, green NDVI (GNDVI), CIG, DVI, optimized soil-adjusted VI (OSAVI), RDVI, nonlinear VI (NLI), and wide-dynamic-range VI (WDRVI) images of the measured area in the apple orchard (grayscale of vegetation index images, convert the index value to 0–255). The formulas to calculate the vegetation indices are shown in Table 3.

2.3. Row and Column Detection Methods of the Orchard Canopy

In this study, the row and column detection method of the orchard was proposed: the row and column detection method based on grayscale projection in orchard index images (RCGP). The PLY data of a 3D point-cloud model of the orchard consist of several 3D coordinates, while the JPG data of a vegetation index is a 2D image. The data formats of the two are thus inconsistent. To achieve unified row and column segmentation across the two types of data, it is necessary to perform a conversion of segmentation positions. In this study, the segmentation performance of the RCGP was analyzed. The RCGP based on grayscale projection in orchard index images was selected, and through the row and column segmentation position conversion, the unified segmentation of the orchard multimodal data was achieved.
In the first step, a vegetation index image was grayscale processed, and the cumulative grayscale value for each row was calculated using Equation (1), while the cumulative grayscale value for each column was calculated using Equation (2). In the second step, the cumulative grayscale values for the rows were searched to find the local peaks by probability density estimation, as shown in Equation (3), based on the principle of kernel density estimation, which were taken as the row segmentation positions and were saved to the array L2D. In the third step, the cumulative grayscale values for the columns were searched to find the local peaks, which were taken as the column segmentation positions and were saved to the cell array C2D{i}:
G L ( i ) = j = 1 n I ( i , j ) ,
G C ( j ) = i = 1 m I ( i , j ) ,
f ^ h ( G L ) = 1 n h i = 1 n K ( G L G L ( i ) h ) ,
where I is the grayscale image of the vegetation index; (i, j) are the image coordinates; m is the number of image rows; n is the number of image columns; GL(i) is the cumulative grayscale value of the ith row; and GC(i) is the cumulative grayscale value of the jth column; f ^ h ( G L ) is the probability density value of GL; h is the bandwidth, and; K is the kernel function.
As shown in Equations (4) and (5), the row segmentation position obtained by the grayscale projection method was converted into a 3D point-cloud model, which could adopt the same row and column segmentation position, thereby realizing the unified row and column segmentation positions with the multimodal data of the orchard. Conversely, the row and column segmentation position obtained by kernel density estimation from the 3D point-cloud model was also converted into vegetation index images:
L 3 D ( i ) = | L 2 D ( i ) | × m 3 D m 2 D ,
C 3 D { i } ( j ) = | C 2 D { i } ( j ) | × n 3 D n 2 D ,
where L2D is the array of row segmentation positions obtained by the grayscale projection method; i is the row segmentation position number; m2D is the number of rows in a vegetation index image, with the unit of px; m3D is the Y-axis length of the 3D point-cloud model, with the unit of m; L3D is the array of row segmentation positions (converted values) of the 3D point-cloud model; C2D{i} (j) is the array of column segmentation positions obtained by the grayscale projection method; j is the column segmentation position number; n2D is the number of columns of the vegetation index image, with the unit of px; n3D is the X-axis length of the 3D point-cloud model, with the unit of m; and C3D{i} (j) is the array of column segmentation positions (converted values) of the 3D point-cloud model.

2.4. Methods for Characteristic Value Extraction for Multimodal Data of the Orchard Canopy

To measure orchard canopy information, the morphological characteristic values of fruit trees were extracted through the 3D point-cloud model of the orchard, and the canopy color and textural characteristic values were extracted through the vegetation indices of the orchard.

2.4.1. Calculation Method for Morphological Characteristic Values of Fruit Trees

In this study, the morphological characteristic values of fruit trees extracted mainly included height H, projected area SXOY, and volume V. The height H is the canopy height of an apple tree, and the maximum vertical height of the Z-axis in a point-cloud model is the canopy height of the highest fruit tree, as shown in Equation (6). The canopy shape of fruit trees is complex, so it is difficult to accurately calculate the canopy volume. There exists a large error in the calculation of the volume using the enclosed volume outside the 3D grids. This is mainly because though the orchard point-cloud models generated by the UAV measurement system have dense point-clouds in the XOY perspective, the point-clouds in the YOZ and XOZ perspectives and other lateral perspectives are relatively sparse. There are also other influencing factors, such as noise interference. In this study, based on the shapes of fruit trees in the measured area, the volume of the canopy was approximated using an outer enclosed ellipsoid circumscribing the canopy. As the point-cloud projection from the XOY perspective had the characteristic of rotation invariance, the point-cloud projection boundary of the XOY perspective was used to calculate the surrounding area SXOY, as shown in Equation (7). Given that the average trunk height of the fruit trees in the measured area was 0.6 m, the volume of the canopy-circumscribing ellipsoid was calculated using Equation (8). The number of canopy point-clouds Pij of each fruit tree was counted by Equation (9).
H = Z max - Z min ,
S X O Y = 1 2 i = 1 N [ x i y i + 1 y i x i + 1 ] ,
V = 4 3 π ( H 0.6 ) 2 S X O Y π ,
P i j = L ( i ) ~ L ( i + 1 ) C { i } ( j ) ~ C { i } ( j + 1 ) P o int C l o u d ,
where H is the canopy height of a fruit tree, with the unit m; Zmax is the maximum value on the Z-axis of the point-cloud model of any single apple tree, with the unit m; Zmin is the minimum value on the Z-axis of the point-cloud model of any single apple tree, with the unit m; SXOY is the projection area of a single apple tree’s canopy point-cloud model in the XOY plane, with the unit m2; N is the number of convex hull vertices; xi is the x coordinate of the ith vertex; xj is the x coordinate of the jth vertex; yi is the y coordinate of the ith vertex; yj is the y coordinate of the jth vertex; xi+1 is the x coordinate of the (i + 1)th vertex; yi+1 is the y coordinate of the (i + 1)th vertex; V is the equivalent volume of the circumscribed ellipsoid of the canopy, with the unit m3; Pij is the number of point-clouds in the ith row and jth column of the fruit tree canopy; C{i} (j) is the value of the initial point on the X-axis of the ith row and jth column; C{i} (j + 1) is the value of the initial point on the X-axis of the ith row and (j + 1)th column;; L(i) is the initial point on the Y-axis of the ith column; and L(i + 1) is the initial point on the Y-axis of the (i + 1)th column.

2.4.2. Calculation Method for the Color Characteristic Values of Fruit Trees

The extracted color characteristic values of fruit trees mainly included the first-, second-, and third-order color moments [30]. The calculation formulas are Equations (10)−(12). The color moment is a simple and effective way to express color characteristic values. Since the color information is mainly distributed in low-order moments, the first-, second-, and third-order moments are sufficient to express the color distributions in vegetation index images. There is good evidence that color moments can effectively represent the color distribution in images. The advantages of this method are that it does not require color space quantization and the characteristic vector has a low dimension.
μ = 1 m × n i = 1 : m j = 1 : n p ( i , j ) ,
σ = ( 1 m × n i = 1 : m j = 1 : n ( p ( i , j ) μ ) 2 ) 1 2 ,
s = ( 1 m × n i = 1 : m j = 1 : n ( p ( i , j ) μ ) 3 ) 1 3 ,
where μ is the first-order color moment of the image; σ is the second-order color moment of the image; S is the third-order color moment of the image; p(i, j) is the color value of the image pixel at the coordinate (i, j); m is the number of rows in the image, and; n is the number of columns in the image.

2.4.3. Calculation Method for the Textural Characteristic Values of Fruit Trees

Grayscale symbiosis matrices or gray-level co-occurrence matrices (GLCMs) were used to express the textural characteristic values of vegetation index images. Haralick et al. [31] were the first to convert grayscale values into textural information based on GLCMs. GLCMs have the advantages of simple processing, good discrimination, and strong adaptability. They have become an important method for analyzing image textural characteristic values [31]. The characteristics of GLCMs mainly include contrast (CON), correlation (COR), angular second-order moment (ASM), and entropy; the calculation formulas are Equations (13)−(16). CON reflects the image clarity and texture depth, and its value indicates the local grayscale correlation in an image. COR is a measure of the similarity of spatial GLCM elements in the row or column direction, and its value indicates the local grayscale correlation in an image. ASM reflects the uniformity of grayscale distribution and the thickness of texture in an image. The more concentrated the grayscale distribution in an image or the coarser the texture, the greater the ASM value; conversely, the more discrete the grayscale distribution or the finer the texture, the lower the ASM value. INM reflects the homogeneity of image texture and measures the local change of image texture; the larger the inverse difference moment, the smaller the change in image texture between different regions, and the greater the local uniformity. These variables are calculated as follows:
C O N = i = 0 : N 1 j = 0 : N 1 | i j | 2 p ( i , j ) ,
C O R = i = 0 : N 1 j = 0 : N 1 ( i × j × p ( i , j ) ) - i = 0 : N 1 j = 0 : N 1 i × p ( i , j ) i = 0 : N 1 j = 0 : N 1 j × p ( i , j ) i = 0 : N 1 j = 0 : N 1 p ( i , j ) ( i i = 0 : N 1 j = 0 : N 1 i × p ( i , j ) ) 2 × i = 0 : N 1 j = 0 : N 1 p ( i , j ) ( j i = 0 : N 1 j = 0 : N 1 j × p ( i , j ) ) 2 ,
A S M = i = 0 : N 1 j = 0 : N 1 p ( i , j ) 2 ,
I N M = i = 0 : N 1 j = 0 : N 1 p ( i , j ) 1 + | i j | ,
where CON is the image contrast; COR is the image correlation; ASM is the angular second moment of the image; INM is the inverse difference moment; (i, j) are two different grayscale values; p(i, j) is the normalized probability of the occurrence of pixel pair (i, j); and N is the grayscale level of the image.

2.5. Measurement Models and Data Analysis Methods for Orchard Canopy Information

2.5.1. Results and Analysis Methods for Row and Column Detection in the Orchard Canopy

Next, the segmentation accuracy (correction rate) and false/erroneous recognition rate of the row and column detection method based on RCGP were statistically analyzed; the calculation formulas are Equations (17) and (18). Specifically, the segmentation performance based on the NG vegetation index images was calculated when GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px.
C = N C N × 100 % ,
E = N E N × 100 % ,
where C is the correction/accuracy rate, reported as %; NC is the number of correctly segmented regions; N is the number of total regions; E is the erroneous recognition (false recognition) rate, reported as %; and NE is the number of erroneously recognized regions (including repeat recognition regions and erroneously recognized regions).

2.5.2. Results and Analysis Methods for the Measurement of Orchard Morphological Characteristic Values

First, a tape measure was used to manually measure the total height of the canopy, trunk height, canopy width, and fruit yield of the apple tree in the measurement area. Each measurement was done three times, and the average value was taken. Based on the average width of a canopy, the area of canopy projection was calculated. According to Equations (6)–(8), the volume of the circumscribing ellipsoid of the apple tree canopy was calculated as the measured circumscribing volume of the apple tree canopy.
Through the methods in Section 2.4.1, the 3D point-cloud model of the orchard was extracted. The height of fruit tree H, the projected area SXOY, and the volume V were extracted. The correctness (accuracy) of fruit tree morphological characteristic values was calculated when GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px. The coefficient of determination (R2), root mean square error (RMSE), and relative average deviation (RAD) were used to analyze the measurement errors of fruit tree morphological characteristic values.

2.5.3. Results and Analysis Methods for Orchard Yield Measurement

In this study, BPANN was used for modeling. Specifically, the independent variables in the model included three morphological characteristic values, three color characteristic values, and four textural characteristic values. NG, NR, NDVI, GNDVI, DVI, CIG, OSAVI, RDVI, WDRVI, and NLI were used for extracting color characteristic values and textural characteristic values. The actual measured value of fruit yield was taken as the dependent variable. Statistical analysis was performed on the model accuracy of fruit yield measurement when GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px. To evaluate the accuracy of the model and test the performance of the model in fruit yield prediction, the R2, RMSE, and RAD were used.

3. Results and Discussion

3.1. Characteristic Extraction from Multimodal Data of the Orchard Canopy

The grayscale image of the NG index was used for row and column detection in the orchard. Figure 2a shows the Y-axis grayscale projection of the orchard vegetation index image, which is the cumulative grayscale value for all the rows in the grayscale image of the index NG. The peak and valley values of the Y-axis cumulative grayscale values are marked with red and blue circles, respectively, and the local peak values of the Y-axis cumulative grayscale values were taken as the row segmentation positions. The row segmentation positions were saved into the array L; the images of each row’s area were also saved. Likewise, the X-axis in the image for each row’s area was in turn grayscale projected, and the local peak values in the cumulative grayscale values and their corresponding X-values were searched. As shown in Figure 2b, the local peak values of the X-axis cumulative grayscale values for each row are marked with red circles, and the local grayscale peak values of the X-axis were taken as the column segmentation positions, which were saved into the cell array C{i}.
The measured area in this study had a total of five rows, and there were, respectively, 30, 31, 32, 33, and 34 apple trees in rows 1 to row 5. This gave a total of 160 apple trees, including eight dead trees that had only a trunk or had few leaves. Figure 3 shows the row and column detection results based on grayscale projection in orchard index images. Based on the RCGP method, the row segmentation positions were accurately detected (accuracy rate 100%) under various GSDs.
In this study, the correction/accuracy rate C and the erroneous-recognition rate E were calculated with Equations (17) and (18). Statistical analysis of the RCGP method was performed, and the performance of column detection when GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px was compared.
In Figure 4, when using the RCGP detection method and GSD values of 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, the average correction/accuracy rates of column detection in the grayscale image of NG index were 100.00%, 98.71%, 98.77%, 99.38%, and 100.00%, respectively, and the average erroneous-recognition rates were 1.94%, 4.42%, 3.78%, 3.76%, and 3.23%. With the RCGP detection method, the accuracy rate and misrecognition rate in column detection were not significantly affected by GSD.
According to the column detection statistics, the main reason for false recognition by RCGP lay in the misidentification of the border area. Because the orchard point-cloud model generated by the UAV was matched according to the key points of the images collected from different waypoints, there were very few leaves on the dead trees, resulting in very few point-clouds generated in these locations/areas, which in turn affected the performance of the 3D detection method. On the other hand, the RCGP detection method detected the dead trees with few leaves, so the column detection performance using this method was optimal, accurate, and stable.

3.2. Calculation and Error Analysis of Orchard Canopy Morphological Information

With the RCGP row and column detection method, the row and column segmentation positions in vegetation index images were determined. The row and column segmentation positions in the 3D point-cloud model were obtained using Equations (4) and (5). The unified segmentation positions in the fruit tree region were used. The morphological parameters of the fruit tree canopy were calculated using Equations (6)–(9), the color characteristic values of the vegetation indices of the fruit tree canopy were calculated using Equations (10)–(12), and the textural characteristic values of the vegetation indices of the fruit tree canopy were calculated using Equations (13)–(16).
Figure 5a shows the detection results for the fruit tree canopy height H, where the highest point and lowest point in each fruit tree area were detected, and the distance between the two points was the height of the fruit tree, as shown in Figure 5b. Figure 5c shows the detection results for SXOY, the projected area of the fruit tree canopy, which is the area enclosed by the projected boundary of the point-cloud model in the XOY perspective, the red circle and the column mark the boundary convex hull point set. Taking the coordinates with average values of the X-axis and Y-axis of the point-cloud model as the center point, the equivalent circle of canopy projection was drawn (the black circle in Figure 5c). The area of this circle is equal to SXOY. In this study, the average value of the trunk of the fruit trees was 0.6 m, and the shape of the canopy was similar to that of an ellipsoid. Hence, based on the tree height H, the height of the trunk, and the projected canopy area SXOY, the volume of the canopy-circumscribing ellipsoid of fruit tree was calculated, as shown in Figure 5d.
Statistical analysis was performed on the measurement results of morphological parameters of the fruit tree canopy, including H, SXOY, and V, as shown in Table 4. When GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, upon comparing the hand-measured H values of the canopy and the values measured with the UAV, the R2 value was, respectively 0.94, 0.92, 0.91, 0.88, and 0.85; the RMSE was 0.08 m, 0.10 m, 0.09 m, 0.10 m, and 0.14 m; and the RADavg was 1.72%, 1.77%, 1.78%, 2.67%, and 3.42%. Upon comparing the hand-measured values of SXOY of the fruit tree canopy and the values measured with the UAV, the R2 value was, respectively, 0.94, 0.91, 0.86, 0.86, and 0.79; the RMSE was 0.72 m2, 0.94 m2, 1.10 m2, 1.13 m2, and 1.39 m2; and the RADavg was 4.33%, 5.98%, 7.19%, 7.75%, and 9.87%. Upon comparing the hand-measured values of V of the fruit tree canopy and the values measured with the UAV, R2 was 0.91, 0.86, 0.85, 0.83, and 0.80, respectively; RMSE was 1.41 m3, 1.63 m3, 1.64 m3, 1.91 m3, and 2.21 m3; and RADavg was 7.90%, 8.28%, 11.90%, 12.61%, and 13.69%. Thus, the statistical results showed that GSD had a significant impact on the accuracy and correlation of the calculated 3D morphological parameters in the point-cloud model of the orchard: as GSD increased, R2 decreased and RMSE increased. When GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, the point-cloud density in the 3D point-cloud model was, respectively, 364.77 points/m3, 75.55 points/m3, 32.01 points/m3, 17.22 points/m3, and 8.65 points/m3. As GSD directly affected the density of the orchard point-cloud model, the average point-cloud density had a significant decreasing trend as GSD increased, resulting in an increasing error in the measurement of canopy morphological parameters. GSD had a greater impact on the measured values of SXOY and V, and a small amount of noise caused a large error in measurements. Specifically, the RADmax range of SXOY was 19.48–35.78% and the RADmax range of V was 37.36–55.99%.
The SXOY of the fruit tree canopy had a higher measurement error than H, which was mainly because the SXOY values were calculated from the actual measured widths of fruit trees. Although the width of the fruit tree was the average of multiple measurements, the width of a fruit tree could easily be measured wrong by hand, leading to a certain difference from the calculated value. In the UAV measurement method, as long as the fruit tree area is accurately segmented, the canopy boundary point set can be quickly found through the boundary point spacing search method. Although SXOY can be calculated from the boundary convex hull point set, it requires an accurate point-cloud model. The fruit tree canopy V had a higher measurement error than H and SXOY, which was mainly due to the use of canopy-circumscribing ellipsoids to approximate the canopy volume of fruit trees. There were errors in both UAV measurements and manual measurements of H and SXOY, which led to a relatively large error in V.

3.3. Orchard Yield Prediction and Error Analysis

Three color moment characteristic values and four textural characteristic values of the fruit tree canopy vegetation indices and four morphological characteristic values of the 3D point-cloud model were used as the input characteristic variables in the orchard yield prediction model. The actual measured yield values of fruit trees were taken as the model output variables for constructing the fruit yield prediction model. The canopy vegetation indices of fruit trees were NG, NR, NDVI, GNDVI, DVI, CIG, OSAVI, RDVI, WDRVI, and NLI (Table 3). Six input modes were used for the fruit yield prediction model, named Inputs 1–6. Specifically, Input 1 included 10 input characteristics, i.e., color moment characteristic values and textural characteristic values of one vegetation index and the 3D morphological characteristic values. Input 2 included four input characteristics: the 3D morphological characteristic values of the fruit trees. Input 3 included 30 input characteristics, i.e., color moment characteristic values of all 10 vegetation indices. Input 4 included 40 input characteristics, i.e., textural moment characteristic values of 10 vegetation indices. Input 5 included 70 input characteristics, i.e., color moment characteristic values and textural moment characteristic values of all 10 vegetation indices. Input 6 included 110 input characteristics, i.e., color moment characteristic values and textural moment characteristic values of all 10 vegetation indices and the morphological characteristic values of the 3D point-cloud model.
BPANN was used to establish the fruit yield prediction model. A three-layer network structure was used, where the excitation function from the input layer to the hidden layer was a tansig function and that from the hidden layer to the output layer was a purelin function. The learning function was a function of gradient descent with momentum weight, and the training function was the Levenberg–Marquardt algorithm. The empirical formula h = sqrt(m + n) + a was used to calculate and select the node number of the hidden layer, where h is the number of hidden layers in the BPANN, m is the number of input layers, n is the number of output layers, and a is the adjustment constant (a = 1–10). The percentages of training, validation, and testing data of BPANN model is 60%, 15%, and 25%, respectively. The performance of the fruit yield prediction model under different GSDs was statistically analyzed.
When GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, for Input 1, in which the characteristic values of vegetation indices NG, NR, NDVI, GNDVI, DVI, CIG, OSAVI, RDVI, WDRVI, and NLI were separately used in the BPANN prediction model, R2 ranged from 0.59–0.73, 0.54–0.63, 0.44–0.59, 0.51–0.67, 0.48–0.68, 0.40–0.65, 0.49–0.66, 0.43–0.77, 0.44–0.59, and 0.48–0.63, respectively (Figure 6a); RAD ranged from 12.18–15.86%, 13.59–18.69%, 15.49–19.33%, 13.12–16.59%, 12.32–18.95%, 12.82–19.48%, 13.60–18.74%, 10.47–17.15%, 16.76–19.45%, and 14.26–18.31% (Figure 6b); RMSE ranged from 9.94–12.23 kg, 11.65–13.10 kg, 12.17–14.68 kg, 10.91–13.90 kg, 10.79–14.21 kg, 11.37–14.74 kg, 12.90–15.45 kg, 9.18–14.10 kg, 12.76–16.85 kg, and 11.53–14.88 kg. The statistical data showed that when the color moment characteristic values, textural characteristic values, and 3D morphological characteristic values of one vegetation index were input into the yield prediction model, the correlation between the predicted yield value and the measured value was not significant, and the differences in the prediction performance between various vegetation indices were not significant. The main reason was that one vegetation index has limited information, and the correlation between fruit yield and fruit tree morphology is not significant enough, which led to the inadequate performance of the fruit yield prediction model.
When GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, for Input 2, the R2 of the BPANN prediction model was, respectively, 0.67, 0.62, 0.61, 0.59, and 0.55 (Figure 7a); RAD was 13.51%, 15.97%, 17.21%, 17.47%, and 17.31% (Figure 7b); and RMSE was 10.92 kg, 11.84 kg, 12.12 kg, 12.78 kg, and 12.85 kg. Statistical analysis showed that when the fruit tree morphological characteristic values were used to predict the fruit yield, the model correlation was not significant, and as GSD increased, the performance of this prediction model worsened, with an increase in the RAD. This was mainly because GSD affected the accuracy of the orchard 3D point-cloud model, which led to an increase in the calculation error of the morphological characteristic values; additionally, the morphological characteristic values are not the only determinants of fruit yield.
When the GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, for Input 3, the R2 of the BPANN prediction model was, respectively, 0.88, 0.83, 0.86, 0.75 and 0.83, as shown in Figure 7a; the RAD values were respectively 8.74%, 9.01%, 8.56%, 12.15% and 10.17%, as shown in Figure 7b; and the RMSE values were respectively 7.71 kg, 7.94 kg, 7.20 kg, 9.69 kg and 8.29 kg. The statistical data showed that when the color characteristic values of fruit tree vegetation indices were used to predict the fruit yield, the model correlation was significant; only when the GSD was 5.43 cm/px, the significance was relatively low. Under different GSDs, the performance of the prediction model was similar. The performance of the prediction model outperformed that of Input 1, where single index characteristic values were used for model prediction. The color characteristic values of the fruit tree canopy vegetation index can be used to predict the yield of fruit trees.
When GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, for Input 4, the R2 of the BPANN prediction model was, respectively, 0.56, 0.57, 0.58, 0.66, and 0.66 (Figure 7a); RAD was 16.03%, 17.59%, 13.66%, 12.91%, and 12.47% (Figure 7b); and RMSE was 13.13 kg, 12.78 kg, 12.71 kg, 11.25 kg, and 12.10 kg. The statistical analysis showed that when the textural characteristic values of fruit tree vegetation indices were input to predict the fruit yield, the model correlation was not significant. As GSD increased, the prediction model performance improved. This was mainly because GSD affects the textures of orchard vegetation index images, so as GSD increased, the canopy area in the vegetation index image became more blurred, while the edges became clearer. However, using the textural characteristic values of vegetation indices to predict the yield of fruit trees has a relatively large error, so textural characteristic values cannot be applied to yield prediction of fruit trees.
When GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, for Input 5, the R2 of the BPANN prediction model was, respectively, 0.88, 0.88, 0.84, 0.83, and 0.83 (Figure 7a); RAD was 8.57%, 7.85%, 9.23%, 10.23%, and 10.10% (Figure 7b); and RMSE was 8.35 kg, 6.74 kg, 8.18 kg, 8.62 kg, and 8.28 kg. The statistical analysis showed that when the color characteristic values and textural characteristic values of fruit tree vegetation indices were input to predict the fruit yield, the model correlation was significant; as GSD increased, the performance of this prediction model worsened. The model performance was similar to that of Input 3. Hence, the color characteristic values and textural characteristic values of canopy vegetation indices can be used to predict fruit yield, but considering the prediction performance, textural characteristic values can be ignored, and the color characteristic values of canopy vegetation indices can be used by themselves to predict fruit yield.
When GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, for Input 6, the R2 of the BPANN prediction model was, respectively, 0.88, 0.87, 0.86, 0.83, and 0.84 (Figure 7a); RAD was 8.10%, 8.05%, 9.03%, 9.56%, and 9.76% (Figure 7b); and RMSE was 6.90 kg, 7.03 kg, 7.25 kg, 7.96 kg, and 7.93 kg. The statistical analysis showed that when the color characteristic values and textural characteristic values of fruit tree vegetation indices and the morphological characteristic values of the point-cloud model were input to predict fruit yield, the model correlation was significant, and the prediction model performance was similar to that of Input 3 and Input 5. Therefore, the input characteristics of Input 6 can be applied to fruit yield prediction.
Based on the above analysis, using the color characteristic values of canopy vegetation indices of fruit trees to predict the yield of fruit trees performed better than using the textural and morphological characteristic values. This was mainly because given the UAV measurement mode used in this study, the generated orthophoto images of orchard canopy had a large amount of color characteristic information, while the textural characteristic values of the canopy were relatively blurred, so the amount of information they bore was relatively small. On the other hand, the morphological characteristic values were affected by the accuracy of the 3D point-cloud model, and the morphological characteristic values could not adequately predict the fruit yield. Thus, using color characteristic values of multiple vegetation indices to predict yield performance is superior to using that of one vegetation index, which is mainly due to the screening of vegetation indices in this study, in which vegetation indices with more obvious characteristic advantages were selected. Multiple vegetation indices contained more canopy information and thus had better prediction performance. This method can be applied to yield prediction of any modern standardized apple orchards.

4. Conclusions

This study proposes a UAV multimodal information-based method for the measurement of canopy morphological information and for yield prediction of apple orchards. The main conclusions are as follows:
  • In this study, a row and column detection method based on grayscale projection in orchard index images, RCGP, is proposed. It allows row and column segmentation using multimodal information of fruit tree canopies in modern standardized apple orchards. The results showed that using the RCGP method, the correction/accuracy rate of row detection in the orchard was 100.00%. Using the RCGP method, when GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, the average correction/accuracy rates of column detection based on the grayscale images of NG index were, respectively, 100.00%, 98.71%, 98.77%, 99.38%, and 100.00%, and the average misrecognition rates were 1.94%, 4.42%, 3.78%, 3.76%, and 3.23%. The RCGP detection method can detect dead trees with few leaves, so the column detection performance using this method was accurate and stable.
  • A method for measuring canopy morphological information of fruit trees based on the 3D point-cloud model of orchards is established. The results show that when GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, comparing the hand-measured values of fruit tree canopy height H and the UAV-measured values yielded an R2 of 0.85–0.94, a RMSE of 0.08–0.14 m, and a RADavg of 1.72–3.42%; comparing the hand-measured values of fruit tree canopy SXOY and the UAV-measured values yielded an R2 of 0.79–0.94, a RMSE of 0.72–1.39 m2, and a RADavg of 4.33–9.87%; comparing the hand-measured values of fruit tree canopy V and the UAV-measured values yielded an R2 of 0.80–0.91, a RMSE of 1.41–2.21 m3, and a RADavg of 7.90–13.69%.
  • The BPANN prediction models for measuring orchard yield are established, when the color moment characteristic values and textural moment characteristic values of all 10 vegetation indices and the morphological characteristic values of the 3D point-cloud model were input to predict fruit yield, the results show that when GSD was 2.13, 3.31, 4.39, 5.43, and 6.69 cm/px, the R2 of the BPANN prediction model was, respectively, 0.88, 0.87, 0.86, 0.83, and 0.84, RAD was 8.10%, 8.05%, 9.03%, 9.56%, and 9.76% and RMSE was 6.90 kg, 7.03 kg, 7.25 kg, 7.96 kg, and 7.93 kg. The model correlation was significant, which can be applied to fruit yield prediction.
The row and column detection method proposed in this study is suitable for segmentation using multimodal canopy data in modern standardized orchards, which lays a good foundation for orchard canopy information measurement. The information of canopy morphological and vegetation index characteristic values could be extracted at the same time by this method, which provided more effective information for fruit yield prediction and improves the accuracy of fruit yield prediction. This study established a canopy morphological measurement method and a yield prediction method for fruit trees, which can provide a theoretical basis and technical reference for canopy information measurements in modern standardized orchards and will be valuable in the production management of modern standardized orchards.

Author Contributions

Conceptualization, G.S. and X.W.; Methodology, G.S. and X.W.; Software, G.S. and H.Y.; Validation, G.S., H.Y. and X.Z.; Formal Analysis, G.S., H.Y. and X.Z.; Investigation, G.S. and H.Y.; Writing-Original Draft Preparation, G.S.; Writing-Review & Editing, G.S.; Project Administration, X.W.; Funding Acquisition, G.S. and X.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the National Key R&D Program of China (Grant No. 2017YFD0701400).

Acknowledgments

The authors also thank the editors and anonymous reviewers for providing helpful suggestions for improving the quality of this manuscript, and American Journal Experts for the English language editing.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Underwood, J.P.; Hung, C.; Whelan, B.; Sukkarieh, S. Mapping almond orchard canopy volume, flowers, fruit and yield using lidar and vision sensors. Comput. Electron. Agric. 2016, 130, 83–96. [Google Scholar] [CrossRef]
  2. Jiménez-Brenes, F.M.; López-Granados, F.; De Castro, A.I.; Torres-Sánchez, J.; Serrano, N.; Peña, J.M. Quantifying pruning impacts on olive tree architecture and annual canopy growth by using UAV-based 3D modelling. Plant Methods 2017, 13, 55. [Google Scholar] [CrossRef] [PubMed]
  3. Johansen, K.; Raharjo, T.; McCabe, M.F. Using multi-spectral UAV imagery to extract tree crop structural properties and assess pruning effects. Remote Sens. 2018, 10, 854. [Google Scholar] [CrossRef]
  4. Pforte, F.; Selbeck, J.; Hensel, O. Comparison of two different measurement techniques for automated determination of plum tree canopy cover. Biosyst. Eng. 2012, 113, 325–333. [Google Scholar] [CrossRef]
  5. Cheng, Z.; Qi, L.; Cheng, Y.; Wu, Y.; Zhang, H. Interlacing orchard canopy separation and assessment using UAV images. Remote Sens. 2020, 12, 767. [Google Scholar] [CrossRef]
  6. Wang, Z.; Underwood, J.; Walsh, K.B. Machine vision assessment of mango orchard flowering. Comput. Electron. Agric. 2018, 151, 501–511. [Google Scholar] [CrossRef]
  7. Rosell, J.R.; Sanz, R. A review of methods and applications of the geometric characterization of tree crops in agricultural activities. Comput. Electron. Agric. 2012, 81, 124–141. [Google Scholar] [CrossRef]
  8. Guo, C.; Zong, Z.; Zhang, X.; Liu, G. Apple tree canopy geometric parameters acquirement based on 3D point clouds. Trans. Chin. Soc. Agric. Eng. 2017, 33, 175–181. [Google Scholar] [CrossRef]
  9. Aasen, H.; Honkavaara, E.; Lucieer, A.; Zarco-Tejada, P. Quantitative remote sensing at ultra-high resolution with UAV spectroscopy: A review of sensor technology, measurement procedures, and data correction workflows. Remote Sens. 2018, 10, 1091. [Google Scholar] [CrossRef]
  10. Kim, J.Y.; Glenn, D.M. Multi-modal sensor system for plant water stress assessment. Comput. Electron. Agric. 2017, 141, 27–34. [Google Scholar] [CrossRef]
  11. Koc-San, D.; Selim, S.; Aslan, N.; San, B.T. Automatic citrus tree extraction from UAV images and digital surface models using circular Hough transform. Comput. Electron. Agric. 2018, 150, 289–301. [Google Scholar] [CrossRef]
  12. Jichen, C.; Wang, X.; Jian, S.; Songlin, W.; Shuo, Y.; Chunjiang, Z. Development of real-time laser-scanning system to detect tree canopy characteristics for variable-rate pesticide application. Int. J. Agric. Biol. Eng. 2017, 10, 155–163. [Google Scholar] [CrossRef]
  13. Nguyen, T.T.; Vandevoorde, K.; Wouters, N.; Kayacan, E.; De Baerdemaeker, J.G.; Saeys, W. Detection of red and bicoloured apples on tree with an RGB-D camera. Biosyst. Eng. 2016, 146, 33–44. [Google Scholar] [CrossRef]
  14. Zarco-Tejada, P.J.; Diaz-Varela, R.; Angileri, V.; Loudjani, P. Tree height quantification using very high resolution imagery acquired from an unmanned aerial vehicle (UAV) and automatic 3D photo-reconstruction methods. Eur. J. Agron. 2014, 55, 89–99. [Google Scholar] [CrossRef]
  15. Bendig, J.; Yu, K.; Aasen, H.; Bolten, A.; Bennertz, S.; Broscheit, J.; Gnyp, M.L.; Bareth, G. Combining UAV-based plant height from crop surface models, visible, and near infrared vegetation indices for biomass monitoring in barley. Int. J. Appl. Earth Obs. Geoinf. 2015, 39, 79–87. [Google Scholar] [CrossRef]
  16. Garcia-Ruiz, F.; Sankaran, S.; Maja, J.M.; Lee, W.S.; Rasmussen, J.; Ehsani, R. Comparison of two aerial imaging platforms for identification of Huanglongbing-infected citrus trees. Comput. Electron. Agric. 2013, 91, 106–115. [Google Scholar] [CrossRef]
  17. Suárez, L.; Zarco-Tejada, P.J.; González-Dugo, V.; Berni, J.A.J.; Sagardoy, R.; Morales, F.; Fereres, E. Detecting water stress effects on fruit quality in orchards with time-series PRI airborne imagery. Remote Sens. Environ. 2010, 114, 286–298. [Google Scholar] [CrossRef]
  18. Ishida, T.; Kurihara, J.; Viray, F.A.; Namuco, S.B.; Paringit, E.C.; Perez, G.J.; Takahashi, Y.; Marciano, J.J. A novel approach for vegetation classification using UAV-based hyperspectral imaging. Comput. Electron. Agric. 2018, 144, 80–85. [Google Scholar] [CrossRef]
  19. Yeom, J.; Jung, J.; Chang, A.; Maeda, M.; Landivar, J. Automated open cotton boll detection for yield estimation using unmanned aircraft vehicle (UAV) Data. Remote Sens. 2018, 10, 1895. [Google Scholar] [CrossRef]
  20. Gong, Y.; Duan, B.; Fang, S.; Zhu, R.; Wu, X.; Ma, Y.; Peng, Y. Remote estimation of rapeseed yield with unmanned aerial vehicle (UAV) imaging and spectral mixture analysis. Plant Methods 2018, 14, 70. [Google Scholar] [CrossRef]
  21. Comba, L.; Biglia, A.; Aimonino, D.R.; Gay, P. Unsupervised detection of vineyards by 3D point-cloud UAV photogrammetry for precision agriculture. Comput. Electron. Agric. 2018, 155, 84–95. [Google Scholar] [CrossRef]
  22. Park, S.; Ryu, D.; Fuentes, S.; Chung, H.; Hernández-Montes, E.; O’Connell, M. Adaptive estimation of crop water stress in nectarine and peach orchards using high-resolution imagery from an unmanned aerial vehicle (UAV). Remote Sens. 2017, 9, 828. [Google Scholar] [CrossRef]
  23. Malambo, L.; Popescu, S.C.; Murray, S.C.; Putman, E.; Pugh, N.A.; Horne, D.W.; Richardson, G.; Sheridan, R.; Rooney, W.L.; Avant, R.; et al. Multitemporal field-based plant height estimation using 3D point clouds generated from small unmanned aerial systems high-resolution imagery. Int. J. Appl. Earth Obs. Geoinf. 2018, 64, 31–42. [Google Scholar] [CrossRef]
  24. Khan, Z.; Chopin, J.; Cai, J.; Eichi, V.R.; Haefele, S.; Miklavcic, S.J. Quantitative estimation of wheat phenotyping traits using ground and aerial imagery. Remote Sens. 2018, 10, 950. [Google Scholar] [CrossRef]
  25. Pugh, N.A.; Horne, D.W.; Murray, S.C.; Carvalho, G., Jr.; Malambo, L.; Jung, J.; Chang, A.; Maeda, M.; Popescu, S.; Chu, T.; et al. Temporal estimates of crop growth in sorghum and maize breeding enabled by unmanned aerial systems. Plant Phenome J. 2018, 1, 170006. [Google Scholar] [CrossRef]
  26. Vega, F.A.; Ramírez, F.C.; Saiz, M.P.; Rosúa, F.O. Multi-temporal imaging using an unmanned aerial vehicle for monitoring a sunflower crop. Biosyst. Eng. 2015, 132, 19–27. [Google Scholar] [CrossRef]
  27. De Castro, A.; Torres-Sánchez, J.; Peña-Barragán, J.M.; Jiménez-Brenes, F.; Csillik, O.; López-Granados, F. An automatic random Forest-OBIA algorithm for early weed mapping between and within crop rows using UAV imagery. Remote Sens. 2018, 10, 285. [Google Scholar] [CrossRef]
  28. Yang, H.; Wang, X.; Sun, G. Three-dimensional morphological measurement method for a fruit tree canopy based on kinect sensor self-calibration. Agronomy 2019, 9, 741. [Google Scholar] [CrossRef]
  29. Han, X.; Thomasson, J.A.; Bagnall, G.C.; Pugh, N.A.; Horne, D.W.; Rooney, W.L.; Jung, J.; Chang, A.; Malambo, L.; Popescu, S.C.; et al. Measurement and calibration of plant-height from fixed-wing UAV images. Sensors 2018, 18, E4092. [Google Scholar] [CrossRef]
  30. Stricker, M.; Orengo, M. Similarity of color images. Proc. SPIE Int. Soc. Opt. Eng. 1970, 2420, 381–392. [Google Scholar]
  31. Haralick, R.M.; Shanmugan, K.; Dinstein, I. Textural features for image classification. IEEE Trans. Syst. Man Cybern. 1973, 3, 610–621. [Google Scholar] [CrossRef]
Figure 1. Three-dimensional point-cloud models and vegetation index images of the apple orchard.
Figure 1. Three-dimensional point-cloud models and vegetation index images of the apple orchard.
Sensors 20 02985 g001aSensors 20 02985 g001bSensors 20 02985 g001c
Figure 2. Row and column detection with multimodal data of orchard canopy.
Figure 2. Row and column detection with multimodal data of orchard canopy.
Sensors 20 02985 g002
Figure 3. Row and column detection results with orchard canopy data.
Figure 3. Row and column detection results with orchard canopy data.
Sensors 20 02985 g003
Figure 4. Performance of column detection with orchard canopy data.
Figure 4. Performance of column detection with orchard canopy data.
Sensors 20 02985 g004
Figure 5. Detection of morphological characteristic values of orchard canopy.
Figure 5. Detection of morphological characteristic values of orchard canopy.
Sensors 20 02985 g005
Figure 6. Performance parameters of the fruit yield prediction model using various vegetation indices.
Figure 6. Performance parameters of the fruit yield prediction model using various vegetation indices.
Sensors 20 02985 g006
Figure 7. Performance parameters of the fruit yield prediction model with various characteristic values.
Figure 7. Performance parameters of the fruit yield prediction model with various characteristic values.
Sensors 20 02985 g007
Table 1. Flight, camera, and ground control parameters.
Table 1. Flight, camera, and ground control parameters.
Pix4Dcapture Values
Flight altitude22 m, 32 m, 43 m, 53 m, 64 m
Flight speedNormal: 22 m (2.6 m/s), 32 m (3.7 m/s), 43 m (4.9 m/s), 53 m (6.1 m/s), 64 m (7.3 m/s)
Image overlapFront overlap: 80%; side overlap: 80%
CameraParrot Sequoia, 16 million pixels, image format: JPG
Ground controlPix4Dcapture: flight planning, Parrot Skycontroller 2
WeatherClear, little wind
Table 2. Summary of parameters used for SfM processing in Pix4Dmapper.
Table 2. Summary of parameters used for SfM processing in Pix4Dmapper.
Pix4DmapperParameters
Coordinate SystemsImage coordinate system: WGS 84 (EGM 96 Geoid); unit: m
Output coordinate system: WGS 84/UTM zone 51N (EGM 96 Geoid)
Initial ProcessingKeypoint image scale: Full
Calibration Method: Standard
Internal Parameters Optimization: All
External Parameters Optimization: All
Rematch: Auto, yes
Point cloud densificationImage scale: 1/2 of original image size
Point density: Optimal
Minimum number of matches: 3
Export format: PLY
DSM, orthographic image, and indicesIndices: NG, NR, NDVI, GNDVI, DVI, CIG, OSAVI, RDVI, WDRVI, NLI
Table 3. Vegetation indices and their calculation formulas.
Table 3. Vegetation indices and their calculation formulas.
Vegetation IndexFormulaVegetation IndexFormula
NGGRE/(NIR + RED + GRE) NRRED/(NIR + RED + GRE)
NDVI(NIR − RED)/(NIR + RED) GNDVI(NIR − GRE)/(NIR + GRE)
DVINIR − REDCIGNIR/GRE − 1
OSAVI1.16 (NIR − RED)/(NIR + RED + 0.16) RDVI(NIR − RED)/(NIR + RED) 1/2
NLI(NIR2 − RED)/(NIR2 + RED) WDRVI(0.2NIR − RED)/(0.2NIR + RED)
Table 4. Analysis of the measurement results of morphological characteristic parameters of the orchard canopy.
Table 4. Analysis of the measurement results of morphological characteristic parameters of the orchard canopy.
Morphological ParameterGSDR2RMSERADmaxRADminRADavg
H2.13 cm/px0.94 0.08 m8.04%0.03%1.72%
3.31 cm/px0.92 0.10 m9.74%0.00%1.77%
4.39 cm/px0.91 0.09 m9.51%0.00%1.78%
5.43 cm/px0.88 0.10 m8.04%0.04%2.67%
6.69 cm/px0.85 0.14 m10.93%0.01%3.42%
SXOY2.13 cm/px0.94 0.72 m219.48%0.09%4.33%
3.31 cm/px0.91 0.94 m224.72%0.00%5.98%
4.39 cm/px0.86 1.10 m2 38.87%0.03%7.19%
5.43 cm/px0.86 1.13 m2 40.54%0.02%7.75%
6.69 cm/px0.79 1.39 m235.78%0.05%9.87%
V2.13 cm/px0.91 1.41 m337.36%0.36%7.90%
3.31 cm/px0.86 1.63 m335.91%0.38%8.28%
4.39 cm/px0.85 1.64 m3 35.78%0.14%11.90%
5.43 cm/px0.83 1.91 m349.79%0.16%12.61%
6.69 cm/px0.80 2.21 m3 55.99%0.35%13.69%
Back to TopTop