Open Access This article is
- freely available
Remote Sens. 2018, 10(5), 805; doi:10.3390/rs10050805
Estimation of Vegetable Crop Parameter by Multi-temporal UAV-Borne Images
Grassland Science and Renewable Plant Resources, Organic Agricultural Sciences, Universität Kassel, D-37213 Witzenhausen, Germany
Department of Earth and Space Sciences, Indian Institute of Space Science and Technology, Valiyamala, Trivandrum 695574, India
Centre for Ecological Economics and Natural Resources, Institute for Social and Economic Change, Dr. VKRV Rao Raod, Nagarabhavi, Bangalore 560072, India
All-India Coordinated Research Project on Agroforestry, University of Agricultural Sciences (UAS), GKVK, Bangalore, Karnataka 560065, India
Organic Plant Production and Agroecosystems Research in the Tropic and Subtropics, Organic Agricultural Sciences, Universität Kassel, D-37213 Witzenhausen, Germany
Author to whom correspondence should be addressed.
Received: 18 April 2018 / Accepted: 17 May 2018 / Published: 22 May 2018
3D point cloud analysis of imagery collected by unmanned aerial vehicles (UAV) has been shown to be a valuable tool for estimation of crop phenotypic traits, such as plant height, in several species. Spatial information about these phenotypic traits can be used to derive information about other important crop characteristics, like fresh biomass yield, which could not be derived directly from the point clouds. Previous approaches have often only considered single date measurements using a single point cloud derived metric for the respective trait. Furthermore, most of the studies focused on plant species with a homogenous canopy surface. The aim of this study was to assess the applicability of UAV imagery for capturing crop height information of three vegetables (crops eggplant, tomato, and cabbage) with a complex vegetation canopy surface during a complete crop growth cycle to infer biomass. Additionally, the effect of crop development stage on the relationship between estimated crop height and field measured crop height was examined. Our study was conducted in an experimental layout at the University of Agricultural Science in Bengaluru, India. For all the crops, the crop height and the biomass was measured at five dates during one crop growth cycle between February and May 2017 (average crop height was 42.5, 35.5, and 16.0 cm for eggplant, tomato, and cabbage). Using a structure from motion approach, a 3D point cloud was created for each crop and sampling date. In total, 14 crop height metrics were extracted from the point clouds. Machine learning methods were used to create prediction models for vegetable crop height. The study demonstrates that the monitoring of crop height using an UAV during an entire growing period results in detailed and precise estimates of crop height and biomass for all three crops (R2 ranging from 0.87 to 0.97, bias ranging from −0.66 to 0.45 cm). The effect of crop development stage on the predicted crop height was found to be substantial (e.g., median deviation increased from 1% to 20% for eggplant) influencing the strength and consistency of the relationship between point cloud metrics and crop height estimates and, thus, should be further investigated. Altogether the results of the study demonstrate that point cloud generated from UAV-based RGB imagery can be used to effectively measure vegetable crop biomass in larger areas (relative error = 17.6%, 19.7%, and 15.2% for eggplant, tomato, and cabbage, respectively) with a similar accuracy as biomass prediction models based on measured crop height (relative error = 21.6, 18.8, and 15.2 for eggplant, tomato, and cabbage).
Keywords:point clouds; biomass; crop height; machine learning; unmanned aerial vehicles; multi-spectral
The ability to non-destructively collect information about crops makes remote sensing a less time consuming and less labor-intensive tool in agricultural science than traditional methods of crop growth monitoring. Unmanned aerial vehicles (UAV) have emerged as a promising remote sensing platform to capture detailed imagery from agriculture crop fields . The ability of UAVs to collect data over wide areas in flexible intervals makes them a superior tool compared to satellite or airborne remote sensing . Especially UAV systems equipped with RGB (red, green, blue) cameras are widely distributed, but systems with other cameras installed (e.g., multi-spectral cameras) are also getting more and more available. Detailed information about crop health , crop biomass development , and crop water status  have been already successfully extracted from UAV remote sensing for various agriculture crops.
Collecting information about phenotypic traits, such as plant height or biomass, at high temporal resolution, is essential for many site-specific management practices or plant physiological studies . Plant height is considered as a good indicator for plant growth and biomass [7,8]. However, the use of classical ground-based height measurements is time consuming, and can rarely be applied non-destructively across larger areas in a repetitive manner . Alternative methods based on LiDAR (light detection and ranging) [9,10], ultrasonic sensors , or high resolution RGB imagery  have been developed recently. While LiDAR sensors provide highly accurate and dense 3D point measurements of crop surfaces, they are still very expensive, and require specific expertise for handling of sensors and the subsequent analysis of the data . Although less cost intensive alternatives exist (Ehlert 2010), they cannot be used to cover large areas due to their limited mobility. Ultrasonic sensors are considered as low-cost and user-friendly approach, but are often limited by their spatial resolution and their susceptibility to wind . RGB image-based detection of crop height is the most recently evolving approach for many different crops, including barley , maize , vineyards , wheat , sorghum , or alfalfa . Especially, 3D point clouds generated from UAV-borne RGB images using SfM (structure from motion) techniques offer new options for deriving crop height information . Despite the successful application of 3D point clouds based on UAV imagery and the SfM procedure, many studies have been limited to single acquisition dates or only short periods in the crop growth cycle. In addition, most of the studies concentrate on crops, which are characterized by homogenous crop surfaces. To our knowledge, no study has yet tested the application of UAV-borne RGB imagery for estimating vegetable crop height over an entire cropping season.
The aim of this study, therefore, was to assess the applicability of UAV-borne RGB imagery for capturing crop height information of three vegetable crops eggplant, tomato, and cabbage over an entire cropping season. These three crops are the major vegetable species in the larger area of Bengaluru, Karnataka, India. They represent three different crop growth forms, and canopy shape pattern ranging from a less complex ball shape for cabbage, to a complex multi-layer appearance for tomato and eggplant. The specific objectives of this study were (1) evaluating the ability of UAV imagery and SfM techniques to model the plant height of three vegetable crops over an entire crop growth cycle; (2) monitoring the effect of the crop development on the model prediction quality of crop height values using UAV imagery; (3) quantifying the effect of crop growth form on the crop height models; and (4) evaluating the applicability of the crop height models for estimating the biomass of vegetable crops.
2.1. Study Site
The study was conducted on the experimental farms of the University of Agricultural Sciences (UAS), Bengaluru (12°58′20.79′′N, 77°34′50.31′′E, 920 asl), India (Figure 1A), where Kandic Paleustalfs and Dystric Nitisols are the dominating soil types. The mean annual temperature is 29.2 °C, with an average precipitation of 923 mm, of which most occurs during the monsoon season from June to October .
2.2. Experimental Design
The field experiment was established in 2016 with a respective rainy and dry season crop rotation. Within the rainy season (July to October) maize (Zea maize L.) (cultivar: Nithyashree), millet (Eleusine coracana Gaertn.) (cultivar: MR 65), and lablab (Lablab purpureaus L. Sweet) (cultivar: HA 4) were grown, while in the dry season (January to May), tomato (Solanum lycopersicum L.) (cultivar: NS-501), eggplant (Solanum melongena L.) (cultivar: Ankur), and cabbage (Brassica oleracea L.) (cultivar: Unnati) were grown on the experimental site as per the design of Figure 1B. The aim was to include the most commonly grown crops in the greater region of Bengaluru in the field experiment. The selection of crops and cultivars was based on recommendations of scientists at UAS. During the dry season, the crops were irrigated with a drip irrigation system. The split-plot experiment comprised 12 main plots (12 × 18 m), whereby each treatment was replicated four times (Figure 1A). In each main plot, three subplots (6 × 12 m) with different levels of nitrogen (N) fertilizer were randomized. Besides a blanket application of phosphorus and potassium (40 kg P ha−1 and 20, 24, and 50 kg K ha−1 for eggplant, tomato, and cabbage), the mean N rate (50 kg N ha−1 as urea for eggplant, 46 kg N ha−1 for tomato, and 60 kg N ha−1 for cabbage) reflected the recommended N dose in the region, whereas the high (+50%) and low (−50%) dose represented additional intensities typical to the study area. Nitrogen application was split into two dressings to reduce leaching losses by the typically strong rainfalls immediately after fertilizer application. As fertilizers were not distributed evenly on the total plot area like previously planned, but directly applied to individual plants, no nutrient deficiency was noticeable in either treatment. To allow for destructive measurements, each plot was divided equally in a sampling sub-subplot (S-plot; Figure 1B), which was used for all destructive biomass samplings and a harvest sub-subplot (H-plot) for undisturbed crop growth and non-destructive measurements, such as determination of plant height. The S- and H-plots had a size of 6 × 6 m each. The layout thus comprised a total of 36 subplots (3 crops × 3 fertilizer treatments × 4 replicates).
2.3. Plant Sampling and Measurements
To calculate average crop height, the height of 30 plants in each S- and H-plot was measured with a ruler to the nearest 0.1 cm (Figure 2) whereby the distance from the ground to the top most vegetative element was determined. The measurements were conducted at five dates throughout the growing season (Table A1 and Figure A1). To measure total biomass (including fruits when present at later growth stages), three randomly selected plants were harvested 5 cm above ground in all S-plots (Figure 2). At each harvest, the biomass (t ha−1) was weighted in the field and averaged for each plot.
2.4. RGB Imagery Sampling
Prior to destructive sampling, each main plot was measured using a RGB camera mounted on a UAV (DJI 3 Professional, DJI, Shenzhen, Guangdong, China). The camera model used was the standard DJI FC300X with a focal length of 3.61mm. At every sampling date, flights were conducted for each plot using the autopilot and the software Pix4D Capture (Pix4D SA, Lausanne, Switzerland). The flight path was kept constant throughout all sampling dates and for all dates. For capturing the images, a gridwise flight path was defined, for which the area of the main plot was overlaid with a grid of several perpendicular flight lines. The flight delivered nadir images, which were used to generate a 3D point cloud. On average ~200 images each with a resolution of 4000 × 3000 pixels were taken during each flight (Figure 2). The images were taken with an 80% forward and side overlap, whereby the flight altitude was set to 20 m (ranging from 17 m to 24 m, due to variations in the built-in GPS of the drone). The flying height represented a compromise of getting enough images (ranging from 189 to 219) for the point cloud processing and still covering the plot in as little time as possible. The variation in the flying height also leads to variation in the mean ground resolution ranging between 6 and 8 mm per pixel. In total, 60 flights (4 main plots × 3 crops × 5 sampling dates) were conducted. The flights were conducted only on days with no or only low wind, to reduce the effect of moving plants during recording. The spatial position of each image was measured by the internal GPS of the UAV (spatial accuracy ~2 m). However, its accuracy is not sufficient for direct georeferencing. Thus, for each main plot, four permanent ground control points (GCPs) were measured at the corners of the plot using a differential GPS (Trimble Inc., Sunnyvale, CA, USA) with a spatial accuracy of ~5 cm. These GCPs were subsequently used for georeferencing, geo-correction and co-registration of the images.
2.5. Point Cloud Processing
To receive point cloud data from the UAV images, a SfM algorithm was applied (Figure A1). The first step of the algorithm is to extract features in each image that can be matched to their corresponding features in other images [20,21]. Prior to this matching step, the SfM performs a bundle adjustment among the images, based on matching features between the overlapped images to estimate interior and exterior orientation of the onboard sensor. Applying a multi-view stereo matching algorithm to the images, a dense point cloud is generated [22,23]. All images were processed using the software Photoscan Pro (AgiSoft LLC, St. Petersburg, Russia). The bundle adjustment was conducted with the high accuracy settings and a key point limit of 100,000 and a tie point limit of 1000. The resulting sparse point cloud was georeferenced using the GCPs to ensure precise alignment and georeferencing of the point cloud. The RMS of the projections ranged from 0.2 to 0.4. All GCPs were manually identified in the raw images, and their coordinates were fed into the Photoscan software. The spatial error of the GCP for all datasets ranged from 5–20 cm. The accuracy decreased towards the end of growing season, due to difficulties in identifying the GCP due to overgrowing (Figure 2). Subsequently, the dense point cloud was processed using the “medium quality” setting. Although higher quality settings were possible, the medium quality was selected to reduce processing time. Depth filtering allows removing outliers within the point cloud and was set to “medium” following . Afterwards, the point clouds were divided into corresponding S- and H-plot.
2.6. Ground Classification
Besides the x-, y-, z-coordinates, each point is associated with a red, green, and blue color information. This color information was used to calculate the Green-Red Vegetation Index (GRVI, Equation (1); ). The GRVI has been shown to be efficient in spectrally discriminating vegetation from soil .with Green and Red indicating the green and red color information for all points of the point cloud. Low GRVI values are characterized by small differences between Green and Red colors, which is typical for soil. Vegetation, on the other hand, is characterized by larger differences between the Green and Red color values, which would lead to higher GRVI values. Using an unsupervised k-means algorithm, all points were classified into vegetation or ground, based on the GRVI values. For this, the k-means algorithm classified all points into k classes (here, k = 2) based on the smallest distance of the points to the mean of the class centroid. This classification was done for all plots and sampling dates individually. The classified ground points were subsequently used for the creation of a digital elevation model (DEM) using Delaunay triangulation (Figure 2). The DEM had a resolution of 1 m, which was a compromise between available ground points and necessary interpolation. To normalize the height information of the vegetation points, from each z-value, the corresponding DEM value was subtracted. This allowed for calculation of the height of each vegetation point above ground surface (Figure 2). The resulting average point density of the vegetation point cloud was 955 points m−2 and 988 points m−2 for the S- and H-plot, respectively. As plant individuals were continuously removed from the S-plot during the growing season, the total standing biomass in the S-plot steadily decreased. This, however, did not affect the presented approach, as the non-vegetated areas were removed using the GRVI values from the drone imagery, and so the average crop height was only based on actual existing plant individuals. For each plot and sampling date, 14 different point cloud metrics were calculated (Table 1). For each height metric only, the points classified as vegetation were used for calculation (Table A2). The selection of the metrics was based on the results from studies focusing on forest biomass estimation [27,28]. The extraction and handling of the point cloud was done using the lidR package  of the program R .
2.7. Statistical Methods
In a first step, the effect of fertilizer treatment and sampling date on the measured crop height was tested using ANOVA. A significant effect of the fertilizer would imply that the crop height modelling should be done separately for each N level. To predict crop height, two machine learning methods were used, namely random forest regression (RFR)  and support vector regression (SVR) . Both methods were chosen because they can handle the high intercorrelation of the point cloud metrics (Table 1). Partial least squares regression (PLSR), another common machine learning method, was also tested for their predictive abilities. However, PLSR did not outperform RFR and SVR, and was thus removed from the further presentations. For model calibration, only point clouds from the S-plot were used, while for validation of the model prediction quality, the point clouds from the H-plot were used. The RFR and SVR calculations were done using the packages randomForest  and e1071  in R .
To evaluate model prediction quality, the explained variance using the pseudo-R2 (Equation (2) was used, as well as the root mean square error (RMSE) of the predicted values, that is, values from the H-plot (Equation (3). Additionally, the bias was calculated (Equation (4). To evaluate the prediction performance for each sampling date, the deviation of the predicted values from the measured height values, scaled by the mean of the measured height values, were used. For the comparison of the models, also, the relative error (rel.err.) was calculated (Equation (5). To assess the applicability of the predicted crop height as an estimator of biomass, a linear regression was calculated between the predicted crop height and the biomass for the H-plot. The model prediction quality was evaluated using the R2 and RMSE.with being the measured crop height values, and the predicted crop height values. indicates the average measured crop height, and n is the number of samples.
To test for the combined effects of sampling date and N-fertilizer on the average crop height, a two-factorial ANOVA was conducted (Table 2). Prior to the ANOVA, the values were tested for normality of residuals and homoscedasticity, and if necessary, were transformed (results not shown). For all three crops, the interaction term of sampling date and N fertilizer was not significant, while the effect of sampling date was highly significant. This is not surprising, as it was assumed that crop height increases similarly with increasing crop age at all N levels, whereby as stated earlier, the range of occurring N supply to crops was limited.
3.1. Crop Height Estimation
To predict mean crop height, 14 point cloud-derived crop height metrics were used (Table 3). The correlation among the metrics ranged from 0.01 to 0.98, with an average intercorrelation of 0.45, indicating a moderate intercorrelation (Table 3). Thus, regression methods, which can handle intercorrelations among the explanatory variables, like RFR or SVR, should be preferred to classical ordinary least squares regression models.
In Figure 3, the measured average crop height values versus the predicted crop height values are shown for: (left) eggplant, (middle) tomato, and (right) cabbage. Two machine learning methods were tested: (top) random forest regression, and (bottom) support vector regression. For RFR, the best model was acquired for cabbage (pseudo-R2 = 0.97, RMSE = 1.3, bias = 0.01). For eggplant and tomato the results were similarly good (eggplant: pseudo-R2 = 0.93, RMSE = 6.86, bias = 0.14, tomato: pseudo-R2 = 0.89, RMSE = 5.49, bias = 0.45) (Figure 3). For SVR, the results were always weaker than for RFR, with a pseudo-R2 for eggplant, tomato, and cabbage of 0.91, 0.87, and 0.89, respectively. The RMSE and the bias, on the other hand, showed larger errors for the SVR models (eggplant: RMSE = 7.36, bias = −0.66, tomato: RMSE = 5.91, bias = 0.91, and cabbage: RMSE = 2.31, bias = −0.24).
3.2. Crop Height Deviation within the Growing Season
To check whether the predictability of the average crop height varied between RFR and SVR along the growing season, the deviation of the predicted from the measured crop height was calculated (Figure 4). For eggplant the median deviation was between 1–20% for all sampling dates for both RFR and SVR. An increase of the relative deviation was visible from sampling date 1 to 5, except for sampling date 3, for which both methods overestimated the measured average crop height. For tomato, the relative deviation shows no clear pattern along the sampling dates for both RFR and SVR. The median deviation for RFR was lowest for sampling date 1 (−0.009) and highest for sampling date 4 (0.16). For SVR, the fluctuation shows a higher amplitude than for RFR, ranging from 0.06 to −0.19. For cabbage, the deviation was lowest for RFR across all sampling dates, ranging from −0.008 in sampling date 2, to 0.10 for sampling date 5. For SVR, the highest deviation was found for sampling date 1 (−0.2). For the other sampling dates, the deviation is comparable to the results from the RFR.
3.3. Crop Biomass Estimation
To test whether the predicted crop height values are reliable estimators for the biomass of the three crops, a log-linear model was calibrated with the predicted crop height values as predictors and the biomass of the H-plot as dependent variable (Figure 5). The predicted crop height values were based on RFR. The models for all three crops showed significant relationships with R2 values from 0.90 (RMSE = 6.01 t ha−1) for eggplant, 0.88 (RMSE = 3.76 t ha−1) for tomato, and 0.95 (RMSE = 9.97 t ha−1) for cabbage. In order to compare the model based on the predicted crop height with a model based on the measured crop height, a log-linear model was also calibrated with the measured average crop height (Figure 5 and Figure 6). For eggplant, the model on the measured crop height showed a higher relative error (rel.err. = 21.6%) than the model based on the predicted crop height. For tomato the model based on the measured crop heights showed a slightly lower error of 18.3% and for cabbage both models the relative error is the same (rel.err. = 15.2%).
The aim of the current study was to assess the applicability of UAV-borne RGB imagery for capturing crop height information for the three vegetable crops (eggplant, tomato, and cabbage), and to assess whether the developed methods can be used over an entire cropping season. Overall, results indicate that a successful estimation of crop height for vegetable crops with very different crop growth forms could be achieved (with R2 value ranging from 0.89 for tomato to 0.97 for cabbage). These results are similar to multi-temporal crop height models for maize and sorghum (ranging from 0.68 to 0.78) , although, in contrast to this study, no systematic bias according to the vegetable growth stage could be found.
Our experiment shows that UAV-borne RGB imagery in combination with SfM techniques can be used to estimate vegetable crop height during the complete growing period. The suggested method using several extracted crop height metrics from 3D point data was used previously in forestry research , and in agroecological research . For example,  used a group of point cloud derived height metrics to estimate variations of maize crop height at landscape scale. The advantage of this method in comparison to the more frequently used single best metric approach is that the information content about crop height differs for each single metric, as can be seen in the correlation among all metrics (Table 3). This indicates that multiple crop height metrics predict biomass information more comprehensively than single crop height metrics. Modern machine learning methods can handle highly intercorrelated variables  better than ordinary least squares regression models, and should therefore be the tool of choice for the prediction of crop height based on multiple crop height metrics from UAV-borne RGB imagery. The results showed no clear superiority of RFR over SVR, although RFR always yielded slightly better results (Figure 3). The slightly better performance might be explained by the lower sensitivity to data skewness and to model overfitting .
UAV-borne RGB imagery-based crop height values indicate a systematic deviation from the field measured crop height values (Figure 4). The apparent increase in the median bias, from early growing stages to the late stages for eggplant and cabbage, indicates an increasing overestimation of the vegetable crop height by the prediction model. A similar trend is evident in  for maize and sorghum. They state that the deviations are mainly due to time lags between the UAV-based measurements and the field-based measurements. For the current study, the sampling date was identical for both UAV and field measurements, indicating that other factors might be the reason for the deviation. The biased estimation is more likely a result from inaccuracies in the point cloud generation process and is, thus, likely to be related to a biomass increase during the crop development, which could complicate the SfM processing . For tomato crop, this trend in bias could not be found, but in contrast to the other two crops tomato was tied up to horizontal threads from sampling date 3, to prevent molding of the fruits. This agricultural practice might also be the reason for the high bias values for tomato at sampling date 4 (Figure 4). In addition to the deviations in estimated crop height from the SfM algorithm, the field-based height measurements are not bias-free. For the present study, the maximum height of any vegetation part at each measurement point was used for the height measurements, which could lead to an overestimated average plot crop height.
The three vegetable crops examined in the present study represent different growth forms with different internal vegetation structures. While the cabbage is characterized by a relatively flat and round shape with compact structure, tomato and eggplants are growing taller, which is characteristic for more complex structures. In comparison to tomato, eggplant creates larger leaves and thicker stems. These vegetation characteristics lead to differences in the biomass and vegetation structure, particularly at the end of the growing season (Figure 5), and could have affected the vegetable crop height estimation . Despite these variations, no difference in model prediction quality was found (Figure 3 and Figure 4). These results agree with the results from , who have shown that, with high spatial resolution UAV-borne RGB imagery, even small structural details can be successfully differentiated.
Crop height is considered as an important indicator for biomass of crops, such as maize , barley , sorghum , and poppy . The predicted vegetable crop height values of the present study show strong and highly significant relationships to the biomass for all three crops (Figure 5). The model prediction quality (measured as R2) was better in the present study than in the study by , who estimated maize aboveground biomass using UAV-borne RGB imagery. However, in addition to being only a single date data , the flying height of the UAV used by  was much higher (130 m) than the flying height (20m) maintained in this study, leading to manifold differences in the spatial resolution of imagery. In another similar study,  found a comparably strong relationship between UAV-borne RGB derived from multi-temporal crop height and barley dry matter yield (R2 = 0.85). The accuracies of the models based on the predicted crop height showed similar prediction qualities than the corresponding models based on field-based crop height values (Figure 5). This supports the idea that point cloud analysis could replace intensive field campaigns for measuring crop height with a similar prediction accuracy for the crop biomass. Both modelling approaches showed clear deviations for the late growing stages (growing stage 4 and 5; Figure 6). These deviations indicate that height is a less accurate estimator of biomass for crops in late growing stages. The results of the present study support the finding of other studies that high resolution multi-temporal crop height information is needed for successful estimation of crop biomass [10,41]. Alternative plant phenotypic traits, such as plant volume, may even yield more accurate predictors for biomass . Thus, future studies should evaluate the further potential of 3D point cloud analysis for estimating key plant phenotypic traits, which can be used as reliable predictors for plant biomass. Furthermore, the results indicate that for a successful estimation of biomass, height information alone is not enough. Evolving sensor fusion approaches might improve the model prediction performance, as it has been shown for example for grassland biomass [42,43].
5. Uncertainties, Errors, and Accuracies
Disturbances of vegetation canopy through environmental conditions (e.g., wind, clouds) can affect point cloud creation and lead to inaccuracies during the point cloud processing. While wind can move the vegetation canopy and, thus, may lead to a distortion of height, clouds can lead to differences in the RGB values which were used for ground classification. However, as the average canopy height was used, we assume these effects are minor, and no UAV flight was conducted during high wind conditions.
For a successful height estimation, a highly precise georeferencing is mandatory. Four GCPs (as used in this study) represent the minimum number of control points for georeferencing. More GCPs could have led to a more accurate representation of the actual crop height. To keep the relative error between the measurements constant, permanent GCPs were used.
Accurate estimation of height for each vegetation point needs an accurate DEM model. While at the beginning of the growing season, plenty of space among individual plants allows for identifying enough ground points for an accurate calculation of the DEM, much of the ground is covered by vegetation at later crop growth stages, which reduces the amount of ground-classified points. This problem is not prominent for cabbage, as there was always enough space between the plant individuals, whereas tomato and eggplant developed a dense vegetation cover with progressing growing season.
Although the accuracies of the biomass prediction models based on the estimated crop height were similar to the predicted biomass based on the measured crop height, circa 10% of the variation in the biomass values remained unexplained. The relationship between height and biomass is exponential, indicating that large values of vegetation height are not a stable estimator of biomass anymore. A reason for this might be that the linear relationship between biomass and crop height is only valid at the beginning of the growing season, whereas with progressing growing season the crop biomass further increases, which is rather due to the growth of fruits than due to height increase.
High resolution subplot or level estimation of crop phenotypic traits from UAV-borne RGB imagery is fast emerging as one time and cost effective remote sensing tool for agricultural crops. The processing and analysis approach developed in this work, based on the extraction of several height parameter from the 3D point clouds and machine learning regression methods, exhibit stable model prediction quality for biomass during the entire growing period and for the three vegetable crops namely, eggplant, tomato, and cabbage. However, the results also show that crop height information, based on UAV imagery analysis, is affected by vegetable crop internal structure and agricultural management. Further research is needed to examine the effect of other crop phenotypic traits on model prediction quality. Overall, the study demonstrated that time-consuming manual height measurement can be replaced by remote sensing approaches for field vegetable crops, and recommend further multi-site and multi-crop studies to better understand the temporal variation between field measured crop height and crop height, based on UAV-borne RGB-based 3D point clouds. In the future, the suggested approach could be used for a real-time evaluation of vegetable biomass.
T.M. and M.W. conceived the idea of the study. T.M. and S.D. conducted the field work. T.M. processed the data and analyzed the results. All authors contributed to the writing and revision of the manuscript.
The authors gratefully acknowledge the financial support provided by the German Research Foundation, DFG, through grant numbers WA 2135/4-1 and BU 1308/13-1 and by the Indian Department of Biotechnology, DBT, through grant number DBT/IN/German/DFG/14/BVCR/2016 as part(s) of the Indo-German consortium of DFG Research Unit FOR2432/1 and DBT (The Rural-Urban Interface of Bangalore: A Space of Transitions in Agriculture, Economics, and Society). We also thank the anonymous reviewers as well as the editors for their valuable comments improving the previous manuscript.
Conflicts of Interest
The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.
Table A1. Sampling dates for studied crops.
|Sampling date 1||8 March 2017||9 March 2017||7 March 2017|
|Sampling date 2||29 March 2017||30 March 2017||28 March 2017|
|Sampling date 3||20 April 2017||18 April 2017||10 April 2017|
|Sampling date 4||16 May 2017||4 May 2017||11 May 2017|
|Sampling date 5||13 June 2017||5 June 2017||7 June 2017|
Table A2. Values for mean and standard deviation (brackets) of parameter used in the present study. The values for the height metrics and the measured canopy height are based on the values from the S- and H-plot (n = 24). The biomass information is based on the values from the H-plot (n = 12).
|Crop||Sampling Date||Measured Crop Height||Biomass [kg m−2]||Hmin||Hmax||Hmean||Hsd||Hmedian||Hskew||Hkurt||Hcv||Hq70||Hq80||Hq90||Hq95||Hq99||Hrelief|
|Cabbage||1||5.856 (0.379)||0.028 (0.028)||0.001 (0)||0.173 (0.417)||0.017 (0.050)||0.028 (0.107)||0.005 (0.001)||3.669 (3.537)||46.453 (97.946)||0.989 (0.437)||0.042 (0.145)||0.066 (0.248)||0.081 (0.306)||0.089 (0.316)||0.109 (0.329)||0.099 (0.056)|
|2||11.936 (1.020)||0.139 (0.139)||0.001 (0)||0.168 (0.508)||0.012 (0.012)||0.02 (0.057)||0.007 (0.001)||1.742 (0.673)||3.857 (3.906)||1.005 (0.688)||0.026 (0.039)||0.034 (0.056)||0.051 (0.116)||0.080 (0.234)||0.108 (0.328)||0.132 (0.035)|
|3||14.947 (1.061)||0.446 (0.446)||0.001 (0)||0.198 (0.446)||0.017 (0.007)||0.021 (0.042)||0.013 (0.001)||1.685 (1.356)||6.123 (12.136)||0.933 (0.737)||0.035 (0.022)||0.042 (0.034)||0.055 (0.063)||0.075 (0.130)||0.132 (0.322)||0.153 (0.052)|
|4||22.296 (0.892)||2.746 (2.746)||0.001 (0)||0.235 (0.374)||0.035 (0.005)||0.032 (0.031)||0.029 (0.002)||1.132 (1.417)||2.738 (12.325)||0.854 (0.462)||0.070 (0.016)||0.080 (0.024)||0.095 (0.044)||0.114 (0.086)||0.177 (0.298)||0.207 (0.047)|
|5||24.967 (1.029)||4.972 (4.972)||0.001 (0)||0.320 (0.643)||0.035 (0.004)||0.030 (0.022)||0.029 (0.002)||1.283 (1.885)||5.397 (22.835)||0.825 (0.401)||0.069 (0.011)||0.078 (0.015)||0.093 (0.027)||0.108 (0.052)||0.166 (0.233)||0.186 (0.054)|
|Eggplant||1||8.744 (0.883)||0.015 (0.015)||0.001 (0)||0.445 (1.299)||0.006 (0.002)||0.014 (0.033)||0.005 (0.001)||4.546 (6.417)||75.084 (171.526)||1.631 (2.860)||0.011 (0.005)||0.013 (0.007)||0.018 (0.016)||0.021 (0.017)||0.076 (0.174)||0.095 (0.056)|
|2||20.020 (1.736)||0.099 (0.099)||0.001 (0)||0.100 (0.013)||0.018 (0.004)||0.016 (0.003)||0.012 (0.003)||1.321 (0.248)||1.451 (1.016)||0.928 (0.067)||0.038 (0.007)||0.044 (0.008)||0.053 (0.009)||0.060 (0.009)||0.074 (0.009)||0.168 (0.030)|
|3||44.833 (3.433)||0.727 (0.727)||0.001 (0)||0.456 (0.260)||0.042 (0.004)||0.034 (0.003)||0.035 (0.004)||1.450 (0.493)||6.031 (9.016)||0.812 (0.038)||0.084 (0.008)||0.095 (0.009)||0.114 (0.010)||0.132 (0.010)||0.176 (0.016)||0.112 (0.047)|
|4||68.798 (4.790)||2.239 (2.239)||0.001 (0)||0.326 (0.103)||0.074 (0.039)||0.049 (0.015)||0.066 (0.042)||0.877 (0.395)||0.894 (1.329)||0.719 (0.108)||0.134 (0.056)||0.149 (0.058)||0.171 (0.061)||0.191 (0.062)||0.229 (0.063)||0.226 (0.097)|
|5||70.315 (4.778)||1.902 (1.902)||0.001 (0)||0.271 (0.042)||0.054 (0.010)||0.042 (0.007)||0.045 (0.009)||1.100 (0.167)||1.185 (0.719)||0.786 (0.036)||0.107 (0.019)||0.121 (0.021)||0.144 (0.024)||0.164 (0.026)||0.204 (0.030)||0.199 (0.033)|
|Crop||Sampling Date||Measured Crop Height||Biomass [kg m−2]||Hmin||Hmax||Hmean||Hsd||Hmedian||Hskew||Hkurt||Hcv||Hq70||Hq80||Hq90||Hq95||Hq99||Hrelief|
|Tomato||1||9.249 (0.592)||0.014 (0.014)||0.001 (0)||0.088 (0.187)||0.008 (0.011)||0.011 (0.033)||0.005 (0)||2.253 (1.805)||14.877 (22.386)||0.833 (0.408)||0.018 (0.035)||0.022 (0.046)||0.035 (0.102)||0.044 (0.132)||0.059 (0.162)||0.129 (0.056)|
|2||24.544 (2.504)||0.050 (0.050)||0.001 (0)||0.131 (0.240)||0.009 (0.006)||0.013 (0.025)||0.006 (0.001)||2.852 (0.848)||13.540 (9.779)||1.079 (0.576)||0.020 (0.018)||0.024 (0.026)||0.037 (0.054)||0.054 (0.106)||0.084 (0.169)||0.091 (0.027)|
|3||43.651 (4.829)||0.360 (0.360)||0.001 (0)||0.500 (0.534)||0.033 (0.006)||0.036 (0.007)||0.021 (0.006)||2.880 (4.579)||69.698 (311.362)||1.078 (0.128)||0.076 (0.012)||0.090 (0.014)||0.113 (0.020)||0.137 (0.031)||0.192 (0.064)||0.089 (0.030)|
|4||53.727 (5.782)||1.140 (1.140)||0.001 (0)||0.611 (0.374)||0.060 (0.030)||0.061 (0.038)||0.038 (0.011)||1.718 (0.403)||4.884 (3.454)||0.974 (0.126)||0.136 (0.080)||0.160 (0.097)||0.198 (0.120)||0.232 (0.137)||0.306 (0.159)||0.108 (0.033)|
|5||46.474 (3.091)||0.827 (0.827)||0.001 (0)||0.305 (0.134)||0.017 (0.005)||0.021 (0.010)||0.010 (0.002)||4.061 (1.885)||37.573 (41.772)||1.220 (0.213)||0.039 (0.013)||0.048 (0.018)||0.065 (0.029)||0.084 (0.045)||0.139 (0.075)||0.060 (0.023)|
Figure A1. Exemplary pictures of three crops used in the study. Top: Point clouds. Bottom: Photographs. From left to right: Tomato, cabbage, and eggplant. All images and point clouds were collected at the second sampling date.
- Malambo, L.; Popescu, S.C.; Murray, S.C.; Putman, E.; Pugh, N.A.; Horne, D.W.; Richardson, G.; Sheridan, R.; Rooney, W.L.; Avant, R.; et al. Multitemporal field-based plant height estimation using 3D point clouds generated from small unmanned aerial systems high-resolution imagery. Int. J. Appl. Earth Observ. Geoinf. 2018, 64, 31–42. [Google Scholar] [CrossRef]
- Zhang, C.; Kovacs, J.M. The application of small unmanned aerial systems for precision agriculture: A review. Precis. Agric. 2012, 13, 693–712. [Google Scholar] [CrossRef]
- Selsam, P.; Schaeper, W.; Brinkmann, K.; Buerkert, A. Acquisition and automated rectification of high-resolution RGB and near-IR aerial photographs to estimate plant biomass and surface topography in arid agro-ecosystems. Exp. Agric. 2017, 53, 144–157. [Google Scholar] [CrossRef]
- Yu, N.; Li, L.; Schmitz, N.; Tian, L.F.; Greenberg, J.A.; Diers, B.W. Development of methods to improve soybean yield estimation and predict plant maturity with an unmanned aerial vehicle based platform. Remote Sens. Environ. 2016, 187, 91–101. [Google Scholar] [CrossRef]
- Park, S.; Ryu, D.; Fuentes, S.; Chung, H.; Hernández-Montes, E.; O’Connell, M. Adaptive Estimation of Crop Water Stress in Nectarine and Peach Orchards Using High-Resolution Imagery from an Unmanned Aerial Vehicle (UAV). Remote Sens. 2017, 9, 828. [Google Scholar] [CrossRef]
- Johnson, C.K.; Mortensen, D.A.; Wienhold, B.J.; Shanahan, J.F.; Doran, J.W. Site-specific management zones based on soil electrical conductivity in a semiarid cropping system. Agron. J. 2003, 95, 303–315. [Google Scholar] [CrossRef]
- Lati, R.N.; Filin, S.; Eizenberg, H. Estimating plant growth parameters using an energy minimization-based stereovision model. Comput. Electron. Agric. 2013, 98, 260–271. [Google Scholar] [CrossRef]
- Madec, S.; Baret, F.; de Solan, B.; Thomas, S.; Dutartre, D.; Jezequel, S.; Hemmerlé, M.; Colombeau, G.; Comar, A. High-throughput phenotyping of plant height: Comparing unmanned aerial vehicles and ground LiDAR estimates. Front. Plant Sci. 2017, 8, 2002. [Google Scholar] [CrossRef] [PubMed]
- Hoffmeister, D.; Waldhoff, G.; Korres, W.; Curdt, C.; Bareth, G. Crop height variability detection in a single field by multi-temporal terrestrial laser scanning. Precis. Agric. 2016, 17, 296–312. [Google Scholar] [CrossRef]
- Tilly, N.; Hoffmeister, D.; Cao, Q.; Huang, S.; Lenz-Wiedemann, V.; Miao, Y.; Bareth, G. Multitemporal crop surface models: Accurate plant height measurement and biomass estimation with terrestrial laser scanning in paddy rice. J. Appl. Remote Sens. 2014, 8, 083671. [Google Scholar] [CrossRef]
- Fricke, T.; Richter, F.; Wachendorf, M. Assessment of forage mass from grassland swards by height measurement using an ultrasonic sensor. Comput. Electron. Agric. 2011, 79, 142–152. [Google Scholar] [CrossRef]
- Bendig, J.; Bolten, A.; Bennertz, S.; Broscheit, J.; Eichfuss, S.; Bareth, G. Estimating biomass of barley using crop surface models (CSMs) derived from UAV-based RGB imaging. Remote Sens. 2014, 6, 10395–10412. [Google Scholar] [CrossRef]
- Leberl, F.; Irschara, A.; Pock, T.; Meixner, P.; Gruber, M.; Scholz, S.; Wiechert, A. Point clouds. Photogramm. Eng. Remote Sens. 2010, 76, 1123–1134. [Google Scholar] [CrossRef]
- Tumbo, S.D.; Salyani, M.; Whitney, J.D.; Wheaton, T.A.; Miller, W.M. Investigation of laser and ultrasonic ranging sensors for measurements of citrus canopy volume. Appl. Eng. Agric. 2002, 18, 367–372. [Google Scholar] [CrossRef]
- Li, W.; Niu, Z.; Chen, H.; Li, D. Characterizing canopy structural complexity for the estimation of maize LAI based on ALS data and UAV stereo images. Int. J. Remote Sens. 2017, 38, 2106–2116. [Google Scholar] [CrossRef]
- Weiss, M.; Baret, F. Using 3D point clouds derived from UAV RGB imagery to describe vineyard 3D macro-structure. Remote Sens. 2017, 9, 111. [Google Scholar] [CrossRef]
- Schirrmann, M.; Giebel, A.; Gleiniger, F.; Pflanz, M.; Lentschke, J.; Dammer, K.-H. Monitoring agronomic parameters of winter wheat crops with low-cost UAV imagery. Remote Sens. 2016, 8, 706. [Google Scholar] [CrossRef]
- Parkes, S.D.; McCabe, M.F.; Al-Mashhawari, S.K.; Rosas, J. Reproducibility of crop surface maps extracted from Unmanned Aerial Vehicle (UAV) derived Digital Surface Maps. In Remote Sensing for Agriculture, Ecosystems, and Hydrology XVIII; Neale, C.M.U., Maltese, A., Eds.; SPIE: Bellingham, WA, USA, 2016. [Google Scholar]
- Prasad, J.V.N.S.; Rao, C.S.; Srinivas, K.; Jyothi, C.N.; Venkateswarlu, B.; Ramachandrappa, B.K.; Dhanapal, G.N.; Ravichandra, K.; Mishra, P.K. Effect of ten years of reduced tillage and recycling of organic matter on crop yields, soil organic carbon and its fractions in Alfisols of semiarid tropics of southern India. Soil Tillage Res. 2016, 156, 131–139. [Google Scholar] [CrossRef]
- Lowe, D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004, 60, 91–110. [Google Scholar] [CrossRef]
- Snavely, N.; Seitz, S.M.; Szeliski, R. Modeling the world from internet photo collections. Int. J. Comput. Vis. 2008, 80, 189–210. [Google Scholar] [CrossRef]
- Westoby, M.J.; Brasington, J.; Glasser, N.F.; Hambrey, M.J.; Reynolds, J.M. ‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications. Geomorphology 2012, 179, 300–314. [Google Scholar] [CrossRef]
- Mesas-Carrascosa, F.-J.; Torres-Sánchez, J.; Clavero-Rumbao, I.; García-Ferrer, A.; Peña, J.-M.; Borra-Serrano, I.; López-Granados, F. Assessing optimal flight parameters for generating accurate multispectral orthomosaicks by uav to support site-specific crop management. Remote Sens. 2015, 7, 12793–12814. [Google Scholar] [CrossRef]
- Röder, M.; Hill, S.; Latifi, H. Best Practice Tutorial: Technical Handling of the UAV “DJI Phantom 3 Professional” and Processing of the Acquired Data; Department of Remote Sensing, University of Würzburg: Würzburg, Germany, 2017. [Google Scholar]
- Tucker, C.J. Red and photographic infrared linear combinations for monitoring vegetation. Remote Sens. Environ. 1979, 8, 127–150. [Google Scholar] [CrossRef]
- Motohka, T.; Nasahara, K.N.; Oguma, H.; Tsuchida, S. Applicability of green-red vegetation index for remote sensing of vegetation phenology. Remote Sens. 2010, 2, 2369–2387. [Google Scholar] [CrossRef]
- Silva, C.A.; Hudak, A.T.; Klauberg, C.; Vierling, L.A.; Gonzalez-Benecke, C.; de Padua Chaves Carvalho, S.; Rodriguez, L.C.E.; Cardil, A. Combined effect of pulse density and grid cell size on predicting and mapping aboveground carbon in fast-growing Eucalyptus forest plantation using airborne LiDAR data. Carbon Balance Manag. 2017, 12, 13. [Google Scholar] [CrossRef] [PubMed]
- Næsset, E.; Økland, T. Estimating tree height and tree crown properties using airborne scanning laser in a boreal nature reserve. Remote Sens. Environ. 2002, 79, 105–115. [Google Scholar] [CrossRef]
- Roussel, J.-R.; Auty, D. lidR: Airborne LiDAR Data Manipulation and Visualization for Forestry Applications. Available online: https://CRAN.R-project.org/package=lidR (accessed on 18 May 2018).
- R Core Team. R: A Language and Environment for Statistical Computing; R Core Team: Vienna, Austria, 2016; Available online: https://www.R-project.org/ (accessed on 18 May 2018).
- Breiman, L. Random Forests. Mach. Learn. 2001, 45, 5–32. [Google Scholar] [CrossRef]
- Cortes, C.; Vapnik, V. Support-vector networks. Mach. Learn. 1995, 20, 273–297. [Google Scholar] [CrossRef]
- Liaw, A.; Wiener, M. Classification and Regression by randomForest. R News 2002, 2, 18–22. [Google Scholar]
- Meyer, D.; Dimitriadou, E.; Hornik, K.; Weingessel, A.; Leisch, F. e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien, 2015. Available online: https://CRAN.R-project.org/package=e1071 (accessed on 18 May 2018).
- Li, W.; Niu, Z.; Chen, H.; Li, D.; Wu, M.; Zhao, W. Remote estimation of canopy height and aboveground biomass of maize using high-resolution stereo images from a low-cost unmanned aerial vehicle system. Ecol. Indic. 2016, 67, 637–648. [Google Scholar] [CrossRef]
- Garcia-Gutierrez, J.; Martínez-Álvarez, F.; Troncoso, A.; Riquelme, J.C. A comparative study of machine learning regression methods on LiDAR data: A case study. In International Joint Conference SOCO’13-CISIS’13-ICEUTE’13; Herrero, Á., Baruque, B., Klett, F., Abraham, A., Snášel, V., de Carvalho, A.C.P.L.F., Bringas, P.G., Zelinka, I., Quintián, H., Corchado, E., Eds.; Springer International Publishing: Cham, Switzerland, 2014; pp. 249–258. [Google Scholar]
- Horning, N. Random forests: An algorithm for image classification and generation of continuous fields data sets. In Proceeding of the International Conference on Geoinformatics for Spatial Infrastructure Development in Earth and Allied Sciences; Hanoi University of Mining and Geology: Hanoi, Vietnam, 2010. [Google Scholar]
- Niederheiser, R.; Rutzinger, M.; Bremer, M.; Wichmann, V. Dense image matching of terrestrial imagery for deriving high-resolution topographic properties of vegetation locations in alpine terrain. Int. J. Appl. Earth Observ. Geoinf. 2018, 66, 146–158. [Google Scholar] [CrossRef]
- Cunliffe, A.M.; Brazier, R.E.; Anderson, K. Ultra-fine grain landscape-scale quantification of dryland vegetation structure with drone-acquired structure-from-motion photogrammetry. Remote Sens. Environ. 2016, 183, 129–143. [Google Scholar] [CrossRef]
- Iqbal, F.; Lucieer, A.; Barry, K.; Wells, R. Poppy Crop Height and Capsule Volume Estimation from a Single UAS Flight. Remote Sens. 2017, 9, 647. [Google Scholar] [CrossRef]
- Bendig, J.; Yu, K.; Aasen, H.; Bolten, A.; Bennertz, S.; Broscheit, J.; Gnyp, M.L.; Bareth, G. Combining UAV-based plant height from crop surface models, visible, and near infrared vegetation indices for biomass monitoring in barley. Int. J. Appl. Earth Observ. Geoinf. 2015, 39, 79–87. [Google Scholar] [CrossRef]
- Moeckel, T.; Safari, H.; Reddersen, B.; Fricke, T.; Wachendorf, M. Fusion of ultrasonic and spectral sensor data for improving the estimation of biomass in grasslands with heterogeneous sward structure. Remote Sens. 2017, 9, 98. [Google Scholar] [CrossRef]
- Safari, H.; Fricke, T.; Reddersen, B.; Moeckel, T.; Wachendorf, M. Comparing mobile and static assessment of biomass in heterogeneous grassland with a multi-sensor system. J. Sens. Sens. Syst. 2016, 5, 301–312. [Google Scholar] [CrossRef]
Figure 1. (A) India’s political map shows the location of Bengaluru; (B) The experimental design of the study. S indicates plots in which all destructive measured parameters were sampled (i.e., biomass harvest). H indicates plots in which all non-destructive measurements were conducted (i.e., spectral sampling). S plots were used for model calibration, while H plots were used for model validation.
Figure 2. Process chain of the analysis. Yellow box (left) describes the biomass sampling and modelling, the green box (centre) shows the crop height sampling, and the blue box (right) describes the point cloud processing. S-plot indicates plots in which all destructive measured parameters were sampled (i.e., biomass harvest). H indicates plots in which only non-destructive measurements were conducted (i.e., spectral sampling). S plots were used for model calibration, while H plots were used for model validation. * Green-Red Vegetation Index.
Figure 3. Field measured average crop height versus predicted average crop height for random forest regression (top) and support vector regression (bottom). From the left to the right, the results are presented for eggplant, tomato, and cabbage.
Figure 4. Relative deviation of the predicted crop height values based on random forest regression (top) and support vector regression (bottom) from the measured crop height values for each sampling date (01–05). From the left to the right, the results are presented for eggplant, tomato, and cabbage.
Figure 5. Biomass versus the (top) field measured crop height and (below) predicted crop height based on random forest regression.
Figure 6. Predicted versus observed biomass values based on (top) field measured crop height and (bottom) predicted crop height. The symbols indicate the five sampling dates: (square) sampling date 1, (circle) sampling date 2, (triangle) sampling date 3, (plus) sampling date 4, and (cross) sampling date 5. The black lines indicate the 1:1 fit of the values.
Table 1. Height metrics derived from point clouds and used as explanatory variables in modelling crop height.
|Hmin||Minimum crop height|
|Hmax||Maximum crop height|
|Hmean||Mean crop height|
|Hsd||Standard deviation of crop height|
|Hmedian||Median crop height|
|Hskew||Skewness of crop height|
|Hkurt||Kurtosis of crop height|
|Hcv||Coefficient of variation crop height|
|Hq70||70th percentile of crop height|
|Hq80||80th percentile of crop height|
|Hq90||90th percentile of crop height|
|Hq95||95th percentile of crop height|
|Hq99||99th percentile of crop height|
|Hrelief||Crop canopy relief height (Hmean-Hmin)/(Hmax-Hmin)|
Table 2. Effects of sampling date and N fertilizer on average crop height for eggplant, tomato, and cabbage grown at UAS Bangalore, India.
|Sampling Date (SD)||N Fertilizer (NF)||SD × NF|
Table 3. Correlation among the point cloud derived crop height metrics.
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).