UAV-Based Computer Vision System for Orchard Apple Tree Detection and Health Assessment

Abstract

Accurate and efficient orchard tree inventories are essential for acquiring up-to-date information, which is necessary for effective treatments and crop insurance purposes. Surveying orchard trees, including tasks such as counting, locating, and assessing health status, plays a vital role in predicting production volumes and facilitating orchard management. However, traditional manual inventories are known to be labor-intensive, expensive, and prone to errors. Motivated by recent advancements in UAV imagery and computer vision methods, we propose a UAV-based computer vision framework for individual tree detection and health assessment. Our proposed approach follows a two-stage process. Firstly, we propose a tree detection model by employing a hard negative mining strategy using RGB UAV images. Subsequently, we address the health classification problem by leveraging multi-band imagery-derived vegetation indices. The proposed framework achieves an F1-score of 86.24% for tree detection and an overall accuracy of 97.52% for tree health assessment. Our study demonstrates the robustness of the proposed framework in accurately assessing orchard tree health from UAV images. Moreover, the proposed approach holds potential for application in various other plantation settings, enabling plant detection and health assessment using UAV imagery.

1. Introduction

Tree diseases can have a significant impact on orchard quality and productivity, which is a major concern for the agricultural industry. Unhealthy or stressed orchard trees are more susceptible to pests, diseases, and environmental stressors, which can reduce the yield and quality of fruit and lead to financial losses for growers. Therefore, developing methods for surveying and monitoring tree health and production quality is essential for orchard management. This can help growers make informed decisions about practices such as irrigation, fertilization, and pest control, optimize orchard yield and quality, reduce input usage, and improve the long-term sustainability of orchard production systems.

Traditional methods of monitoring orchard tree health, such as manual inspection and visual examination, rely on human expertise to determine quantitative orchard tree parameters. These methods are labor-intensive, time-consuming, costly, and subject to errors. In recent years, remote sensing platforms such as satellites, airplanes, and unmanned aerial vehicles (UAVs) [1,2,3,4] have provided new tools that offer an alternative to traditional methods. Deep neural networks (DNNs) [5] have also emerged as a powerful tool in the field of machine learning. The high spatial resolution provided by UAV images combined with computer vision algorithms have made tremendous advances in several domains such as forestry [6], agriculture [7], geology [8], surveillance [9], and traffic monitoring [10].

Motivated by the latest advances in machine-learning-based computer vision systems, this paper proposes a new framework to automatically detect orchard apple trees and assess their health from UAV multi-band imagery. The proposed framework adopts a two-stage approach. First, the tree localization problem is addressed using visual object detection. The second stage deals with tree health assessment through image patch classification.

1.1. Tree Detection

Over the years, both classical machine learning and deep learning methods have been extensively explored to address the tree detection problem.

Classical object detection methods often involve the utilization of handcrafted features and machine learning algorithms. Local binary pattern (LBP) [11], scale-invariant feature transform (SIFT) [12,13], and histogram of oriented gradients (HOG) [14,15] are the most frequently used handcrafted features in object detection. For example, the work in [16] presented a traditional method for walnut, mango, orange, and apple tree detection. It adopts the template matching image processing approach to very high resolution (VHR) Google Earth images acquired over a variety of orchard trees. The template is based on a geometrical optical model created from a series of parameters, such as illumination angles, maximum and ambient radiance, and tree size specifications. In [17,18], the authors detected palm trees on UAV RGB images by extracting a set of key points using the scale-invariant feature transform (SIFT). The key points are then analyzed with an extreme learning machine (ELM) classifier, which is a priori trained on a set of palm and no-palm tree keypoints. Similarly, ref. [19] employed a support vector machine (SVM) for image classification into vegetation and non-vegetation patches. Subsequently, the HOG feature extractor was applied on vegetation patches for feature extraction. These extracted features were then used to train a SVM to recognize palm tree images from background regions. The study in [20] proposed an object detection method using shape features for detecting and counting palm trees. The authors employed circular autocorrelation of the polar shape (CAPS) matrix representation as the shape feature and the linear SVM to standardize and reduce the dimensions of the feature. Finally, the study uses a local maximum detection algorithm based on the spatial distribution of standardized features to detect palm trees. The work in [7] presented a method to detect apple trees using multispectral UAV images. The authors identified trees using thresholding techniques applied on the Normalized Difference Vegetation Index (NDVI) and entropy images, as trees are chlorophyllous bodies that have high NDVI values and are heterogeneous with high entropy. The work in [21] proposed an automated approach to detect and count individual palm trees from UAV images. It is based on two processing steps: first, the authors employed the NDVI to perform the classification of image features as trees and non-trees. Then, palm trees were identified based on texture analysis using the Circular Hough Transform (CHT) and the morphological operators. In [22], the authors applied k-means to perform color-based clustering followed by a thresholding technique to segment out the green portion of the image. Then, trees were identified by applying an entropy filter and morphological operations on the segmented image.

On the other hand, numerous studies have investigated the use of deep-learning algorithms to detect trees in UAV RGB imagery. For instance, ref. [23] detected citrus and other crop trees from UAV images using a CNN algorithm applied to four spectral bands (i.e., green, red, near infrared and red edge). The initial detection was followed by a classification refinement procedure using superpixels derived from a Simple Linear Iterative Clustering (SLIC) algorithm and a thresholding technique to address the confusion between trees and weeds and deal with the difficulty in distinguishing small trees from large trees. In [24], the authors adopted a sliding window approach for oil palm tree detection. A sliding window was integrated with a pre-trained AlexNet classifier to scan the input image and identify regions containing trees. The work in [25] exploited the use of state-of-the-art CNNs, including YOLO-v5 with its four sub-versions, YOLO-v3, YOLO-v4, and SSD300 in detecting date palm trees. Similarly, in [26], the authors explored the use of the YOLO-v5 with its subversions and DeepForest for the detection of orchard trees. In [27], three state-of-the-art object detection methods were evaluated for the detection of law-protected tree species: Faster Region-based Convolutional Neural Network (Faster R-CNN) [28], YOLOv3 [29], and RetinaNet [30]. Similarly, the work in [31] explored the use of Faster R-CNN, Single Shot Multi-Box Detector (SSD) [32], and R-FCN [33] architectures to detect seedlings.

Most of these works explored fine-tuning state-of-the-art object detectors for tree detection by taking an object detection model that is pre-trained on Benchmark datasets [34,35,36] and adapting it specifically for the task of detecting trees. However, applying these methods to UAV images has particular challenges [37] compared to conventional object detection tasks. For example, UAV images often have a large field of view with complex background regions, which can significantly disrupt detection accuracy. Furthermore, the objects of interest are often not uniformly distributed with respect to the background regions, creating an imbalance between positive and negative examples. Data imbalance can also be observed between easy and hard negative examples, since with UAV images, a large part of the background has regular patterns and can be easily analyzed for detection. We believe that applying deep learning detection algorithms directly in these situations is not an optimal choice [38], as they mostly assign the same weight to all training examples, so that during the training step easy examples may dominate the total loss and reduce training efficiency.

To mitigate this issue, hard negative mining (HNM) can be adopted for object detection. Various HNM approaches [37,39,40] involve iteratively bootstrapping a small set of negative examples, by selecting those that trigger a false positive alarm in the detector. For example, ref. [41] presented a training process of a state-of-the-art face detector by exploiting the idea of hard negative mining and iteratively updating the Faster R-CNN-based face detector with hard negatives harvested from a large set of background examples. Their method outperforms state-of-the-art detectors on the Face Detection Data Set and Benchmark (FDDB). Similarly, an improved version of faster R-CNN is proposed in [42], by using hard negative sample mining for object detection using PASCAL VOC dataset [36]. Likewise, ref. [43] used the bootstrapping of hard negatives to improve the performance of face detection on WIDER FACE dataset [44]. The authors pre-trained Faster R-CNN to mine hard negatives, before retraining the model. The work of [45] presented a cascaded Boosted Forest for pedestrian detection, which performs effective hard negative mining and sample reweighting, to classify the region proposals generated by RPN. The A-Fast-RCNN method, described in [46], adopts a different approach for generating hard negative samples, by using occlusion and spatial deformations through an adversarial process. The authors conducted their experiments on PASCAL VOC and MS-COCO datasets. Another approach to apply HNM using Single Shot multi-box Detector (SSD) is proposed in [47], where the authors use medium priors, anchor boxes with 20% to 50% overlap with ground truth boxes, to enhance object detector performance on the PASCAL VOC dataset. The proposed framework updates the loss function so that it considers the anchor boxes with partial and marginal overlap.

In our work, we propose a HNM approach for the tree detection stage, where the mined hard negative samples are included in the initial tree dataset to be considered during training. Then, we retrain the object detector using the true positive and false positive examples to enhance the discrimination power of the model. To the best of our knowledge, our work is the first to use a HNM approach for tree detection.

1.2. Tree Health Classification

Vegetation indices (VIs) have been introduced as indicators of vegetation status, as they provide information on the physiological and biochemical status of trees. These mathematical combinations of reflectance measurements are sensitive to different vegetation parameters, such as chlorophyll content, leaf area, and water stress. It has been shown through many studies [48,49,50] that by analyzing these indices, we can gain insights into the health and vitality of trees.

For example, the work in [51] presented a framework for orchard tree segmentation and health assessment. The proposed approach is applied to five different orchard tree species, namely plum, apricot, walnut, olive, and almond. Two vegetation indices, including visible atmospherically resistant index (VARI) and green leaf index (GLI) were used with the standard score (which is also known as z-score) for tree health assessment. The study in [52] proposed a process workflow for mapping and monitoring olive orchards at tree scale detail. Five VIs were investigated, including normalized difference vegetation index (NDVI), modified soil adjusted vegetation index 2 (MSAVI 2), normalized difference red edge Vegetation index (NDRE), modified chlorophyll absorption ratio index improved (MCARI2), and NDVI2. The authors applied statistical analyses to all calculated VIs. Similarly, ref. [53] presented an approach for Huanglongbing (HLB) disease detection on citrus trees. First, the trees were segmented using thresholding techniques applied on the normalized difference vegetation index (NDVI). Then, for each segmented tree, a total number of thirteen spectral features was computed, which include six spectral bands and seven vegetation indices. The indices studied were: NDVI, green normalized difference vegetation index (GNDVI), soil-adjusted vegetation index (SAVI), near infrared (NIR)—red(R), R/NIR, green (G)/R and NIR/R. A SVM classifier is then applied to distinguish between healthy and HLB-infected trees. The work in [54] presented a method for the identification of stress in olive trees. An SVM model was applied to VIs to classify each tree pixel into two categories: healthy and stressed. The work in [48] presented a method to monitor grapevine diseases affecting European vineyards. The authors explored the use of different features including spectral bands, vegetation indices and biophysical parameters. They conducted a statistical analysis for the selection of the best discriminating variables to separate between symptomatic vines including FD and GTD from asymptomatic vines (Case 1) and FD vines from GTD ones (Case 2).

In our work, we conducted a comprehensive investigation on the assessment of apple tree health using twelve different vegetation indices derived from muli-band UAV imagery. We also explored a set of machine learning classifiers to perform health classification based on vegetation indices.

The main contributions of this paper are summarized as follows:

• We propose a novel framework for automatic apple orchard tree detection and health assessment from UAV images. The proposed framework could be generalized for a wide range of other UAV applications that involve a detection/classification process.
• We adopt a hard negative mining approach for tree detection to improve the performance of the detection model.
• We formulate the tree health assessment problem as a supervised classification task based on vegetation indices calculated from multi-band images.
• We present an extensive experimental analysis covering various aspects of the proposed framework. Our analysis includes an ablation study demonstrating the importance of the HNM technique for tree detection, an exploration of several classification methods for health assessment, and a feature importance analysis within the context of health classification.

The rest of the paper is structured as follows. Section 2 provides a detailed description of the proposed framework. The experimental results are presented in Section 3 and discussed in Section 4, where we also suggest directions for future work. Finally, Section 5 concludes the paper.

2. Materials and Methods

2.1. Study Area

Images were captured during the summer of 2018 over two apple orchards in Souris, Prince Edward Island, Canada (Lat. 46.44633N, Long. 62.08151W), as shown in Figure 1. The surveyed orchard consists of 18 distinct types of apple trees, such as Cortland, Gala, Sunrise, Virginia Gold, Honey Gold, a mixed variety, Jona Gold, Russet, and Spygold. These trees vary in age, with some being young while others are older. The UAV images were taken using a MicaSense RedEdge3 multispectral camera (MicaSense Inc., Seattle, WA, USA). It has five sensors, including one sensor for each of the following spectral bands: B1 (blue) centered at 475 nm; B2 (green) centered at 560 nm; B3 (red) centered at 668 nm; B4 (RE) centered at 717 nm, and B5 (NIR) centered at 840 nm. The camera was mounted under a DJI Matrice 100 light Unmanned Aerial Vehicle (UAV). Its weight is slightly less than 2.0 kg. Both the camera and the UAV were connected to mission planner software from MicaSense to fly at 100 m above the ground with 70% overlap between adjacent images. Wind speed during image acquisition was less than 20 km/h and the weather condition was sunny-cloudy. The captured images were orthorectified and mosaicked together using Pix4D to cover the whole study area.

2.2. Dataset Construction

In order to prepare the data for object detection, the orthomosaic was subdivided into 515 × 512-pixel patches using a regular grid. Then, we performed manual annotation for patches, which consists of localizing our objects of interest (apple trees) manually through bounding boxes using the open-source VGG Image Annotator (VIA) tool [55]. We used the field observations performed by specialists to validate our data annotation. These patches were split into three subsets: training, validation, and testing, using 3-fold cross-validation to ensure an impartial evaluation of object detection models.

To prepare the dataset for tree health classification, we utilized field inventory data. This involved gathering tree localization information through GPS, measuring tree parameters using specialized instruments, and recording all observations in a field notebook. By mapping these findings to the acquired images, two different tree health status were assigned: healthy and unhealthy to each individual tree. Finally, we performed data splitting using stratified cross-validation to divide the dataset into training, validation, and testing subsets.

Figure 2 illustrates two instances of trees captured at ground level: one depicting a healthy tree and the other representing an unhealthy tree. Stressed apple trees exhibit a reduction in chlorophyll content, red or brown discoloration on apple leaves, and infected flowers turning black, produced by a loss of photosynthetic metabolism due to bacterial disease or fungus [56,57].

Our final dataset contained 2828 trees in total, where 2240 are healthy and 588 are unhealthy. Using stratified 10-fold cross-validation, the tree images were divided into training, validation, and testing sets, ensuring that the ratio of healthy to stressed trees remains consistent across subsets.

2.3. Proposed Framework

The objective of this research is to propose an automated framework for assessing tree health using UAV images. The proposed framework, as depicted in Figure 3, employs a two-stage approach to sequentially address two tasks: tree detection and tree health classification.

In the first stage, the input orthomosaic is subdivided into multiple patches of 512 by 512 pixels using a regular grid to fit the framework input. Then, each patch is subjected to a tree detection procedure, which aims to identify and locate trees within that patch. This is accomplished by outlining the detected trees using bounding boxes. Once the trees are identified through the tree detection module, the corresponding boxes containing the detected trees are extracted and cropped. These cropped boxes are then subjected to tree classification to determine their health status.

Due to potentially distracting complex backgrounds, tree recognition is a challenging task. As shown in Figure 4, some background objects (red rectangles) have a similar color, shape, and texture as the target tree objects (blue rectangles).

For tree health assessment, we investigated the use of vegetation ondices as it has been shown that these spectral features are of great importance to identify the tree health status.

2.3.1. Tree Detection Stage

The proposed method for tree detection, depicted in Figure 5, starts by training a baseline object detector on UAV RGB images manually annotated for tree detection. The performance of the baseline model is then assessed to identify false detections, corresponding to objects incorrectly detected as trees. In the object detection task, a detection is considered as false if its overlap with the ground truth object is lower than an overlap threshold. In our work, we select the samples that have no overlap with the target object and consider them as the hard negative samples. These hard negative examples are then used to create a new class, which is added to the training data. Afterward, we perform fine-tuning of the baseline tree detector using the updated training dataset that contains the two classes. Using false positive detections with no overlap with target objects as a new class is motivated by the fact that these samples represent a source of disturbance in the training data, which can lead to inaccurate detection. By doing this, the model can learn to distinguish between true positive (trees) and false positive (non-trees) examples. To address the imbalance problem between the target class and the hard negative class, we use the focal loss [30] as the objective function during fine-tuning of the object detector. This method helps to improve the accuracy of the tree detector by reducing the impact of the noise caused by hard negative samples in the training data.

In our work, we explored two state-of-the-art baseline models for object detection: YOLO [58] using its COCO pre-trained model and DeepForest prebuilt model [59] trained on The National Ecological Observatory Network (NEON [60]) crowns data set. The two baseline architectures are explained further in the Appendix A. The choice of YOLO-v5 and DeepForest as baseline models is motivated by their impressive performance, popularity, and suitability for the task. These models have achieved state-of-the-art accuracy and efficiency in object detection, making them strong candidates for comparison. Note that we finally adopted DeepForest as a baseline detection model in our framework. This choice is based on a performance analysis of the two architectures, as discussed further.

2.3.2. Tree Health Assessment

Figure 6 summarizes the proposed method for tree health mapping. It follows a machine learning pipeline. Each detected tree image from the first stage is extracted and cropped by selecting pixels within a rectangular area around the tree center using a 32-pixel box size.

First, we compute vegetation indices from the five-band raw reflectance tree images. Among the large number of vegetation indices that are typically used as indicators of vegetation status [61], there is a subset of VIs that are commonly applied for health assessment [62,63,64]. Because of their high sensitivity to changes in moisture content, pigment indices, and vegetation health, red, green, blue, red edge, and near-infrared bands are frequently used to evaluate the status of vegetation [65]. Using these bands, a total of 12 vegetation indices (

Table 1. Vegetation indices derived from UAV band reflectance.

Vegetation Index	Equation	Reference
Difference Vegetation Index	DVI = Near-infrared (NIR)-Red	[70]
Generalized Difference Vegetation Index	GDVI = NIR − Green	[71]
Green Normalized Difference Vegetation Index	GNDVI = (NIR − Green)/(NIR + Green)	[72]
Green-Red Vegetation Index	GRVI = NIR/Green	[71]
Normalized Difference Aquatic Vegetation Index	NDAVI = (NIR − Blue)/(NIR + Blue)	[73]
Normalized Difference Vegetation Index	NDVI = (NIR − Red)/(NIR + Red)	[70]
Normalized Difference Red-Edge	NDRE = (NIR − RedEdge)/(NIR + RedEdge)	[74]
Normalized Green	NG = Green/(NIR + Red + Green)	[69]
Normalized Red	NR = Red/(NIR + Red + Green)	[69]
Normalized NIR	NNIR = NIR/(NIR + Red + Green)	[69]
Red simple ratio Vegetation Index	RVI = NIR/Red	[75]
Water Adjusted Vegetation Index	WAVI = (1.5*(NIR − Blue))/((NIR + Blue) + 0.5)	[73]

) were calculated based on their ability to reveal stress. The difference vegetation index (DVI) and the normalized difference vegetation index (NDVI) were estimated based on the evidence of correlation with tree stress [53,66]. The green normalized difference vegetation index (GNDVI) and normalized difference red-edge (NDRE) were also estimated, as both have demonstrated high sensitivity to variations in chlorophyll concentrations [65,67,68]. As an ideal vegetation index should be most sensitive to vegetation dynamics and not sensitive to changes in soil background or variable illumination or atmospheric effects, The intensity-normalization method [69] was introduced to minimize errors due to these changes. The normalized Green (NG), normalized Red (NR), and normalized NIR (NNIR) were calculated using this method and included with features.

Each detected tree image is represented by a feature vector that contains the average value of each vegetation index map. This feature vector comprises 12 values, where each value represents the average of a specific vegetation index map. To address class unbalance, we applied the synthetic minority oversampling technique (SMOTE) to oversample the examples in the minority class.

Finally, we applied a set of classifiers to perform the classification of the health status of apple trees. These classifiers were chosen based on their suitability for handling multivariate data and their recognized performance in similar studies. In our work, we explored the use of support vector machines (SVM) [76], decision tree [77], random forests (RF) [78], k-nearest neighbors (KNN) [79], and light gradient boosting machine (LightGBM) [80], which are explained further in Appendix B.

2.4. Implementation

All experiments were conducted on a PC with Intel Core i7-7700 CPU, NVIDIA GeForce GTX1080 GPU, and 64 GB of RAM. The operating system used by the PC was Windows 10. For the tree detection stage, our approach involved using two benchmark models. We applied transfer learning to the YOLO model (with 300 epochs and a batch size of 16) and fine-tuned the DeepForest model (with 100 epochs and a batch size of 16). After mining hard negatives and updating our dataset, we further fine-tuned the detection model using the updated dataset. We experimented with freezing the entire backbone and varying numbers of layers. YOLO model (with 4 frozen backbone layers) yielded the best results. In the case of the DeepForest model, we performed a full retraining process using the updated dataset. For the tree health classification, we explored the use of various supervised classifiers. To handle the issue of class imbalance, we employed the SMOTE technique. A grid search strategy was used to identify the best parameters for each classifier. With respect to our chosen model, Random Forest, we selected the Gini index as the splitting criterion.

2.5. Evaluation Metrics

To evaluate the performance of our framework for tree detection, we use the following metrics.

• Precision $P_{\mathrm{d}}$ (Equation (1)) is the percentage of correct detections among all the detected trees.

  $$P_d=\frac{{\mathrm{TP}}_d}{{\mathrm{TP}}_d+{\mathrm{FP}}_d}$$

  (1)
• Recall $R_{\mathrm{d}}$ (Equation (2)) is the percentage of correctly detected trees over the total number of trees in the ground truth.

  $$R_d=\frac{{\mathrm{TP}}_d}{{\mathrm{TP}}_d+{\mathrm{FN}}_d}$$

  (2)
• $F1{\text{-}\mathrm{score}}_d$ (Equation (3)) is the harmonic average of precision and recall.

  $$F1{\text{-}\mathrm{score}}_d=2\ast \frac{{\mathrm{P}}_d\ast {\mathrm{R}}_d}{{\mathrm{P}}_d+{\mathrm{R}}_d}$$

  (3)

In Equations (1)–(3), the subscript d denotes detection, ${\mathrm{TP}}_d$ is the number of true positives (i.e., correctly detected trees), ${\mathrm{FP}}_d$ is the number of false positives (i.e., regions incorrectly detected as trees), and ${\mathrm{FN}}_d$ denotes the number of false negatives (i.e., the number of missed trees). On a test image, a detection is considered as correct if the overlap between the detected tree and the tree in the ground truth was greater than 50%. The overlap between the detection and the ground truth is computed using the Intersection Over Union (IOU) metric (Equation (4)).

$$\mathrm{IOU}=\frac{Area(B_1\cap B_2)}{Area(B_1\cup B_2)},$$

(4)

here $B_1$ is the ground truth bounding box and $B_2$ is the predicted bounding box.

To evaluate the performance of our framework for tree health classification, we use the following metrics.

• Precision $P_{\mathrm{c}}$ (Equation (5)) is defined as the ratio of correct classifications for a given class to the total number of classifications made for that class.

  $$P_c=\frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FP}}_c}$$

  (5)
• Recall $R_{\mathrm{c}}$ (Equation (6)) is defined as the ratio of correct classifications for a given class to the total number of instances that actually belong to that class.

  $$R_c=\frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FN}}_c}$$

  (6)
• $F1{\text{-}\mathrm{score}}_c$ is the harmonic average of $P_{\mathrm{c}}$ and $R_{\mathrm{c}}$ of a given class.
• Accuracy (Equation (7)) is defined as the ratio of the correct classifications to the total number of tree instances classified.

  $$Accuracy=\frac{\mathrm{Number}\mathrm{of}\mathrm{correct}\mathrm{classifications}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{trees}\mathrm{classified}}$$

  (7)

In Equations (5)–(7), the subscript c denotes classification, ${\mathrm{TP}}_c$ is the number of correctly classified instances of a given class, ${\mathrm{FP}}_c$ is the number of instances incorrectly classified as belonging to a given class, and ${\mathrm{FN}}_c$ is the number of instances incorrectly classified as not belonging to a given class.

3. Results

3.1. Tree Detection

Table 2. Detailed and overall cross-validation results of the tree detection stage using hard negative mining approach in terms of precision, recall, and F1-score. Values in bold font correspond to the best-achieved results.

	Baseline 1: DeepForest			Baseline 2: YOLO
Fold	${\mathit{P}}_{\mathrm{d}}$(%)	${\mathit{R}}_{\mathrm{d}}$(%)	$\mathbf{F}\mathbf{1}{\text{-}\mathbf{Score}}_{\mathit{d}}$(%)	${\mathit{P}}_{\mathrm{d}}$(%)	${\mathit{R}}_{\mathrm{d}}$(%)	$\mathbf{F}\mathbf{1}{\text{-}\mathbf{Score}}_{\mathit{d}}$(%)
Fold 1	82.25	87.24	84.67	83.89	87.27	85.15
Fold 2	87.57	88.06	87.82	79.12	92.35	84.91
Fold 3	87.87	84.73	86.27	83.09	87.45	84.38
Average	85.85	86.67	86.24	82.01	88.99	84.81

shows the detailed and overall cross-validation results of the proposed tree detection approach using a hard negative mining approach for each baseline model. We can notice that the proposed detection method using the two baselines is stable across all folds. This demonstrates the robustness of our model, which is able to perform well and consistently on different partitions. We can also see that the DeepForest model is slightly better than YOLO, achieving a higher F1-score average of 86.24%.

DeepForest’s superior performance could be explained by the fact that it is specifically designed for tree detection in aerial imagery. It offers several kernel-specific advantages over YOLO for tree detection. DeepForest is trained on a large dataset that includes UAV images of different tree species, ages, and environmental conditions. This specialized training allows it to leverage domain-specific knowledge and detect trees with high accuracy and specificity, even in challenging scenarios such as partial occlusions or low contrast with the background.

We conducted an ablation study to evaluate the importance of mining hard examples for tree detection. We reported the results of training baseline object detectors without hard negative samples by comparison to our HNM-based approach.

Table 3. Ablation study result of the detection step: overall 3-fold cross-validation results of two baseline models applied with and without HNM. Values in bold font correspond to the best results.

Baseline Model	${\mathit{P}}_{\mathbf{d}}$ (%)	${\mathit{R}}_{\mathbf{d}}$ (%)	$\mathbf{F}1{\text{-}\mathbf{Score}}_{\mathit{d}}$ (%)
Without HNM
Baseline 1: DeepForest	84.82	86.18	85.46
Baseline 2: YOLO	79.40	88.05	82.64
With HNM
Baseline 1: DeepForest	85.85	86.67	86.24
Baseline 2: YOLO	82.01	88.99	84.81

reports the overall 3-fold cross-validation results using two baseline models: DeepForest and YOLO-v5 [81]. We train YOLO-v5 using the COCO pre-trained model and DeepForest using its prebuilt model. The reported results show that both YOLO and DeepForest benefited from hard negative mining.

YOLO fine-tuned with the mined hard negatives achieved an F1-score of 84.81%, outperforming the YOLO baseline by 2.17%, which suggests that the detector learns to eliminate a number of false detections. Using the DeepForest model, the inclusion of hard negatives in training improves the performance compared to the baseline, with an improvement of 0.78% based on the F1-score. Overall, our results demonstrate the importance of HNM in tree detection to improve the detection ability in both baseline models.

3.2. Health Classification

The tree health assessment stage is formulated as a patch image classification problem. We applied different classifiers including SVM, decision tree, and Random Forest using twelve spectral features computed from five-band imagery (Red, green, blue, near-infrared, and red edge).

Table 4. Overall cross-validation results of different ML classifiers for tree health classification using vegetation indices in terms of precision, recall, F1-score, and overall accuracy. Values in bold font correspond to the best-achieved accuracy.

Health Status	${\mathit{P}}_{\mathbf{c}}$ (%)	${\mathit{R}}_{\mathbf{c}}$ (%)	$\mathbf{F}1{\text{-}\mathbf{Score}}_{\mathit{c}}$ (%)	Accuracy (%)
Random Forest Classifier (RF)
Healthy	98.8	98.8	98.8	97.52
Unhealthy	95.5	95.5	95.5
Light Gradient Boosting Machine (LightGBM)
Healthy	99	98.8	98.9	97.47
Unhealthy	95.6	96.1	95.8
K-nearest neighbors (KNN)
Healthy	98.6	98.1	98.4	97.07
Unhealthy	92.9	95	93.9
Support Vector Machine (SVM)
Healthy	99.4	95.8	97.6	96.31
Unhealthy	86.2	97.8	91.6
Decision Tree Classifier (DT)
Healthy	98.4	97.9	98.1	95.91
Unhealthy	92.3	93.9	93.1

presents the performance of each classifier for each class in terms of precision, recall, F1-score, and overall accuracy.

We can notice that all the explored classifiers performed well in the two classes. Specifically, they achieved higher results for the healthy class. Based on the overall accuracy, RandomForest [78] is the best classifier, achieving 97.52%. Random forests classifier adopts an ensemble approach by combining multiple decision trees to improve the accuracy and robustness of predictions. It randomly selects subsets of features and samples from the dataset to create a set of decision trees. Each tree independently makes predictions based on the selected features. The final prediction is obtained by combining the predictions of all the trees in the forest.

These results show the effectiveness of random forest in accurately classifying tree health status based on the selected vegetation indices.

In

Table 5. Feature importance using Random Forest.

Ranking	Feature	Contribution
1	NR	34%
2	NNIR	25%
3	RVI	16%
4	NDAVI	15%
5	DVI	10%

, we present the feature importance ranking generated by the Random Forest classifier. It allows us to assess the contribution of each feature to the overall accuracy of the model. The feature importance is determined using a Gini importance metric [82], which measures the frequency with which a particular feature is utilized for data splitting across all trees in the forest. This metric is weighted by the number of samples in each node.

We can conclude that the Normalized Red (NR) and Normalized Near-Infrared (NNIR) features are the most important features used by RandomForest to distinguish between healthy and stressed trees.

The Normalized Red (NR) index measures the reflectance of red light in vegetation. It is often used to estimate the amount of chlorophyll in plant leaves, which is an indicator of plant health. Chlorophyll is responsible for photosynthesis, and healthy trees typically have higher chlorophyll content than unhealthy trees. Therefore, the contribution of NR suggests that chlorophyll content is a key factor in determining tree health.

On the other hand, the Normalized Near-Infrared (NNIR) index measures the reflectance of near-infrared light in vegetation. It is often used to estimate the amount of vegetation biomass, which is an indicator of tree productivity. Healthy trees typically have a higher biomass than unhealthy trees. Therefore, the importance of NNIR suggests that biomass is also a key factor in determining tree health.

Taken together, the importance of NR and NNIR indicates that tree health status is determined by a combination of factors related to chlorophyll content and biomass.

4. Discussion

Over the years, the use of unmanned aerial vehicles (UAVs) in precision agriculture has experienced significant growth. Initially used for crop monitoring, UAV applications have expanded to include pest and disease control, treatment, and even providing real-time decision support. Our study specifically focuses on mapping individual tree crowns and evaluating their health using high-resolution UAV imagery and computer vision techniques. Our proposed method utilizes hard negative mining strategy to perform individual tree detection. To assess the health of the detected trees, vegetation indices derived from multi-band UAV images were employed.

4.1. Tree Detection

Our proposed method for individual tree detection based on the HNM technique demonstrated its efficiency compared to the use of a baseline object detector. The inclusion of hard negative samples during model training improves the model’s ability to distinguish between the target object (apple trees in our case) and other background objects. We believe that the application of the hard negative mining (HNM) learning strategy could be promising on UAV images as this kind of image displays the challenge of a complex background that may distract the visual object detection as shown in Figure 4. To the best of our knowledge, this is the first work to apply the HNM strategy for tree detection. In comparison to existing works in the literature, our proposed tree detection method achieves interesting results that are comparable or better than other studies. Our F1-score of 86.24% is comparable to the F1-score of [83] (86%), who uses a local maximum algorithm to detect pine trees on UAV-derived canopy height model imagery. Similarly, ref. [20] obtained an F1-score of 75.75% using a SVM based on shape features to detect palm trees on UAV RGB images, which is also slightly lower than our achieved F1-score.

Overall, our results demonstrate the effectiveness of hard negative mining approach for tree detection and highlight the importance of using specialized models like DeepForest for this task.

4.2. Trees Health Assessment

In comparison to previous studies carried out to address tree health assessment, our classification accuracy of 98.06% was higher than the one obtained by [84] (93.17%) using an SVM classifier to map the degree of damage caused by forest pests on UAV-based hyperspectral images. It was also higher than the one obtained by [85], who mapped tree health status using multispectral imagery (95%) or hyperspectral imagery (91%). Our method also outperforms statistical methods reviewed in [86] using generalized linear models (GLM), maximum entropy (ME), and random forests (RF), achieving an overall accuracy of 87.3%, 93.9%, and 97.1%, respectively, to identify areas affected by bark beetles on satellite imagery. Generally, these results showed that our approach utilizing supervised classification via Random Forests provides similar or better levels of accuracy to comparable studies. Furthermore, the conducted feature importance analysis underscores the contribution on NR and NNIR indices in assessing apple trees’ health. This highlights the value and efficiency of vegetation indices derived from remote sensing imagery in autonomously assessing vegetation health, eliminating the need for manual human intervention.

In the context of precision agriculture, these findings align with numerous studies that have leveraged Unmanned Aerial Vehicles (UAVs) for efficient and precise applications such as monitoring water stress [87,88], production volume estimation [89], and assessment of resource efficiencies [90]. This reinforces the paradigm shift towards technology-driven agricultural practices, promoting enhanced productivity and sustainable resource use.

Our future work aims to investigate the use of other bands such as red edge and near-infrared for tree detection, as they may contain relevant information for accurate detection. Moreover, hyperspectral imaging technology can also be explored for tree detection. It can provide both spatial geometric information, such as the tree crown shape, size, and texture, as well as the relationship between adjacent tree crowns, and precise spectral information. Forestry and agriculture have benefited from the use of this technology [91] for tree identification, species classification [92], tree health monitoring [93], and biodiversity assessment [93].

5. Conclusions

In this paper, an effective two-stage framework is proposed to automate orchard apple tree health assessment from UAV images. The first stage addresses the tree detection problem using a hard negative mining approach to improve tree detection performance. The second stage deals with tree health classification, where we use Random forest based on twelve selected vegetation indices. The proposed framework has the potential to reduce errors associated with manual field inventory, and eliminate the need for time-consuming and expensive field surveys. Furthermore, our framework produces maps showcasing stressed trees across orchards. Utilizing a tree health map for an orchard provides multiple advantages for growers. It enables the identification of potential issues at an early stage, precise intervention, strategic planning for harvesting, and enhanced crop management. This map can be used as a guide for further investigation by growers. Thus, orchard managers can benefit from the early detection of stress to perform targeted plantation management strategies. By leveraging this valuable information, growers can enhance productivity, minimize risks, and promote the long-term health and success of their orchard.