Plot-Level Maize Early Stage Stand Counting and Spacing Detection Using Advanced Deep Learning Algorithms Based on UAV Imagery

Abstract

Phenotyping is one of the most important processes in modern breeding, especially for maize, which is an important crop for food, feeds, and industrial uses. Breeders invest considerable time in identifying genotypes with high productivity and stress tolerance. Plant spacing plays a critical role in determining the yield of crops in production settings to provide useful management information. In this study, we propose an automated solution using unmanned aerial vehicle (UAV) imagery and deep learning algorithms to provide accurate stand counting and plant-level spacing variabilities (PSV) in order to facilitate the breeders’ decision making. A high-resolution UAV was used to train three deep learning models, namely, YOLOv5, YOLOX, and YOLOR, for both maize stand counting and PSV detection. The results indicate that after optimizing the non-maximum suppression (NMS) intersection of union (IoU) threshold, YOLOv5 obtained the best stand counting accuracy, with a coefficient of determination (R²) of 0.936 and mean absolute error (MAE) of 1.958. Furthermore, the YOLOX model subsequently achieved an F1-score value of 0.896 for PSV detection. This study shows the promising accuracy and reliability of processed UAV imagery for automating stand counting and spacing evaluation and its potential to be implemented further into real-time breeding decision making.

1. Introduction

Crop breeding is a critical process underlying global food security and raw material production [1]. Data of high quality and accuracy produced in breeding field trials are a fundamental component of breeding success. Variation is regularly observed on the field, region, and country levels due to genetic variation, environmental effects, and the differential responses of genotypes grown in multiple environments on the expression of a given phenotype (e.g., grain yield), known as genotype by environment (G × E) interaction [2]. Non-genetic error variation and G × E impact almost every aspect of the decision-making process involved in plant-breeding programs [3]. Therefore, accounting for and modeling variation are important for improving the breeding process [4].

Maize (Zea mays ssp. Mays) is a major crop grown in the United States and one of the most widely consumed grains in the world. Since the 1930s, the combination of improved genetics and production practices has resulted in consistent increases in yield per acre, with increments in grain yields of around 2.5 bu/ac per year [5]. The annual average maize grain yield fluctuates due to yearly weather patterns, resulting in variable impacts on the economy and global food security. It has been reported that these fluctuations are mainly due to variability in growing conditions from year to year [6,7].

Crops respond differently to environmental factors, including agronomic management practices. For maize, both planting density and plant-level spacing variabilities (PSV) have significant effects on grain yield [8,9], and different hybrids have different optimal densities required to achieve the maximum yield. A reduction in the number of plants or an increase in the variability of plant spacing will result in the decline in the grain yield achieved per unit of planted land [10,11]. Unlike the grain yield, which is determined by harvesting an entire plot with a plot combine, the number of plants per plot and the variability in space in each breeding row can only be obtained through visual inspections at the plot level. This activity is time-consuming and labor-intensive. Furthermore, while the stand count is recorded more regularly, PSV is a much more difficult trait to quantify, as it requires measuring the distance between every pair of plants in a row [9] and is thus very rarely measured or reported. PSV is also an important source of error persisting in the analysis of maize trials, as statistical models that account for patterns of variation across fields cannot account for plot-specific factors such as PSV.

Unmanned aerial vehicles (UAVs) have vast potential for applications in agriculture and provide an opportunity to support a high-throughput method with which to detect plant stands [12]. UAV-based sensing platforms cover large areas in a short amount of time compared with ground vehicles or robots. They can provide data with higher spatial and temporal resolutions than satellite imagery [13]. With a superior data collection speed and resolution, UAVs have become increasingly important tools used to acquire high-resolution RGB (visible light spectrum) [14], multispectral (specific wavelength ranges across the spectrum) [15], and hyperspectral (light emitted outside of the visible spectrum) [16,17] imagery for plant phenotyping in the digital agriculture field.

To date, there have been successful applications combining UAV imagery and deep learning in precision agriculture, such as estimations of crop counts and estimations of plant density in field conditions. Kuznetsova et al. [18] compared the performance of You Only Look Once (YOLO) v3 and v5 in orchard apple detection. In the experiment, YOLOv5 outperformed YOLOv3 with a false negative rate (FNR) of 2.8% and false positive Rate of 3.5% (FPR). Ong et al. [19] used RGB UAV imagery and a convolutional neural network (CNN) to detect weeds in Chinese cabbage, and the overall accuracy was 92.41%. Fend et al. [20] developed a nearly real-time UAV image-processing algorithm based on a convolutional neural network (CNN) and obtained encouraging results for cotton stand counting (R² = 0.95). Lin et al. [21] compared the MobileNet and CenterNet algorithms and concluded that CenterNet has a better performance in cotton plant counting. Ranđelović et al. [22] applied RGB UAV imagery and a machine learning model to predict soybean density and achieved promising results, with a correlation coefficient equal to 0.87. Mhango et al. applied a faster region-based convolutional neural network (FRCNN) to estimate potato densities using UAV imagery. The estimated results had high correlation with the manually determined ground truth (R² = 0.80) [23]. To the best of our knowledge, previous studies mostly focused on applying UAV imagery and deep learning techniques in calculating either the stand count or plant density but not PSV.

To build an object detection model that is efficient and reliable in detecting both maize stands and PSV, we compared the performance of two YOLO series models, namely, YOLOv5 and YOLOX, and one unified network for multiple tasks, called YOLOR (You Only Learn One Representation) [24,25]. The YOLO model is the best-known and most popular one-stage object detection algorithm due to its small model size and fast calculation speed [26]. It was first introduced by Redmon et al. in 2015 [27]. Enhanced through its generations, YOLOv5 has more flexible control of the model size without a loss of accuracy. The YOLOX model is built on the YOLOv3 baseline with a DarkNet53 model [28] as the backbone and has been shown to have a better detection accuracy [24]. The YOLOR was designed to mimic human learning; it can learn from both features and implicit knowledge and has demonstrated promising object detection results. Therefore, this study aimed to evaluate the performance of the three different deep learning object detection models and optimize their thresholds in estimating stand counting for maize-breeding plots and quantifying the PSV.

2. Materials and Methods

2.1. Experimental Sites and Design

The experiments were conducted at the West Madison Agricultural Research Station (WMARS) in Verona, Wisconsin, USA (43°03’37” N 89°31’54” W), and the Arlington Agricultural Research Station (AARS) in Arlington, Wisconsin, USA (43°18’13” N 89°23’04” W). Five maize hybrids with good germination rates were selected as the experimental materials, among which three were from the Field Corn Breeding Program and two from the private sector (Dekalb: DKC46-60 and DKC48-12RIB). The five hybrids are in the 96–102 relative maturity group. The experimental materials were planted following a split-plot design on 11 May 2021 at the WMARS and 18 May 2021 at the AARS, respectively. Each plot was 6.7 m in length and 0.97 m in row spacing. At each of the sites, the plots received one of the five plot-level spacing treatments (Figure 1), and the subplots accommodated the five maize hybrids with five replicates, leading to 125 plots in total.

Each plot was planted as a two-row plot with 40 seeds evenly distributed every 0.15 m at each row, resulting in approximately 80 seeds in each plot. Thirteen days after planting (DAP), the maize seedlings at the West Madison site were manually removed to create varied spacing distances between pairs so as to form the four plot-level treatment conditions shown in Figure 1. The plot-level treatment conditions 1 to 4, shown in Figure 1, were Perfect, Not Enough Seeds (NES), Skips, and Big Gaps. The plot-level treatment condition 5 (Misplacements) was manual planting with two plants next to each other, which increased the planting density by 10%. The average seedling numbers of all the plots at this site were 65, 63, 64, 80, and 94 for Skips, Not Enough Seeds, Big Gaps, Perfect Spacings, and Misplacements, respectively. In Arlington, the plots were planted in the same way as above but did not receive treatments for NES, Skips, or Big Gaps. Only the PSV for Perfect and Misplacements is counted here, resulting in 100 and 25 plots for the conditions of Perfect Spacing and Misplacements, respectively.

2.2. UAV Image Acquisition

Aerial image data were collected using a Phantom 4 Pro V2.0 (DJI, Shenzhen, Guangdong, China) with an onboard RGB camera. The flight camera had a resolution of 4864 × 3648 pixels and was set to take timelapse photos at 2 frames per second. The drone was set to fly at 12 m above the ground with 80% image overlapping, and the ground sampling distance (GSD) was 0.37 cm. The flight plan was pre-defined in a flight control app called FieldAgent (Sentera, Saint Paul, MN, USA).

The aerial images were taken on 3 June 2021, at both sites, when the maize plants developed the third leaf collar (roughly in the V3 stage). The leaves could be clearly seen in the images. Plots with a plant density not meeting our experiment design were discarded, leading to a total of 249 images for validating the deep learning methods.

2.3. Image Processing and Labeling

Each maize plot was visually identified and cropped from the center of the raw RGB image that was taken directly above the plot. Thus, no orientated adjustment was needed for the plot image. The image dimensions for individual plots were, on average, 1900 × 400 pixels. A free and open-source image-labeling package, LabelImg [29], was used to manually label the maize stands and PSV for individual plots. All targets (either maize stands or PSV) were visually detected and labeled by drawing rectangular bounding boxes around the edges of the targets. In stand counting, each bounding box corresponds to an individual maize plant. If the plants overlap with each other, the bounding boxes will also overlap. For PSV, each individual box represents a specific spacing variation. A single bounding box for the Misplacements contains all the maize plants in a particular location, while Skips and Big Gaps include plants both before and after the spacing, along with the gap itself. In the case of Big Gaps, the spacing length typically accommodates more than two plants, whereas the Skips involve only a single plant. Two lists each containing the pixel positions of the four vertices of the labeled bounding boxes were saved as separate Extensible Markup Language (.xml) files for the maize stands and PSV in each plot. Figure 2a shows a single labeled maize stand in an example maize plot, and Figure 2b shows its labeled PSV spots.

2.4. Model Development

2.4.1. Detecting and Counting Maize Stands

Three models were developed for detecting maize stands, namely, YOLOv5, YOLOX, and YOLOR. The YOLOv5 model has a similar structure compared to the previous YOLO–YOLOv3 that uses Mosaic data augmentation for the input data. However, the addition of an auto-anchor check in YOLOv5 enables YOLOv5 to converge faster than the previous YOLO series and maintain high accuracy.

Based on the modified version of YOLOv3 with Darkent-53 as its backbone, YOLOX is an anchor-free object detection algorithm with a decoupled head to improve object detection accuracy. This mechanism ensures that YOLOX has a better performance compared with the other YOLO series, which use coupled detection heads. Similarly, the YOLOX model was built on the version with minimum parameters.

Though it has a similar name, YOLOR does not adopt any of the architectures in the YOLO series. YOLOR is a unified network used to encode implicit knowledge and explicit knowledge together. Given the small sample size of our dataset, all three models were configured with the version of the smallest model size, including YOLOv5s, YOLOX-s, and YOLOR-P6, to prevent potential overfitting.

The models were developed and trained using Google Colab. The images of 249 maize plots at the two sites and their bounding boxes for the maize stands were randomly split into training, testing, and validation sets with proportions of 7:2:1. In order to enhance the training process, all three models were trained using transfer learning, leveraging the pre-trained weights from the MS COCO dataset [24,25,30]. During training, the dataset was divided into mini-batches with a batch size of 16. An epoch was counted when all the minibatches were fed to the model, and three epoch numbers (500, 1000, 1500) were used for comparing the training and testing performances. As increasing the epoch may yield a better accuracy but can also lead to overfitting, it is important to choose the best weights during the whole training process. Thus, the best weights will be saved and used to evaluate the accuracy of each model and further compare their performances.

All other parameters, such as the learning rate, were kept with the original settings that each algorithm was designed with, which are shown in

Table 1. Parameters used for the three models for training. All the parameters are the original settings.

Model	Parameter	Values
YOLOv5	Learning rate	1 × 10−3
	Momentum	0.98
	Learning decay	1 × 10−2
YOLOX	Learning rate	0.01 × BatchSize/64
	Momentum	0.9
	Learning decay	5 × 10−4
YOLOR	Learning rate	2.61 × 10−3
	Momentum	0.949
	Learning decay	5 × 10−4

.

2.4.2. Detecting the PSV

The three model architectures introduced in Section 2.4 were also used for detecting the PSV. Instead of taking the positions of the maize stands as training responses, the models for different PSV were trained to output the locations of the spacing variation spots (plant pairs with abnormal distances) and of the categories (Big Gaps, Skips, and Misplacements) for each spot. Compared with the stand counting, the data only focused on PSV, while Perfect and NES were not counted in the dataset, so that the size of the training and testing data was smaller. Thus, the training epochs needed to be decreased, and each of the models was trained with 200, 500, 800, and 1000 epochs.

The workflow of both detecting maize stands and the PSV is shown in Figure 3.

2.5. Evaluation of Model Performance

The performance of the developed models in detecting the maize stands was evaluated based on the classification confusion matrix between the model outputs and the manual labels of the testing dataset [31]. Precision, recall, and F1-scores are the indicators most widely used to summarize object detection performance, using Equations (1)–(3). Equation (4) computes the average precision by taking the area under the precision–recall curve p(r). The precision and recall values of the model are plotted as a function of the model to obtain the p(r) curve.

$$Precision=\frac{TP}{TP+FP}$$

(1)

$$Recall=\frac{TP}{TP+FN}$$

(2)

$$F1=2\times \frac{Precision\times Recall}{Precision+Recall}$$

(3)

$$AP={{\displaystyle \int }}_{r=0}^{1}p\left(r\right)dr$$

(4)

The mean absolute error (MAE) and coefficient of determination (R²) between the model and manual counts were used as metrics for evaluating the performance in counting the maize stands from images, computed as shown in Equations (5) and (6), where TP, FP, and FN stand for true positive, false positive, and false negative, respectively. They were obtained from the confusion matrices between true classes and model outputs.

$$MAE=\frac{1}{n}{\displaystyle \sum }_1^n\left|y_i-{\widehat{y}}_i\right|$$

(5)

$${\mathrm{R}}^2=1-\frac{{{\displaystyle \sum }}_1^n{\left(y_i-{\widehat{y}}_i\right)}^2}{{{\displaystyle \sum }}_1^n{\left(y_i-{\overline{y}}_i\right)}^2}$$

(6)

where n is the number of images, and y_i and ${\widehat{y}}_i$ are the manual and model counts (i.e., number of bounding boxes) of the i^th image, respectively. ${\overline{y}}_i$ is the mean manual count of the images. Pearson’s correlation coefficient was used to further investigate the linear relationship between the maize stand counts and ground truth under the Misplacement treatment using Equation (7).

$${\rho }_{xy}=\frac{Cov\left(x,y\right)}{{\sigma }_x{\sigma }_y}$$

(7)

Similarly, the performance in detecting the PSV was evaluated using the precision, recall, and F1-scores of each algorithm. F1-score is sufficient for performance evaluation due to the lack of overlapping bounding boxes for each variability in PSV detection.

3. Results

3.1. Detecting and Counting Maize Stands from UAV-Based Imagery

The performance of the three models in detecting the maize stands from the UAV canopy images is shown in

Table 2. Performance in maize stand detection and counting.

Model	Training Epoch	Average Precision	Coefficient of Determination (R2)	Mean Absolute Error (MAE)
YOLOv5	500	0.917	0.621	6.208
	1000	0.931	0.708	5.688
	1500	0.921	0.724	5.333
YOLOX	500	0.898	0.805	3.542
	1000	0.889	0.773	3.792
	1500	0.882	0.710	4.354
YOLOR	500	0.920	0.790	4.583
	1000	0.904	0.767	4.958
	1500	0.902	0.838	4.104

. The best AP were 0.931, 0.898, and 0.920 for YOLOv5, YOLOX, and YOLOR. These findings align with a previous study that achieved a mean average precision (mAP) of 86% by employing YOLOv3 and training it with 200 annotated images for estimating cotton stand count using UAS (unmanned aerial system) images [32].

Regarding the performance in counting the maize stands, the YOLOX model under 500 training epochs achieved the smallest MAE = 3.542, resulting from the strong agreement between the image-based and manual counts, especially for the groups of Skips, NES, Big Gaps, and Perfect Spacings. The errors of YOLOX were mainly caused by the underestimated stand counts of the Misplacement group. The underestimation of overlapping plants is highlighted with orange dots in Figure 4 for YOLOX, deviating from the red line. This issue is due to the large proportion of overlapping areas between the maize stands in the Misplacement group when the data were collected. It is also notable that the further training of the model (under 1000 and 1500 epochs) could not aid in reducing the underestimation of overlapping plants caused by Misplacement.

Table 2 reveals inconsistencies among the three evaluation criteria: AP, R², and MAE. YOLOv5 has the highest AP, while YOLOR performs best in terms of R². It is important to note that MAE has the nature of measuring the average error, in contrast to the others [33]. R² may negatively impact YOLOX due to the underestimated Misplacement group. AP only emphasizes the accuracy in the detection of the bounding boxes and fails to reflect the counting accuracy. Hence, MAE serves as the most appropriate evaluation metric here.

Compared with YOLOX, both the YOLOv5 and YOLOR outputs mostly agreed with those in the Misplacement group’s detection. However, clear offsets were observed between the image-based and manual counts for the other four groups. These overestimations were caused by the fact that the threshold setting of the intersection over union (IoU) of non-maximum suppression (NMS) for determining the detection bounding boxes as positive or negative outputs for each stand (Section 3.2) was not optimized. Therefore, it was observed that the YOLOX model was more suitable for counting maize stands under the Perfect, Skips, NES or Big Gaps conditions, where the plants were not crowded and did not severely overlap with each other.

Through visual observation of the testing dataset for all 48 images, it was found that both YOLOv5 and YOLOR produced redundant bounding boxes under the Perfect, Skips, NES, and Big Gaps spacing treatments compared with YOLOX. However, YOLOX detected a value less than the true value under the Misplacement treatment. Figure 5 shows the original plot-level image and the maize stand counting demo for the three deep learning models under the Misplacement spacing treatment. From the demo, we can see that both YOLOv5 and YOLOR have overlapping bounding boxes that constitute multiple entries at the same location, whereas YOLOX tends to detect the plant without overlapping. Compared with Figure 6, which presents another spacing treatment, namely, Skips, both YOLOv5 and YOLOR count 84 maize stands, as compared with ground truth 76. Both YOLOv5 and YOLOR have some redundant counts because both models output more than one bounding box for a single maize stand, and the defined IoU threshold was not able to differentiate between false and true positive boxes. Consequently, though the average precision (AP) of YOLOv5 and YOLOR were higher than that of YOLOX, the false positive boxes caused a higher MAE, as shown in Table 1.

3.2. Determining the Optimal NMS IoU Threshold

In the previous sections, we demonstrated that both the YOLOv5 and YOLOR models overestimated the samples from the Perfect, Skips, NES, and Big Gaps groups, in which the maize plants were not crowded or severely overlapping with each other. The YOLOX performed well for these four groups but showed underestimation for the Misplacement group. This was due to varied ability in eliminating redundant detection boxes among the three models in different PSV scenarios, which was controlled using the NMS IoU threshold for the three models. The NMS IoU is a method commonly used to select single entity (i.e., the bounding boxes in this case) out of many overlapping entities by discarding those that are below a given thresholding parameter. The thresholding parameter is calculated as the IoU between the pairs of detected bounding boxes.

The PSV treatments with varied overlapping conditions might favor different IoU values. The Misplacement group underestimated by YOLOX might benefit from a looser threshold (a higher IoU) to retain the bounding boxes that were previously removed, while the groups overestimated by YOLOv5 and YOLOR might favor a more restrictive threshold (lower IoU values) for eliminating redundancies. Therefore, the determination of the optimal NMS IoU value in each scenario for the models plays a key role in improving the overall counting accuracy.

The results in the previous section were based on the default NMS IoU thresholds of 0.5, 0.5, and 0.3 for YOLOv5, YOLOR, and YOLOX, respectively. Nonetheless, as a result of the unique treatment of spacing variabilities in the dataset. Misplacement led to the presence of some bounding boxes that overlap with one another more than 90% in the training dataset. Both YOLOv5 and YOLOR learned this special feature and tended to detect more stand counts under the other four PSV. Thus, maize stand detection under these special spacing variabilities required further evaluations of the NMS IoU thresholds.

Table 3. Average precision under different NMS IoU thresholds.

Model	Training Epoch	NMS IoU Threshold	AP
YOLOv5	500	0.1	0.921
		0.2	0.924
		0.3	0.923
		0.4	0.920
		0.5	0.917
	1000	0.1	0.934
		0.2	0.934
		0.3	0.933
		0.4	0.931
		0.5	0.931
	1500	0.1	0.922
		0.2	0.929
		0.3	0.929
		0.4	0.926
		0.5	0.921
YOLOX	500	0.1	0.897
		0.2	0.898
		0.3	0.898
		0.4	0.898
		0.5	0.898
	1000	0.1	0.806
		0.2	0.892
		0.3	0.891
		0.4	0.891
		0.5	0.891
	1500	0.1	0.800
		0.2	0.799
		0.3	0.799
		0.4	0.878
		0.5	0.882
YOLOR	500	0.1	0.927
		0.2	0.931
		0.3	0.932
		0.4	0.930
		0.5	0.920
	1000	0.1	0.931
		0.2	0.935
		0.3	0.936
		0.4	0.937
		0.5	0.904
	1500	0.1	0.927
		0.2	0.931
		0.3	0.933
		0.4	0.932
		0.5	0.902

shows the initial evaluation of the performance when changing the NMS thresholds from 0.1 to 0.5.

In general, the average precision values for all three models did not differ significantly across the different NMS IoU thresholds. However, there is still an improvement in the AP results when comparing the numbers in Table 3 with those in Table 2. Since the main purpose of optimizing the NMS IoU threshold was to help the models to have the best performance under both the special spacing treatment, Misplacement, and the other four groups (Perfect, Skips, NES, and Big Gaps), instead of demonstrating the models’ performance based on all the spacing variabilities, Misplacement was separated from the other four groups.

Table 4. AP, Pearson’s correlation, and MAE under various NMS IoU thresholds.

Model	NMS IoU	Misplacement			Perfect, Skips, NES, or Big Gaps
		AP	Pearson’s Correlation	MAE	AP	Pearson’s Correlation	MAE
YOLOv5-1000	0.1	0.851	0.162	15.5	0.955	0.978	2.711
	0.2	0.851	0.503	12.4	0.958	0.986	1.605
	0.3	0.85	0.358	10.6	0.957	0.990	1.316
	0.4	0.846	0.470	8.1	0.957	0.986	1.184
	0.5	0.84	0.319	4.9	0.956	0.964	5.895
	0.6	0.84	0.379	12.2	0.957	0.932	12.421
	0.7	0.836	0.441	23.4	0.957	0.888	19.447
	0.8	0.817	0.440	42.6	0.953	0.781	34.316
	0.9	0.708	0.570	95.0	0.868	0.624	88.105
YOLOX-500	0.1	0.716	0.569	13.1	0.902	0.969	2.868
	0.2	0.799	0.920	10.9	0.902	0.967	2.053
	0.3	0.799	0.853	10.2	0.902	0.975	1.789
	0.4	0.798	0.791	8.7	0.902	0.976	1.553
	0.5	0.798	0.833	7.0	0.902	0.973	1.579
	0.6	0.798	0.615	4.9	0.902	0.968	1.763
	0.7	0.798	0.610	5.0	0.902	0.965	1.947
	0.8	0.796	0.615	12.2	0.902	0.952	4.000
	0.9	0.776	0.723	43.8	0.893	0.824	25.053
YOLOR-1000	0.1	0.856	0.182	14.5	0.955	0.983	2.632
	0.2	0.857	0.395	12.4	0.958	0.979	2.000
	0.3	0.861	0.688	10.2	0.959	0.979	1.763
	0.4	0.858	0.692	8.9	0.961	0.979	1.579
	0.5	0.853	0.707	3.9	0.959	0.959	5.289
	0.6	0.848	0.529	7.7	0.959	0.945	9.974
	0.7	0.842	0.596	14.2	0.959	0.932	13.158
	0.8	0.834	0.663	24.5	0.957	0.926	19.974
	0.9	0.774	0.323	68.5	0.924	0.848	59.184

shows the performance for the Misplacement group and the other four groups under changing NMS IoU thresholds. R² is not used here to evaluate the models' performance, since the selection of the sample size can significantly impact the anticipated R². In this case, Misplacement only has 10 samples. With small numbers of samples, the mean of the coefficient of determination may exhibit a high value, even if there is no correlation between the variables [34]. Here, the MAE is a more meaningful metric with which to assess accuracy, as it solely quantifies the differences in quality between the predicted stand count value and the ground truth. Pearson’s correlation is used here to demonstrate the linear correlation between the updated stand counts and the ground truth.

Table 4 reveals that the models’ accuracy is influenced by the NMS IoU, as determined by the MAE. For the Misplacement PSV, both YOLOv5 and YOLOR display an optimal performance when the threshold is set to 0.5, yielding MAE values of 4.9 and 3.9, respectively. As described in Section 3.1, YOLOX tends to underestimate the maize stand counts but can achieve better results when the NMS IoU is set to 0.6, with an MAE equal to 4.9. These outcomes indicate that the NMS IoU threshold must be optimized and can have different values under different PSV. For example, while YOLOv5 achieves optimal results for Misplacement when the threshold is set to 0.5, it shows diminished accuracy for the other four groups. Similarly, with different optimized NMS IoU thresholds under various PSV, the detection results can be enhanced, as shown for YOLOX and YOLOR. The correlation coefficient for all three models across various thresholds also demonstrates a linear correlation between the detected maize stand counts and the ground truth.

Table 5. Stand counting results after choosing the best average precision NMS IoU threshold.

Model	Training Epoch	NMS IoU Threshold (Misplacement/Other Four)	Coefficient of Determination (R2)	Mean Absolute Error (MAE)
YOLOv5	1000	0.5/0.4	0.936	1.958
YOLOX	500	0.6/0.4	0.918	2.417
YOLOR	1000	0.5/0.4	0.946	2.063

shows the re-evaluation results after choosing the optimized NMS IoU threshold for the PSV of Misplacement and the other four groups. The R² and MAE are improved dramatically compared with the results shown in Section 3.1, with all three models having an R² above 0.9 and an MAE of approximately 2.0. In Figure 7, the Misplacement (orange dots) for YOLOX shift towards the ground truth. Additionally, for both YOLOv5 and YOLOR, Figure 7 shows that the problem of overestimating for the other four groups can be resolved by choosing an optimized NMS IoU threshold, as compared with the results in Figure 4. The experiment results indicate that after carefully choosing the optimized NMS IoU threshold, all three models could yield promising results under different PSVs.

3.3. Detecting and Visualizing the PSV

The performance in the detection of PSV from UAV images on the plant level is shown in

Table 6. Precision, Recall, and F1-score for maize spacing treatment detection with three different deep learning models under different epochs.

Model	Training Epoch	Precision	Recall	F1
YOLOv5	200	0.861	0.921	0.890
	500	0.862	0.89	0.876
	800	0.886	0.862	0.876
	1000	0.845	0.834	0.840
YOLOX	200	0.941	0.854	0.896
	500	0.903	0.825	0.862
	800	0.831	0.864	0.847
	1000	0.899	0.856	0.877
YOLOR	200	0.785	0.877	0.829
	500	0.876	0.867	0.872
	800	0.879	0.876	0.878
	1000	0.875	0.852	0.863

. Overall, all three models displayed a promising performance, with all showing an F1 above 0.80. YOLOX under 200 training epochs achieved the highest accuracy, with a precision of 0.941 and F1-score of 0.896, followed by YOLOv5 under 200 training epochs. For both YOLOv5 and YOLOX, increasing the number of training epochs did not increase the F1-score, which was caused by the fact that the size of the training dataset was even smaller than that for maize stand counting. However, YOLOR under 800 training epochs reached the best performance, with an F1-score equal to 0.878, which may mean that YOLOR needs more training epochs to learn the features from the training data.

It is difficult to determine the best model by only evaluating the F1-scores in the table, since all the F1-scores are close to each other. This is mainly caused by the fact that when detecting the PSV, one will only locate the variabilities in a single bounding box, instead of multiple bounding boxes as in the previous maize stand counting. The examples in Figure 8 demonstrate the performance of the three models in detecting PSV. In general, all three models could successfully detect multiple spacing variabilities (indicated by the color and label of each bounding box) in the maize plots. The detected positions and corresponding categories closely matched the manual labels. However, it can be noted from Figure 8 that YOLOv5 and YOLOR tended to detect empty spaces at the end of each plot as Big Gaps, while YOLOX could avoid this issue.

From the visualization results, we can consider the developed method to be promising in detecting PSV in maize plots for practical use.

4. Discussion

It was proven in previous studies that crop stand counting could be achieved using deep learning object detection algorithms [21,35,36]. In the case of our study, YOLOX initially achieved the lowest MAE among all the advanced deep learning models that were used, the others being YOLOv5 and YOLOR, for plot-level maize stand counting. After optimizing the NMS IoU threshold, all three models demonstrated promising results, with an R² above 0.9 and MAE of around 2. YOLOX outperformed YOLOv5 and YOLOR in the PSV detection applications.

In the field of object detection, newly developed algorithms such as the ones used in our study have been shown to have much more accurate results than the traditionally adopted methods, such as Faster-RCNN or YOLOv3. For traditional object detection, YOLOX-s [24] outperforms YOLOv5-s with a 2.9% AP, and YOLOR [25] proves that a single model architecture is still effective for multi-task learning. Despite their seemingly better performance in traditional object detection, there are only a few studies showing that these methods have the potential to be used in the field of precision agriculture or related areas. Zhang et al. [37] adopted YOLOX for the fruit-counting area and achieved a value of 99.519% with respect to the best detection rate. Additionally, Song et al. [38] used the improved YOLOX-tiny for tree height estimation based on fisheye images, and the highest relative error of the tree measurements was 4.06%. Zhong et al. [39] successfully adopted YOLOR to detect real-time marine animals in a coral reef ecosystem, and it achieved an AP above 0.79 in both the fish and turtle categories.

Plant spacing is an important topic in maize production, and it has been reported that with well-tuned planters and uniform plant spacing, the yields could be 20% greater than those of fields with PSV [9]. Wang et al. [40] presented a vision-based method to measure maize plant spacing and population in the early growth stage. Tang et al. [41] proposed different methods to detect and measure the spacing between corn plant stem centers through mosaic images. Nevertheless, these works focused on normal spacing situations and the distance between each plant, which requires the information regarding the center of the maize to be known before the measurements. This can be problematic, since this information can sometimes be very vague in the case of special spacings and can introduce additional uncertainties. Additionally, the plant distance for certain types of plant spacings can vary considerably. Thus, our experiment not only proved that YOLOv5, YOLOX, and YOLOR can be applied in small crop object detection and have a better accuracy than the previous advanced models but also demonstrated that the different successfully detected PSV can be used for further analysis.

However, some limitations should be noted. First, the flying height is an important factor that will influence the detection accuracy, since the object of interest is relatively small [42]. This presents a challenging problem, as small objects may lack the information required to distinguish them from background objects [43,44] such as weeds and soil. The ideal strategy for maximizing the information collected requires the UAVs to fly at a relatively low altitude. The tradeoff between the flying height and image quality is that drones can cover a larger area by flying at higher altitudes or a smaller area at a lower altitude. Additionally, most deep learning algorithms require a large training dataset, and the current traditional object detection datasets, such as COCO [45], contain 328 k images. Additionally, researchers [46] showed that a small training dataset may cause an unstable deep learning algorithm performance. Based on our experiment results, when the training epoch increased, the results were not guaranteed to be improved.

In future studies, platforms other than UAVs could be applied to stand counting in order to acquire higher-resolution imagery, as in the study of Wang et al. [36], who mounted a camera on a cart and collected video sequences. This study showed a promising stand counting result with an accuracy over 98%. Flying height derivation may influence the results of deep learning because the imagery may contain different information. Thus, different flying heights should be accounted for in further analyses. Since this experiment only focused on the V3 stage, it was possible to acquire the earlier stage of maize growth in order to avoid leaf overlapping. Additionally, data augmentation is another tool that could be considered as a solution to enlarge the number of training datasets in order to prevent model overfitting due to limited data, as stated in [47]. Data augmentation could improve deep learning accuracy in traditional object detection tasks. Therefore, more studies need to be conducted to examine the height influence in order to find the optimal flying altitude as well as the optimal size of the dataset and thus obtain a stable and, therefore, more reliable result.

5. Conclusions

In this study, three deep learning models, YOLOv5, YOLOX, and YOLOR, were applied to detect and count maize and spacing variabilities in the V3 stage using UAV imagery. Overall, all three deep learning models were able to conduct plot-level stand counting and determine PSV at a satisfactory level. Under different spacing variabilities, the NMS IoU threshold is an important parameter that needs to be optimized in order to obtain a more robust model. For the Misplacement, the NMS IoU should be set to around 0.5 to 0.6, compared with the other PSV, which can have a lower threshold. After adjusting the optimal NMS IoU threshold, the YOLOv5 model had a better overall performance for the different spacing treatments, indicated by its better value of R² = 0.936 and lowest MAE value = 1.958. All three algorithms achieved closely satisfied results in detecting PSV, with the best F1-score above 0.85. When detecting PSV through visual examination, YOLOX outperformed the other models, with an F1-score = 0.896. In our experiment, we proved that deep learning could potentially be used to accurately and efficiently detect maize stands and PSV from UAVs imagery. The detection outcomes could be used in breeding experiments as phenotypic information in future analyses for genotype–phenotype mapping, understanding genotype by environment interaction, or breeding selection. The proposed method could also be adopted in crop production settings, providing a reference to maize growers for precision management activities.