Computer Vision and Deep Learning as Tools for Leveraging Dynamic Phenological Classification in Vegetable Crops

Abstract

The efficiency of agricultural practices depends on the timing of their execution. Environmental conditions, such as rainfall, and crop-related traits, such as plant phenology, determine the success of practices such as irrigation. Moreover, plant phenology, the seasonal timing of biological events (e.g., cotyledon emergence), is strongly influenced by genetic, environmental, and management conditions. Therefore, assessing the timing the of crops’ phenological events and their spatiotemporal variability can improve decision making, allowing the thorough planning and timely execution of agricultural operations. Conventional techniques for crop phenology monitoring, such as field observations, can be prone to error, labour-intensive, and inefficient, particularly for crops with rapid growth and not very defined phenophases, such as vegetable crops. Thus, developing an accurate phenology monitoring system for vegetable crops is an important step towards sustainable practices. This paper evaluates the ability of computer vision (CV) techniques coupled with deep learning (DL) (CV_DL) as tools for the dynamic phenological classification of multiple vegetable crops at the subfield level, i.e., within the plot. Three DL models from the Single Shot Multibox Detector (SSD) architecture (SSD Inception v2, SSD MobileNet v2, and SSD ResNet 50) and one from You Only Look Once (YOLO) architecture (YOLO v4) were benchmarked through a custom dataset containing images of eight vegetable crops between emergence and harvest. The proposed benchmark includes the individual pairing of each model with the images of each crop. On average, YOLO v4 performed better than the SSD models, reaching an F1-Score of 85.5%, a mean average precision of 79.9%, and a balanced accuracy of 87.0%. In addition, YOLO v4 was tested with all available data approaching a real mixed cropping system. Hence, the same model can classify multiple vegetable crops across the growing season, allowing the accurate mapping of phenological dynamics. This study is the first to evaluate the potential of CV_DL for vegetable crops’ phenological research, a pivotal step towards automating decision support systems for precision horticulture.

1. Introduction

Plant phenology comprises the seasonal timing of biological events (e.g., cotyledon emergence) and the causes of their timing concerning biotic and abiotic interactions (agricultural practices included) [1,2].

The timing of crops’ phenological phases (phenophases) provides valuable information for monitoring and simulating plant growth and development. Therefore, it allows the thorough planning and timely execution of agricultural practices, which are usually carried out according to specific phenophases (e.g., timing of irrigation and harvest), leading to higher and more stable crop yields, sustainable practices, and improved food quality [3,4].

Field observation, crop growth models and remote sensing approaches are the most common methods for phenology monitoring. Field observation involves in situ plant assessment and the manual recording of phenophases (e.g., leaf unfolding). This observational approach is laborious and prone to error due to the large spatiotemporal variability of phenophases among plants [5,6]. Consequently, an increased sampling density is needed to adequately represent crop phenology at the field scale. Although crop growth models can simulate the timing of the phenophases of a particular genotype (individual), they cannot transpose the simulation to different spatial scales, as the heterogeneities in climate and management conditions are neglected [7,8,9]. Moreover, near-surface remote sensing techniques, such as digital repeat photography [10,11], and satellite remote sensing linked with vegetation indices (e.g., leaf area index) provide valuable data about phenology dynamics at a regional scale [12]. However, the ability of these methods to monitor phenology at the field or subfield scale is limited, especially in crops with hardly perceivable phenophase transitions such as vegetable crops [13].

Vegetable crops (see https://www.ishs.org/defining-horticulture, last accessed: 18 September 2022), are characterised by rapid growth, which leads to not very defined phenophases. Since most of them are fresh food, the timing of sensitive phenological events, such as cotyledon emergence, is of economic and technical concern to establish accurate practices, such as weed removal [14,15]. Therefore, it is important to set alternative methods suitable for reliably and operationally assessing vegetable crops’ phenophases to support precision horticulture (PH) production systems.

The dynamics of crop phenophases result from a set of structural, physiological, and performance-related phenotypic traits that characterise the given genotype and its interactions with the environment [16]. Computer vision (CV) has attracted growing interest in the agricultural domain as it comprises techniques that allow systems to automatically collect images and extract valuable information from them towards accurate and efficient practices [17]. Usually, CV-based systems include an image acquisition phase and image analysis techniques that can distinguish the regions of interest to be detected and classified (e.g., whole plant, leaves) [18,19]. Among the plethora of image analysis techniques already developed, the one that best performs object detection in agriculture (e.g., disease detection [20], crop and weed detection [21], and fruit detection [22]) is deep learning (DL) [23]. DL is based on machine learning and has led to breakthroughs in image analysis, given its ability to automatically extract features from unstructured data [24]. In particular, convolutional neural networks (CNNs) are being tested for various tasks to support PH production systems, including phenology monitoring [25,26]. Thereby, it is hypothesised that high-throughput plant phenotyping techniques [27] based on CV coupled with DL (CV_DL) can assess the spatiotemporal dynamics of crop traits related to its phenophases in the context of PH [28].

Nevertheless, applications of CV and DL in phenology monitoring are mostly focused on arable crops (e.g., rice, barley, maize) [29,30,31,32], orchards [33,34], and forest trees [35,36,37] with limited studies into vegetable crops. Additionally, the developed models are crop-specific and tend to be based on well-defined phenophases, such as flowering [13,33,35], neglecting the spatiotemporal phenology dynamics. Vegetables usually grow as annual crops and are harvested during the vegetative phase. Thus, the phenotypic traits for phenological identification are restricted to leaves (e.g., number, size, and colour). Furthermore, considering that vegetable crops are often sown as mixed cropping, and the morphological traits of leaves are very similar between plants, especially in the early growth stages, it is difficult to identify the specific phenotypic traits in order to classify the phenophases of the corresponding plant (crops and weeds included) [38,39,40].

This study primarily aims to develop an automatic approach for the dynamic phenological classification of vegetable crops using CV_DL techniques. Four state-of-the-art DL models were tested through a custom dataset that consists of RGB and greyscale annotated images corresponding to the main phenophases of eight vegetable crops between emergence and harvest. Thus, this work contributes to the state-of-the-art introducing a vegetable crop classification system that can identify multiple crops considering phenology dynamics throughout the growing season at the subfield scale.

2. Materials and Methods

2.1. Dataset Acquisition and Processing

Eight vegetable crops were selected to build a varied dataset of phenophases (

Table 1. Vegetable crops selected for the phenophase image dataset construction.

Common Name	Binomial Name
Arugula	Eruca vesicaria subsp. sativa
Carrot	Daucus carota subsp. sativus
Coriander	Coriandrum sativum
Lettuce	Lactuca sativa
Radish	Raphanus sativus
Spinach	Spinacia oleracea
Swiss chard	Beta vulgaris subsp. vulgaris var. cicla
Turnip	Brassica rapa

), taking into account the length of the growing season, the intensity of agricultural practices (mainly weed removal), and the resistance to pests and diseases.

Each crop was manually sown in a section of a 4.5 m² plot at a greenhouse in Vairão, Vila do Conde, Portugal, according to the recommended sowing density. Weeds were manually removed at an early growth stage (less than four unfolded leaves), and plants were irrigated twice a day. Image acquisition occurred before the weed removal operation.

Images in the RGB colour space were acquired between March and July 2021 by a smartphone (Huawei Mate 10 Lite, 16 MP resolution in Pro Mode) in different view angles and sunlight conditions, both procedures increasing the data representativeness.

Depending on the crop, one or several growing seasons occurred during the image acquisition operation, and images were collected covering the main vegetative phenophases of the crops studied. The classification of the phenophases of each crop was based on the BBCH-scale [41]. In total, there were collected 4123 images with a resolution of 3456 × 4608 px each.

Table 2. Number of images acquired for each crop and the corresponding number of plants for each phenophase defined.

Crop	Phenophase	BBCH 1-Scale Code	No. of Plants	No. of Images
Arugula	coty_arugula	10	281	312
	minus9_arugula	11–18	631
	plus9_arugula	19 2	66
Carrot	coty_carrot	10	774	533
	smallleaves_carrot	11–13	830
	carrot	14–49	955
Coriander	coty_coriander	10	743	321
	smallleaves_coriander	11–13	382
	coriander	14–19 2	654
Lettuce	coty_lettuce	10	856	1426
	minus9_lettuce	11–18	3878
	plus9_lettuce	19–48	428
	ready_lettuce	49	625
Radish	coty_radish	10	869	494
	smallleaves_radish	11–13	904
	bigleaves_radish	14–49	314
	root_radish	45–49	244
Spinach	spinach	10–18	200	270
	big_spinach	19 2	93
Swiss chard	coty_chard	10	55	454
	chard	10–19 2	633
Turnip	coty_turnip	10	283	313
	smallleaves_turnip	11–13	400
	turnip	14–49	550
Total				4123

¹ Biologische Bundesanstalt, Bundessortenamt und CHemische Industrie. ² Leaf harvest during growing season.

depicts the dataset: the number of images and plants of each crop and phenophase, accordingly.

All images were rescaled four times to a resolution of 864 × 1152 px, improving the processing operations. To evaluate the versatility of the DL models studied, the RGB images were also transformed into greyscale using OpenCV (see https://opencv.org, last accessed: 12 September 2022) library. With this transformation, the complexity and the bias are reduced, as the model is not influenced by the colour in the images, while it is forced to learn features that are not specific to colour, improving the generalisation ability. By applying the luminosity method, it is possible to assign a weighted average of the colour components to each RGB channel (see https://docs.opencv.org/4.x/de/d25/imgproc_color_conversions.html, last accessed: 12 September 2022) (Equation (1)). An example of this transformation is presented in Figure 1.

$$GreyValue=(0.299\times Red)+(0.587\times Green)+(0.114\times Blue)$$

(1)

Since the training and evaluation of DL models involves supervised learning, the images need to be annotated. Following the BBCH-scale [41] and the phenophases defined for each crop (Table 2), images were manually annotated using the open source annotation tool CVAT (see https://www.cvat.ai/, last accessed: 18 September 2022), indicating by rectangular bounding boxes the position and phenophase of each plant. After annotation, the images and the corresponding annotations were exported under the Pascal VOC [42] and YOLO formats [43] to train the SSD and YOLO models, respectively. All annotated images (RGB and greyscale) are publicly available at the open access digital repository Zenodo (see https://doi.org/10.5281/zenodo.7433286, last accessed: 5 November 2022) [44].

2.2. Deep Learning Approach

2.2.1. Object Detection Models

Object detection models are specialised DL models designed to detect the position and size of objects in an image. There are different kinds of DL architectures to detect objects, but the most common ones in the state of the art are based on single-shot CNN, such as SSD and YOLO.

SSD was first introduced by Liu et al. [45]. This architecture comprises three main components: the feature extractor, the classifier, and the regressor. The feature extractor is a CNN frequently designated as the backbone and can be any classification network. Three state-of-the-art backbones were selected for this work: Inception v2, MobileNet v2, and ResNet50. These networks were selected due to the good results obtained in previous works on object detection in agriculture [46,47]. The image was then split into different sizes and positioned windows during the image processing stage. The classifier element will predict the object in each window, and the regressor will adjust the window (also called the bounding box) to the object’s position and size. Combining the three elements (feature extractor, classifier and regressor) leads to an object detection model.

Inception V2, MobileNet v2, and ResNet 50 are three state-of-the-art CNNs. Inception v2 [48] is an improvement of GoogLeNet, also named Inception v1 [49]. This network is made of inception modules and batch normalisation layers. MobileNet v2 is a CNN classifier designed for mobile and embedded applications [50]. In this network, the authors replaced the full convolutional operators with a factorised version that splits the convolution into two layers. The first layer is a lightweight filter, while the other layer is a point-wise operator which creates new features. As such, MobileNet v2 reduced the number of trainable parameters and training complexity. To improve the results of VGG DL models, He et al. [51] studied residual neural networks. ResNet is based on plain architectures with shortcuts for residual learning. This strategy reduced the number of trainable parameters and the training complexity against VGG. ResNet models allow depths between 18 and 152 layers, but ResNet 50 is the most common in the literature.

You Only Look Once (YOLO) is also a well-known single-shot detector, serving most of the time as a reference object detector. YOLO v4 is the fourth generation of YOLO models [52], which delivered multiple features to the original YOLO version, improving the model’s performance and keeping the inference speed. YOLO uses a backbone of DarkNet (see https://pjreddie.com/darknet/, last accessed: 14 October 2022). Among the selected models, YOLO v4 aims to be the one which provides the best balance between accuracy and inference speed. Therefore, this study covers the most current object detection architectures in the literature, especially concerning DL applications in agriculture.

2.2.2. Models Training

For training purposes, the dataset was divided into three sets: training, validation, and test sets.

Table 3. Dataset split of each crop and phenophase images and plants for the training and evaluation of deep learning models (considering only RGB images).

Crop	Phenophase	Train		Validation		Test
		Plants	Images	Plants	Images	Plants	Images
Arugula	coty_arugula	172	188	40	62	69	62
	minus9_arugula	440		93		98
	plus9_arugula	45		12		9
Carrot	coty_carrot	486	321	157	106	131	106
	smallleaves_carrot	492		173		165
	carrot	563		180		212
Coriander	coty_coriander	331	163	188	64	224	94
	smallleaves_coriander	191		73		118
	coriander	326		158		170
Lettuce	coty_lettuce	486	665	226	471	144	290
	minus9_lettuce	2420		889		569
	plus9_lettuce	248		99		81
	ready_lettuce	348		166		111
Radish	coty_radish	650	347	166	98	53	49
	smallleaves_radish	664		190		52
	bigleaves_radish	218		57		39
	root_radish	163		47		34
Spinach	spinach	122	162	37	54	41	54
	big_spinach	53		24		16
Swiss chard	coty_chard	32	319	12	90	11	45
	chard	441		129		63
Turnip	coty_turnip	101	158	66	93	116	62
	smallleaves_turnip	197		111		92
	turnip	246		224		80

depicts the number of images and plants of each crop and phenophase, respectively, assigned for each set (considering only RGB images). The dataset split rate was determined by considering the corresponding number of images and plants in each phenophase.

Data augmentation was used to increase the number of images, improving the overall learning procedure and performance by inputting variability into the dataset [23]. The transformations were carefully chosen, applying those that could happen under actual conditions, as displayed in Figure 2. The data transformations were applied to training and validation sets. The same transformations were applied to RGB images and the corresponding greyscale images (

Table 4. Training and validation image sets’ composition after the data augmentation step, considering only RGB images.

Crop	Train	Validation
Arugula	2750	682
Carrot	2889	954
Coriander	1789	703
Lettuce	9292	3798
Radish	3461	980
Spinach	1620	540
Swiss chard	3190	1800
Turnip	2054	1209

).

TensorFLow r.1.15.0 (see https://www.tensorflow.org/, last accessed: 13 October 2022) was used for the training and evaluation scripts of SSD models, whereas Darknet (see https://pjreddie.com/darknet/, last accessed: 14 October 2022) was used for YOLO v4. A GPU RTX3090 (VRAM of 24 GB) and a CPU Ryzen 9 5900X (12-core 3.7 GHz with a Turbo 4.8 GHz 70 MB SktAM4 with a RAM of 32 GB) were available to run the scripts.

Through transfer learning, pre-trained models with Microsoft’s COCO dataset (see https://cocodataset.org/, last accessed: 8 October 2022) were fine-tuned to classify vegetable crops’ phenophases. Some changes to the default training pipeline were made to optimise the training dynamics and generalisation ability, such as adjusting the batch size. The standard input size for each model was maintained with the addition of a variation of SSD MobileNet v2 with an input size of 300 × 300 px. This information is shown in

Table 5. Input and batch size for the training pipeline of the deep learning (DL) models tested.

DL Model	Input Size (px)	Batch Size
SSD Inception v2	$300\times 300$	32
SSD MobileNet v2	$300\times 300$	24
SSD MobileNet v2	$640\times 640$	24
SSD ResNet 50	$640\times 640$	12
YOLO v4	$416\times 416$	64

.

Since data augmentation has already taken place, it was removed from the training pipeline. The SSD models’ training sessions ran for 50 000 steps, while the YOLO v4 training was conditioned by the number of classes of each crop (see https://github.com/AlexeyAB/darknet, last accessed: 4 November 2022). For example, YOLO v4 trained with lettuce (four classes) images ran for 8000 steps. An evaluation session occurred in the validation set according to the standard value used by the pre-trained models. These evaluation sessions are useful because they monitored the evolution of the training, i.e., detecting if the evaluation loss begins to increase while the training loss decreases or stays constant, which means that the model is excessively complex and cannot be well generalised to new data.

For benchmarking purposes, initially, training was carried out individually only with the RGB images corresponding to each crop and then replicated with greyscale images. After the individual performance evaluation of each model, the one that presented the best performance results was selected to be trained with all available data (RGB and greyscale images).

2.2.3. Models Performance Evaluation

The performance of the DL models was evaluated through a comparison between the predictions made by the DL model and the ground truth data of the position and phenophase of each plant. The following evaluation metrics were considered: F1-Score, mean average precision (mAP), and balanced accuracy (BA) [42,43,53].

The cross-validation technique was applied according to Magalhães et al. [46] to optimise the DL model’s performance by optimising the confidence threshold. In the validation set, augmentations were removed, and the F1-Score was calculated for all the confidence thresholds from 0% to 100% into steps of 1%. The confidence threshold with the best F1-score output was selected for the model’s normal operation. The performance evaluation procedure considered the test set images and occurred in a GPU RTX3090 with a VRAM of 24 GB.

3. Results

Table 6. Confidence threshold that optimises the F1-score metric for each deep learning (DL) model tested in the validation set without augmentation.

DL Model	Crop	RGB		Greyscale
		Confidence Threshold (%)≥	F1-Score (%)	Confidence Threshold (%)≥	F1-Score (%)
SSD Inception v2 (300 × 300 px)	Arugula	23	87.8	42	69.6
	Carrot	20	58.8	87	48.4
	Coriander	22	66.5	18	46.2
	Lettuce	41	74.7	57	90.9
	Radish	18	86.3	10	67.3
	Spinach	35	93.2	13	94.0
	Swiss chard	14	92.5	37	96.0
	Turnip	17	65.6	10	51.9
Mean			78.2		70.5
SSD MobileNet v2 (300 × 300 px)	Arugula	32	80.3	82	74.9
	Carrot	24	58.4	93	43.0
	Coriander	38	68.2	33	32.0
	Lettuce	85	73.9	39	87.0
	Radish	34	82.2	52	56.2
	Spinach	7	98.4	75	96.2
	Swiss chard	37	92.9	98	93.3
	Turnip	28	69.7	38	46.6
Mean			78.0		66.2
SSD MobileNet v2 (640 × 640 px)	Arugula	31	81.6	92	59.7
	Carrot	32	62.5	91	40.5
	Coriander	36	67.3	55	46.3
	Lettuce	75	84.9	53	87.4
	Radish	43	81.7	83	67.0
	Spinach	89	98.3	70	95.5
	Swiss chard	68	92.1	100	92.2
	Turnip	53	67.9	11	50.4
Mean			79.5		67.4
SSD ResNet 50 (640 × 640 px)	Arugula	60	88.6	53	83.2
	Carrot	42	69.7	41	64.5
	Coriander	52	75.7	61	67.7
	Lettuce	66	78.0	55	95.2
	Radish	52	91.6	56	84.8
	Spinach	85	98.3	90	98.2
	Swiss chard	78	94.2	85	94.5
	Turnip	55	70.3	43	64.0
Mean			83.3		81.5
YOLO v4 (416 × 416 px)	Arugula	34	92.0	27	92.4
	Carrot	39	72.0	64	78.3
	Coriander	53	78.3	49	83.0
	Lettuce	75	64.9	75	93.2
	Radish	60	82.7	79	92.8
	Spinach	92	97.2	37	100
	Swiss chard	49	94.4	51	95.4
	Turnip	49	78.1	66	80.8
Mean			82.4		89.5
Overall Mean			80.3		75.0

shows the results of the cross-validation technique applied to the validation set for both the models trained with RGB and greyscale images. The highest F1-score values are reported by SSD MobileNet v2 (300 × 300 px) and YOLO v4 trained with RGB and greyscale spinach images, respectively. SSD MobileNet v2 (300 × 300 px) trained with RGB carrot images and greyscale coriander images presented the lowest F1-scores of 58.4% and 32.0%, respectively. On average, the models with a higher F1-score are SSD ResNet 50 and YOLO v4, trained with RGB and greyscale images, respectively.

Figure 3 reports the mAP results for the best-computed confidence threshold. On average, this metric presents a value close to 70%, and the models trained with RGB images are slightly above (71.6%). Indeed, 25 of the 40 models trained with RGB images (solid black bars, Figure 3) present an mAP equal to or higher than the corresponding model trained with greyscale images (dashed bars, Figure 3).

The models trained with carrot images present a lower mAP: close to 53% (RGB images) and 38% (greyscale images). Figure 4 shows the low performance of the SSD MobileNet v2 model (300 × 300 px) applied to carrot classification phenophases. This crop generally has erect and compound leaves (low soil cover), which facilitates weed growth and makes it more difficult to distinguish between the soil and plants (crop and weeds included).

The confusion matrix (Figure 5) for SSD MobileNet v2 (300 × 300 px) shows low performance. The model had considerable difficulty locating the carrot plants, especially when trained with greyscale images, given the number of undetected ground truths (false negatives).

On the other hand, the models trained with spinach images are those with higher mAP (98.3% for RGB and 98.1% for greyscale images). The phenotypic traits of this crop test set may contribute to explaining the results. Figure 6 represents the spinach test set: images containing only one plant, with little weed or other plant presence and a clear distinction between the soil and the plant. Figure 6 is an example performed by ResNet 50, which was able to correctly classify all 57 ground truths of the test set.

The BA metric presented in Figure 7 is close to 77% (RGB images) and 79.2% (greyscale images), which are similar to the results of mAP (Figure 3).

The importance of BA relies on two aspects: it allows one to overcome biased conclusions due to unbalanced datasets [53] and it enables the analysis of correctly located objects, regardless of whether they are accurately classified. The latter is of particular interest in this work, as it is difficult, even for a skilled observer, to identify plants as small as cotyledons (Figure 8a) or to distinguish whether a plant has less (Figure 8b) or more (Figure 8c) than nine leaves.

SSD ResNet 50 was the best-performing SSD model for both image groups, which is reflected not only in mAP results (76.0% for RGB and 73.3% for greyscale images) but also in BA results (81.4% for RGB and 79.8% for greyscale images).

However, SSD models have been outperformed by YOLO v4. This DL model presents, for both groups of images, not only a better performance when the analysis is based on mAP (76.2% for RGB and 83.5% for greyscale images) but also when the evaluation relies on BA (85.2% for RGB and 88.8% for greyscale images). Supported by these results, it was decided to train the YOLO v4 (416 × 416 px) model with all the images of the 24 defined phenophases, representing the growing season of the eight vegetable crops selected for the dataset construction. The dataset split resulted in 54,090 images for training, 19,532 images for validation, and 1524 images for test sets. The training and validation sets include the images resulting from the augmentation procedure.

As previously mentioned, Darknet was used for the training and evaluation scripts, and the model training session ran for 48,000 steps. Furthermore, cross-validation (Figure 9) was applied with the model achieving an F1-Score of 83.6% for a confidence threshold higher than 53%.

The following performance analysis, summarised in

Table 7. Different evaluation metrics results for YOLO v4 model variations.

Model	F1-Score (%)	mAP (%)	BA (%)
RGB	82.8	76.2	85.2
Greyscale	88.1	83.5	88.8
RGB + greyscale	83.0	76.6	81.7

, was conducted on the test set, considering the best-computed confidence rate (53%). The overall performance remained consistent, except for the slight differences compared with the average values of the YOLO v4 variations trained with the independent image groups.

The confusion matrix in Figure 10 shows YOLO v4 performance. This DL model could discretise the phenotypic traits of the different phenophases for successful classification. The model was able to correctly classify 4123 plants out of a total of 5396 ground truths (Table 2).

Nevertheless, some phenophases have similar phenotypic traits, leading to misclassifications. For instance, the coty class is present in all crops except spinach, which presents similar morphological traits as those seen in Figure 11. Nevertheless, the number of false positives only between predictions in this class is relatively low, yet the respective number of false negatives cannot be neglected. Hence, the resulting mAP for coty is approximately 67%.

4. Discussion

Overall, both architectures were generic enough to successfully classify the vegetables’ phenophases. SSD ResNet 50 was the best-performing SSD model for both image groups individually, and SSD MobileNet v2 (640 × 640 px) performed slightly better than the variation SSD MobileNet v2 (300 × 300 px), suggesting that the performance of this architecture is favoured by increasing the input size. The proposed SSD MobileNet v2 models outperformed the one presented by d’Andrimont et al. [31], which obtained an mAP close to 55%.

The results presented by the SSD ResNet 50 model align with those shown by Ofori et al. [39], where this model performed better than other DL models, including SSD Inception v2. Pearse et al. [35] and Samiei et al. [40] presented the ResNet model variations that performed slightly better than that proposed in this study (98% and 90%, respectively). However, the former application was for the well-defined Metrosideros excelsa flowering phenophase classification, and the latter used a dataset that did not represent actual conditions since the soil background was removed from the images.

YOLO v4 was the model that presented a more consistent performance in both image groups. Hence, it was trained with all the images available in the dataset. The model performance was not significantly impacted, remaining close to 77% and 82% for mAP and BA, respectively. To the best of our knowledge, no work has tested a YOLO model for phenology monitoring. Correia et al. [36] is the only study where an object detector was developed, with lower performance when compared with the proposed YOLO v4 model.

The greyscale transformation did not significantly hinder the models’ performance, the differences being more notorious on the SSD Inception v2 and smaller on YOLO v4, where even the models trained with greyscale images showed a better performance.

Furthermore, the DL models’ performance was not homogeneous with regard to different crops, with the models trained with carrot images and those trained with spinach images being the lowest and highest extremes, respectively. In this case, and in line with different authors [29,30], the images’ quantity and quality, particularly the specific phenotypic traits of each crop, may compromise the results.

The models could classify vegetables at different phenological phases, even in non-structured environments, considering overlaps and variable sunlight conditions. For the comprehensive tracking of the crop cycle, it is crucial to monitor the phenology right after the cotyledons’ appearance and opening. Regarding coty (cotyledons’ opening), YOLO v4 classified this phenophase with an mAP of approximately 67%. Although slightly lower than the results reported by the literature [39,40], it is important to highlight that, in contrast to these, the images applied in this study represent actual conditions, without pre-processing operations, such as segmentation, to differentiate the plants.

5. Conclusions

This paper presents a benchmark of four DL models for assessing the capability of CNN to leverage phenological classification in vegetable crops. A dataset with 4123 images was manually collected during various growing seasons of eight vegetable crops representing the main phenophases from emergence to harvest. For benchmarking purposes, each model was tested for the classification of each crop individually. YOLO v4 was the best-performing model compared to the SSD models. Therefore, this model was tested with all data obtaining a mAP of 76.6% and a BA of 81.7%.

All assessed models delivered promising results, and were able to classify the phenological phases of individual plants between emergence and harvest. Furthermore, a single DL model could classify multiple vegetable crops considering phenology dynamics throughout the growing season at the subfield scale. As far as we have been able to ascertain, this is the first study to evaluate the potential of CV_DL for vegetable crops’ phenological research. Notwithstanding some limitations, this work is an important step towards sustainable agricultural practices.

Further work needs to be done to (i) enlarge the dataset, balancing it with more images within less populated phenophases and increasing the number of vegetable crops; and (ii) assess the inference speed of the proposed models before transferring the detection framework to a robotic-assisted CV platform, thus allowing a more rigorous image collection as well as a more accurate ground truth identification; (iii) extend the benchmark with a new and more optimised backbone network, such as YOLO v7, and different DL architectures, such as faster R-CNN, confirming whether the SSD and YOLO architectures maintain better performance; and (iv) perform an ablation study in the best-performing model, YOLO v4, to understand the model behaviour and measure the contribution of each model component to the overall model performance.

Furthermore, the proposed CV_DL framework can be framed in a modelling approach estimating the extent to which genetic, environmental, and management conditions impact the timing of phenophases and how they affect the efficiency of agricultural practices and crop yield.