Using Deep Neural Networks to Evaluate Leafminer Fly Attacks on Tomato Plants

Abstract

Among the most common and serious tomato plant pests, leafminer flies (Liriomyza sativae) are considered one of the major tomato-plant-damaging pests worldwide. Detecting the infestation and quantifying the severity of these pests are essential for reducing their outbreaks through effective management and ensuring successful tomato production. Traditionally, detection and quantification are performed manually in the field. This is time-consuming and leads to inaccurate plant protection management practices owing to the subjectivity of the evaluation process. Therefore, the objective of this study was to develop a machine learning model for the detection and automatic estimation of the severity of tomato leaf symptoms of leafminer fly attacks. The dataset used in the present study comprised images of pest symptoms on tomato leaves acquired under field conditions. Manual annotation was performed to classify the acquired images into three groups: background, tomato leaf, and leaf symptoms from leafminer flies. Three models and four different backbones were compared for a multiclass semantic segmentation task using accuracy, precision, recall, and intersection over union metrics. A comparison of the segmentation results revealed that the U-Net model with the Inceptionv3 backbone achieved the best results. For estimation of symptom severity, the best model was FPN with the ResNet34 and DenseNet121 backbones, which exhibited lower root mean square error values. The computational models used proved promising mainly because of their capacity to automatically segment small objects in images captured in the field under challenging lighting conditions and with complex backgrounds.

1. Introduction

Agricultural pests are considered an important cause of losses in world agricultural production [1]. Among agricultural pests, insects can be particularly harmful, as they feed on various parts of the plant, affecting their development, as well as acting as vectors of various devastating plant diseases [2]. In some situations, losses can reach 100% of the production if detection and control are ineffective or nonexistent [3,4].

The tomato plant is one of the vegetable crops most susceptible to pests and diseases [5,6]. Leafminer flies of the genus Liriomyza are among the main pests afflicting the tomato crop, with the largest number of host plants reported within the Agromyzidae family [7]. The larvae feed by opening galleries or mines in the leaf parenchyma, which is why they are called leafminer larvae. It is estimated that the larvae of Liriomyza sativae, the species most associated with tomatoes, reduce plant photosynthesis levels by up to 65% because of stippling and tunneling [8]. This leads to yield reduction, causing significant economic losses for producers.

The detection and quantification of the severity of pest infestation are important tools in sampling plants that assist in decision making for controlling pests and protecting crops [9]. Frequent visual inspection of plantations, aimed at detecting and identifying pests, requires specialized labor. In most cases, inspection consists of manual and visual sampling along the plot using a beating sheet [10]. The problem with this traditional pest sampling method is that, by the time the infestation is detected, the damage has already been caused, and control will be more difficult [11,12]. In addition, the method is subject to human error and may become unfeasible depending on the size of the area [13,14,15]. Therefore, developing an automatic pest identification system could benefit producers by controlling pests and reducing yield loss.

An increasing number of solutions combining digital images with machine learning techniques have been developed for problems related to pest identification. Image processing coupled with machine learning algorithms helps identify and map the most infested areas, assisting in integrated pest management. Among the techniques used for the identification and mapping of pests, there is a predominance of studies using traps. However, one problem with this approach is that only adults and flying insects are captured [16]. An alternative could be direct identification in the field through images of the symptoms caused by pests observable in plant leaves [17,18,19]. Nevertheless, the challenge of this technique is related to the heterogeneity of field conditions, such as the complex background, variability of lighting, and presence of shadows [15,20,21,22].

Currently, artificial intelligence algorithms, such as convolutional neural networks (CNNs), have been successfully deployed in the segmentation of digital images for pest and disease detection [23,24,25,26,27]. Some models currently in evidence for semantic segmentation are U-Net, created for the segmentation of biomedical microscopy images [28,29]; FPN [30], developed primarily for object detection and image segmentation for urban planning and natural landscape monitoring [31]; and LinkNet [32], developed for the segmentation of high-resolution images of urban landscapes.

Creating an automatic identification system requires developing an image segmentation method for leaves attacked by leafminer flies in a natural environment. This automatic system will allow the development of robotic monitoring devices that can be used by pesticide application machinery, reducing operational costs, losses from pest attack, and the environmental impact associated with the control of leafminer flies. Therefore, the present study is aimed at developing and evaluating models of CNNs for the detection and automatic estimation of the severity of symptoms of leafminer fly attacks in tomato leaf digital images captured under field conditions.

2. Material and Methods

For better organization and understanding, the research design of the study was divided into several steps: creating the database, annotating the images and creating reference masks, preprocessing the data, configuring the experiment, applying the model evaluation metrics, and estimating the severity.

2.1. Database Creation

The image database used in this study consisted of 90 images (with a size of 2268 × 4032 pixels) of tomato leaves with symptoms of attack by the leafminer fly (Liriomyza spp.). Images were obtained from two different tomato fields located in the city of Coimbra, Minas Gerais, Brazil. A Huawei P20 Lite smartphone (Huawei Technologies Co., Shenzhen, Guangdong, China) with a 13-megapixel-resolution camera was used to acquire leaf images. An attempt was made to focus on one leaf in each image. The images were captured under field conditions during the daytime, with variations in the lighting and background.

The image bank obtained was then randomly divided into 80% for the training set and 20% for the testing set. The testing set was only used to evaluate the performance of the model. In the training set, 10% of the images were used for model validation during the model training process. The validation process helps determine the best modeling hyperparameters (features) during the model training process, which refers to the percentage of accurately elucidated data samples [33].

2.2. Annotation of Images and Creation of Reference Masks

Image annotation is the process of labeling images to outline the target characteristics of the dataset representing the way a human would perceive them. The resulting dataset is then considered a functional dataset to train the machine learning models. In this study, each image was manually annotated for training the machine learning algorithm by sampling all the classes present, the leaf, and the symptom. At the same time, the background was automatically segmented after demarcation of the target of interest. After annotation, a binary mask containing three classes was created, as shown in Figure 1.

2.3. Data Preprocessing

In the preprocessing step, performed before the CNN model training, patches of size 256 × 256 pixels were extracted from source images and their respective masks. The image patch process is an intermediate solution between feature-based and direct-image methods, enabling precise image localization and a more extensive set of training data [29]. After the patches from the images and masks were extracted, selection was performed on the training patches. Patches containing >95% of their pixels corresponding to the background class were discarded from the training set. This was performed to improve the balance among classes, as much of the original image was the background. In the end, 1900 images (256 × 256 pixels) were left for training and 200 images (256 × 256 pixels) for testing. Finally, to increase the generalization capacity and prevent the model from overfitting, all images and reference masks in the training database were subjected to a data augmentation process with several transformations, including horizontal and vertical rotation and zooming in and out [13,34,35].

2.4. Configuration of the Experiment

Three CNN models, U-Net, FPN [36], and LinkNet, were selected for this study. Transfer learning was used for all the models. In transfer learning, the weights of the pretrained models, called backbones, are defined in the encoder block of the models (Figure 2). The backbones used in this study were VGG16, ResNet34, Inceptionv3, and DenseNet121. These backbones have a significant capacity to generalize predictions for images outside the database used for training. All the backbones were trained using the ImageNet database [37], which is an extensive visual database designed for use in the development of visual object recognition.

The Google Colaboratory platform was used to conduct the experiment using a virtual machine. The configuration of the virtual machine consisted of an Intel Xeon 2.2 GHz processor, an Nvidia Tesla P100 video card with 16 GB of dedicated memory, and 32 GB of RAM.

The training process was performed with 20, 30, 40, 50, and 100 epochs, and it was verified that the model with 30 epochs presented the best results. Then, the models were trained using the following configurations: 30 epochs from scratch, intersection over union (IoU) of the validation database as early stopping, and patience for 5 epochs. Early stopping is a regularization technique widely used in machine learning to avoid model overfitting. The batch size was 16, and Softmax was defined as the activation function for all the output neurons of the last layer. The Adam optimization algorithm was used to adjust the neural network parameters with a learning rate of 0.001.

For improved inference and comparison of the results obtained in this study, all hyperparameters were standardized for all trained models. The loss function initially used for multiclass semantic segmentation tasks was the categorical cross entropy between the predicted value and the reference value for each pixel [38,39]. However, the imbalance among sampled classes also indicates an imbalance in the learning of these different classes. Hence, focal loss was used; this is an adjusted categorical cross entropy highly recommended for semantic segmentation tasks with a database with an imbalance among classes [40,41]. Focal loss is characterized by a categorical cross entropy with the addition of a weighting factor (γ) to the equation, which increases the reliability of the prediction for more sampled classes, making it possible to focus the training of the model for the less sampled classes—in this case, the symptoms under study, which had defined γ = 2 [41]. The Dice coefficient is quite similar to the IoU and is often used as a metric to assess the similarity between two images, but here, it was applied as a loss function (Dice loss).

2.5. Model Evaluation Metrics

The performance of the models employed in this study was evaluated based on accuracy A, precision P, recall R, and IoU, also known as the Jaccard similarity coefficient. This coefficient is precisely the intersection of the reference mask with that predicted by the model over the union of the reference mask with the model prediction. It is widely used to calculate the similarity between two images [40]. These metrics are defined by the following equations:

$$A=\frac{TP+TN}{TP+TN+FP+FN}$$

(1)

$$P=\frac{TP}{TP+FP}$$

(2)

$$R=\frac{TP}{TP+FN}$$

(3)

$$IoU=\frac{TP}{TP+FP+FN}$$

(4)

where

TP =

true positive is the number of pixels correctly assigned to the evaluated semantic class (background, healthy leaf, and injured leaf);

FP =

false positive is the number of pixels incorrectly classified in the semantic class, although they belong to another class;

FN =

false negative is the number of pixels belonging to the semantic class assigned to another class.

The values of these metrics range from 0 to 1. A value of 0 means no overlap, whereas a value of 1 represents a complete overlap of the classes between the reference mask and that predicted by the model.

2.6. Severity Estimation

The severity of pest attack in plants is traditionally estimated by the proportion of the attacked tissue of the plant relative to the total leaf area [14,42,43,44]. The precision of these estimates ensures a closer-to-reality analysis, which enables their use as indicators for decision making by producers to control pests. The severity was estimated using the ratio between the number of pixels contained in the lesion class and the total area of the leaf focused on in the image [23]:

$$\mathrm{severity}=\frac{S}{S+L}100$$

(5)

where

L = area of pixels contained in the leaf class;

S = area of pixels contained in the symptom class.

3. Results

3.1. Comparison of Models and Backbones

The performance of U-Net, FPN, and LinkNet CNN models with different backbones was investigated to quantify the severity of leafminer fly attacks on tomato leaves. The IoU’s of the validation data during the training for each epoch were compared to evaluate the quality of training of the models. All trained models converged with six epochs, except for the FPN model with the VGG16 backbone, which converged with seven epochs.

Table 1. Performance summary of the best and worst models used for the prediction of class 1 (background area), class 2 (leaf area), and class 3 (leafminer fly symptom) in the validation data during the training processing.

Category	Deep Learning Model	Backbone	Class	Average IoU (%)
Best models	U-Net	Inceptionv3	Background	86
	U-Net	Inceptionv3	Leaf	87
	FPN	DenseNet121	Symptoms	61
Worst models	U-Net	VGG16	Background	65
	LinkNet	VGG16	Leaf	69
	LinkNet	VGG16	Symptoms	25

shows the results of the best models and worst model for each class separately, enabling a better understanding of the performance of each model for each class. For class 1 (i.e., background) and class 2 (i.e., leaf area), the best result was obtained using the U-Net model and the Inceptionv3 backbone, and the worst result was obtained using the LinkNet model and the VGG16 backbone. However, for class 3 (i.e., symptoms), the best model was obtained using the FPN model and the DenseNet121 backbone, and the worst model was obtained using the LinkNet model and the VGG16 backbone. Among all the semantic segmentation backbones used in this study, Inceptionv3 achieved the best result for segmenting the pixels of classes 1 and 2, and the DenseNet121 backbone achieved the best result for segmenting the pixels of class 3.

Table 2. Performance summary of the U-Net, LinkNet, and FPN models and backbones in the testing database.

Deep Learning Model	Backbone	Test Accuracy (%)	Average Precision (%)	Average Recall (%)	Average IoU (%)
U-Net	VGG16	83.90	80.63	78.31	61.76
	ResNet34	90.00	86.63	85.81	72.73
	Inceptionv3	91.58	87.84	87.59	77.71
	DenseNet121	91.25	87.83	87.74	76.33
LinkNet	VGG16	81.23	73.94	67.87	53.03
	ResNet34	88.66	84.62	84.77	73.21
	Inceptionv3	91.06	87.09	87.39	75.67
	DenseNet121	90.72	87.00	86.79	75.99
FPN	VGG16	89.27	85.58	84.31	73.12
	ResNet34	90.53	87.29	86.27	74.61
	Inceptionv3	91.10	87.98	86.82	75.12
	DenseNet121	91.56	88.63	87.71	76.62

presents the performance of the models employed in this semantic segmentation study using transfer learning. In general, the models performed well for the segmentation task in the testing database. The U-Net model with the Inceptionv3 backbone had the best IoU (77.71%). The second-best model with performance in terms of IoU (76.62%) was obtained using FPN with the DenseNet121 backbone. LinkNet with the VGG16 backbone performed the worst (IoU = 53.03%).

Figure 3, Figure 4 and Figure 5 show the results of the multiclass semantic segmentation performed by the different CNN models and backbones on the testing database. This visual comparison with the reference mask, which was manually annotated, and the images predicted by the models reveals a better segmentation of classes 1 and 2 by the U-Net model with the Inceptionv3 backbone. For class 3, the FPN model with the DenseNet121 backbone achieved better segmentation owing to the higher IoU values obtained by this model. For the tested backbones, the best segmentation was obtained using DenseNet121 and Inceptionv3. The worst result was obtained using VGG16 for all three models.

3.2. Severity Estimated by Models

The severity of the leaf symptoms caused by the leafminer fly in tomato crops was evaluated using masks manually annotated with masks predicted at the pixel level by the U-Net, FPN, and LinkNet segmentation models. The relationship between the total area comprising the pixels of the leaf class and that of the symptom class was obtained, as determined by Equation (5). Figure 6, Figure 7 and Figure 8 show predicted severity and reference severity (observed) scatter plots. In addition, the figures show the R², RMSE, and MAE metrics. The R² metrics were calculated based on a linear regression between prediction and reference severity. Greater values of R² and lower values of RMSE and MAE indicate better models. The FPN models gave better estimates of leaf symptom severity when the reference masks (i.e., manually annotated) were used as input data for training the models, with the Inceptionv3 backbone having the lowest root mean square error (RMSE) value (Figure 6).

The U-Net models obtained the second-best result among the trained models, exhibiting intermediate RMSE values when compared with those of the others (Figure 7). Inceptionv3 and ResNet34 backbones for the U-Net model had the lowest RMSE values. The LinkNet model exhibited the worst performance (Figure 8). The LinkNet models overestimated the severity of leaf symptoms and erroneously classified the pixels that constituted the classes.

4. Discussion

Among the models analyzed in this study, U-Net gave the best results on the testing database. It is a full CNN with two blocks that uses line-by-line convolution. Therefore, its performance was equal to that of the dense layer of a traditional CNN. In the contraction block, each pooling operation degrades the resolution of the image, and because this is an irreversible operation, there is a loss of spatial information of small targets [45], which is probably the reason for the lower performance compared to that of the other networks.

The FPN model achieved the second-best performance in the overall score, although the IoU results obtained for each symptom class were in question. A possible explanation is that the FPN utilizes a descending stochastic gradient and a horizontally connected structure that combines low resolution, robust semantic features with high resolution, and weak semantic features for the detection and segmentation of small targets, such as those sampled by the symptom class [30]. Compared with the results reported in the literature, the results obtained in this study are satisfactory for the same operation in a considerably smaller database for the semantic segmentation task. For instance, Gonçalves et al. [46] obtained IoU values from 64.7% to 79.4% for leaf symptoms caused by coffee leafminers, soybean rust, and wheat tan spot.

The LinkNet model gave the worst results for overall segmentation. However, when evaluating the IoU for the symptom class, it produced a slightly inferior result compared with the FPN model. LinkNet is a fully convolutional network that is widely used for semantic segmentation tasks, focusing on rapid prediction based on an encoder–decoder structure. To ensure no loss of pixel location information in the encoding part of the model, LinkNet directly propagates the spatial information from the encoder to the decoder at the same level, which may explain this result [32]. In addition, the time and operations required to relearn lost features are reduced, thus leading to a significant reduction in processing time.

The experiments revealed that the DenseNet121 backbone achieved better IoU results in all segmentation models, except for U-Net, because it was able to sample more pixels referring to the symptom class. Theoretically, deeper networks are expected to perform better in semantic segmentation tasks. However, the experimental results showed the opposite when comparing the number of trainable parameters per model and the backbone (

Table 3. Number of trainable parameters per model and backbone.

Backbone	Deep Learning Model
	U-Net	LinkNet	FPN
	Trainable Parameters	Trainable Parameters	Trainable Parameters
VGG16	23,748,531	20,318,611	17,572,547
ResNet34	24,439,094	21,620,118	23,915,590
Inceptionv3	29,896,979	26,228,243	24,994,851
DenseNet121	12,059,635	8,267,411	9,828,099

). A possible explanation is that the size of the database used in this study was small, requiring deeper networks [47].

In terms of RMSE values, the best severity estimate of leafminer fly symptoms was obtained using the FPN model with the backbones ResNet34 and Inceptionv3. The models trained with U-Net and the Inceptionv3 backbone estimated the second-best severity of leafminer fly symptoms. These RMSE results also confirm the higher values of IoU obtained for both the FPN and U-Net models. The LinkNet models overestimated the severity, as verified by the IoU values found for the classes presented and by the visual comparison made in the present study, which indicated poor classification of the classes in the masks predicted by the model. The LinkNet model was developed to process high-resolution aerial images, which may explain this result. The resolution used in this study may have been insufficient, causing a decrease in performance. In addition, the number of targets used was relatively small, which affected the detection accuracy [14,48].

The results obtained demonstrate that it is possible to perform segmentation and estimation of the severity of leaf symptoms in images with the current challenges existing in semantic segmentation tasks, such as capturing images directly in fields with complex backgrounds and variable lighting [20]. In addition, the proposed method is nondestructive, with no need to remove the leaves to capture images in an environment with controlled lighting and homogeneous background, unlike what has been addressed in the literature [49].

The values of the mostly used evaluation metrics in machine learning studies, such as accuracy, precision, and recall, were higher than those of IoU, which was expected. None of these metrics were considered adequate for multiclass semantic segmentation precisely because of the imbalance of the sampled classes, which is common in semantic segmentation tasks [50,51] in which the precision of the most sampled classes ends up shadowing the imprecision of the smallest classes. Therefore, IoU has been recommended for leafminer fly symptom severity estimation, as it better reflects the quality of the segmentation [23]. It was possible to confirm the difficulty of segmentation of fewer sampled classes, despite the use of focal loss, by using the IoU values separately.

The importance of transfer learning in machine learning, specifically in semantic segmentation using a small database, is evident. The lack of data, especially annotated data, is one of the limitations of CNN models [13]. The annotation of images is a supervised process and can be very time-consuming and challenging, depending on the size and quantity of objects to be annotated in the images and the number of classes present. Collecting images under real field conditions, previous annotation, and subsequent sharing of data with the scientific community would encourage the consolidation of machine and deep learning in various areas of agriculture.

Future research should consider the use of (i) videos rather than digital images and (ii) CNN YOLO v2 or v3 (You Only Look Once), which has several benefits and is widely used for instance segmentation. This segmentation method combines object detection and semantic segmentation to distinguish different objects in a scene. It provides different identifiers for different objects and can simultaneously evaluate different symptoms. In addition, using different backbones trained in different datasets and a considerably more extensive database could improve model performance.

5. Conclusions

In this study, the severity of leafminer fly symptoms in tomato leaves was experimentally compared using different deep CNNs and backbones. The results show that the U-Net network achieves better performance in segmenting the background and leaf area classes. The FPN model has a greater capacity for segmentation of the symptom class, exhibiting slightly better performance than the others in predicting small and less sampled targets, even when using a small database. The experimental results also demonstrate that Inceptionv3 and DenseNet121 backbones could provide the best performance for the semantic segmentation network of small targets in complex backgrounds with variations in lighting and multiple occlusions. The backbones VGG16 and ResNet34 exhibited the worst results.

In conclusion, the segmentation of symptoms under the complex conditions presented in this study, with images captured in a field with a complex background and irregular lighting, is feasible. This can serve as front-end decision support for developing an automatic pest identification system attached to tractors and farm implements directly in the field.