Pesticide-Free Robotic Control of Aphids as Crop Pests

Abstract

Because our civilization has relied on pesticides to fight weeds, insects, and diseases since antiquity, the use of these chemicals has become natural and exclusive. Unfortunately, the use of pesticides has progressively had alarming effects on water quality, biodiversity, and human health. This paper proposes to improve farming practices by replacing pesticides with a laser-based robotic approach. This study focused on the neutralization of aphids, as they are among the most harmful pests for crops and complex to control. With the help of deep learning, we developed a mobile robot that spans crop rows, locates aphids, and neutralizes them with laser beams. We have built a prototype with the sole purpose of validating the localization-neutralization loop on a single seedling row. The experiments performed in our laboratory demonstrate the feasibility of detecting different lines of aphids (50% detected at 3 cm/s) and of neutralizing them (90% mortality) without impacting the growth of their host plants. The results are encouraging since aphids are one of the most challenging crop pests to eradicate. However, enhancements in detection and mainly in targeting are necessary to be useful in a real farming context. Moreover, robustness regarding field conditions should be evaluated.

1. Introduction

The use of pesticides has become natural and exclusive to us because our civilization has relied on them since antiquity to fight weeds, insects, and diseases, which would generate a mean 50% production loss [1]. Unfortunately, this has progressively led to alarming consequences on water quality, biodiversity, and human health. Exposure of the European population to endocrine-disrupting pesticides alone cost up to €270B in health expenses in 2017 [2]. Tang et al. [3] concluded that 64% of global agricultural land (25.106 km${}^2$) is at risk of pesticide pollution, and 31% is at high risk. The class of neonicotinoids, pesticides responsible for the deaths of 300,000 bee colonies annually, has even been recently banned from use [4]. In its “Ecophyto” plan, the French government has decided to reduce the use of agrochemicals by 50% by 2018. Unfortunately, due to alternatives to these chemicals being too scarce, the objective has been postponed to 2025.

For ten years, progress in computer science, the Internet of Things (IoT), robotics, and agronomic modes have improved farming practices, provoking two consecutive revolutions: smart farming [5,6] and now, robotic farming [7]. A major line of research in this revolution is the improvement of crop pest management thanks to the development of new tools for automated pest detection and their neutralization by more environmentally friendly means.

Thus, several studies have demonstrated the possibility of the automated detection of multiple agricultural pests in laboratory or field conditions with a good level of accuracy thanks to image analysis techniques associated with artificial intelligence. Images are captured using RGB (red-green-blue), multi-spectral or hyperspectral cameras with variable dimensions and depths (3 dimensions, RGB-depth, etc.) [8,9]. Spectral cameras provide high quality and quantity of data for pest and disease monitoring on crops [10,11,12], detection of insect infestations on stored food [13], mosquito gender and age evaluation, and taxonomical applications [14], with excellent results provided by automated solutions. Nevertheless, they require complex technology, mainly tested under laboratory conditions, which requires more research for efficient use in the field. On the contrary, RGB image processing has been widely used to develop pest detection systems on traps or plants in the field based on the analysis of the color composition, gray levels, shape or texture of objects in the image [8,15]. Detection methods performed with 3D cameras, computer-based classification, and notably, deep learning algorithms [16,17,18,19,20] have also been investigated. Some optical solutions for automated pest detection have been mounted on robots to explore the feasibility of using such technology in real conditions in greenhouses [21] or in the fields [22]. In addition, some of these robots can multitask, offering detection of pests on crops and their neutralization. They perform localized spraying of pesticides in small quantities, which is preferable to the traditional method that consists of spraying the entire crop when infested areas are discovered [9,23,24].

Both mechanical and electrical-based works have already been explored [25,26]. However, very few works have been conducted on the laser-based neutralization of insect pests as an alternative to pesticides. Hori et al. investigated the effect of UV irradiation on eggs, larvae, pupae and adults of Drosophila melanogaster. High doses were effective in order to neutralize them, but genetic variation in insect resistance could appear in the long term. Laser irradiation with a green light was also effective on mosquitoes, but it has never been tested on an autonomous robot [27]. Furthermore, weed processing is now actively commercialized [28]. Weed destruction is typically carried out by a targeted herbicide spray or mechanically based methods. Thermal methods are sometimes used, and laser-based destruction is also still under study [29]. As soon as 2001, laser-burning could reduce weed biomass by 90% with only 20% of the energy usually used for thermal weeding [30].

To date, technologies for automated spraying or mechanical destruction of weeds are appearing on the market either as large and costly farming robots [31] or as small-sized ones, such as Oz (Naïo Tech). Yet, no non-chemical solution exists concerning crop pests. This is why the Greenshield (see https://anr-greenshield.insa-lyon.eu/, accessed on 9 September 2022) project’s objective was to demonstrate the feasibility of using a laser-based detection-neutralisation module (DNM) embedded in a mobile robot dedicated to this purpose on aphids. This robot managed to do demonstrate its feasibility in lab conditions in September 2021. It also demonstrated the feasibility of using the DNM with a commercially available robot (Dino, Naio Tech) to combat weeds.

Detecting aphids in crop fields is a complex task due to their small size and confounding color. As with numerous other applications, a deep learning approach has been envisaged. Indeed, this artificial intelligence approach is nowadays widespread for face recognition [32], image segmentation [33], signal analysis and automatic interpretation, and the control of autonomous cars. More particularly, deep learning and image processing are two disciplines whose applications are very varied and encompass areas such as medicine, robotics, security, and surveillance. In the frame of image processing, deep learning is very powerful at extracting complex information from input images. Such algorithms’ structures are often based on convolutional neural networks that can capture the spatial dependencies at a different scale in an image. They consist of stacking layers of convolution and mathematical functions with several adjustable parameters, which seek to identify the presence of particular patterns on the image at different scales. The tuning and the optimization of these parameters are performed during a preliminary training phase, which relies on the availability of a large quantity of data, thanks to specific cost functions (loss).

The originality of this project lies in the embedding of various technologies in a single farming robot necessitating a multidisciplinary approach including pest biology, laser photonics, visual servoing, and mobile robotics. As aforementioned, the main objective was to prove the feasibility of such an approach to replace pesticides. Therefore, this paper describes the main requirements of this system and the related solutions we envisage. It exposes the experimental approach to validate their consistency, and it discusses the results. It is organized as follows. The next section details materials and methods for the detection and neutralization of insect pests on crops. Experimental results are provided in Section 3 and discussed in Section 4. We then conclude this paper, which summarizes the achievements of the Greenshield project, and we evoke future directions.

2. Materials and Methods

2.1. Prototype Robot Structure

The main subsystems dedicated to the detection and neutralization of aphids are all embedded in a mobile robot. In the future, this mobile robot should be able to autonomously proceed in crop fields, globally browse them and permit a global detection/neutralization action. In the frame of this study, a prototype (see Figure 1) has been designed with the requirement of validating the detection/neutralization functions. This prototype had to proceed at a minimum speed of 10 m per minute, work 3 h between two battery recharges, and weigh less than 100 kg to avoid compacting the ground. To span rows, the overall width of the wheels had to be less than 1 m, and the inner space between wheels had to be greater than 30 cm. The robot had to be empty between the wheels to host the detection/neutralization hardware.

We actually could not find any mobile robot off-the-shelf fitting all the requirements. We then designed it only for detection/neutralization experimental purposes. In this version, maintainability requirements limited the number of coverings and so drastically decreased its ability to work in humid environments. We only covered its bottom part according to local laser-based security regulations to avoid hitting and wounding surrounding people. During the experiments, it was located on rails to travel only in straight lines over one plant row. It is therefore not optimized for real external conditions (with humidity, rocks, obstacles, U-turns at the end of rows…). To free up some room on the bottom part of the robot, wheels with embedded brushless motors and position encoders (MOTTROTXFS6-OZOV1 by OZO) were chosen. The four wheels were connected to 2 dsPIC33EP512GM710 (Microchip) motor control boards to synchronize their actions and globally control the robot displacements. These boards were programmed with Matlab-Simulink (Mathworks) software using the “MPlab for Simulink” library furnished by Microchip.

2.2. Insects and Plants

This study focused on the neutralization of aphids, as they are among the most harmful pests for crops and complex to control [34]. A solution able to control their spread should most likely work on other invertebrates such as cockroaches, mosquitoes and fruit flies. To do so, we chose three aphid lines from parthenogenetic females in the fields, different in color and size, providing a diverse database to develop our DNM. Thus, we used two pea aphid lines, Acyrthosiphon pisum LL01 (green) and YR2 (pink), and one bird cherry-oat aphid line Rhopalosiphum padi LR01 (black, 2 to 3 times smaller than A. pisum).

Aphids were kept under obligate parthenogenesis in laboratory conditions to rear unsynchronised parthenogenetic aphid populations and synchronized nymphs N1 (first instar) as described by Simonet et al. [35] according to study requirements. A. pisum lines were maintained on young broad bean plants (Vicia faba. The L. cv. Aquadulce) and R. padi lines were maintained on wheat seedlings (Triticum aestivum (Linnaeus) cv. Orvantis). Insects and plants were reared in controlled conditions at 21 ± 2 $°$C and 65% relative humidity (RH) ±5% with a 16 h photoperiod.

Organic seeds of V. faba and T. aestivum were supplied by Graines-Voltz^® (Loire-Authion, France) and Nature et Progrès^® (Alès, France), respectively. All plants in this study were grown in a commercial peat substrate Florentaise^® TRH400 (Saint-Mars-du-Désert, France) for 10 days from seed to 2–3 leaves (seedling stage for broad bean and emergence stage before tillering for wheat). Then, the plants were used for a week for study purposes, including insect rearing, plant and aphid detection, aphid mortality monitoring and evaluation of the potential damage caused by laser shots on plants. The method used for these experiments will be detailed in the following sections.

2.3. Global Process

A trolley-like robot with three embedded subsystems is proposed: the RGB-D (red-green-blue plus depth) acquisition system and light sources, the aphid localization algorithm using deep learning, and the neutralization system based on the laser.

Figure 2 illustrates the whole system organization inside the robot. The light sources, the camera, and the seedlings featuring several aphids were collocated in a black box. In the following, we detail each subsystem and start with a global overview of the neutralization process.

The mobile robot proceeds in a right line over a single plant row and detects aphids during motion. When detected aphids are out of reach, the robot stops and triggers the neutralization tasks. In the long term, the neutralization task should run continuously. For the real-time detection of aphids, we use a 3D camera and artificial intelligence object recognition techniques (Section 2.5). The coordinates of the aphids visible on the acquired image are transmitted to the neutralization module.

The aphid neutralization module is mounted on the robot and consists of a (harmless) laser pointer and a power laser source, taking the same optical path toward a pair of micro-mirrors (see Figure 3b) that position beams in the robot’s workspace (see Figure 2). The laser pointer is used to realize a visual servoing (detailed in Section 2.6) to target the aphids located in the images. Once the pointer spot is stabilized on the target, the power laser emits a pulse carrying a sufficient amount of energy to neutralize the target aphid (by raising its temperature; see Section 2.8). It then moves on to the next targeted aphid in the image. We describe the path optimization in Section 2.7. When all detected aphids are treated, the robot restarts its motion with pest scanning. Several types of low-power laser sources that can be mounted on a mobile robot have been tested to identify the best beam tracking on the leaves. Experiments were performed allowing us to correlate the energy emitted by the laser beam with the fecundity and mortality rates of aphids and the efficiency of weed destruction. The detection/localization/neutralization software was linked to the robot locomotion to dynamically enable proceeding. ROS middleware [36] was used to manage the communication between modules.

2.4. RGB-D Acquisition

To localize aphids, a RGB-D camera (ZED mini, Stereo labs, see https://www.stereolabs.com/zed-mini/, accessed on 9 September 2022) is used (see Figure 3a). Its settings were adjusted to limit light reflections and maximize the contrast (maximum contrast and saturation, reduced gain; see

Table 1. Configuration of various parameters of the ZED camera.

Parameters	Values	Explanation
Resolution	HD1080 (1920 × 1080)	A 2K resolution would consume a lot of resources and therefore would slow down the detection algorithm.
Capture Speed	30 FPS	Maximum available in the HD1080 mode.
Brightness	1	Low brightness to limit light reflections on the surface of the leaves.
Contrast	6	High contrast makes it easier to detect pink aphids on green leaves.
Hue	0	Default value that matches the colors perceived by human eyes.
Saturation	8	Maximized to let the aphids appear on green leaves.
Gamma	2	A low gamma level limits the white light in the picture.
Acuity	4	Average value as high values generate noise on the back plane.
White Balance	auto	To adapt it taking into account fixed other color parameters (hue, saturation).
Exposition	75%	Set to keep the brightness at an acceptable level.
Gain	10	Adjusted to keep the consistency between the other settings with minimal add noise addition.

) to ensure the best image acquisition in the lighting condition. The light sources used were white LEDs, and, to reduce the impact of the foreign light sources, we added some light protections, as visible in Figure 4. These protections will also serve to limit laser exposure.

2.5. Detection and Localization

The objective of this module is to detect aphids on images of crops that the robot scans from above, localize them, and send these coordinates to the neutralization module described below. In terms of specifications, it must meet the following three criteria:

• Detection accuracy: a maximum error of 3 mm must exist between the center of detected aphids and their true location in the image so that the laser spot always overlaps a part of the targeted aphid;
• Detection sensitivity: at least 60% of aphids present in the image (this level has been set arbitrarily taking into account that natural predators should finish the work, but it requires experimental validation);
• Real-time operation: the entire program (detection algorithm + laser control) must run at a speed greater than 10 frames per second to permit the robot to cover a 1 ha crop field in 24 h (in the case of a field where rows are located every 40 cm, the robot will have to travel 25 km during 12 h, corresponding to a mean speed of 1 km/h or 29 cm/s).

The problem of detecting and localizing aphids of different sizes, colors and positions in natural image sequences of various crops is particularly difficult. From our knowledge, only artificial neural networks based on deep learning detection architectures can perform this task in real-time. Therefore, to detect aphids in each image acquired by the camera located under the robot as the latter proceeds over each crop row, we selected from the current arsenal of detection architectures (EfficientDet, Yolo, FasterRCNN, U-Net with Hausdorff distance [37]) YOLO version 4 [38] for its speed and its small-size object performance in images. Such approaches are supervised; this means that an important data set with manual annotations is needed for the training phase. When trained, the network was transferred and run on an embedded GPU board (Nvidia Jetson Xavier AGX).

To do that, we first built an image database featuring the two pea aphid lines, A. pisum LL01 (green) and YR2 (pink) on broad bean seedlings. We chose A. pisum on broad bean for the training set because of the color diversity of the aphid lines occurring on the same crop, which has an optimal leaf surface for plant and aphid detection. Furthermore, we used young plants for the detection of plants and aphids because of their vulnerability to pest attacks and their compactness compatible with the regular passage of a robot above the rows. Thus, this technology could be useful for the detection and neutralization of pests that can cause serious damage and economic losses to vulnerable young plants, such as soybean aphids [39] and black cutworms on corn [40], or crops with a compact habit, such as black bean aphids on sugar beets [41].

The training set of images was annotated using LabelMe (see https://github.com/wkentaro/labelme, accessed on 9 September 2022) software. Around 700 images were manually labeled, with each one featuring 20 to 30 aphids. It was necessary to manually draw a bounding box around each aphid as well as to put a dot on the center of each aphid. Since we did not have a sufficient amount of images for training the neural networks from scratch, we fine-tuned our network from an already trained network (trained from MS COCO data set) and we used data augmentation on our images during training to avoid the phenomenon of over-fitting and to improve the generalization ability of the trained model to new images. The following image transformations were applied randomly to generate modified images during training:

• Horizontal or vertical image flip;
• Rotation of 10° in the positive or negative direction;
• Brightness modification;
• Hue and saturation modification;
• Contrast modification;
• Grayscale modification.

We started to develop this localization function with YOLOv4 [38] (whose code is available at https://github.com/AlexeyAB/darknet, accessed on 9 September 2022) as it is very efficient in the detection of objects in real-time and is broadly used in applications requiring fast detection speed, such as in drones or autonomous car vision. YOLOv4 is a one-stage detector. Its backbone is CSPDarknet53 [42], a model tuned for image recognition. The CSPNet framework improves the residual structure of the neural network [43] to diminish the issues of the gradient-vanishing problem during back-propagation of the gradient, encourage the network to reuse feature maps (with residual layers) at different hierarchies, and reduce the set of optimization parameters. The secondary part of the network (the neck) features a PANet (path aggregation network) [44] to mix and combine the maps of characteristics generated by the backbone. It is indeed a preparation step for the following detection. The YOLO detector predicts the class of the object, its bounding box, and the probability of its presence in its bounding box at three different scales. Because of its small characteristic maps, this kind of multi-scale detector struggles to detect small objects in the image. Thus, in practice, it could not detect aphids located at a distance greater than 30 cm from the camera.

We tested several input image sizes (640 × 640 and 512 × 512) knowing that the size of the images acquired by the camera is 2208 × 1242 pixels to determine the best choice. We also optimized the network process using batch inference, which was not yet implemented in the original YOLOv4 code. To speed-up the inference time, we chose tkDNN (the source code for tkDNN is available on Github: https://github.com/ceccocats/tkDNN, accessed on 9 September 2022), which is a neural network library specifically designed to work on NVIDIA Jetson boards. It is built with the primitives of cuDNN, a library from NVIDIA regarding neural networks, and TensorRT, an inference platform high performance that enables high throughput and low latency for inference applications related to deep learning. TensorRT engine optimizes models by quantifying all network layer parameters and converting them into formats that occupy less memory space. The tkDNN library allows inference with network parameters in three different numeric formats: FP32 (a 32-bit floating point format representation), FP16 (a 16-bits floating point format), and INT8 (integers represented with 8 bits). In this project, we chose the FP16 format as it is displayed the most accurate and efficient format (see https://github.com/ceccocats/tkDNN#fps-results, accessed on 9 September 2022). The data degradation caused by format conversion remained meaner than 0.02%.

We also studied an approach to detect smaller objects without a bounding box based on the weighted Hausdorff distance [37] (code available on Github: https://github.com/javiribera/locating-objects-without-bboxes, accessed on 9 September 2022). The chosen neural network architecture was a U-Net optimized with a loss function based on this distance. Rather than taking bounding boxes as references (ground truth), the network seeks to localize objects according to reference points.

Once the bounding box center or an aphid point was well located on the image in the form of pixel coordinates, it was then possible to convert these points into a 3D reference using the Software Development Kit of the ZED mini camera. Indeed, the 2D image is taken by only one of the two objectives of the ZED camera, and the other one is used to interpret the points in 3 dimensions. As the captured images were in RGBXYZ format (2D image with depth), we could obtain the cloud measurement of neighborhood points in XYZ (3D) from a detected point on the RGB (2D) image.

2.6. Laser-Based Targeting

Targeting aphids from their localization in acquired images requires visual servoing. Indeed, the 3D positions of the detected aphids furnished by the RGB-D camera were not precise enough to efficiently target aphids. Several works have dealt with visual servoing. The closer ones are analyzed here. In 1996, Hutchinson et al. proposed an IBVS (image-based visual servo) approach with a PID control law. They obtained a mean and maximum error of 2 and 3 pixels, respectively, on flying mosquito targets [45]. In 2016, Andreff et al. used the same IBVS approach in laser micro-surgery, also using a 2DOF piezoelectric mirror to steer the laser ray [46] with a simple proportional gain in the control law, resulting in an exponential convergence of the error. In 2018, Kudryavtsev et al. developed a visual servoing system for a three-tube concentric tube robot, using an adaptive proportional gain approach that was successfully validated in simulation using the visual servoing platform ViSP and then in an experimental setup [47]. The adaptive gain increased the convergence speed while ensuring stability, avoiding overshoot for small errors and limiting the maximum speed.

Finally, we based this study on the works described in [48]. In this specific case, the system consisted, on the one hand, of a coarse tracking system using a pair of stereoscopic cameras to identify the approximate 3D location of a mosquito pest target. On the other hand, the system featured a visual servoing system based on a single high-speed camera and a fast scanning mirror to guide the tracking laser to the target pest aiming at minimizing pixel tracking error. The annihilating laser dose was then administrated through the same optical path. To orientate the laser beams towards the targets, we chose a 2-axis system that orientates two micro-mirrors and permits ±25° of rotation range with a resolution meaner than 5 µrad in a closed loop and a repeatability of 100 µrad. This system features sensors with a bandwidth of 350 Hz for rotations meaner than 0.1°. The IBVS approach was chosen because of the properties of the laser-mirror system and the robot operation. The exact geometric relationship between the target and the mirrors is not known given the evident variability of the pest locations and the complicated potential estimation of their 3D position using the camera, in contrast to an industrial application where the target can be fixed. Three control laws were studied, consisting of two variants of an adaptive proportional gain (AG1 and AG2) and a PID. These control laws were implemented in the visual servoing controller in Figure 5. The PID controller, expressed in z space, was:

$$C_{PID}\left(z\right)=K_p+K_i\frac{T}{1-z^{-1}}+K_d\frac{1-z^{-1}}{T}$$

(1)

where $K_p$, $K_i$, and $K_d$ are constant parameters, and T is the sampling period. The AG1 controller was:

$$C_{AG1}\left[k\right]=\lambda \left[k\right]\mathrm{with}\lambda \left[k\right]=\lambda \left[k-1\right]+c_1·e^{-c_2·\epsilon {\left[k\right]}^2}$$

(2)

where $\lambda \left[k\right]$ is the adaptive gain, $c_1$ and $c_2$ are constant parameters, and $\epsilon \left[k\right]$ is the error signal ($\epsilon =p_d-p_l$). The AG2 controller was:

$$C_{AG2}\left[k\right]=\lambda \left[k\right]\mathrm{with}\lambda \left[k\right]=({\lambda }_0-{\lambda }_{\infty })·e^{-\frac{{\dot{\lambda }}_0}{{\lambda }_0-{\lambda }_{\infty }}\epsilon {\left[k\right]}^2}+{\lambda }_{\infty }$$

(3)

where ${\lambda }_0$ (resp. ${\lambda }_{\infty }$) is the gain when $\epsilon =0$ (resp. ∞), and ${\dot{\lambda }}_0$ is the slope of $\lambda \left(\epsilon \right)$ when $\epsilon =0$ [49].

The proposed control scheme runs on the Jetson Xavier board. We first tested the targeting system with 2D scenes of virtual aphids (to isolate the targeting performance from the aphid detection performance). We input in real-time the coordinates of virtual aphids randomly located in the images of the camera. We did the same to evaluate the performance of the system with moving targets by varying the coordinates of virtual aphids in the images. The camera was located on top of a 40 cm radius disk that was explored by the visual servoing system with a low-power laser beam (see Figure 6). During experiments, the system orientated the micro-mirrors to make the laser spot, visible on the plane, travel as quickly as possible to each target.

2.7. Multiple Target Optimization

When multiple pest targets are present in the image, optimization is required to minimize the global distance traveled and thus to reduce the time taken to destroy all the targets. An example of a popular algorithm in literature is the traveling salesman problem (TSP). It is a combinatorial optimization algorithm that computes the shortest possible path to reach a set of objects. However, the TSP literature mainly considers only stationary targets and ignores any time-dependent behavior, which is critical in this application since every target moves relative to the camera. It results in a delayed disappearance from the image and thus their potential miss. Helvig et al. [50] proposed a generalization of a time-dependent TSP, which they called mving-target TSP. It was related to several applications where a pursuer must intercept in a minimum time a set of targets moving with constant velocities. The following theorem was developed for a mixture of approaching and receding targets from the origin, which would be analogous to this application where aphids are approaching and receding from the origin of the image. Two types of optimization algorithms were developed and tested: moving targets TSP, and a hybrid moving-targets TSP and nearest neighbors.

The first one consisted of prioritizing the targets in the function of their distance to the image origin in increasing order. This is equivalent to prioritizing the targets with the greatest Y-coordinate in the image axis (see Figure 7). An alternative solution was tested where a secondary criterion was applied, giving priority to the nearest target from the current position, selected among the lowermost targets.

To evaluate the performance of the system with moving targets, we made the virtual aphids translate (towards the bottom of the image) in real-time in the images at a constant velocity of 5 pixels per second. We also performed experiments with blue LED stripes located in a 40 cm × 20 cm on a rotating disc to test the system by including some target detection. We could not use living aphids as they constantly move in such a situation. We similarly could not use dead ones as they do not keep their initial color for enough time.

2.8. Laser Choice and Dimensioning

To select the most effective laser, the correlation between laser energy and aphid mortality was evaluated and published in a previous work [51]. For that purpose, aphids treated with different laser energies were put back on the leaves and observed every day for mortality counts. Careful examinations made it possible to distinguish between live and dead aphids. As long as an aphid moved, even slightly, it was considered alive. In contrast, dead aphids were those with darkened abdomens due to laser burns and stiff spread feet; they were unresponsive to any stimuli.

A first experiment to determine the lethal dose (LD) necessary to achieve a 90% mortality (LD90) at Day + 1 was set up as in [27,52].

Three wavelengths were used, with each one covering a different part of the electromagnetic spectrum: 532 nm (visible), 1090 nm (short wavelength infrared (SWIR)), and 10.6 µm (long wavelength infrared (LWIR)) known as a CO${}_2$ laser, manufactured by CNI Lasers, IPG and an Access Laser, respectively. Samples of 48 aphids originating from the three lines aforementioned were used. The aphids were random unsynchronized adults, representative of what would be found in a field without an early detection system. The Hosmer–Lemeshow and the area under the receiver operating characteristic curve tests were performed on logistical fits to ensure that the retrieved fit was correct and accurate.

Then, to analyze the mortality dynamics of aphids during their development, the same experiment was conducted on synchronized one-day-old nymphs (N1) with a mortality counting every day until they reached their adult stage on the 7th day, thus considering an LD90 at Day + 7. Given the great results of the previous experiment, we focused on the 10.6 µm laser and used the A. pisum line LL01 so we could rely on their well-known life-history traits [35].

Going further, potential transgenerational laser-induced effects were addressed as well. To do this, we reproduced the previous experiment, but this time using an LD50 to have a good balance between the number of survivors and quantifiable laser-induced effects and to ensure a robust statistic. The survivors of the irradiated aphid generation (F0) at Day + 7 gave birth to nymphs of the second generation (F1) every day for 15 days. For the two generations F0 and F1, mortality was assessed every day during their development. Then, we measured their mass and developmental delay at Day + 7. Finally, we monitored the fecundity for 15 days of F0 aphids and F1 aphids born on the 1st, 5th, 10th, and 15th days of the reproduction period of F0 aphids [51]. All the treatments were repeated five times. Statistical tests were performed using R Studio software (Boston, MA, USA). The mortality rate of the synchronized N1 aphids is presented here with a GLM Gaussian test with a 95% confidence interval.

Moreover, the laser beam must be innocuous for host plants. Indeed, in case of false-positive detection, the entire laser energy would strike them. Hence, the impact of the energy delivered in our experiments has also been investigated on host plants in our previous work [51].

V. faba and T. aestivum host plants were chosen for the A. pisum and R. padi aphid lines, respectively. The plants were cultivated as described in Section 2.2 and shot at the seedling stage of 2–3 leaves. We shot the host plants 4 times with different values found in each experiment with the 10.6 µm laser. First, to ensure no underestimation, we used the highest fluences corresponding to LD90 at Day + 1 on A. pisum LL01 and R. padi adult aphids. Then, we also used the fluence that is the most reasonable for in-field use, corresponding to LD90 at Day + 7 on N1 LL01 aphids. According to aphids’ location on their host plants, we shot different organs, namely the adaxial and abaxial leaf surfaces, the stem, and the apex. Samples of 20 plants were shot by fluence level and by targeted organ. On Day + 7, height, fresh mass, and leaf surface were measured. The statistical analyses detailed in the previous study [51] did not show any significant difference between control and irradiated plants.

3. Results

A prototype has been designed and built with the only purpose to validate the localization-neutralization loop on a single seedling row (see Figure 8). The experimental results performed in the laboratory with this prototype demonstrate the feasibility of detecting different lines of aphids (50% detected at 3 cm/s) and of neutralizing them without impacting the growth of their host plants.

3.1. Aphid Detection and Localization

3.1.1. Lighting Conditions

We could observe that the best results in aphid detection were obtained with the white LEDs. Hence, white LED strips were installed in the robot to create a homogeneous lighting environment, reducing the sensitivity to external conditions (see Figure 4). The camera settings were also adjusted to limit light reflections and maximize the contrast (maximum contrast and saturation, reduced gain; see Table 1). These results were obtained with the robot over an artificial ground reproducing near field conditions. They should be confirmed by real in-field experiments with various natural ambient light conditions.

3.1.2. Localization Performance

We evaluated the performance of the two convolutional neural networks (Darknet YoloV4 and U-Net with Hausdorff) using precision and recall criteria. Accuracy is defined by the proportion of correct detection among all the proposed detected aphids. Recall (or sensitivity) is the proportion of correct detection among all the relevant elements. A detection was considered a true positive when the distance between its predicted position and the reference was less than 5 pixels. This tolerance was defined by considering that the average size of aphids on the image was 10 × 10 pixels. This corresponds to a maximum error of 2 mm when the aphids are close to the camera (≈25 cm) and 4 mm when they are at the camera detection limit (≈50 cm). Both algorithms were evaluated based on 84 test images. The networks were trained with the same data set of 400 training images plus 200 validation images.

Table 2. Performance comparison between YOLOv4 and U-Net with Hausdorff distance (U-Net-HD) for aphid detection. Values in bold indicate the best results for each criterion.

	YOLOv4	U-Net-HD
FPS (Nvidia Quadro 400)	10-11	2-3
True Positive (TP)	238	278
False Positive (FP)	490	997
False Negative (FN)	1371	1349
Precision	0.37	0.15
Recall	0.21	0.17

summarizes the experimental results obtained with both proposed networks. We manually optimized the U-Net-HD architecture and its hyper-parameters to obtain the best results in terms of true positive detection (TP). Such optimizations were not performed on YOLO. Despite the manual optimizations, YOLOv4 outperforms the U-Net-HD network considering the speed, precision, and sensibility criteria. Therefore, we worked on the optimization of YOLO.

The effect of cropping the camera images and keeping only the relevant area (800 × 800 pixels) led to a rise of 6% in accuracy and 5% in sensitivity compared to a whole picture. Indeed, the YOLO network has an input dimension of 512 × 512 pixels. Sub-sampling the input image (2208 × 1242 pixels) to a 512 × 512 pixels image blurred the objects and caused a reduction in precision.

To get better accuracy, we also tiled the area of 800 × 800 pixels into 4 or 16 sub-parts. The inference speed (frames per second) was computed when the network ended up processing all these areas. We observed an increase in sensitivity when using 400 × 400 pixel tiles (0.73 versus 0.51 for 800 × 800 pixel tiles). In the case of very small images (200 × 200 pixels), the sensitivity falls to 0.53. We also noticed that this method greatly reduces the inference speed of the model because of the large number of images to be processed for each capture. We also observed that the code developed in C++ (gcc version 7.5.0-ubuntu 18.04) was more efficient than that of Python API (Python 3.6.9, ubuntu (about 1.5 times faster in terms of FPS speed). The averaged results on our test set are visible in

Table 3. Performance comparison for YOLOv4 with smaller images.

Network Input Size	Input Image Size	FPS (Quadro 400)		Precision	Recall
		Python API	C++
640 × 640	2208 × 1242	10–11	16	0.35	0.49
512 × 512	2208 × 1242	13–14	21	0.34	0.46
512 × 512	800 × 600	13–14	21	0.4	0.51

. Some results are visible in Figure 9. This sample frame illustrates the results obtained with YOLOv4. It shows a single frame from the test set, (a) showing the initial image, (b) the known position of aphids, and (c) the resulting prediction. One can observe that 2 aphids were not found while another non-existing one was found.

We could then raise the model inference speed from 9 FPS to 19 FPS without any loss of precision. We used the tkDNN engine, FP16 floating point precision, and the C++ API on the Jetson Xavier AGX embedded board with cropped images of 320 × 320 pixels. Some resulting films are visible on https://www.creatis.insa-lyon.fr/~grenier/research/GreenShield/, (accessed on 9 September 2022). It was more than our real-time detection expectations.

3.2. Laser-Based Neutralization

3.2.1. Pest Neutralization

The efficiency of the laser treatment on unsynchronized adult aphids has been studied and published in [51]. We analyzed the effect of lasers with wavelengths in the visible (532 nm wavelength), in the near infra-red (1070 nm) and in the far infra-red (10.6 $\mathsf{\mu }$m). We demonstrated that the 1070 nm wavelength is inefficient for killing R. padi aphids, requiring lethal doses four orders of magnitude higher than the two other wavelengths. For A. pisum LL01 (green) and YR2 (pink) aphids, the 10.6 µm wavelength is more effective than the 532 nm one. For instance, with green aphids, LD90 fluence values of 53.55 J·cm${}^{-2}$ and 12.91 J·cm${}^{-2}$ are retrieved at 532 nm and 10.6 µm, respectively. Indeed, going from one laser to another results in an LD90 more than four times lower. Hence, the 10.6 µm wavelength was chosen for the rest of the study. These values are quite the same for the pink aphids. However, this is not the case for R. padi (black) aphids. In this case, both wavelengths exhibit the same efficiency. The smaller size of this specie can explain this difference.

By changing the targeted aphid stage and extending the observation period, the LD90 can be lowered again. Consequently, a lower LD90 fluence means faster, more secure, and more energy-saving treatment. Thus, targeting nymphs results in an LD90 at Day + 1 that is reduced by four times compared to adults. Moreover, counting the nymphs’ mortality at Day + 7 instead of Day + 1 leads to a 2.6 times decreased LD90 value. Globally, targeting nymphs and observing them at Day + 7 allows for decreasing the fluence to 1.15 J·cm${}^{-2}$. This value is ten times lower than the previous value, while still maintaining a high level 90% neutralization rate. Moreover, the mortality dynamics of aphids over 7 days are not linear [51]. Indeed, most aphids are already neutralized after 3 days, and then a plateau is reached. This means that the LD90s at Day + 7 and Day + 3 are almost identical. Consequently, the irradiated aphids with a low fluence will already be dead after 3 days, thus reducing the duration of their attack on the plants.

Regarding potential transgenerational effects, results show that laser treatment does not induce any significant effect on the mortality of the next F1 generation [51]. Hence, the laser-based strategy must be based on direct laser effects.

3.2.2. Targeting

The IBVS algorithm featuring a PID controller obtained the best results versus the two variants of AG. We could obtain a mean response time $t_{5\%}$ = 490 ms (vs. 830 and 920 for AG), corresponding to a mean $t_{5\%}$ = 490 ms (see Figure 10). With moving virtual targets, the PID control showed slightly better performance with a mean $t_{5\%}$ = 420 ms (vs. 590 and 870 for AG). We also performed experiments with velocities ranging from 5 to 12 pixels/s. We observed that the system did not have the time to travel to all targets at 12 pixels/s and over (corresponding approximately to 1 cm/s at a distance of 30 cm). Experiments with blue LED stripes showed a mean response time $t_{5\%}$ that greatly increased to 620 ms.

4. Discussion

Concerning the detection of aphids, by inferring with 2208 × 1242 pixel resolution images, we found that the speed of the Hausdorff-optimized U-Net detection was not suitable for real-time application. As the images were captured at 15 FPS by the ZED mini camera, and this network can process only 2 or 3 images per second, many images were not taken into account when the robot was moving. This was due to the excessive depth of this model, which was set to increase the ability to detect small objects. This indeed resulted in a gain in precision but a loss in efficiency. Moreover, the accuracy of U-Net was not better than that of YOLOv4 (accuracy 0.15 vs. 0.21 of YOLOv4) as it generated much false detection (997 false positives versus 490 false positives for YOLOv4). We then simplified the architecture of U-Net by reducing the number of layers and by training it with cropped images of smaller sizes. However, its accuracy remained poor compared to the model of Darknet YOLOv4. The simplified U-Net model failed to achieve 10% accuracy on cropped areas of 400 × 400 pixels. Taking into account these results, we, therefore, decided to apply YOLOv4 for the detection of aphids, with the aforementioned accuracy and speed optimizations.

Concerning the targeting process, aiming at sending the laser spot onto detected aphids, we could obtain medium performance in 2D scenes with mean travel times of 620 ms between two targets. This is much too low a value and was moreover obtained in laboratory conditions with virtual aphids, knowing that plants infected with tens of aphids should be treated faster to enable a 1 ha field treatment in 24 h at 29 cm/s. Speed improvement should be researched for a robot spanning several rows with several detection/neutralization modules working in parallel. Faster micro-mirrors should be envisaged, taking into account the cost and affordability of such a robot.

Additionally, to perform a precise localization of the laser spot, it was necessary to be in a black ambiance, while aphid detection required a good luminosity. We then had to alternate detection and targeting and synchronize the lighting so that each detection was performed in the best lighting conditions. In the robot, as we employed LED stripes, we could efficiently perform the switching, but these working conditions decreased the operational flow. On-robot experiments showed that the performance decreases as soon as the lighting conditions deteriorate (which may be the case during sunny days with external rays of light coming through the entrance for plants) and when 3D objects are present in the scene, necessitating stopping the robot. Indeed, in 3D scenes, the spot was sometimes not visible due to the non-alignment of the laser source and the camera axis and the presence of leaves in different depths of field. The algorithm had to work blindly during the time the spot remains invisible. When this period was too long, the system skipped this target and selected the next one. This greatly slowed the servoing and reduced the performance. Therefore, during the experiments, to decrease these periods, even if we could target aphids with the robot moving, we preferred to stop it during the targeting phases. A study enhancing the robustness of visual servoing in the presence of occlusions is mandatory.

At last, Newscale micro-mirrors are too fragile for farming conditions. Other stronger solutions have to be studied, knowing that we did not study the impact of vibrations due to the locomotion of the robot on the ground yet. Regrettably, we could not validate (for logistical reasons) the whole chain simultaneously (from aphid detection to neutralization with the power laser): detection/localization/targeting has been validated independently from targeting/neutralization. Globally, we could make the robot proceed at a velocity of 3 cm/s and have it detect 50% of the present aphids, then target 83% of the detected aphids on 3 successive broad bean plants, with each one featuring 8 aphids. This is far from the requirements but it provides the first proof of concept validating the feasibility of such an approach.

Regarding aphid neutralization, we found that the CO${}_2$ laser is best suited to neutralize almost all aphids with low fluences by raising their cell water temperature with LWIR at 10.6 µm. Indeed, the energy needed to have an LD90 after one day in an adult population equal to 12.91 J·cm${}^{-2}$ per shot was 4 times lower than the other lasers tested. Additionally, targeting younger aphids as N1 and extending the observation period by a few days can lower the energy required by 10 times, resulting in 1.15 J·cm${}^{-2}$. Keller et al. found a value quite similar for mosquitoes using the same wavelength (1.1 J·cm${}^{-2}$) [27]. Additionally, they concluded that the LD90 value is roughly stable when changing the beam size. These results and our estimates of energy consumed for a field treatment detailed in our previous work [51] show that the laser is not the most power-consuming component in our robot. However, the detection must be very precise to target aphids at the youngest stages and to differentiate them from other insects beneficial to crops. Moreover, most aphids are localized on the abaxial leaf surface. They are not visible by the camera that films the leaves from above. We thought of putting the camera in a lower position, aiming upwards, but we fear that such a position would be risky in case of obstacles, and it would be more sensible to dust accumulation. Another (more interesting) approach would consist of agitating the leaves using airflow or a mechanical system so that aphids leave them to fall (fainting being dead). We would then just have to detect and shoot them on the ground.

Finally, in case of false positive detection, we ensured that hitting leaves would not impact plant growth [51].

5. Conclusions

The experimental results (a video is available at https://www.creatis.insa-lyon.fr/~grenier/research/GreenShield/, accessed on 9 September 2022) globally demonstrate, in lab conditions, the feasibility of detecting different lines of aphids (50% detected at 3 cm/s) and of neutralizing them by CO${}_2$ laser shots with high efficiency (90% mortality after 3 days with 1.15 J·cm${}^{-2}$ for A. pisum LL01 N1) without impacting the growth of their host plants in case of a missed shot. Showing the feasibility of this approach is encouraging as aphids are one of the most difficult crop pests to combat. Thanks to a black environment reproduced under the robot, the latter could be used particularly on vulnerable young plants such as soybeans and corn or crops with a compact habit such as sugar beets. However, aphid neutralization is closely dependent on the quality of the detection, since it is recommended to detect aphids at the nymphal stage and to differentiate them from the beneficial insects that one wishes to preserve on the crops. Other pests such as fall armyworms Spodoptera frugiperda and dark sword grass Agrotis ipsilon larvae that are bigger should pose fewer detection issues. Future directions consist of enhancing the performance of the detection and targeting phases to obtain characteristics closer to the requirements, making it useful in a real farming context. Additionally, the robustness concerning field conditions is to be studied. The most worrying aspect is the presence of vibrations due to the irregular ground, which may decrease the performance of the positioning of the micro-mirrors and so the quality of the pest targeting.