Automated Remote Insect Surveillance at a Global Scale and the Internet of Things

Abstract

The concept of remote insect surveillance at large spatial scales for many serious insect pests of agricultural and medical importance has been introduced in a series of our papers. We augment typical, low-cost plastic traps for many insect pests with the necessary optoelectronic sensors to guard the entrance of the trap to detect, time-stamp, GPS tag, and—in relevant cases—identify the species of the incoming insect from their wingbeat. For every important crop pest, there are monitoring protocols to be followed to decide when to initiate a treatment procedure before a serious infestation occurs. Monitoring protocols are mainly based on specifically designed insect traps. Traditional insect monitoring suffers in that the scope of such monitoring: is curtailed by its cost, requires intensive labor, is time consuming, and an expert is often needed for sufficient accuracy which can sometimes raise safety issues for humans. These disadvantages reduce the extent to which manual insect monitoring is applied and therefore its accuracy, which finally results in significant crop loss due to damage caused by pests. With the term ‘surveillance’ we intend to push the monitoring idea to unprecedented levels of information extraction regarding the presence, time-stamping detection events, species identification, and population density of targeted insect pests. Insect counts, as well as environmental parameters that correlate with insects’ population development, are wirelessly transmitted to the central monitoring agency in real time and are visualized and streamed to statistical methods to assist enforcement of security control to insect pests. In this work, we emphasize how the traps can be self-organized in networks that collectively report data at local, regional, country, continental, and global scales using the emerging technology of the Internet of Things (IoT). This research is necessarily interdisciplinary and falls at the intersection of entomology, optoelectronic engineering, data-science, and crop science and encompasses the design and implementation of low-cost, low-power technology to help reduce the extent of quantitative and qualitative crop losses by many of the most significant agricultural pests. We argue that smart traps communicating through IoT to report in real-time the level of the pest population from the field straight to a human controlled agency can, in the very near future, have a profound impact on the decision-making process in crop protection and will be disruptive of existing manual practices. In the present study, three cases are investigated: monitoring Rhynchophorus ferrugineus (Olivier) (Coleoptera: Curculionidae) using (a) Picusan and (b) Lindgren trap; and (c) monitoring various stored grain beetle pests using the stored-grain pitfall trap. Our approach is very accurate, reaching 98–99% accuracy on automatic counts compared with real detected numbers of insects in each type of trap.

1. Introduction

Given the urgency to increase world food production to satisfy the needs of an increasing population, it is vital that we assist farmers to make decisions to mitigate high rates of crop loss due to insect pests and thus implicitly increase food production. Crop losses due to insect pests can be substantial and may be prevented, or reduced, by crop protection measures [1,2]. Farmers rarely have the quality of information needed to make timely decisions about insecticidal treatments. We attribute this to the fact that farmers’ knowledge on when and where the infestation initially occurred and how effective the treatment was, is based on manual inspection of monitoring traps deployed on a limited spatial and temporal scale. Scouters performing visual identification and counting of insect species are not always confident experts making reliable identifications and the protocols they follow often result in compromises. Manual inspection of traps shows considerable heterogeneity in geographic and temporal coverage. Since inspection of insect traps, is concentrated in a few sites, this highly aggregated distribution of information, limits our ability to understand the large-scale dynamic of insect distribution and to benefit from its knowledge.

From both a conceptual and management perspective, there is an urgent need to increase the information flow over a large area and through time from the field-traps straight to a central monitoring agency, as well as to visualize and summarize this flow in a statistically reliable sense. To this end, we develop technologies to improve, expand and automate global monitoring of insects of economic importance to thousands of nodes around the world.

Innovative use of sensors and networks targeting animals are starting to be translated into new ecological knowledge [3,4]. Automatic monitoring of biodiversity [5,6] mainly in the form of automated species identification of vocalizations of birds [7], bats [8], insects [9,10], whales [11] and amphibians [12] is a developing trend in ecology. This knowledge, however, is still fragmented and isolated to small scale paradigms that neither communicate nor are integrated with a universal view of biodiversity monitoring.

In our vision, a trap is a ‘thing’ in the context of the IoT, i.e., a typical plastic trap augmented with a sensor that records the insect’s presence upon its entrance in the trap and with wireless communication capability to broadcast the sensed data. These individual entities are single nodes that can possibly interact with other nodes to establish their own network or they can report straight to the conventional internet highway through the ubiquitous mobile phone coverage (i.e., through the GPRS functionality). We are currently investigating a multitude of sensors to detect the insects’ presence in a distributed fashion and in a cost- and power-effective way. As an example of distributed monitoring, in the stored-grain pitfall case several traps are placed in the silo. These e-pitfall traps exchange information and collectively report insect counts to the management service of the silo. Several ways have already been identified to detect insects’ presence:

(a)

photo-interruption of either entering or falling insects in several types of traps (e.g., Red-palm weevil traps, pitfall traps, funnel traps). Photo-interruption is also used in electronic gates installed in beehives. A low power emitter of infrared light and a coupled photodiode form a sheet of light covering the entrance of the trap. The flow of light is interrupted from an insect entering and thus it is counted,

(b)

Analysis of the wingbeat of insects flying in traps (as in McPhail type and mosquito traps). The flow of light is modulated by the wingbeat of the insect flying-in. The wingbeat is recorded and constitutes a biometric signature of a specific species [9,13],

(c)

Picking up their vibrations due to locomotion and feeding (chewing) in grains (only for stored grain pests [10,14,15]).

Traps equipped with a detection sensor and wireless communication abilities have some distinct advantages compared to manual monitoring. They can monitor insect populations 24 h a day, upon their entrance to the trap, every day of the year, in dispersed nodes across a variety of fields, simultaneously, and all counts and recordings can be permanently stored in a cloud service. Another distinct advantage is the determination of the precise onset of an infestation. The time-schedule of trips to the field for trap inspection may not coincide with the initiation of a pest population increase, whereas automatic monitoring and reporting can be set in real-time (or more practically in a daily schedule). Real-time reporting, in our opinion, opens up new grounds in agricultural research and mainly in crop protection as—besides a timely control action in response to a pest infestation—it can help in the evaluation of the impact of a control treatment (e.g., chemical spraying, release of beneficial entomophagous insects etc.), and therefore schedule future actions accordingly.

The electronic traps can naturally include a time stamp of each insect incident and the information provided by the e-traps formulates new services: to carry out studies that cannot be practically performed manually, as explained hereinafter: There have been numerous studies demonstrating the periodicity of trap captures [16,17,18,19,20,21]. Some insect species appear to respond to pheromone during daylight while others are active during the night because of complicated mechanisms of insect physiology and reproduction [22,23]. A record including the trap location, time, date, temperature and humidity each time an insect enters the trap, would provide a significant amount of data that would help us understand better the chemical ecology of a pest. It is very difficult to run these studies with direct observation and they are not replicated through time or across sites to any great extent because of the manpower requirement.

Another advantage of automated insect surveillance is that long-term population and distribution data for insect species of interest can be logged. The logging of adequate historical data can help us to understand the population dynamics of the pest and use predictive models to estimate statistically meaningful risks of an infestation, its evolution and the possibility of future outbreaks. Finally, communication on a global scale can bring researchers to work together across large distances on the same pest by navigating themselves through the IoT to distant traps or receiving data summaries across all sensors in the network. Infestation data at global scales can be exploited by commercial and colonial interests to forecast prices in crop production.

The reported literature on electronic insect traps that employ optical sensors is sparse. In Hendricks [20,24] the first integrated synergy of a trap with electronic elements is reported with a view of transferring recorded data to a computer. The approach is interesting given the means of that time. In [25,26], the authors presented a stand-alone device that would count and transmit counts from the field of a very destructive fruit pest, the oriental fruit fly, Bactrocera dorsalis (Hendel) (Diptera: Tephritidae).

Our approach aims at reducing the necessity of human-in-the loop in any intermediate processing stage of the workflow and reserves the need of expert entomologists only for the highest abstraction layer: the interpretation of the data received (trap catches) normally presented in the form of georeferenced maps and the corresponding decision making and action planning based on pest Economic Injury Levels (EIL) population thresholds that are applied in the frames of Integrated Pest Management (IPM). Our work focuses on leveraging the quality of service of remote surveillance of pest populations to a better and cost-effective status than sparsely applied human inspection.

2. Materials and Methods

We have embedded our electronics in traps monitoring the population of insect pests of olive, cotton, grapes, fruit trees, stored cereals and pulses, pine trees, and palm plantations. Mosquitoes and beehives are a category of their own that have also been integrated within our framework. Our approach is not constrained to a specific brand or type of trap. However, we need types of traps that protect to a certain extent the exposure of the electronics and have a shape that allows the insects to pass through a funnel entrance so that they constrain their movement pattern to be counted and/or recorded. In this work, we provide field results for three traps: The Picusan (SANSAN Prodecing SL, Valencia, Spain), the Stored-grain pitfall (EDIALUX, Bornem, Belgium), and the Lindgren (Forestry Distributing, Inc., Boulder, CO, USA) type, as the evaluation of traps in the field is a laborious process. In time, results of all common trap types will be presented in detail.

The philosophy in all trap types is common: There is always an emitter of light opposite to a receiver of light and the path of the incoming insect stands in between. The interruption of the path of light effects a voltage drop that exceeds a threshold and constitutes a count. The technology that does not analyze wingbeat (e.g., as in the case of fruit flies and mosquitoes) is simpler than the one that senses only the presence of insects. All three traps presented in this study sense the presence of an incoming insect. Both receiving and emitting elements are deployed as 1D linear arrays that are long enough to cover the entrance to the trap. There are small variations among these three different traps mainly due to the size of the insects and the peculiarities of the trap (see

Table 1. Optical elements of the sensor embedded in the insect traps.

Trap Type	LEDs/Photodiodes	Diffuser
Picusan	5	NO
Stored-grain pitfall	16	YES
Lindgren	5	NO

). Picusan is custom made for R. ferrugineus that is a relatively large insect whereas the stored-grain pitfall trap needs to count insects possibly smaller than 1 mm. In the Picusan and Lindgren traps the light field is composed of five parallel light-beams (5 LEDs with a small emitting angle of ±10 degrees) opposite to five photodiodes connected in row. Therefore, if there is an interruption of light in any of the five beams no light passes through the photodiodes and the insect is detected by comparing the voltage to a threshold. The distance of one beam from the consecutive one is 7 mm. Therefore, it detects insects larger than 6 mm. In the stored-grain pitfall case, an insect can enter from any hole of the lid. To avoid blind spots in the field of view we need to have uniform field sensing insect sizes ≤0.5 mm. We used 16 LEDs and the same number of photodiodes and both emitter and receiver have a light diffuser. All sensors are operated in pulse mode i.e., there is no constant flow of light from emitter to receiver but a pulse train is emitted. This is the key element to a long-lasting operation and this is analyzed in detail in [9].

In Figure 1 we envisage how insect surveillance can be applied at local, regional, country, continental, and global scales using the emerging technology of IoT. Local networks become themselves nodes in larger networks until reaching global coverage.

The detected events due to photo-interruption need to be combined with the use of species specific pheromone attractants. This is because photo-interruption senses the presence of the insect during its entrance to the trap but is blind to species identity. Therefore, we must ensure that in the clear majority of cases the insect is attracted by a species-specific pheromone. The actual power sufficiency of these traps is three months expandable to 12 months with additional batteries. Although the human presence is again vital to replace pheromone lures it is possible to greatly reduce the visiting frequency at least to a bimonthly basis and simplify the attendance work during trap check. On the other hand, devices equipped with sensors that can analyze the wingbeat, besides sex-pheromones, can use food baits as well, as the wingbeat is a biometric information that can be used as evidence to identify species in an automated fashion.

2.1. Trap Type #1: The Picusan Trap

The red palm weevil, Rhynchophorus ferrugineus (Olivier) (Coleoptera: Curculionidae) is the most dangerous and devastating pest of the date palm as it can weaken and eventually kill the tree [27]. Given the importance of the red palm weevil, efforts are being made to develop new monitoring tools, such as the deployment of a new black pyramidal trap design (Picusan trap Figure 2) [28]. Traps of this type, containing aggregation pheromones of R. ferrugineus are modified to include an optical sensor that senses adult pests falling into the trap. Counts, as well as other environmental parameters are transmitted straight to the internet through GPRS. The aggregation pheromone 4-methyl-5-nonanol or ferrugineol with ethyl acetate, was mixed with a combination of food lures specialized to attract R. ferrugineus, capturing adults of both sexes [28]. Though it did not occur in our experiments, it is known that another species very similar to R. ferrugineus, the sisal weevil Scyphophorus acupunctatus (Gyllenhal) (Coleoptera: Curculionidae) may be attracted to the specific pheromone and enter into the trap [29]. This insect is an important pest of agave, yucca, and various other plants of the families Agavaceae and Dracenaceae [30]. In such a case, the trap would not discern the entrance of S. acupunctatus against R. ferrugineus. In case of palms absence in the monitored area, the same smart trap configuration can be used for S. acupunctatus.

2.2. Trap Type #2: The Stored-Grain Pitfall Trap

Stored-grain pitfall traps are typically used for monitoring several species of stored-grain beetles (Coleoptera) in silos, warehouses and processing plants [31]. They are placed inside the bulk grain near the surface (Figure 3). The cone-shaped device is made of clear plastic and has a removable perforated lid, which allows insects to enter, but not escape. As in the case of the funnel trap, various pheromone lures targeting different species may be used.

Many destructive beetle pests of stored grain may be monitored by this type of trap: the flour beetles Tribolium spp. (Tenebrionidae), the grain weevils Sitophilus sp. (Curculionidae), the lesser grain borer Rhyzopertha dominica (F.) (Bostrichidae), the cigarette beetle Lasioderma serricorne (F.) (Anobiidae), and the khapra beetle Trogoderma granarium Everts (Dermestidae) [32,33,34,35].

For the purposes of our study, a prototype equipped with a linear array of five Light Emitting Diodes (LED) opposite to five receiving photodiodes was evaluated. The prototype trap was put in a large plastic barrel (120 L) with 80 kg maize. Adult beetles of various species were collected from laboratory rearings and transferred to the experimental barrel. In order to ensure trap catches a large number of adult beetles was used resulting in an infestation level of more than 15 adults per kg maize. Caught beetle adults were checked and counted after 24 h and were compared with the counts from the electronic system.

2.3. Trap Type #3: The Lindgren Trap

Pine Beetle Lindgren Trap (Figure 4) is a form of a funnel trap. Lindgren pheromone traps are widely used to attract the pine beetle Dendroctonus ponderosae Hopkins (Coleoptera: Curculionidae) [36,37]. They are used either as monitoring traps or for mass trapping to reduce the populations of pine beetles.

They come with different numbers of funnels that form a tree-mimicking silhouette. We have used them successfully to count R. ferrugineus beetles as well. The same attractants were used as with the Picusan trap.

3. Results & Discussion

Results from the evaluation of the prototype traps are presented in

Table 2. Number of insects (manual inspection) and automatically counted (electronic sensors) adult beetles in three trap types.

Trap Type	Species	Manually Verified	Automatically Counted	Correlation Coefficient (r)
Picusan 1	R. ferrugineus	37	35	0.9966
		42	42
		59	58
Pitfall 2	C. ferrugineus.	59	62	0.9912
		45	49
		67	74
	O. surinamensis	31	34	0.9978
		11	12
		24	25
	R. dominica	15	15	0.9976
		23	24
		24	26
	S. oryzae	21	21	0.9900
		32	36
		29	30
	T. confusum	13	13	0.9912
		26	30
		34	36
Lindgren 3	R. ferrugineus.	14	14	0.9999
		45	49
		67	74

Monitoring period from 1 September 2016 to 5 December 2016; ¹ Number of traps: 3; ² Single trap inside grain mass, insect density >15 adults/kg grain; ³ Single trap hung from a wall externally to a laboratory.

and Figure 5. As is clearly concluded from our data, our system is very accurate, reaching 98–99% accuracy on automatic counts compared with real detected numbers of adult beetles in each trap. The accuracy of our system in detecting adult beetle catches is also shown by the very high (r > 0.99 in all cases) correlation between the generated signals and actual numbers of insects caught in the trap.

Only a few remote pest monitoring systems, based on wireless communication technology, have been evaluated in the past, with varying accuracy. The oriental leafworm moth Spodoptera litura (Fabricius) (Lepidoptera: Noctuidae) was effectively monitored by an ecological monitoring system combining GSM transmission technologies with mechatronics with accuracy ranging from 71% to 100% [38]. Average accuracies of 78.1% [25], 96.3% [39], and 94.9% [40] were demonstrated by automatic monitoring systems counting the catches of the oriental fruit fly B. dorsalis. Other automated systems with image analysis technology also proved to be reliable in detecting mainly whiteflies and moths, with accuracies ranging from 70% to 100% [41,42,43,44,45]. The accuracy of our system is higher than almost all of the abovementioned monitoring systems.

4. Data Processing and the IoT

The IoT has made its appearance in agricultural tasks [46,47,48,49] but is not connected, up to now, directly to the vision of insect monitoring at large spatial scales as this work suggests. Though one may think that the most valuable part of a service based on a network of traps is the trading of the hardware or the software, the priceless product is the transmitted numbers. Granting access to a cloud service visualizing and interpreting data is a business of its own. Data can be used to influence decisions, it can be exchanged, hired or sold and become input in predictive analytics tools whose predictions can lead to new services (e.g., price prediction of food products, prediction for possible pest population outbreaks and crop losses).

The data delivered can be split into three distinct subsets:

(a)

Counts delivered on a pre-scheduled basis along with the time-stamps of each insect entrance to the traps.

(b)

Environmental data (mainly humidity, temperature and GPS tag).

(c)

Wingbeat recordings uploaded to a server (in the case of McPhail and mosquito traps).

Once the data are collected and delivered to the server there are different levels of data processing abstractions that we can apply. A data-collection interface with inference based data analysis provides the basis for predictive ecological models and mining of events for agricultural management. In Figure 6 we show the visualization interface that can be used for streamlining data collection and management. The following mode of engagement between data representations and the human expert is to set counts on maps and interpolate measurement between nodes to form pest population level maps to assess the current situation and respond timely with a focused localized treatment. Additionally, one can validate causal hypotheses between the timing of treatments and after treatment insect counts. Moreover, the distribution of time stamps can be related to the efficacy of different attractants in the pheromone cycle of insects. Regarding the wingbeat recordings, these become a permanent record in the database and can be subjected to different feature extraction and classification practices to identify the source of the audio data at the species level.

Regarding long term data abstractions, once historical data are piled up over time, they provide the basis for predictive models of infestations and outbreaks of epidemics (e.g., as in the case of mosquitoes).

The GPRS modem of the trap, once connected to the mobile provider, is capable of having internet connectivity. The trap has the domain name of the backend site embedded in its EPROM and, through the GPRS modem and the mobile network that supports GPRS, makes a TCP/IP connection to the webserver of the backend, via an HTTP request, essentially mimicking a web browser (client). At this point, the trap inserts its data as parameters for the page that it wants to access. Once the HTTP request reaches the web server, the latter receives the data from the request of the trap (via the appropriate code, written in PHP) and logs the information in the database.

The web application consists of two parts: The backend that manages the information and the database in the server and a frontend that visualizes the information at the browser of the user and the interface with the user. The site is based on a web hosting provider running Apache, with PHP 5.5 support. The database is setup using MySQL, which is open source. The backend is written in Laravel 5.1 PHP 5.5 and the frontend is written in HTML 5, making use of the Angularjs Javascript and JQuery javascript Framework. The data follow the JSON formalisation. The maps are provided via the Google Maps API.

Technological advances do not always manage to penetrate the routine practice of cultivators. To succeed in altering the habits of decades, we must offer a working solution to the real needs of practitioners. Moreover, the cost of altering standard traps to electronic ones must be low and the cost/benefit trade off due to their use carefully calculated. In order to help assessing this ratio we report that the current production cost of an insect counter as in the Stored-grain pitfall trap is 65 Euros for a single device falling to 40 Euros for 100 traps and 32 Euros for 1000. Regarding Picusan and Lindgren the corresponding costs are 40 Euros falling to 26 and 19 Euros respectively (as per 26 May 2017). The power sufficiency of Picusan/Lindgren trap is 3.5 months with a single 3.7/3000 mAh rechargeable lithium battery and as regards stored-grain pitfall 2.5 months using a 3400 mAh battery. It is an encouraging fact that, long term autonomous deployment is feasible due to low-power electronics and scheduling of operations to minimize power consumption and in the long run the cost of electronics can only drop.

Agricultural entomologists master the knowledge of their field and cannot have the technical expertise of disparate and interdisciplinary knowledge requiring the cooperation of diverse technologies. These technologies include wireless communication networks, security of data, quality control protocols, data processing and management. Therefore, the whole sensor-network setup must be offered by engineers and data scientists to stakeholders as a ‘plug and play’ installation. The collaboration of agricultural entomologists is mainly needed for the species identification of trapped insects, assessing the accuracy of the whole set-up, and translating summarized information to ecological knowledge.

Trained practitioners and qualified experts return high quality data but are not easily available to attain field-data from traps. Even when some kind of a sensing modality is available, they are often unwilling to spend their time and expertise to carry out detrimental tasks such as inspecting endless information queues (e.g., microphone recordings or pictures from cameras). Unattended remote surveillance solves the problem of collecting sparse manual data and overcomes scalability limitations, but produces large quantities of data that can prove of limited use if not sorted out and summarized automatically. In this section, we clarify what information we can obtain from a collection of distributed trap nodes and if the visualization and aggregated indices of such data suffice to augment policy decisions.

In this work, we establish a connection between sensors’ readings, pest population level, and predictive models to ensure timely and effective control treatments. Acceptance of automated monitoring practices will raise doubts about the reliability of data collected without an expert’s intervention. The optoelectronics need to reduce their errors (i.e., false alarms and misses) to reach comparable analysis to that done by experts. A long-term field operation is needed to identify the cause of possible false alarms, detection misses, and sensor failures in sometimes harsh conditions before applying the output of such data-collection schemes to modeling and policy. To give a lucid example, we discovered a source of false alarms in the Picusan trap after a rainstorm where some raindrops entered through the top entrance of the trap. This was not observed during the operation of a trap from spring to fall and was observed only during winter time. This problem was easily solved by modifying the top of the trap, but was only solved because of the long-term deployment that indicated a problem. Our perspective is that the potential for big data collected from the collaboration of nodes at large spatial scales can overcome random local errors and this combined with their streaming to data visualization tools is sufficient to grant an advantage over the current manual practices. We believe current results are sufficient to warrant further exploration on insect surveillance. Insect surveillance can provide insight into the effects of insecticide efficiency, reduce their use and shape our understanding of pest problems in agriculture. Provided we continue improving the reliability of devices and services and perform real-field, long-term trials, we can upgrade automated practices to the level of being indispensable to farmers, policy makers and stakeholders.