TinyML Classification for Agriculture Objects with ESP32

Abstract

Using systems with machine learning technologies for process automation is a global trend in agriculture. However, implementing this technology comes with challenges, such as the need for a large amount of computing resources under conditions of limited energy consumption and the high cost of hardware for intelligent systems. This article presents the possibility of applying a modern ESP32 microcontroller platform in the agro-industrial sector to create intelligent devices based on the Internet of Things. CNN models are implemented based on the TensorFlow architecture in hardware and software solutions based on the ESP32 microcontroller from Espressif company to classify objects in crop fields. The purpose of this work is to create a hardware–software complex for local energy-efficient classification of images with support for IoT protocols. The results of this research allow for the automatic classification of field surfaces with the presence of “high attention” and optimal growth zones. This article shows that classification accuracy exceeding 87% can be achieved in small, energy-efficient systems, even for low-resolution images, depending on the CNN architecture and its quantization algorithm. The application of such technologies and methods of their optimization for energy-efficient devices, such as ESP32, will allow us to create an Intelligent Internet of Things network.

1. Introduction

Over the past decade, methods based on machine learning have intensively been under development in agriculture. This is due to the possibility of processing and analyzing big data on the state of crop areas, allowing a more complete look at the process of exploiting land resources and optimizing the efficiency of agricultural methods [1,2]. In response to a growing need for food security in countries, the use of hardware–software systems with integrated machine learning algorithms is especially relevant for effective monitoring of crop yields and crop area conditions [3,4,5,6].

Although modern methods of field monitoring are useful for crop forecasting and optimization of agricultural processes [7], there are limitations to the practical large-scale implementation of these systems. These include

• Insufficient spatial and temporal resolution of data [8];
• Incorrect use of neural network models [9];
• Poor reliability of systems in various operating conditions [10];
• Low energy efficiency [11].

The high cost of hardware, which is based on the use of NVIDIA Jetson platforms in many cases [12,13,14,15,16], affects the agricultural sector; although many promising AI technologies exist locally, they have different levels of implementation and efficiency. These reasons are slowing the growth of digital vision-based solutions for crop monitoring and implementing strategies to enhance food security and avert food crisis.

A review of existing systems for detecting and classifying objects of the agro-industrial complex in field conditions shows that most agrotechnical systems are redundant in terms of hardware and computing power. Currently, there is a large number of neural network models for solving problems in the agro-industrial complex. Most of these software solutions are built on resource-intensive architectures, such as YOLO [17,18], SSD [19], and Fast R-CNN [20], which can at least be launched on single-board computers such as NVIDIA Jetson and Raspberry Pi [21,22]. However, the cost of such solutions is quite high compared to microcontroller-based systems. Therefore, the development of new CNN architectures and their optimization for low-power computing devices have become an emerging trend in machine learning, known as Tiny Machine Learning (TinyML).

TinyML was pioneered by Google during the development of optimized speech recognition algorithms for deployment on mobile devices. Numerous research groups have recently begun adapting neural network models to the TinyML approach for various tasks in industrial sectors [23,24,25]. Research shows a detailed application of the TinyML approach with CNN optimization on an energy-efficient platform for the task of recognizing crops and weeds on agricultural land. Applied research in this specific area is scarce. The study most relevant to our work is that of Dennis Agyemanh Nana Gookyi et al. [26], who investigated the application of TinyML for corn leaf disease detection. The results suggest that this novel approach to plant classification could also enable the development of cost-effective, durable IoT devices for field monitoring, reducing both energy consumption and maintenance costs.

In addition, technologies based on the Internet of Things have great potential in agriculture [2,27]. It has been shown that Internet of Things technologies, in combination with the modern computing capabilities of microcontrollers, form a stable trend towards increasing the efficiency of data processing in environmental monitoring tasks based on devices with limited resources [28]. Therefore, this study presents an approach to creating a hardware and software complex for local energy-efficient classification of images in the agricultural sector with support for Internet of Things protocols. The results of classification of objects in agricultural fields are shown when working in scenarios with poor energy constraints and a low-quality dataset. The goal of this work is to analyze the benefits of energy-efficient deployment of machine learning in agricultural object classification tasks for a resource-constrained scenario.

This research uses the ESP32-CAM microcontroller for the hardware, with the resolution of images used to train the neural network ranging from 32 × 32 to 800 × 600 pixels. The work was carried out in several main stages. The first stage included preparing images of crop areas and images of plant crops and weeds. The second stage involved training the model to classify objects in the image. The third stage involves the transfer of model parameters for work in the microcontroller programming environment with subsequent quantization. Finally, the fourth stage involved testing and validating the resulting model on the microcontroller.

The main contributions of this work include the following:

1.

A lightweight hardware and software complex for local energy-efficient image classification with support for Internet of Things protocols is presented.

2.

An evaluation of the effectiveness of the TinyML model for the task of classifying agricultural objects based on low-resolution data ranging from 32 × 32 to 800 × 600 pixels is conducted.

3.

The dependences of the dynamics of neural network training for data of different quality in a system with limited resources are shown.

The remainder of this paper is organized as follows: Section 2 presents the design stages of the neural network architecture and algorithms of the software package for classifying agricultural crop objects. The structure of the hardware part of the package is described. Section 3 presents the results of training two neural network architectures for a dataset with different resolutions. In Section 4, a conclusion is given on the possibility of using TinyML technologies in the agro-industrial sector for classifying objects in crop fields.

2. Materials and Methods

2.1. Consideration the Terms of Reference

When working with microcontroller systems, programmers have to deal with continuous optimization, especially if the problem to be solved is complex and requires a wide range of user functions at a minimum unit cost. This should be understood when creating realistic technical requirements for the system being developed.

Thus, the developed system should provide

• The means to locally capture images of sufficient quality to be analyzed by a neural network.
• Non-blocking code to asynchronously acquire data from output neurons and broadcast the data to the user.
• A user interface displaying the captured image and its classification (using Wi-Fi or Bluetooth to communicate with the user).
• The ability to detect three classes: plant-free zone, weed, and wheat. In classification, the classes “plant-free zone” and “weed” will belong to an enlarged classification for the user-“attention zone.” Thus, support for neural computing with CNN architecture is required. The classification accuracy should not be less than 90%, considering the computational power of the microcontroller.
• A subroutine for local accumulation of the dataset, which will allow us to generate unique data for additional training of the applied neural model.

2.2. Definition of Hardware and Software Complex

The ESP32-CAM microcontroller platform was chosen for debugging of the software system. It consists of the ESP32 microcontroller itself, which has two physical 32-bit cores on Xtensa LX6 architecture. The board also includes a 4 MB PSRAM chip to relieve the controller’s memory when capturing high-quality images with resolutions higher than 800 × 600 pixels, which greatly improves the system’s performance and response. Support for an external MicroSD card is provided for data storage. The computational module was chosen after evaluating hardware platforms with superior performance to the ESP32 microcontroller. A review of the research literature confirmed the effectiveness of the ESP32-CAM for object recognition and classification tasks [29]. The low weight (less than 10 g) and low power consumption (which in peaks does not exceed 500 mA) of this hardware platform allows for integration of the system both on wheeled and tracked vehicles, as well as on aircraft (e.g., quadrocopters) for analyzing images from the air. Furthermore, market analysis also demonstrated its economic viability, as no alternative board provides higher performance for under USD 10. Thus, the ESP32-CAM platform was adopted for this applied study.

The basic camera for this platform is the OV2640, which enables JPEG images with a maximum resolution of 1600 × 1200 pixels to be captured with a capture angle of 66°. The camera provides the following output formats: YUV(422/420)/YCbCr422, 8-bit compressed data, RGB565/555, and 8-/10-bit Raw RGB data. This project uses the 8-bit RGB format. The image resolution and color reproduction quality are sufficient for image analysis. These modules are often used in conjunction with more powerful single-board computers, such as the Raspberry Pi [30], which enable future computational power scaling without any modifications to the optical hardware’s specifications or properties.

When integrating the system as an attachment for agricultural machinery, synchronization of multiple devices during system operation in the field can be implemented based on the ESP-MESH protocol over Wi-Fi. This ensures accurate periods of photo fixation and collection of information from multiple nodes.

This layout is used in tandem with the popular GPS/GLONASS module Beitian BN-220. This is necessary to add information on photo-fixation coordinates to the collected data arrays. Then, the information is entered into the mapping system, which allows for a basic analysis to be performed after data collection and for a heat map of the attention zones of the cultivated areas to be created. This system is not this paper’s object of research, but a continuation of the formed hardware–software complex of weediness analysis of sown areas with the use of artificial intelligence algorithms. This study is expected to allow users to conduct this work more efficiently and purposefully in future visits to the fields.

For the transition to the program complex, it is necessary to collect data arrays for each analyzed class of images. Data was collected using various types of field equipment ranging from a height of 0.8 to 2 m above the ground using an ESP32-CAM-based device, the schematic diagram of which is shown in Figure 1.

This layout was also used to debug image classification algorithms. This layout was powered either from a step-down converter with a power ranging from 24–12 V to 5 V in a moving vehicle or from a USB-TTL converter based on the CH340 chip.

As a result, several hundred unique photos were collected, divided by the classes described above. However, a preliminary review of the obtained data showed their low diversity and the need for refinement by increasing their base to find characteristic features. Therefore, we decided to take advantage of existing datasets used for deep learning in larger projects. As a result, the following datasets with information useful for solving our problem were partially considered and applied [31,32,33,34,35,36], and the total number of photos before augmentation was 12,272.

As TinyML algorithms improve, we plan to release competing microcontroller software to perform real-time (or close to it, so that the processing time is less than 2 s) weed classification for point field processing. This will change approaches to the engineering of equipment like WeedSeeker 2 [37] and, most importantly, dramatically reduce their power consumption and cost.

Engineering of the neural network architecture and algorithms of the program complex was carried out in several stages:

• Extraction and classification of data samples from external datasets for subsequent use in neural network training;
• Preprocessing and augmentation of data, including resizing to multiple resolutions ranging from 512 × 512 to 80 × 60 pixels;
• Construction of training, testing, and validation datasets;
• Visual inspection of compiled datasets to ensure correctness and consistency;
• Definition and implementation of the neural network architecture using TensorFlow, as well as model compilation and training;
• Analysis of training dynamics and evaluation of prediction accuracy;
• Selection of representative datasets, followed by model optimization and quantization for deployment efficiency;
• Assessment of the accuracy of the optimized model;
• Conversion of trained model parameters into a binary format suitable for use in embedded C++ environments;
• Development of microcontroller firmware supporting TensorFlow Lite [38] and implementation of a basic web interface for interaction;
• Integration of the trained neural network into the firmware codebase;
• Testing of image classification performance on a physical hardware prototype.

An example of a random dataset after generating class datasets with a resolution of 160 × 120 pixels is shown in Figure 2.

The resolution of the image fed to the neural network affects the number of features that will be found.

A representative sample was formed and then the model was quantized, and the transition to an eight-bit integer data type was carried out. This means that the input and output values are now scaled to values ranging from −128 to +127. After that, the already quantized model of the developed program was ported to the ESP32 microcontroller.

We also noticed that, with input data of low resolution (below 100 × 100 pixels), quantization of trained models leads to critical loss of classification accuracy. Thus, if a normal model gave 69–71% classification accuracy, the accuracy decreased to 50–54% after quantization.

When trained on datasets of a higher resolution, such as those ranging from 256 × 256 pixels up to 800 × 600 pixels, the classification accuracy improved but, when trying to run these models on a microcontroller, the required memory stack exceeded the existing free space. An example of memory allocation requests at different image resolutions is presented below (Figure 3). It should be noted that the microcontroller platform used has only 4 MB of memory, both for the main program and for the neural network computation stack.

It should be noted that the amount of memory needed for microcontroller systems is an important parameter to consider when selecting the architecture of a neural network, since any changes in the architecture will affect the required memory stack and the classification time (Figure 4). The following two convolutional neural network architectures (Figure 5 and Figure 6) were chosen for this research. Figure 3 shows the characteristics of the first architecture (Figure 5).

The first architecture was applied in one of the existing projects aiming to classify visually described spectrograms after windowed Fourier transform for sound classification based on an ESP32 microcontroller [39].

The second architecture was obtained by investigating the learning dynamics of different neural network architectures and their resulting accuracy.

Several versions of the microcontroller program were tested. The first version was based on the classical library for microcontrollers, “tflite-micro” [38], and the second version was based on “ArduTFLite” [40]. Since no differences were observed in the performance of tasks, the program based on “ArduTFLite” was determined to be the main one to increase the readability of the code. The structure of the program is based on previously conducted research on the classification of audio recording spectrograms [39,41].

3. Results and Discussion

Datasets with different training resolutions were investigated. At resolutions lower than 70 × 70 pixels, the features that can be used to separate photos visually disappear or become implicit, so we decided to train the models at higher resolutions (Figure 7).

The standard metrics of the TensorFlow package were used to describe the learning processes of the models. More precisely, the “categorical_accuracy” method was applied to evaluate accuracy:

$$Accuracy={\displaystyle \frac{1}{n}}\sum _{i-1}^n\left(y_i=={\widehat{y}}_i\right)$$

(1)

where $y_i$ is the true class label, ${\widehat{y}}_i$ is the predicted class label, and $n$ is the total number of samples.

To calculate losses, the “categorical_crossentropy” method was used:

$$Loss= -\sum _{i-1}^n\left(y_i·log\left({\widehat{y}}_i\right)\right)$$

(2)

The resulting metrics are shown in the figures below. “Training Accuracy” is a metric that determines how well a model predicts the correct class in a photo from a training dataset. For example, a score of 90% or higher means that the model has memorized the images from the training sample well. “Testing Accuracy” determines how accurately the model predicts the correct class based on a validation dataset that it has never considered in training. For example, a score of 10% means that when submitting a new image, the model does not recognize what it is. “Training Loss” determines the number of model errors on the training data, in numerical terms, through the loss function. “Testing Loss” reveals how many mistakes the model makes on the test data; a drop in this value means that the model is learning to understand the essence (key features of classes) and is not just memorizing images.

Figure 8 illustrates the training dynamics of the convolutional neural network (CNN) based on the architecture presented in Figure 5. This architecture demonstrated suboptimal performance, particularly at higher image resolutions, which led to the decision to adopt an alternative model, as shown in Figure 6. Training with the second architecture proved to be more efficient and yielded significantly better results, as reflected in the training graphs presented in Figure 9 and Figure 10.

The resulting classification accuracies for both CNN architectures—standard and quantized—at various input resolutions are summarized in

Table 1. Classification accuracy at different resolutions of input data.

Model	Quantized	Accuracy (320 × 240), %	Accuracy (160 × 120), %	Accuracy (80 × 60), %
The first CNN architecture	No	82.5	79.32	74.25
The first CNN architecture	Yes	76.6	71.43	60.98
The second CNN architecture	No	94.83	90.21	84.58
The second CNN architecture	Yes	96.15	92.86	87.50

. The quantized version of the second architecture (Figure 6) achieved the highest accuracy across all resolutions, including the lowest resolution of 80 × 60 pixels.

An estimate of the memory consumption for each CNN configuration when deployed on the ESP32 microcontroller is provided in

Table 2. Memory usage of the CNN models on ESP32.

Model	Memory (320 × 240), KB	Memory (160 × 120), KB	Memory (80 × 60), KB
The first CNN architecture	3584	1536	384
The second CNN architecture	5120	2304	562

.

Figure 11 shows an example of image classification performed using the second CNN architecture. The results are presented as pairs of ground truth and predicted classes, allowing for a direct comparison between the expected and actual outputs of the trained model.

The classification results of the finished system with an integrated neural network are shown in the figure below. The microcontroller initializes a Wi-Fi access point with a static IP address for quick connection to a local HTTP server with WebSocket support. When a user clicks on the “classification” button, the controller captures the image, and from 4 to 15 s, it generates a result with a distribution of classes from −128 to 127. This result is then stored in an external MicroSD card. The response is sent to the user via WebSocket in the form of an array with integer values of the “byte” type (Figure 12 and Figure 13).

The obtained results demonstrate the feasibility of classifying plant objects in the field without relying on high-performance computing equipment or prolonged data processing time, particularly during CNN training, which currently remains a limitation for real-time monitoring systems [42]. The accuracy of the results depends on the quality of training images for each data class; however, classification accuracy exceeding 87% can be achieved even for low-resolution images depending on the CNN architecture and its quantization algorithm (Table 1). These findings are inferior to solutions based on the NVIDIA Jetson Nano platform. In the work by Kaldarova M. et al. [15], the training accuracy of the weed recognition model reaches 0.98 with an inference time of 0.2 s. The model was trained using images with a resolution of 500 × 350 pixels, which is six times higher than the image resolution used in the present study. This factor also impacts recognition quality and performance hardware. Finally, the superior results reported in [15] were achieved with significantly higher power consumption (typically 5–15 W compared to 1 W for the ESP32) and substantially greater hardware cost (approximately USD 200 compared to less than USD 10 for the ESP32). The optimal solution depends on the specific requirements for power availability, scalability, and computational demand. Our work demonstrates that for plant object classification in agricultural fields, microcontroller-tier hardware is not only viable but optimal when energy efficiency and cost are the primary constraints.

Thus, even when implemented on low-power microcontrollers, the CNN enables plant object classification across crop fields while reducing the energy consumption of field monitoring hardware–software systems without compromising accuracy.

4. Conclusions

This article investigated the possibility of using TinyML technologies in the agro-industrial sector to classify objects in crop fields. We used a CNN in a low-power microcontroller, which showed good accuracy in determining classes of data with different quality.

As a result, the architecture used in practice was chosen (Figure 6) and the algorithm operated at a resolution of 80 × 60, which ensured an accuracy of 87% with a maximum classification time of up to 8 s. Data collection and classification are implemented locally on the micro-controller, which allows for mobile analysis of the surfaces of crop fields and the creation of data arrays on their condition. We can also plot a map showing the weediness and crop yields of fields. Implementing such a system can help to optimize the use of resources; for example, when treating fields with fertilizers and herbicides.

The results of this study can be used to create databases with a pre-classified set of photos of a crop area with a set of geographic coordinates on the field for the cultivation of grain crops. The system developed for classifying plant objects based on a microcontroller and the TinyML approach will be used to create maps of field weediness to determine areas resistant to herbicides.

Further research will focus on conducting a deeper analysis of the TinyML approach in classifying plant objects on fields. We will consider applying hybrid KAN-MIXER and KAN-BiLSTM architectures that surpass traditional CNNs in both data interpretability and feature extraction efficiency [43,44]. The adoption of these models will be driven by the next phase of our research, which focuses on classifying more complex datasets to develop integrated systems for agricultural field mapping. The result of the considered neural network deployed on a microcontroller will be compared with the most prominent network architectures deployed on the NVIDIA Jetson platform. We will also prioritize integrating the TinyML approach with Internet of Things systems to ensure the best solution to the problem of classifying objects in real time and to minimize computing resources.