Enhancing Agricultural Efficiency: Integrating YOLOv5 and SORT Tracker for Precise Weed and Cabbage Detection ^†

Abstract

This paper presents an innovative approach to agricultural management by integrating YOLOv5 for real-time detection and a modified SORT tracker for tracking two critical classes in agriculture: cabbage and weed. By deploying YOLOv5 on each video frame and updating the SORT tracker with current detections, we uniquely identify every cabbage and weed plant, allowing for precise monitoring and analysis. This method not only enables the collection of detailed data on the area covered by cabbage and weeds but also facilitates the calculation of free area, providing invaluable insights for optimizing crop management and reducing weed-related losses.

1. Introduction

The challenge of effective weed control in agricultural settings is critical for ensuring crop health and maximizing yield. Conventional weed management methods, such as chemical herbicides and manual labor, while effective, pose significant concerns related to environmental sustainability, economic viability, and human health risks. The need for innovative, sustainable, and efficient weed management strategies is apparent, prompting the exploration of advanced technological solutions.

Recent advancements in robotics and artificial intelligence have led to the development of mobile agricultural robots equipped with deep learning capabilities. These robots offer a promising alternative to traditional weed control methods. Equipped with high-resolution cameras and sophisticated sensors, mobile robots can navigate through fields, identifying and distinguishing between crops and weeds with high precision.

The core of this innovation lies in the application of Convolutional Neural Networks (CNNs), a class of deep learning algorithms renowned for their ability to process and classify visual imagery. By training CNNs on vast datasets of agricultural images, mobile robots can learn to recognize various plant species, enabling them to target weeds selectively without harming the surrounding crops [1,2].

This precision targeting not only reduces the dependency on chemical herbicides but also minimizes physical damage to crops, thereby supporting healthier crop growth and contributing to higher yields. Moreover, the automation of weed detection and removal processes significantly reduces the labor requirements and operational costs associated with manual weed control, making it an economically viable solution for large-scale agricultural operations [3].

The application of mobile robots for weed control also offers adaptability and scalability. These robots can be programmed to deal with various crop types and growth stages, as well as different weed species. Furthermore, their deployment can be scaled according to the size of the agricultural operation, from small farms to large agricultural fields, providing a flexible and efficient solution for weed management [4].

2. Related Work

Recent literature on weed detection showcases a variety of methods employing visual recognition algorithms, with a shift towards integrating deep learning models for enhanced accuracy. Among them, Convolutional Neural Networks (CNNs) have proven effective in recognizing complex patterns within agricultural environments, enabling precise weed and crop differentiation [5,6].

However, limitations exist within current robotic systems for weed control, particularly in terms of operational flexibility across different crop types and environmental conditions. These challenges underscore the need for more adaptable and robust systems capable of real-time, efficient weed detection and eradication [6,7].

In addressing these limitations, the integration of SORT tracker and YOLOv5 presents an advanced solution for object tracking and detection. SORT, with its simple online and real-time tracking capabilities, when coupled with the high-speed, accurate object detection of YOLOv5, significantly improves the performance of agricultural robots in identifying and tracking weeds among crops. This combination not only enhances the precision of weed control methods but also contributes to the development of autonomous agricultural machinery capable of sustainable weed management [6,7].

These advancements illustrate a promising direction for agricultural technology, emphasizing the importance of integrating sophisticated object detection and tracking systems for improved weed management strategies.

3. Methodology

In the prevailing methodologies, object detection algorithms are typically employed to process entire video recordings, subsequent to which an object tracking mechanism is iteratively applied to all identified detections, as illustrated in (Figure 1a) [8,9,10]. In our distinct approach, depicted in Figure 1b, it is imperative for the object tracking algorithm to be operational on each current frame, thereby facilitating immediate interventions on each detected weed. To this end, we deploy an object detection model configured with a batch size of one for every frame, concurrently updating our object tracking framework. Moreover, we aggregate all bounding boxes associated with plant detections. Upon the completion of the video analysis, the largest bounding box for each unique plant identifier is selected. This assembled data constitutes our statistical dataset, encompassing both plants and weeds, which serves as the basis for calculating the ratio of the area occupied by beneficial plants relative to the areas that remain unweeded and the extents of bare soil. This dataset holds potential for diverse applications, including the analysis of plant growth dynamics and other agronomical assessments.

Figure 1. (a) Traditional usage of object detectors and trackers for recorded videos. (b) Modification of the usage of an object detector and tracker for video streams.

4. Implementation

Essential to the construction of machine learning algorithms is the availability of high-quality data. Researchers often encounter issues such as insufficient data, data not available in the desired format, low-quality data, and data cluttered with irrelevant attributes.

Footage of cabbages and weeds is initially captured in video form. These videos are then processed into still images by extracting one frame out of every ten. Each image is meticulously annotated using the VoTT v1.7.0 (Visual Object Tagging Tool), an open-source software that assists in labeling images for object detection as is shown in Figure 2 [11].

Figure 2. Annotating cabbage and weed dataset.

In support of developing a mobile robot aimed at the automated removal of weeds, a targeted dataset featuring cabbages and weeds is assembled. The robot is designed to function within cabbage fields. The dataset is prepared by filming a sequence that captures both cabbages and weeds under a variety of environmental conditions, such as different growth stages of the cabbage, varying camera angles, and different lighting conditions.

5. Results and Discussion

5.1. Accuracy of Detection and Tracking Performance

Our experimental results demonstrate that the integration of YOLOv5 and the SORT tracker achieves real-time performance on a laptop with mid-range specifications. The YOLOv5 model, renowned for its speed and accuracy in object detection, successfully identifies cabbage and weed in each frame of the video. The SORT tracker, when used in conjunction with YOLOv5, ensures that even when the object detector misses some detections, the tracker smooths these gaps by maintaining the identity of objects across frames.

5.2. Detection Precision and Recall

The YOLOv5 model trained on our dataset, which includes annotated images of cabbages and weeds, achieves a high precision and recall. This implies that the model can effectively distinguish between cabbages and weeds, reducing the likelihood of false positives and negatives. The detection performance metrics indicate that our model can reliably identify the targeted classes under various conditions.

5.3. Tracking Robustness

The SORT tracker enhances the robustness of the detection system. By associating detected objects across successive frames, the tracker provides continuous identification of cabbages and weeds, ensuring consistent tracking even without transient occlusions or rapid movements. This continuity is crucial for accurate monitoring and analysis over time.

5.4. Data Analysis on Cabbage and Weed Areas

The data collected through this approach offers detailed insights into crop management. By analyzing the bounding boxes of cabbages and weeds, we can calculate the area covered by each category. This information allows us to quantify the ratio of weed-covered areas to cabbage-covered areas, providing valuable feedback for optimizing weed management strategies. The ability to identify and quantify free areas also aids in understanding the spatial distribution of crops and weeds, which is essential for planning interventions and maximizing crop yield.

5.5. Comparison with Traditional Methods

Compared to traditional weed control methods, our approach offers several advantages. Conventional methods, such as manual weeding and chemical herbicides, are labor-intensive, environmentally harmful, and often less precise. Automated methods using YOLO v5 and SORT provide a scalable and efficient alternative, reducing the need for manual labor and minimizing the use of chemicals. Furthermore, the real-time data collection and analysis capabilities of our system enable more informed decision-making and timely interventions, enhancing overall agricultural efficiency.

6. Conclusions and Future Work

6.1. Summary of Contributions and Benefits

This paper presents a novel approach to agricultural weed and crop management by integrating YOLO v5 for real-time detection and SORT for tracking. Our system effectively identifies and tracks cabbages and weeds, offering a precise and automated solution for monitoring and managing crop fields. The results indicate that this method is capable of real-time operation on mid-range laptops, demonstrating its practicality and efficiency. The ability to work with small datasets highlights the system’s robustness and adaptability to varying conditions.

6.2. Potential Improvements and Future Directions

Future work will focus on expanding the dataset to include images captured under different weather conditions and at various growth stages of the vegetables. This will enhance the model’s generalizability and performance across a wider range of scenarios. Additionally, we aim to test the system with vegetables and fruits that are ready for harvesting to further validate its effectiveness. Continued development will also involve integrating this detection and tracking system with robotic weed control mechanisms, enabling fully autonomous agricultural robots capable of not only identifying but also removing weeds.

6.3. Broader Implications for Precision Agriculture

The integration of advanced object detection and tracking algorithms in agricultural robots represents a significant step towards precision agriculture. By providing real-time, accurate, and detailed data on crop and weed distribution, our system supports more sustainable and efficient farming practices. This technology has the potential to reduce dependency on chemical herbicides, lower labor costs, and increase crop yields, contributing to more sustainable and profitable agricultural operations. The broader adoption of such technologies could revolutionize agricultural practices, promoting healthier crops and more environmentally friendly farming methods.