Improvement of YOLO v8 Segmentation Algorithm and Its Study in the Identification of Hazards in Plateau Pika

Abstract

Rodent infestation has become one of the important factors in grassland degradation on the Qinghai–Tibet Plateau, one of the hindrances to ecological and environmental protection, and a threat to the balance and development of the ecosystem in the Sanjiangyuan region. Based on the need for the scientific planning for ecological protection, this paper designs a method for detecting rodent infestation in plateau scenarios. Firstly, data were collected and annotated, and a dataset of plateau rodent distribution in the Qinghai region was constructed. The collected data include videos captured through drone-based field surveys, which were processed using OpenCV and annotated with LabelMe. The dataset is categorized into four specific types: ungobbled rat holes, gobbled rat holes, rocks, and cow dung. This categorization allows the model to effectively differentiate between rodent-related features and other environmental elements, which is crucial for the segmentation task. Secondly, the latest segmentation algorithm provided by YOLO v8 is improved to design a segmentation algorithm that can accurately detect the distribution of rodent infestation in the plateau scene. The specific improvements are as follows: firstly, the Contextual Transformer module is introduced in YOLO v8 to improve the global modeling capability; secondly, the DRConv dynamic region-aware convolution is introduced in YOLO v8 to improve the convolutional representation capability; thirdly, the attention mechanism is incorporated in the backbone of YOLO v8 to enhance the feature extraction capability of the network capability. A comparison test with the original algorithm on the plateau rodent distribution dataset showed that the new algorithm improved the detection accuracy from 77.9% to 82.74% and MIoU from 67.65% to 72.69% on the plateau rodent distribution dataset. The accuracy of the evaluation of plateau rodent damage levels has been greatly improved.

1. Introduction

The Qinghai–Tibet Plateau is the birthplace of the Yangtze and Yellow Rivers and is known as the water tower of Asia. Protecting the ecological environment of the Qinghai–Tibet Plateau region (Zhao, Chen, and Fan 2020) is related to the water security of many Asian countries [1]. In recent years, grassland degradation (Jiyuan, Xinliang, and Quanqin 2008) and soil erosion have become serious in Qinghai Province [2], and the grassland diversity (Wen et al., 2013) is rapidly decreasing [3]. In the new era, in order to implement the concepts of green water, green mountains, golden mountains, and the system of mountain, water, forest, lake, lake, grass, and sand management, it is especially important to scientifically and effectively manage the problem of grassland degradation in the Sanjiangyuan area.

In grassland ecosystems, rodents occupy an extremely important position. A moderate number of rats and rabbits play a positive role in the biological chain, but an excess of rats and rabbits can cause damage to grasslands and thus lead to rodent infestations. The proliferation of grassland rodents is often accompanied by many hazards, such as the degradation of grasslands, with reduced livestock carrying capacity; the sanding of grasslands, with severe soil erosion; and the spread of diseases, which threatens the health of farmers and herders (Zhou Yang et al., 2021) [4]. The highland sage-grouse is widely distributed in the Qilian Mountains, and excessive feeding and digging activities threaten the grassland ecosystem in the region (HAN Ruiying et al., 2022) [5]. In addition to the large amount of forage and grazing animals competing for food, the damaging effects of the grassland are mainly due to the damage to the soil structure on which the grass plants grow and the reduction in the available area for grazing (LIU Wei et al., 2003) [6]. Therefore, there is a need for a more scientific program to determine the level of damage caused by plateau sage-grouse and to take timely measures for precise management.

Deep learning plays an essential role in the vast field of ecological resource monitoring. For instance, in classification tasks, Mohanty et al. utilized deep learning methods to classify images of 38 plant diseases, some of which may be related to rodent infestations, thereby establishing a potential connection for detecting rodent-related damage in plants (Mohanty, Hughes, and Salathe, 2016) [7]. In another study, Norouzzadeh et al. (2018) applied deep learning to automatically classify and count images captured by camera traps [8], effectively enabling animal population monitoring, which could be adapted for rodent monitoring in the future [9]. In terms of detection and localization, Pound et al. (2017) focused on detecting and localizing parts like wheat ears and leaf tips, showcasing deep learning’s potential in precise plant feature recognition [10]. This approach has applications in identifying rodent-damaged regions within vegetation, such as recognizing plant features impacted by rodent activity. Additionally, Weinstein et al. (2019) used RetinaNet (Lin et al., 2017) to achieve canopy detection, attaining significant improvements in speed and accuracy, which may prove helpful in the large-scale monitoring of rodent disruptions [11,12]. Furthermore, deep learning has made notable advances in animal monitoring (Kellenberger, Marcos, and Tuia, 2018), cloud detection (Shi et al., 2016), and plant disease detection Mohanty, Hughes, and Salathe, 2016), supporting the efficient processing and accurate identification of ecological data [7,13,14]. In semantic segmentation, Douarre et al. (2019) achieved segmentation of soil and plant roots in images, providing an effective approach for analyzing plant–soil interactions [15]. Although instance segmentation has been relatively underexplored, Romera-Paredes and Torr (2016) successfully achieved segmentation and counting of each leaf of Arabidopsis thaliana based on Fully Convolutional Networks (FCN) and Long Short-Term Memory (LSTM) methods [16]. This approach could be applied to identify rodent-related features, such as burrows and damaged vegetation. The YOLO family, a very mature framework in target detection, has incorporated instance segmentation in the version of YOLO v5, and in the latest YOLO v8 algorithm, the performance of instance segmentation has become outstanding. The authors’ team has been working on the ecology of grasslands in the Qinghai region and has used deep learning techniques to study wolfsbane flowers. Most of the studies on plateau rodent infestation (Wu et al., 2019; Qi et al., 2022) are based on the Maxent model in machine learning, which cannot determine the current erosion status of grasslands through images [17,18]. Therefore, this paper aims to achieve plateau rodent hazard recognition through the YOLO v8 algorithm, combined with the rodent damage situation in the Qinghai region.

The population density of plateau sage-grouse is closely related to the number of holes (Yan et al., 2019) [19]. Using deep learning techniques (Christin, Hervet, and Lecomte 2019), for the plateau grassland scenario in the Qinghai region, the proportion of effective sage-grouse holes was used to infer the degree of sage-grassland encroachment by sage-grouse and to give the corresponding hazard level, to provide a reference for grassland degradation management in Qinghai region (Yang Xingkang 2011) [20,21]. In this context, the authors collected and established a dataset of the distribution of sage-grouse holes in the plateau grassland through fieldwork and designed an improved sage-grouse hole segmentation algorithm based on the YOLO v8 segmentation algorithm. In this paper, firstly, the Contextual Transformer module is introduced to improve the global modeling capability; secondly, the DRConv dynamic region-aware convolution is introduced to improve the representation capability of convolution; finally, the attention mechanism was fused in the backbone to enhance the feature extraction capability of the network. After experimental comparison, the effectiveness of the improved algorithm was thus verified and applied to practical scenarios.

2. Improvements to the YOLO v8 Segmentation Algorithm

The YOLO family is a very popular set of algorithmic frameworks in the field of target detection and has been widely used in various fields such as traffic (Lin and Sun 2018; Dong et al., 2021) [22,23], medicine (Sindhu et al., 2018; Huang et al., 2020; Ronneberger, Fischer, and Brox 2015) [24,25,26], and water monitoring (Wu Xinghui et al.) [27]. In recent years, many researchers have tried to apply the YOLO algorithm to segmentation (Diaz-Zapata, Erkent, and Laugier 2021; Mohamed et al., 2021) [28,29], and the YOLO v5 team integrated the segmentation module into the official project when they released version 7.0 and stated that YOLO v5 segmentation was fast and good. YOLO v8 is one of the latest versions of the YOLO family, developed by the v5 team, and it includes the image segmentation module. Therefore, this paper chooses to improve on YOLO v8 and apply the latest technology to the environmental protection of the Sanjiangyuan region.

Recent modifications to YOLO have significantly enhanced its performance across various application scenarios. A notable example is YOLO-Facev2 (Yu et al., 2024), which introduces the Receptive Field Enhancement (RFE) module and the Separation and Enhancement Attention Module (SEAM) [30]. These innovations address challenges such as scale variation and occlusion, improving face detection accuracy in complex environments. This approach could inspire similar improvements for rodent detection in ecological environments, especially in the presence of scale variation and occlusion. Additionally, YOLO-FD (Li et al., 2024) integrates a multi-task learning framework, combining object detection and semantic segmentation tasks, achieving excellent performance in fish disease detection, surpassing traditional models like YOLO v8 and Deeplabv3+ [31]. This method has significantly improved the performance in complex multi-task applications and incorporates techniques like PCGrad and weight uncertainty optimization, offering valuable insights for enhancing the multi-task framework for rodent detection and segmentation on the Qinghai Plateau. Furthermore, studies such as drone-based weed detection and Sunflower-YOLO illustrate how YOLO architectures can be further optimized for specific applications. For instance, Rehman et al. (2024) improved real-time weed detection accuracy through enriched feature extraction layers, such as Ghost Convolutions and BottleNeckCSP [32]. This innovation inspired us to integrate similar techniques into our YOLO v8 modifications to enhance the model’s ability to detect small features, such as rodent burrows or plant damage, in large-scale ecological environments. Similarly, Sunflower-YOLO Jing et al., 2024) addresses challenges posed by different plant growth stages and environmental conditions through the introduction of deformable convolutions and the SimAM attention mechanism [33]. While this method was not directly applied in our study, it provided valuable inspiration for optimizing our approach to detecting rodent-induced damage in various vegetation types across the Qinghai Plateau. These recent studies demonstrate how YOLO versions have been tailored to specific fields to achieve substantial performance improvements. For example, YOLO v8 itself includes several upgrades, such as replacing the C3 module with the gradient-rich C2f module and shifting from anchor-based detection to anchor-free detection. These changes collectively enhance the overall performance of the model.

The YOLO series has used mosaic data augmentation since v4 (Bochkovskiy, Wang, and Liao 2020) to improve data diversity [34]. v8 also uses this data augmentation series and turns off mosaic data augmentation in the last 10 epochs, thus effectively improving the accuracy. The backbone and neck section of YOLO v8 replaces the C3 Module in v5 with the more gradient-rich C2f Module and designs different parameters for different scales of the model to significantly improve the performance of the model. The classification and detection heads are separated, and the anchor-based idea is abandoned and replaced with the anchor-free idea.

These recent advancements highlight the importance of adjusting YOLO architectures to meet specific application needs. By drawing on these modifications, this study leverages the latest improvements in YOLO v8 and further customizes the architecture for detecting rodent damage in grasslands. The improvements in YOLO v8—such as the integration of dynamic region-aware convolution and Contextual Transformer modules—are aimed at enhancing feature extraction and modeling capabilities, making the model more adaptable and effective for ecological protection tasks in the Sanjiangyuan region.

In this paper, a series of improvements are made to the original model, and the structure of the improved model is shown in Figure 1. Based on the original model, all the convolution modules are replaced with dynamic region-aware convolution modules. Such a change improves the representation capability of the convolution while maintaining the computational cost and the translation invariance of the standard convolution. The Bottleneck in the c2f module is then replaced with the CoTBottleneck, i.e., the Contextual Transformer module is introduced into the backbone network to make full use of the contextual information and improve the global modeling capability. Finally, the Convolutional Block Attention Mechanism module (CBAM) is added to the backbone network of YOLO v8 to enhance the feature extraction capability of the backbone network.

2.1. Introduction of the Contextual Transformer Module

In recent years, transformers have been used in numerous computer vision (Dosovitskiy et al., 2020) fields with good results [35]. However, most designs directly use self-attention on the feature map to obtain the attention matrix of isolated queries and keys at each spatial location, without making full use of the contextual information between adjacent keys. Therefore, a new Transformer-based Contextual Transformer (CoT) block has been designed by some researchers (Lin et al., 2017) [12]. This design makes full use of contextual information to guide the learning of dynamic attention matrices, thereby improving the visual representation.

In the traditional self-attentive mechanism, the learning between query and key is independent of each other, but the adjacent contextual information is ignored, thus making the 2D feature graph under-expressed. Therefore, researchers (Li et al., 2021) designed a CoT module, as shown in Figure 2, to efficiently facilitate self-attentive learning and enhance feature graph representation [36]. As shown in Figure 2, the input 2D feature graph $X\in {\mathbb{R}}^{H\times W\times C}$, and keys, queries, and values are denoted as K, Q, and V, respectively, where $K=X$, $Q=K$, and $V=X{W}_v$. The CoT block first extracts contextual information using a k × k group convolution to Obtain $K^1$, a static contextual representation that responds to the contextual information between adjacent key values. Next, $K^1$ is concatenated with Q. The result after concatenation is obtained by two successive 1 × 1 convolutions to obtain the attention matrix based on query and keys learning:

$$A=\left[\begin{array}{c}{K}^1,Q\end{array}\right]W_{\theta }{W}_{\delta }.$$

(1)

In this way the learning capacity of the self-attentive mechanism is enhanced, resulting in the feature mapping:

$$K^2=V⊛A.$$

(2)

That is the dynamic context representation. Finally, the output Y of the CoT module is obtained by fusing the static context representation and the dynamic context representation through the attention mechanism.

2.2. Introduction of DRConv Dynamic Region-Aware Convolution

DRConv (Chen et al., 2021) can automatically assign multiple filters to the corresponding spatial domains with similar feature representations; so, DRConv outperforms standard convolution in modeling semantic variation [37]. The standard convolution can extract more visual features by adding filters to achieve better results but at a higher computational cost. DRConv improves the representation of the convolution by adding channel filters while maintaining similar computational cost and translation invariance as the standard convolution. Moreover, the plug-and-play feature of DRConv can directly replace the standard convolution in existing networks, thus achieving a performance gain.

The DRConv dynamic region-aware convolution model is shown in Figure 3, and it is divided into two main steps, i.e., using a learnable guided mask module to divide the features into regions in the spatial dimension and then customizing the filters in each shared region to perform the convolution operation. In Figure 3, the number of regions is m. First, the guided feature is obtained from X using the standard k × k convolution, and then m filters are obtained using the filter generator module. The region of the guided mask is divided into m regions, each of which has a corresponding filter. Finally, a k × k convolution is performed in these regions using the respective filters, and the final output Y is obtained.

2.3. Inclusion of CBAM Attention Mechanism Module

Due to the influence of various flora and fauna in the grassland scene, the recognition of rat holes is affected by various complex backgrounds, and because the unchewed rat holes and cow dung are similar in appearance and color, the two instances are often confused during segmentation; so, the CBAM attention mechanism (Woo et al., 2018) is added to the backbone network of YOLO v8 again to improve the feature extraction ability of the segmentation algorithm and reduce the influence of irrelevant elements on the accuracy of the model, thus improving the accuracy of the segmentation algorithm [38].

The structure of the CBAM attention module is shown in Figure 4. The channel attention mechanism compresses the feature map in the spatial dimension, and the compression uses average pooling and maximum pooling to aggregate the spatial information of the feature map to produce the channel attention map. In this process, the average pooling has feedback on all pixel points on the feature map, while the maximum pooling only has feedback on the gradients in the feature map where the response is greatest. The spatial attention mechanism, on the other hand, compresses the channels and uses average pooling and maximum pooling in the channel dimension, extracting the high times, the width submaximum, and the mean, respectively, on the channels and merging the extracted feature maps, both with a channel count of 1, into a single feature map with a channel count of 2. After two attention mechanisms, the results were weighted to obtain the results.

3. Introduction to the Dataset

Under the leadership of experts from the Qinghai Provincial Grassland General Station, the author’s team carried out fieldwork and filming by drones, screened the data, and finally obtained a set of video data with suitable angles, obvious features, and relative clarity; then, we used OpenCV to process the video, labelme to label the processed data, and integrated them to obtain the plateau sage-grouse hole dataset used in this paper.

The data distribution of the training and validation sets is shown in

Table 1. Distribution of the number of upland sage-grouse hole datasets.

Data	Train	Val
Total images	5840	648
All labels	41,211	4764
Mousehole0	16,533	1852
Mousehole1	15,282	1804
Stone	7479	902
Cow dung	1917	206

, where there are 5840 images in the training set and 648 images in the validation set. In addition, several sets of test images were prepared to test the effect of the model. The four categories from top to bottom are ungobbled rat holes, gobbled rat holes, rocks, and cow dung. Examples of the data are shown in Figure 5, from top to bottom, stone, ungobbled rat hole, cow dung, and gobbled rat hole. As can be seen from Table 1, the number of tags varies considerably between categories, with cow dung being less numerous; however, the shape of cow dung is extremely similar to that of a gnawed mouse hole, which also places a high demand on the improvement of the algorithm.

4. Analysis of Experimental Results

4.1. Experimental Conditions

This experiment was based on the PyTorch 1.13 framework to build a deep learning model, dividing the dataset into a training set and a testing set in the ratio of 9:1. The GPU was Nvidia RTX3060, the CUDA version was 11.6, and the CPU model was i7-12700F.

For training, the mosaic data enhancement strategy used in YOLO v8 was still retained, with the epoch set to 120 and the batch_size set to 32, and DFL_loss was used to allow the network to quickly focus on the distribution of locations that were closer to the target location. To effectively prevent overfitting, we employed the hyperparameter settings summarized in

Table 2. Hyperparameter settings.

Hyperparameter	Value
Initial learning rate (lr0)	0.01
Final OneCycleLR learning rate (lrf)	0.01
Momentum	0.937
Weight decay	0.0005
Warmup epochs	3.0
Image augmentations	HSV hue, saturation, and value enhancements

.

Experiments were conducted to evaluate the recognition ability of the model by the evaluation metrics mPrecision, mAP50, mRecall, and mIoU in the example segmentation.

4.2. Analysis of Experimental Results

By monitoring the loss and accuracy curves during training, we further validated the effectiveness of the hyperparameter settings in preventing overfitting.

To better understand the computational complexity and performance trade-offs of different YOLOv8 versions tested in this study, we summarize the key metrics (parameters, gradients, GFLOPs) for each model version in

Table 3. Model comparison for YOLOv8 variants.

Model Name	Parameters	Gradients	GFLOPs (FLOPs per Second)
YOLOv8n	2,975,439	2,975,423	11.9
COT	3,017,711	3,017,695	12.0
DRConv	2,968,531	2,968,515	46.1

. This comparison highlights the impact of additional modules on the model size and computational demand.

In Table 3, the standard lightweight version of YOLOv8n has fewer parameters, making it suitable for real-time applications. By adding the CoT module to YOLOv8n, the model’s ability to capture contextual information is enhanced, although this slightly increases the parameters and computational load. Additionally, using the DRConv module in two convolutional layers significantly improves the model’s performance while maintaining reasonable computational complexity. As observed from Table 3, adding the CoT and DRConv modules increases the computational complexity in terms of parameters and GFLOPs. Specifically, the DRConv module significantly raises the GFLOPs (from 11.9 to 46.1), reflecting the added computational cost. Despite this, both modules contribute to improved performance metrics, as shown in the results of

Table 4. Experimental data.

Algorithm	mPrecision	mAP50	mRecall	mIoU
YOLO v8	77.9%	76.55%	71.53%	67.65%
CoT	78.18%	78.51%	73.80%	68.26%
CBAM	79.52%	79.83%	75.11%	70.19%
DRCONV	79.35%	79.61%	74.98%	70.34%
ALL	82.74%	83.54%	81.39%	72.69%

and Figure 6, Figure 7 and Figure 8. This trade-off between computational cost and performance improvement is critical in real-world applications where model efficiency is as important as accuracy.

These findings highlight the importance of balancing the computational complexity with the performance gains. While the additional computational cost might seem significant, the improvements in model accuracy and feature recognition capabilities justify the trade-off, especially in the context of complex tasks such as rodent detection and environmental monitoring. The ability to enhance performance without severely compromising efficiency is essential for deploying models in large-scale real-time applications.

Figure 6. Training loss curve.

Figure 6. Training loss curve.

This figure illustrates the various losses on the training set, including the box loss, segmentation loss, classification loss, and DFL loss. These curves show that the model effectively learned the data features during training, with all losses significantly decreasing.

Figure 7. Validation loss curve.

Figure 7. Validation loss curve.

This figure presents the variations in various losses on the validation set, including the box loss, segmentation loss, classification loss, and DFL loss. As training progresses, the losses gradually decrease, demonstrating good convergence of the model.

This figure shows the changes in precision, recall, and mean average precision (mAP50 and mAP50-95) over time during training. As the training progresses, these metrics steadily improve, indicating that the model’s detection capability continues to be enhanced.

In this work, three improvements were made to the YOLO v8 model, the network was trained in the order of improvement, the best weights obtained from the training were tested, and the experimental results are shown in Table 4.

Figure 8. Precision and recall curves.

Figure 8. Precision and recall curves.

As can be seen from Table 4, the three improvement strategies proposed in this paper led to a certain degree of improvement in the YOLO v8 model on the plateau mouse rabbit hole dataset. Among them, the CBAM attention mechanism achieved the most obvious improvement, and according to the experimental data demonstration, the CBAM module mainly improved the recognition rate of gnawed mouse holes, and the boundary depiction of gnawed mouse holes in the validation set was closer to the actual value. The DRConv module also improves the model, as it improves the representation of the convolution so that the recognition accuracy of all categories is improved accordingly. Using the three improvement strategies simultaneously, the best performance was achieved for all metrics, with the average accuracy rate increasing from 77.9% to 82.74%, the mAP50 increasing from 76.55% to 83.54%, and the average recall rate increasing from 71.53% to 81.39%. The mIoU increased from 67.65% to 72.69%. The change in mIou for each of these categories is shown in

Table 5. Lifting effects by category.

Labels	Mousehole0	Mousehole1	Stone	Cow Dung
Original model	60.74%	59.44%	66.93%	83.5%
New Model	65.65%	68.38%	67.41%	89.32%

.

In Table 5, the segmentation accuracy of each category has been improved to some extent, among which the improvement in the gnawed mouse holes is the most obvious, but the improvement in the stone category is not satisfactory. This paper mainly focuses on mouse holes; so, we will not investigate the reason why the stone category has no obvious improvement here.

Figure 9 shows the comparison of the prediction effect of the algorithm before and after the improvement of Figure 5. From the figure, it can be seen that the accuracy of all categories of segmentation has been improved, and the small stones that were not originally identified in the upper left corner are identified in the improved model. It can therefore be seen that the improved algorithm has a higher detection rate and has a much better overall segmentation performance.

4.3. Real-World Testing

After completing the debugging of the algorithm model, the model was deployed to the server and traveled to the field to test the model’s effectiveness. Under the guidance of experts from the Qinghai Grassland General Station, the author’s team visited areas where plateau sage-grouse are active, photographed them with a drone, uploaded the results to the server, and tested the model. Comparing the two figures, there is not much difference in the number of sage-grouse holes present in the same height shot, but most of the holes in Figure 10 are ungnawed sage-grouse holes, whereas in Figure 10 there are more gnawed sage-grouse holes and a larger area of gnawed sage-grouse holes, accounting for more than 35% of the area, which is a severely eroded grassland. For the corresponding area in Figure 11, prevention and reasonable guidance are needed, while for the corresponding area in Figure 11, measures such as grassland restoration and rodent eradication are needed.

To evaluate the model’s generalizability and robustness, we also conducted cross-dataset validation, testing the model on multiple datasets to verify its applicability in various scenarios. These datasets include a private dataset containing three plant species: Thermopsis lanceolata, jjc, and zmc. Although the dataset is currently confidential, we have provided sample images in Figure 12, Figure 13 and Figure 14 to demonstrate the model’s detection performance across different plant types. By testing the model on these datasets with varying environmental conditions and object characteristics, we validated the model’s generalizability across diverse real-world scenarios. The results show that the model effectively handles different plant species and demonstrates strong robustness in diverse conditions, further enhancing our confidence in its broad applicability in practical use.

5. Conclusions

This paper focused on the degradation management of plateau grassland and developed a dataset specifically for plateau sage-grouse holes in the Qinghai region. Three different improvements were made to the YOLO v8 algorithm to enhance its effectiveness in practical tests. The improved model demonstrated superior performance compared to the original model and proved effective in detecting the degree of sage-grouse erosion in real-world scenarios. However, these improvements introduced additional parameters, which increased the model’s complexity and made it less lightweight.

In future research, the paper acknowledges the need for more lightweight improvement modules to strike a balance between model performance and real-time segmentation with low latency. The goal is to explore techniques that can further enhance the model’s performance while maintaining efficient and real-time segmentation capabilities. By incorporating lightweight improvements, the model can be optimized for practical applications in the field of plateau grassland degradation management.