Gully Erosion Monitoring Based on Semi-Supervised Semantic Segmentation with Boundary-Guided Pseudo-Label Generation Strategy and Adaptive Loss Function

Abstract

Gully erosion is a major threat to ecosystems, potentially leading to desertification, land degradation, and crop loss. Developing viable gully erosion prevention and remediation strategies requires regular monitoring of the gullies. Nevertheless, it is highly challenging to automatically access the monitoring results of the gullies from the latest monitoring data by training historical data acquired by different sensors at different times. To this end, this paper presents a novel semi-supervised semantic segmentation with boundary-guided pseudo-label generation strategy and adaptive loss function method. This method takes full advantage of the historical data with labels and the latest monitoring data without labels to obtain the latest monitoring results of the gullies. The boundary-guided pseudo-label generation strategy (BPGS), guided by the inherent boundary maps of real geographic objects, fuses multiple evidence data to generate reliable pseudo-labels. Additionally, we propose an adaptive loss function based on centroid similarity (CSIM) to further alleviate the impact of pseudo-label noise. To verify the proposed method, two datasets for gully erosion monitoring are constructed according to the satellite data acquired in northeastern China. Extensive experiments demonstrate that the proposed method is more appropriate for automatic gully erosion monitoring than four state-of-the-art methods, including supervised methods and semi-supervised methods.

1. Introduction

As one of the forms of soil degradation, gully erosion not only incises the land surface and nibbles the field but also washes out the fertile soil and degrades the efficiency of the large cultivator [1,2,3]. Regular investigation of gullies is conducive to understanding their evolution regularity and formulating appropriate restoration strategies [4,5,6]. At present, gullies have been investigated in many countries around the world [7,8,9,10]. Nevertheless, these investigations mainly rely on field surveys or manual interpretation of aerial/satellite images, which requires enormous human, material, and financial resources. In the past few years, artificial intelligence (AI) in the form of deep learning (DL) could automatically extract representative abstract features, thus obtaining remarkable achievements in many fields [11,12]. Therefore, there is an urgent need to design a DL-based monitoring method that allows automatic access to the latest monitoring results of the gullies relying on the existing historical data.

As shown in Figure 1, the automatic monitoring process of gully erosion is illustrated by taking northeastern China as an example. In this figure, D^L is the historical data with labels, in which the labels come from the previous manual investigation process, whereas D^U is the latest monitoring data without labels and needs to be automatically obtained the monitoring results. Moreover, D^L and D^U were acquired over the same area at different times. Nevertheless, due to the different acquisition times, it is also difficult to ensure the consistency of the sensors collecting the two types of data. As a result, the same geographical object from D^L and D^U could have significant feature differences. Under the above conditions, the automatic monitoring method aims to accurately attain the latest monitoring results (the monitoring results of D^U).

In the field of DL, supervised semantic segmentation methods based on convolutional neural networks (CNNs) have attracted extensive research and application [13,14,15]. For instance, Wang et al. [16] proposed a segmentation method based on object context and boundary enhanced loss for earthquake-damaged buildings, aimed at enhancing the feature representation ability and refining the segmented boundaries of earthquake-damaged buildings; Yu et al. [17] researched the impact of the attention mechanism on the segmentation model and designed an attention-gates Unetwork (AGs-Unet) for buildings segmentation; Zhu et al. [18] introduced a multiscale-aware and segmentation-prior conditional random fields (MSCRF) to solve the problem of excessive smoothing of building edges in segmentation results. In fact, for the latest monitoring data (test data), it is difficult to achieve the desired monitoring results relying only on the historical data (training data) acquired by different sensors at different times.

To address this issue, we shift our focus to semi-supervised semantic segmentation methods. Compared with supervised methods, semi-supervised methods can provide pseudo-labels for the latest monitoring data to reduce the impact of feature differences [19,20]. Nevertheless, one inherent problem exists, in that the pseudo-labels inevitably contain noise. Therefore, some methods filter the predictions with confidence [21,22,23,24]. In other words, only highly confident predictions are used as pseudo-labels, whereas ambiguous ones are discarded. Obviously, these methods ignore numerous correct predictions with low confidence. Thus, many improved methods have been proposed. For example, Zhang et al. [25] exploited virtual adversarial perturbation and density-aware entropy to find valuable predictions with low confidence as the candidate samples; Wang et al. [26] argued that every pixel matters to the model training and proposed a category-wise queue that consists of negative samples to filter unreliable pixels; Yao et al. [27] measured the variance of pseudo-labels and regularized the network to learn with more confident pseudo-labels; He et al. [28] presented an effective distribution alignment and random sampling method to create unbiased pseudo-labels that match the true class distribution estimated from the labeled data.

Although many achievements have been made in semi-supervised methods, several limitations still remain for high-precision gully erosion monitoring. (1) Due to the enormous differences between historical data and the latest monitoring data, it is a challenge to provide credible pseudo-labels for the latest monitoring data. Thus, how to mine more valuable information to improve the reliability of pseudo-labels is a limitation that needs to be addressed. (2) The pseudo-labels inevitably contain noise, and even state-of-the-art methods cannot solve this problem. Therefore, it is necessary to further reduce the impact of pseudo-label noise. In response to these limitations, a novel semi-supervised semantic segmentation with boundary-guided pseudo-label generation strategy and adaptive loss function method is proposed. The main contributions of this paper are summarized as follows:

(1)

To improve the reliability of pseudo-labels, we propose a boundary-guided pseudo-label generation strategy (BPGS), which is composed of an object boundary generator and a multi-evidence fusion strategy. First, an object boundary generator based on CNNs and superpixel segmentation is proposed to output boundary maps of geographic objects, aiming to exploit the structural prior information and neighborhood correlation of pixels. On this basis, guided by these boundary maps, a multi-evidence fusion strategy is designed to fully utilize historical labels as well as the output of the student model and the teacher model to generate high-quality pseudo-labels.

(2)

To alleviate the impact of pseudo-label noise, an adaptive loss function based on centroid similarity is developed. In this loss function, centroid similarity (CSIM) is designed to measure the reliability of pseudo-labels and adjust the loss value. In this manner, the loss value is weighted lower when the reliability is lower. Consequently, we can endure pseudo-label noise through this loss function during the training process.

(3)

To validate our method, two benchmark datasets for gully erosion monitoring are constructed according to the satellite data acquired in northeastern China.

The rest of the paper is organized as follows: Section 2 introduces the proposed method in detail; Section 3 displays the datasets, experiments, and results; Section 4 presents a discussion on this paper; Section 5 draws the conclusions and offers some prospects.

2. Method

For gully erosion monitoring, we propose a semi-supervised semantic segmentation with boundary-guided pseudo-label generation strategy and adaptive loss function method, illustrated in Figure 2. Our method contains two models with the same backbone, including a student model (e.g., ${\mathsf{\Psi }}_A$ in Figure 2) and a teacher model (e.g., ${\mathsf{\Psi }}_B$ in Figure 2). The historical data with labels (D^L) are fed directly into ${\mathsf{\Psi }}_A$ for supervised training. Nevertheless, as for the latest monitoring data (D^U), the training process consists of four steps. First, we employ the designed object boundary generator to obtain the boundary maps of D^U. Next, by using ${\mathsf{\Psi }}_A$ and ${\mathsf{\Psi }}_B$ to predict D^U, we can obtain the predictions and confidences of two models for D^U. Then, guided by the boundary maps, a multi-evidence fusion strategy combines historical labels and the output of two models to generate reliable pseudo-labels. Finally, the proposed centroid similarity automatically adjusts the loss value by measuring the reliability of pseudo-labels during the training process. Based on the above operations, the training of ${\mathsf{\Psi }}_A$ is completed. Furthermore, the proposed method is converted to an iterative training framework for better performance, and a detailed introduction can be found in Section 2.3.

2.1. Boundary-Guided Pseudo-Label Generation Strategy

This section describes the process of generating high-quality pseudo-labels through the proposed BPGS. The BPGS consists of an object boundary generator and a multi-evidence fusion strategy, corresponding to serial numbers 1–3 in Figure 2. Then, we will describe the object boundary generator and the multi-evidence fusion strategy in detail.

2.1.1. Object Boundary Generator

Most semi-supervised methods ignore structural prior information and pixel neighborhood information when generating pseudo-labels, thus often producing noisy predictions around the object boundaries. To address this, we propose the object boundary generator to extract the boundary maps of geographic objects, thus providing guidance for the subsequent pseudo-label generation.

An object instance often includes similar colors or texture features [29,30], so grouping spatially continuous pixels that have similar features into the same object is a reasonable strategy. Based on this principle, we first designed a dual attention network based on CNNs to extract pixels with similar features, in which two attention modules enhance the ability to focus on the gullies. On this basis, a superpixel segmentation method, the SLIC method [31,32], is applied to further identify spatially continuous pixels. Based on the above introduction, the specific flowchart of the object boundary generator is shown in Figure 3.

As shown in Figure 3, the input is an image of gully erosion, and the output is the boundary map of this image. We first input this image into the dual attention network. In this way, pixels with similar features are aggregated into an object. Then, the SLIC method is employed to refine the output of the dual attention network by mining the spatial relationships between pixels. On this basis, the refined results are used as labels for updating the parameters of the dual attention network. Repeat the above operations until the set termination condition is reached. The termination condition can be found in Section 3.2.2. Finally, we extract the boundary pixels in the final refined result as the boundary map. Based on the above steps, we can extract the boundary map of the input image. To illustrate the object boundary generator in more detail, we will introduce the dual attention network and the SLIC refinement.

(1)

Dual attention network: As shown in Figure 3, the dual attention network consists of three parts, taking an image as the input and the predictions of this image as the output. The first part is features encoding, consisting of three convolution blocks. Each block includes a convolution layer, a batch normalization layer, and a ReLU activation function layer. The second part is dual attention enhancement, which is beneficial for focusing on the information of interest and suppressing irrelevant background information. In this part, we adopt a channel attention module and a spatial attention module, as shown in Figure 4. As for the third part, a convolution layer, a batch normalization layer, and an argmax method are used to decode the features. With the above three parts, pixels with similar features can be aggregated into an object.

(2)

SLIC refinement: The dual attention network aims to aggregate pixels with similar features into one object. However, it is also preferable for objects to be spatially continuous [29]. Meanwhile, the SLIC method considers the spatial correlation of pixels in images when generating irregular pixel regions (superpixels). Therefore, we adopt the SLIC method to refine the output of the dual attention network. Specifically, based on the SLIC method, we first extract the superpixel set $S={\left\{S_o\right\}}_{o=1}^O$ from the input image, where O is the total number of superpixels, and $S_o$ denotes a set of the index of pixels belonging to the oth superpixel. Then, on the basis of the predictions obtained from the dual attention network and $S_o$, we can obtain $L_o$ (the predictions for pixels belonging to $S_o$). Next, we force the most frequent category in $L_o$ as the category for all pixels of $S_o$. Finally, iterate the above steps until all superpixels are updated. Based on the above operations, continuous pixels with similar features are aggregated into the same object.

2.1.2. Multi-Evidence Fusion Strategy

Although the object boundary generator is able to exploit the structural prior information and neighborhood correlations of pixels, more useful information should also be mined to improve the reliability of pseudo-labels. Compared with other disasters, such as earthquakes and floods, most gullies will not change significantly in the short term. As shown in Figure 1, although D^L and D^U were acquired over the same area at different times, the gullies in them had not altered dramatically. This means that historical labels contain a wealth of valuable information for generating pseudo-labels. Meanwhile, some scholars argue that different model initialization could help models describe the same data from different aspects or perspectives and significantly improve performance [33]. Inspired by this view, ${\mathsf{\Psi }}_A$ and ${\mathsf{\Psi }}_B$ were pre-trained on historical data from two different initializations, thus providing different supports for pseudo-labels. In summary, we take the historical labels and the output of two models with different initializations as evidence data to generate pseudo-labels. However, with the introduction of more evidence, this also brings evidence conflict. In other words, as a basis for determining whether a pixel belongs to gully erosion, three evidence data may give opposite conclusions, and the experimental results in Section 4.1 verify that it is difficult to achieve reliable results for the simple practice of taking the intersections of all evidence data. Therefore, a multi-evidence fusion strategy is designed to combine three evidence data in a holistic way.

The multi-evidence fusion strategy is based on the Dempster/Shafer (D–S) evidence theory [34,35,36] and generates reliable labels under the guidance of boundary maps. Specifically, for an input image, we first denote its boundary map as $R={\left\{R_i\right\}}_{i=1}^I$. Here, I is the total number of objects in the boundary map, and $R_i$ is the ith object. According to the D–S evidence theory, the decision fusion framework can be defined as $\Theta :${GE, NG} for any object $R_i$, where GE and NG denote gully erosion and non-gully erosion, respectively. Thus, the non-empty subset $\psi$ of $\Theta$ should contain {GE}, {NG}, and {GE, NG}.

Then, we define the basic probability assignment formula (BPAF) of $\psi$ as $m(\psi )$, which can be represented as follows:

$$m(\psi )={\displaystyle \sum _{\cap {\psi }^{\prime }=\psi }{\displaystyle \prod _{1\le g\le G}{m}_g}}({\psi }^{\prime })/{\displaystyle \sum _{\cap {\psi }^{\prime }\ne \varnothing }{\displaystyle \prod _{1\le g\le G}{m}_g}}({\psi }^{\prime })$$

(1)

where G is the total number of evidence data, $m_g(\cdot )$ is the BPAF of gth evidence. On this basis, the corresponding BPAF of $R_i$ is established through the following equations:

$$m_{ig}[\{GE\}]=C_{ig}\times P_{ig}$$

(2)

$$m_{ig}[\{NG\}]=(1-C_{ig})\times (1-P_{ig})$$

(3)

$$m_{ig}[\{GE,NG\}]=(C_{ig}+P_{ig})-2C_{ig}{P}_{ig}$$

(4)

where $C_{ig}$ and $P_{ig}$ are the overall confidence indicators that $R_i$ belongs to GE in the gth evidence and can be represented as

$$C_{ig}=\frac{1}{N(R_i)}\times {\displaystyle \sum _{q=1}^{N(R_i)}{E}_{ig}^q}$$

(5)

$$P_{ig}=\frac{1}{N(R_i)}\times {\varphi }_{ig}$$

(6)

Here, $N(R_i)$ is the total number of pixels of $R_i$, $E_{ig}^q$ is the confidence that the qth pixel of $R_i$ belongs to GE in the projection of $R_i$ on the confidences of gth evidence, ${\varphi }_{ig}$ is the total number of pixels belonging to GE in the projection of $R_i$ on the predictions of gth evidence. In addition, it should be noted that we set the confidence to 1 for pixels belonging to GE and 0 for pixels belonging to NG when adopting history labels to calculate $E_{ig}^q$.

Finally, calculate the $m_i[\{GE\}]$, $m_i[\{NG\}]$, and $m_i[\{GE,NG\}]$ by using Equations (1)–(6). If $m_i[\{GE\}]$>$m_i[\{NG\}]$ and $m_i[\{GE\}]$>$m_i[\{GE,NG\}]$ are satisfied, $R_i$ belongs to gully erosion; otherwise, $R_i$ belongs to non-gully erosion. On this basis, iterate over all objects of $R$ to obtain the pseudo-label of the input image.

2.2. The Adaptive Loss Function Based on Centroid Similarity

Since it is unlikely that all evidence data will make mistakes at the same time, their similarity also reflects the reliability of pseudo-labels. Moreover, the centroid reflects the mass distribution of an object, so the closer the distance between the pixel and the centroid, the more reliable it will be as proof of monitoring [37,38]. Thus, the similarity should also be more focused on the representative pixels of each object. Based on the above analysis, the adaptive loss function based on CSIM is proposed. The adaptive loss $\zeta$ includes a supervised loss ${\zeta }_{sl}$ and a pseudo-supervised loss ${\zeta }_{pl}$ and can be represented as

$$\zeta ={\zeta }_{sl}+{\zeta }_{pl}$$

(7)

The supervised loss ${\zeta }_{sl}$ is formulated using the standard binary cross-entropy loss in D^L:

$${\zeta }_{sl}=\frac{1}{\left|D^L\right|}{\displaystyle \sum _{Z\in D^L}-\frac{1}{W\times H}{\displaystyle \sum _{r=1}^{W\times H}[{\upsilon }_r^Z\times \log (F_r^Z)+(1-{\upsilon }_r^Z)\times \log (1-F_r^Z)]}}$$

(8)

where Z is an image in D^L, W and H represent the width and height of Z, respectively. $F_r^Z\in [0,1]$ and ${\upsilon }_r^Z\in \left\{0,1\right\}$ represent the prediction and label for the rth pixel of Z, respectively. Meanwhile, it should be noted that if the pixel belongs to gullies, we set its label to 1. Otherwise, its label is 0. Similarly, the pseudo-supervised loss ${\zeta }_{pl}$ in D^U can be represented as

$${\zeta }_{pl}=\frac{1}{\left|D^U\right|}{\displaystyle \sum _{X\in D^U}-\frac{1}{W\times H}{\displaystyle \sum _{r=1}^{W\times H}[{\upsilon }_r^X\times \log (F_r^X)+(1-{\upsilon }_r^X)\times \log (1-F_r^X)]\times CSI{M}_r^X}}$$

(9)

where X is an image in D^U, $F_r^X\in [0,1]$ and ${\upsilon }_r^X\in \left\{0,1\right\}$ represent the prediction and pseudo-label for rth pixel of X, respectively. $CSI{M}_r^X$ is the CSIM for rth pixel of X, i.e., the reliability of pseudo-labels for rth pixel of X. The specific introduction of $CSI{M}_r^X$ is as follows.

For ease of introduction, we let the rth pixel of X be pixel j, and its projection on the boundary map belongs to $R_i$. Meanwhile, $R_{iX}$ and $R_{iT}$ represent the projection of $R_i$ on X and historical labels, respectively. $R_{iA}$ and $R_{iB}$ represent the projection of $R_i$ on the predictions of ${\mathsf{\Psi }}_A$ and ${\mathsf{\Psi }}_B$ for X, respectively. On this basis, the specific steps for calculating $CSI{M}_r^X$ are as follows.

Step 1: Define $(\epsilon ,\mu )$ as the centroid coordinates of $R_{iX}$, and the calculation of centroid coordinates can be defined as

$$\epsilon =\frac{{\displaystyle \sum _{q=1}^{N(R_{iX})}{b}_q{\epsilon }_q}}{{\displaystyle \sum _{q=1}^{N(R_{iX})}{b}_q}}$$

(10)

$$\mu =\frac{{\displaystyle \sum _{q=1}^{N(R_{iX})}{b}_q{\mu }_q}}{{\displaystyle \sum _{q=1}^{N(R_{iX})}{b}_q}}$$

(11)

where $N\left(R_{iX}\right)$ is the total number of pixels of $R_{iX}$, $b_q$ is the pixel value of the qth pixel in $R_{iX}$. ${\epsilon }_q$ and ${\mu }_q$ are the X-axis and Y-axis coordinates of the qth pixel, respectively.

Step 2: Compute the Euclidean distance of each pixel in $R_{iX}$ from the centroid coordinates, thus obtaining the centroid distance set ED.

Step 3: Calculation of the pixel value after the centroid distance constraint, and the specific formula can be defined as

$$R_{iA}^{*}\left(j\right)=R_{iA}\left(j\right)\times e^{\left(1-\left[ED(j)/\max (ED)\right]\right)}$$

(12)

where $e^{(·)}$ is the exponential function, $R_{iA}(j)$ is the value of pixel j in $R_{iA}$, $R_{iA}^{*}(j)$ is the constrained value, and $ED(j)$ is the distance from pixel j to the centroid. On this basis, it can be seen that the centroid distance constraint ($e^{\left(1-\left[ED(j)/\max (ED)\right]\right)}$) varies according to the distance to the centroid and reaches a maximum at the centroid. Therefore, the centroid distance constraint can increase the divergence around the centroid (more representative predictions) and correspondingly alleviate the effect of predictions far from the centroid.

Step 4: Repeating Step2, the constrained object $R_{iA}^{*}$ can be obtained by traversing each pixel of $R_{iA}$. Similarly, $R_{iB}^{*}$ and $R_{iT}^{*}$ can be obtained.

Step 5: Calculation of the reliability for the rth pixel, and the specific calculation is as follows:

$$CSI{M}_r^X=(1-\kappa )\times S{M}_{iAiB}^{*}+\kappa \times [(S{M}_{iAiT}^{*}+S{M}_{iBiT}^{*})/2]$$

(13)

where $S{M}_{iAiB}^{*}$ is the structural similarity (SSIM) of $R_{iA}^{*}$ and $R_{iB}^{*}$, and the specific calculation process of SSIM can be referred to [39,40]. Similarly, $S{M}_{iAiT}^{*}$ and $S{M}_{iBiT}^{*}$ can be calculated. Furthermore, $\kappa$ is the similarity regulation indicator. When $\kappa$ is small, the student model will focus more on the predictions of ${\mathsf{\Psi }}_A$ and ${\mathsf{\Psi }}_B$. Otherwise, it will pay more attention to historical labels.

2.3. Model Training and Testing

2.3.1. Iterative Training Framework

In this section, our method is converted to an iterative training framework for better performance. Among them, both ${\mathsf{\Psi }}_A$ and ${\mathsf{\Psi }}_B$ can be used as the student model first. The pseudo-code using ${\mathsf{\Psi }}_A$ first as the student model is shown in Algorithm 1.

Algorithm 1 Iterative Training Framework

Input: Historical data (labeled data) DL, the latest monitoring data (unlabeled data) DU. Output: Trained model ${\mathsf{\Psi }}_A^V$ and ${\mathsf{\Psi }}_B^V$. 1 Initialize ${\mathsf{\Psi }}_A^0$ and ${\mathsf{\Psi }}_B^0$ with different pre-trained weights. 2 Train ${\mathsf{\Psi }}_A^0$ on DL. 3 Train ${\mathsf{\Psi }}_B^0$ on DL. 4 for $n\in \left\{1,\cdot \cdot \cdot ,V\right\}$, do 5    Predict on DU with ${\mathsf{\Psi }}_A^{n-1}$. 6    Predict on DU with ${\mathsf{\Psi }}_B^{n-1}$. 7    Use BPGS to fuse historical labels and the output of ${\mathsf{\Psi }}_A^{n-1}$ and ${\mathsf{\Psi }}_B^{n-1}$$\to$ pseudo-labeled data DPL. 8    Fine tune ${\mathsf{\Psi }}_A^n$ from ${\mathsf{\Psi }}_A^{n-1}$ on both DL and latest DPL with the adaptive loss $\zeta$. 9    Predict on DU with ${\mathsf{\Psi }}_A^n$. 10  Use BPGS to fuse historical labels and the output of ${\mathsf{\Psi }}_A^n$ and ${\mathsf{\Psi }}_B^{n-1}$$\to$ pseudo-labeled data DPL. 11  Fine tune ${\mathsf{\Psi }}_B^n$ from ${\mathsf{\Psi }}_B^{n-1}$ on both DL and latest DPL with the adaptive loss $\zeta$. 12 end for

2.3.2. Testing Process

In the testing phase, we first employ ${\mathsf{\Psi }}_A^V$ and ${\mathsf{\Psi }}_B^V$ to predict the test images. Then, the fusion results of the two models can be attained using Equations (1)–(6). Furthermore, the test labels are pixel wise and contain boundary pixels, so the closed operation is used to process these fusion results. Based on the above steps, we can obtain the final monitoring results.

3. Experiments and Results

3.1. Datasets

3.1.1. Dataset Description

Due to the lack of publicly available datasets for gully erosion monitoring, we constructed two benchmark datasets (including HC2012 and HC2020) to verify the performance of the proposed method. The HC2012 dataset was carried out based on the remote-sensing images of Huachuan County, Heilongjiang Province, China, collected by the ZY-3 satellite in 2012. The images included the panchromatic and multispectral (blue, green, red, and near-infrared) bands, with spatial resolutions of 2.1 m and 5.8 m, respectively. By using ENVI software, the images were fused to pan-sharpened RGB images with a spatial resolution of 2.1 m, as shown in Figure 5a. The HC2020 dataset was based on the same-site (Huachuan County) remote-sensing images collected by the GF-2 satellite in 2020. The images also covered the panchromatic and multispectral bands, with spatial resolutions of 1 m and 4 m, respectively. Then, the images were fused to pan-sharpened RGB images with a spatial resolution of 1 m, as shown in Figure 5b. In addition, during the experiments, the pan-sharpened RGB images of two datasets were also rigorously registered to facilitate the implementation of the proposed method.

3.1.2. Label of Dataset

In view of the experimental hardware condition and keeping abundant spatial contexts of the gullies, as far as possible, we segmented the original pan-sharpened RGB images into sub-images with 384 × 384 pixels. As a result, the HC2012 dataset obtained 981 samples. In addition, since two datasets were registered, the samples with the same number and size were obtained for the HC2020 dataset. Based on this, we annotated these samples with the assistance of the water department. Figure 6 shows a sample from the HC2020 dataset and its corresponding label.

3.2. Experimental Design and Implementation Details

3.2.1. Experimental Design

To evaluate our proposed method, we designed two experiments. In Experiment 1 (HC2012 to HC2020), the HC2012 dataset was employed as the training set (historical data), while the HC2020 dataset was used as the test set (the latest monitoring data). In Experiment 2 (HC2020 to HC2012), we reversed Experiment 1. In other words, the HC2020 dataset and the HC2012 dataset were treated as the training set and the test set, respectively. Moreover, based on the practical monitoring needs, we also provided the images of the test set as unlabeled data to the semi-supervised methods, whereas their labels were only used in the test phase to calculate the evaluation metrics. By carrying out the above two experiments, it is beneficial to validate the effectiveness and extensiveness of our method.

In these two experiments, our method was compared with four state-of-the-art methods, including two supervised and two semi-supervised methods. The supervised methods include Bisenetv2 [41] and Segformer [42], and the semi-supervised ones include U²PL [26] and DMT [33]. All comparison methods were reproduced using the open code of the original authors. Due to different data sources and hardware conditions, we made some necessary modifications, such as image size, batch size, number of categories, etc. To ensure the fairness of the experiments, the same data augmentation, learning rate, iterations, and optimizer were set, whereas the other hyperparameters followed the settings recommended by the original authors. Moreover, we evaluated different methods with four evaluation indicators, including precision, recall, F1 score, and intersection over union (IoU). The specific calculation formulae of these indicators can be found in Refs [43,44,45,46]. Compared with precision and recall, the F1 score and IoU are two comprehensive indicators. Hence, we will pay more attention to the IoU and F1 score in the experiments.

3.2.2. Implementation Details

The proposed method was implemented based on the PyTorch-1.7.0 framework in the Ubuntu 18.04 environment. All experiments were performed on a workstation. Its model was the Dell Precision T7920, which consists mainly of an Intel Xeon Silver 4210R CPU with 40G RAM and an Nvidia GeForce RTX 3090 GPU with 24G RAM. In the experiments, ${\mathsf{\Psi }}_A^0$ and ${\mathsf{\Psi }}_B^0$ used DeepLab-v3 with Resnet101 as the backbone but separately used pre-trained weights from COCO [47] and ImageNet [48]. In the training phase, ${\mathsf{\Psi }}_A$ was first used as the student model, V was set to 5, each iteration had 20 epochs, and the SGD optimizer with the poly learning rate schedule was used to optimize the network. Some common data augmentation methods were adopted, including the horizontal flip of the image, random rotation of the image within the angle range of [–15,15], and random scaling was set at 0.75 and 1.5. The termination condition of the object boundary generator in Section 2.1.1 was 128 iterations. Furthermore, the similarity regulation indicator $\kappa$ in Equation (11) was set to 0.6, and its detailed introduction can be found in Section 4.2.

3.3. Experiment 1: HC2012 to HC2020

In Experiment 1, the HC2012 dataset was employed as the historical data (training set), while the HC2020 dataset was treated as the latest monitoring data (test set). Figure 7 displays the latest monitoring data and the corresponding labels. Among them, some representative lands are marked with orange, yellow, blue, and red rectangles. On this basis, the monitoring results of five different methods for representative lands are shown in Figure 8.

As shown in Figure 8, two supervised methods (Bisenetv2 and Segformer) manifest poor performance in all representative lands, which is consistent with the previous analysis of supervised methods. Compared with supervised methods, semi-supervised methods can provide pseudo-labels for the latest monitoring data to reduce the impact of feature differences. Thus, three semi-supervised methods (U²PL, DMT, and the proposed method) monitor most of the gully erosion pixels. However, there are plenty of false positive pixels in the results of U²PL and DMT. For example, in the field roads (the golden rectangles in the second and third rows of Figure 8), wasteland (the golden rectangle in the first row), and the edge regions of the gullies (the golden rectangle in the fourth row), U²PL and DMT all present large areas of false positive pixels. These results suggest that it is difficult to acquire reliable monitoring results through the teacher model alone. With the introduction of BPGS and the adaptive loss function, the proposed method can better handle the impact of feature differences on automatic monitoring, thus presenting the best visual results.

To further evaluate the performance of the five methods, precision, recall, F1 score, and IoU were selected as the evaluation indicators for gully erosion monitoring. On this basis, the three times average quantitative evaluation results of five different methods are shown in

Table 1. Quantitative evaluation results of Experiment 1. The entries in bold denote the best results in Experiment 1.

Methods	Precision	Recall	F1 score	IoU
Bisenetv2 [41]	50.9%	40.9%	45.4%	18.9%
Segformer [42]	52.7%	39.4%	45.1%	18.6%
U2PL [26]	43.2%	83.8%	57%	41.1%
DMT [33]	44.8%	85%	58.7%	42.4%
The proposed method	58.7%	60.4%	59.5%	64.5%

.

As shown in Table 1, the proposed method outperforms the other methods in terms of the IoU, precision, and F1 score. In particular, it raises the IOU by 22.1% compared with the second ranked method (DMT). As state-of-the-art supervised methods, Bisenetv2 and Segformer manifest poor performance in all evaluation indicators, which is consistent with the visual analysis. Moreover, the recalls of U²PL and DMT are higher than 83%, while their precisions are lower than 45%. This suggests that these two methods improve recall by sacrificing precision. Thus, they perform worse than the proposed method in terms of more balanced indicators (F1 score and IoU). Nevertheless, BPGS generates reliable pseudo-labels, and the adaptive loss function reduces the impact of pseudo-label noise, so the proposed method can acquire the best quantitative evaluation results. In summary, our method is more competent for complex gully erosion monitoring than other methods.

3.4. Experiment 2: HC2020 to HC2012

In Experiment 2, the HC2020 dataset and the HC2012 dataset were employed as the historical data and the latest monitoring data, respectively. Figure 9 displays the latest monitoring data and the corresponding labels. On this basis, the monitoring results of five different methods for representative lands are shown in Figure 10.

As shown in Figure 10, the supervised methods display massive false negative results, and the three semi-supervised ones monitor most of the gully erosion pixels, which is similar to Experiment 1. As in the golden rectangle in the first row of Figure 10, the three semi-supervised methods achieve satisfactory results, while the supervised ones still have a mass of false negative pixels. These results again justify the choice of semi-supervised methods for gully monitoring. When monitoring some field roads with similar characteristics to gullies (the golden rectangles in the second and third rows of Figure 10), U²PL and DMT show false positive results, while the proposed method is able to monitor correctly. Meanwhile, compared with Experiment 1, the edge regions of the gullies are more likely to show errors due to the reduced image resolution. For example, in the golden rectangle in the fourth row of Figure 10, three semi-supervised methods display different degrees of false positive and false negative results. Nevertheless, the proposed method still achieves the best visual results in this region. To further evaluate the performance of the methods, the three times average quantitative evaluation results of five different methods are shown in

Table 2. Quantitative evaluation results of Experiment 2. The entries in bold denote the best results in Experiment 2.

Methods	Precision	Recall	F1 score	IoU
Bisenetv2	58.8%	26.6%	36.6%	18.1%
Segformer	57.4%	25.9 %	35.7%	16.2 %
U2PL	41.9%	82.8%	55.6%	39.3%
DMT	44.5%	85.5%	58.5%	42.1%
The proposed method	63.9%	65.2%	64.5%	60.4%

.

As shown in Table 2, the proposed method outperforms the other methods in terms of the IoU, precision, and F1 score. Among them, it attains an IoU improved by 18.3% and an F1 score improved by 6% in comparison with the second ranked method (DMT). Due to the feature differences between historical data and the latest monitoring data, Bisenetv2 and Segformer once again present bad performance in all evaluation indicators. As for U²PL and DMT, these methods exhibit similar evaluation results as in Experiment 1, i.e., higher recall and lower precision. In addition, compared with Experiment 1, the IoUs of all methods show different degrees of degradation due to the reduced image resolution. Nevertheless, because BPGS can merge more valuable information to generate reliable pseudo-labels, and the adaptive loss can further reduce the impact of pseudo-label noise, the IoU of the proposed method is still higher than 60%. As a result, the proposed method can better address the impact of feature differences on automatic monitoring and possesses a higher superiority in complex gully erosion monitoring tasks.

4. Discussion

4.1. Ablation Study

In order to analyze the effectiveness of BPGS and adaptive loss function in the proposed method, an ablation study was conducted. First, we constructed a baseline method. Based on the iterative framework in Section 2.3, this baseline method used the intersection of evidence data as the pseudo-labels of the latest monitoring data. Additionally, this baseline method applied the standard binary cross-entropy loss function. On this basis, we take the IoU indicator as the evaluation criterion, and the three times average evaluation results of the ablation study are illustrated in

Table 3. The results of the ablation study. √ and — separately represent used and not used; the entries in bold denote the best results in the corresponding experiment; and Δ(IoU) is the IoU change compared with the baseline.

Experiment	Method	BPGS	Adaptive Loss	IoU	Δ(IoU)
Experiment 1	Baseline	–	–	44.7%	+0.0%
	Baseline	√	–	56.1%	+11.4%
	Baseline	√	√	64.5%	+19.8%
Experiment 2	Baseline	–	–	43.6%	+0.0%
	Baseline	√	–	53.3%	+9.7%
	Baseline	√	√	60.4%	+16.8%

.

As shown in Table 3, Baseline + BPGS improves the IoU by 11.4% and 9.7%, respectively, compared with Baseline. This demonstrates that the proposed BPGS can effectively improve the reliability of pseudo-labels by mining more useful information. Additionally, compared with Baseline+ BPGS, the proposed method (Baseline + BPGS + Adaptive Loss) attains preferable results. Hence, it is feasible to adjust the loss function according to the similarity of evidence data, thereby effectively mitigating the impact of pseudo-label noise. Based on the above analysis, it can be concluded that BPGS and the adaptive loss function are effective and necessary.

4.2. Analysis of the Setting of Similarity Regulation Indicator

In the adaptive loss function, the similarity regulation indicator $\kappa$ in Equation (11) is used to determine the level of attention to different evidence data. To clarify the setting basis of $\kappa$, the relationship between parameter $\kappa$ and the IoU is analyzed, as shown in Figure 11.

As shown in Figure 11, the horizontal coordinates are the similarity regulation indicator $\kappa$, the interval is 0.05, the longitudinal coordinates are the IoU, and the results of the two experiments are expressed by two curves with different styles. In addition, we also marked separately the highest IoU of the two experiments. With the increase in $\kappa$, the IoU curves of the two experiments first gradually increase and then decrease after reaching the highest point. Among them, $\kappa$ = 0.6 and $\kappa$ = 0.55 correspond to the highest points of the IoU curves, with 64.5% and 60.9% in Experiment 1 and Experiment 2, respectively. The detailed $\kappa -\mathrm{IoU}$ values in the two experiments are shown in

Table 4. Detailed $\kappa -\mathrm{IoU}$ values in the two experiments. The entries in bold denote the best results in the corresponding experiment.

Experiment 1	$\kappa$	0	0.05	0.1	0.15	0.2	0.25	0.3	0.35	0.4	0.45	0.5
	IoU (%)	51.4	52.1	53.5	54.7	56.3	57.5	59.8	60.7	62.2	63.4	62.7
	$\kappa$	0.55	0.6	0.65	0.7	0.75	0.8	0.85	0.9	0.95	1
	IoU (%)	63.3	64.5	62.9	60.3	58.7	56.8	55.4	54.8	54.3	53.6
Experiment 2	$\kappa$	0	0.05	0.1	0.15	0.2	0.25	0.3	0.35	0.4	0.45	0.5
	IoU (%)	49.8	49.6	50.7	51.2	53.9	55.3	57.1	59.5	58.3	59.4	60.1
	$\kappa$	0.55	0.6	0.65	0.7	0.75	0.8	0.85	0.9	0.95	1
	IoU (%)	60.9	60.4	59.7	58.1	57.8	55.9	53.4	53.6	52.5	52.1

.

By analyzing Table 4, we discover that when $\kappa$ was set at 0.6, the IoU could reach 60.4%, which is only 0.5% lower than the corresponding highest IoU in Experiment 2. That is to say, the ideal monitoring results could be obtained in the two experiments by setting $\kappa$ at 0.6. Therefore, to avoid excessive hyperparameter tuning, the similarity regulation indicator $\kappa$ is advised to be set directly at 0.6 in practical applications.

5. Conclusions

In this paper, we propose a semi-supervised semantic segmentation with boundary-guided pseudo-label generation strategy and adaptive loss function method. To the best of our knowledge, this is the first paper to implement automatic gully erosion monitoring from data acquired by different sensors at different times. In this method, the boundary-guided pseudo-label generation strategy (BPGS) composed of the object boundary generator and the multi-evidence fusion strategy is designed to enhance the reliability of pseudo-labels. Meanwhile, the adaptive loss function based on centroid similarity (CSIM) is proposed to further alleviate the impact of pseudo-label noise. Two experiments carried out on the HC2012 and HC2020 datasets show that the proposed method can better cope with the impact of feature differences on automatic monitoring than other state-of-the-art methods and present an IoU above 64% and 60%, respectively. Therefore, our method is more suitable for complex gully erosion monitoring. Furthermore, the ablation study demonstrates that BPGS and the adaptive loss function are effective and necessary. Concerning future work, we will further improve the performance and reliability of gully erosion monitoring, thereby continuing to make contributions to the research on automatic gully erosion monitoring.