A Novel Weakly Supervised Remote Sensing Landslide Semantic Segmentation Method: Combining CAM and cycleGAN Algorithms

Abstract

With the development of deep learning algorithms, more and more deep learning algorithms are being applied to remote sensing image classification, detection, and semantic segmentation. The landslide semantic segmentation of a remote sensing image based on deep learning mainly uses supervised learning, the accuracy of which depends on a large number of training data and high-quality data annotation. At this stage, high-quality data annotation often requires the investment of significant human effort. Therefore, the high cost of remote sensing landslide image data annotation greatly restricts the development of a landslide semantic segmentation algorithm. Aiming to resolve the problem of the high labeling cost of landslide semantic segmentation with a supervised learning method, we proposed a remote sensing landslide semantic segmentation with weakly supervised learning method combing class activation maps (CAMs) and cycle generative adversarial network (cycleGAN). In this method, we used the image level annotation data to replace pixel level annotation data as the training data. Firstly, the CAM method was used to determine the approximate position of the landslide area. Then, the cycleGAN method was used to generate the fake image without a landslide, and to make the difference with the real image to obtain the accurate segmentation of the landslide area. Finally, the pixel-level segmentation of the landslide area on remote sensing image was realized. We used mean intersection-over-union (mIOU) to evaluate the proposed method, and compared it with the method based on CAM, whose mIOU was 0.157, and we obtain better result with mIOU 0.237 on the same test dataset. Furthermore, we made a comparative experiment using the supervised learning method of a u-net network, and the mIOU result was 0.408. The experimental results show that it is feasible to realize landslide semantic segmentation in a remote sensing image by using weakly supervised learning. This method can greatly reduce the workload of data annotation.

1. Introduction

Landslides are one of the most widespread natural hazards [1]. The early spotting of landslide disasters in remote areas through remote sensing images can effectively reduce the occurrence of secondary disasters. Due to the problems of slow speed and the expertise required for the manual interpretation of remote sensing images, researchers have started to apply deep learning techniques to the automatic interpretation of remote sensing images [2].

Since the release of the ImageNet dataset [3] in 2009, deep learning has been rapidly developed in the field of computer vision. With the development of deep learning techniques in the fields of image classification, detection, and segmentation, the detection or segmentation of landslide areas in remote sensing images has also started to explore the use of deep learning methods to solve this problem. However, unlike other remote sensing image recognition tasks, there have been relatively few studies using deep learning to detect or segment landslide areas at this stage [4]. Sameen et al. [5] used deep residual detection based on a feature fusion network to detect the remote sensing images of landslides in the Kinmallan plateau, Malaysia, and improved the F1 score by 0.13 and mIOU by 0.1296 compared with the method of convolutional layer stacking. Chen et al. [6] proposed a change detection method based on deep convolutional network and obtained a false recognition rate of 0.176. Cheng et al. [7] proposed a YOLO-SA model using Qiaojia and Ludian counties in Yunnan Province, China, as the study area, and improved the recognition accuracy to 0.9408.

The remote sensing landslide detection algorithm can output the coordinate position of the landslide, while the remote sensing landslide segmentation algorithm can accurately segment the boundary of the landslide. Therefore, the remote sensing landslide segmentation algorithm can be used to study the landslide area change and landslide area calculation. Additionally, there have been even fewer studies related to remote sensing image segmentation tasks due to the lack of datasets. Soares et al. [8] used DEM information as training data and used the U-Net model to automatically segment landslides in the city of Novo Fribourg, located in the mountains of Rio de Janeiro, southeastern Brazil, and obtained F1 scores of 0.55 and 0.58 on two different test sets. Du et al. [9] compared six commonly used deep learning semantic segmentation models based on a self-built Yangtze River coastal landslide dataset, and finally obtained 0.542 and 0.740 mIOU accuracy on the GCN and deeplabv3 models, respectively. Prakash et al. [10] used an improved u-net network to experiment on the state-wide landslide dataset in Oregon. This method finally achieved a detection rate of 0.72, which is better than the traditional method. Bo et al. [11] used the deep learning semantic segmentation model to accurately detect the landslide area of remote sensing images in Nepal in 2015, and achieved a recall rate of 0.65 and an accuracy rate of 0.55.

The remote sensing landslide segmentation algorithm is identical to the ordinary semantic segmentation task. At present, the mainstream method is supervised learning, which requires a large number of pixel-level labeled training data. Pixel-level annotated datasets often require significant labor costs. For example, in the Bijie landslide dataset [12], the annotators used irregular polygons to annotate the landslide area in each remote sensing image. Annotators need to mark an average of 29 coordinate points on each image. If more accurate annotation results are required, the annotator needs to annotate more coordinate points and more accurate coordinate point locations. The datasets of landslide remote sensing images are too expensive to produce, which is a great challenge for performing research with remote sensing landslide images. The weakly supervised semantic segmentation algorithm based on image-level annotation can realize the study of landslide region segmentation on image-level annotated datasets, which provides a new idea for the study of a landslide region segmentation algorithm in remote sensing images.

As a commonly used weakly supervised learning algorithm, CAM can be used for extracting the geographic objects supervised by image-level labels [13]. Feng et al. [14] used self-matching CAM to provide a novel and accurate explanation of CNN for SAR image interpretation. While cycleGAN is usually used for virtual data generation, Park et al. [15] used cycleGAN to generate wildfire images as the training data for Densenet, and demonstrated a high wildfire detection accuracy. However, there has been no research to date that has used CAM and cycleGAN in the field of remote sensing landslides via image segmentation.

To this end, we propose a weakly supervised approach to landslide image segmentation training using only image-level labeled data as training data. First, we use the CAM method to locate the approximate location of the landslide area. Then, we use the cycleGAN method to generate the image in which there is no landslide and make the difference with the current image to realize the fine segmentation of the landslide area. Finally, the pixel-level segmentation of the landslide area on the remote sensing image is performed on the dataset based on image-level annotation.

Our main contributions are the following:

(1)

Proposing a cycleGAN-based method for the weakly supervised training of image-level labeled remote sensing landslide images, and achieving the fine segmentation of landslide regions;

(2)

Combining the CAM and cycleGAN methods to improve the segmentation accuracy of weakly supervised learning algorithms on remote sensing landslide images.

2. Dataset

2.1. Data Sources

The remote sensing landslide image dataset we used was from the Bijie landslide dataset [12] (Figure 1). This dataset uses remote sensing images of Bijie City of Guizhou Province, in China. The dataset was acquired by the TripleSat remote sensing satellite between May and August 2018, and the spatial resolution of the images is 1 meter. This dataset contains 770 remote sensing landslide images and 2003 remote sensing non-landslide images.

The Bijie dataset consists of slices of landslide images and non-landslide images, but the town buildings are only included in the non-landslide images. Since town building images in non-landslide images would affect the training of cycleGAN, we manually removed the non-landslide images which contain town buildings, and finally, 958 non-landslide remote sensing images were left as negative samples and 770 remote sensing landslide images were left as positive samples of the dataset. We randomly chose 78 images (approximately 10% of the total) among the positive samples and 97 images among the negative samples as the test set.

2.2. Pixel-Level Annotation and Image-Level Annotation

Pixel-level annotation is to determine whether each pixel in an image belongs a certain category, with the aim of annotating specific category regions in the image. Since the class to be labeled usually has irregular shapes, the labeling is labor-intensive, such as the COCO dataset [16]. Pixel-level labeled datasets are usually used for the training of semantic segmentation networks. The pixel-level labels that we used are shown in the pixel label column in Figure 1.

Image-level annotation only needs to annotate whether the image contains the category, and usually one image corresponds to one or more labels, so the annotation is relatively simple. Image-level annotated datasets are often used to train classification models, such as the ImageNet dataset [3]. The weakly supervised learning approach we used trains a semantic segmentation task with image-level labeled datasets. The image-level label we used is shown in the image label column in Figure 1.

3. Method

3.1. CAM-Based Weakly Supervised Algorithm

The class activation mapping (CAM) approach was first proposed by Zhou et al. [17] for the study of weakly supervised target localization and neural network visualization.

As shown in Figure 2, for a trained landslide classification network, the final classification result depends on the weight corresponding to the category class of the landslide multiplied by the final feature value, which in turn, is obtained by averaging the feature layers. Therefore, the weights of the feature layers directly affect the final classification results.

We used the weights of the landslide category to linearly weight the feature layer to obtain the class activation map for landslide classification, upsampled the class activation map to the size of the original image, and then superimposed it with the original image so that we could observe which regions of the original image are used by the network to make judgments about the category to which the image belongs.

Because the CAM method can draw the heat map of the target class of interest on the dataset based on image-level annotation, we can realize the rough segmentation of the remote sensing landslide image based on the heat map obtained by CAM.

CAM can be expressed by Equation (1).

$$M_c\left(x,y\right)={\displaystyle \sum }_k{w}_k^c{f}_k\left(x,y\right)$$

(1)

where $M_c \left(x,y\right)$ denotes the final obtained class activation map of category c, k denotes the number of channels of the feature map after the last convolution layer, $w{ }_k^c$ denotes the weight of k channels of the fully connected layer corresponding to category c, and $f_k \left(x,y\right)$ denotes the feature map after the last convolution layer.

Since the average image size of the Bijie dataset is 282 × 272, we resized the images to 256 × 256 as model inputs. When using the CAM approach, we first trained the training dataset for classification using the ResNet 50 model [18], and then calculated the class activation mapping maps for the test images based on the trained classification network. Since we trained the classification model with only two classes, namely landslide and non-landslide, we substituted the class corresponding to the landslide class directly into Equation (1) to obtain the class activation map corresponding to the landslide image.

3.2. cycleGAN-Based Weakly Supervised Algorithm

3.2.1. Generate Images before Landslides with cycleGAN

GAN networks [19] were first used to randomly generate images of a specific style, and cycleGAN networks [20] can achieve the conversion of styles between different domains while keeping the subject content of the original image unchanged.

Inspired by the function of style migration implemented by the cycleGAN network, we consider the images with landslide as one style of domain and the images without landslide as another style of domain. Additionally, we can train a generation network that can convert the landslide images to non-landslide images, and finally realize the generation of virtual non-landslide images from the landslide images.

Compared to the pix 2 pix adversarial generative network algorithm [21], the training data used by cycleGAN does not require the paired images of two domains, which provides a great convenience in obtaining training data. As we used the landslide images and non-landslide images, we do not need to obtain pairs of remote sensing images before and after landslides, but only one part of the dataset is landslide images and the other part is non-landslide images.

The goal of cycleGAN is to obtain, by training the data of two domains X and Y, the generator G, which can realize the style migration from domain X to domain Y; the generator F, which can realize the style migration from domain Y to domain X; the discriminator D_x, which can identify whether the image is a real image in X or a fake image generated by the image in Y through F; and the discriminator D_y, which can identify whether the image is a real image in Y or a fake image generated by the image in X through G. The flow chart of cycleGAN training is shown in Figure 3.

The loss function of the cycleGAN network mainly consists of two parts: one part is the adversarial loss, which is similar to the loss function of the classical GAN network and is designed to make the style of the fake samples generated by generators G and F close to the style of the real samples; and the other part is the cyclic consistency loss, which is used to constrain that the fake sample G(x) (obtained by real sample x through generator G) can be recovered into x again through generator F, i.e., F(G(x)) = x. Similarly, the real sample y can be recovered into y by G(F(y)). Therefore, the total loss function of the cycleGAN network can be expressed as Equations (2)–(4).

$$L\left(G,F,D_X,D_Y\right)=L_{GAN}\left(G,D_Y,X,Y\right)+L_{GAN}\left(F,D_X,Y,X\right)+\lambda L_{cyc}\left(G,F\right)$$

(2)

Additionally:

$$L_{GAN}\left(G,D_Y,X,Y\right)={\mathbb{E}}_{y~p_{data}\left(y\right)}\left|\log D_Y\left(y\right)\right|+{\mathbb{E}}_{x~p_{data}\left(x\right)}\left|\log {\displaystyle (}1-D_Y{\displaystyle (}G\left(x\right){\displaystyle )}{\displaystyle )}\right|$$

(3)

$$L_{cyc}\left(G,F\right)={\mathbb{E}}_{x~p_{data}\left(x\right)}\left[\left|\right|F\left(G\left(x\right)\right)-{x\left|\right|}_1\right]+{\mathbb{E}}_{y~p_{data}\left(y\right)}\left[\left|\right|G\left(F\left(y\right)\right)-{y\left|\right|}_1\right]$$

(4)

When using cycleGAN to train remote sensing images, we consider the landslide image as domain X and the non-landslide image as domain Y. Through the loss function, we obtain the generator G that can convert the landslide images into the non-landslide image and the generator F that can convert the non-landslide image into the landslide image.

3.2.2. Difference Method to Obtain Landslide Area

The difference method can analyze the degree of difference between two images by finding the difference value between the two images pixel by pixel. By setting a proper threshold value, the part of the two images with large differences can be further analyzed.

From Figure 4, we can see that the difference between the real landslide image and the generated non-landslide image mainly exists in the landslide area. Therefore, we can obtain the specific area where the landslide occurs in the landslide image by a simple difference method.

Before segmenting the landslide image using the difference method, the color image needs to be converted to grayscale, as shown in Equation (5):

$$Gray=R\times 0.3+G\times 0.59+B\times 0.11$$

(5)

Then, the pixel-by-pixel difference between the landslide image and the non-landslide image is obtained to obtain the difference image of the two images, as shown in Equation (6).

$$D\left(x,y\right)=\left|f_{landslide}\left(x,y\right)-f_{non-landslide}\left(x,y\right)\right|$$

(6)

Finally, a proper threshold T is chosen to perform a pixel-by-pixel analysis of the differential image to find the landslide area in the remote sensing image, as shown in Equation (7).

$$R\left(x,y\right)=\{\begin{array}{c}1,D\left(x,y\right)>T\\ 0, others\end{array}$$

(7)

3.3. Method of Combining cycleGAN and CAM

By comparing the segmentation results of the CAM-based method and the cycleGAN-based method, we can find that the CAM-based method can determine the approximate location of the landslide region in the image, but it is not very accurate in segmenting the edges of the landslide; however, the cycleGAN-based method is better at handling the details of the image, and thus more accurately segments the boundary of the landslide region in the image, although it can easily mis-segment the non-landslide region (such as the road in the images).

Additionally, we sought to verify whether the results of both methods contained the correct segmentation results. Therefore, we combined the CAM and cycleGAN methods and made an intersection of the results of both so as to improve the accuracy of the segmentation results.

The flow of the remote sensing landslide image segmentation using the combined CAM-based and cycleGAN-based methods is shown in Figure 4. First, we used the original image to obtain the result using the CAM-based method in Section 3.1; then, we used the original image to obtain the result using the cycleGAN-based method in Section 3.2; finally, we intersected the CAM result and the cycleGAN result pixel by pixel to obtain the final result.

4. Results and Discussion

4.1. Model Evaluation Method

In landslide remote sensing image semantic segmentation tasks, test results are usually evaluated using metrics such as precision, recall, and mIOU [8]. Precision, recall, and mIOU are mainly obtained by calculating TP, FP, TN, and FN [22]. The true positives (TPs) denote positive samples correctly predicted by the model; the false positives (FPs) denote positive samples which were incorrectly predicted by the model; true negatives (TNs) denote negative samples correctly predicted by the model; and false negatives (FNs) denote negative samples which were incorrectly predicted by the model.

Precision: indicates how many of the positive samples predicted by the model are correct, i.e., the percentage of the predicted landslide area that was correctly predicted. Precision is calculated using Equation (8).

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(8)

Recall: indicates how many of the actual positive samples were correctly predicted, i.e., the percentage of the area where the landslide actually occurred that the model was able to correctly predict. Recall is calculated using Equation (9).

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(9)

mIOU: it is usually used to comprehensively evaluate the performance of the segmentation model. IOU is calculated by dividing the intersection set by the union set of ground truth and prediction results, and mIOU requires averaging the calculated results for each category [23]. mIOU can be expressed by Equation (10).

$$\mathrm{MIoU}=\frac{\mathrm{A}{{\displaystyle \cap }}^{ }\mathrm{B}}{\mathrm{A}{{\displaystyle \cup }}^{ }\mathrm{B}}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$$

(10)

False positive rate: in the process of the landslide segmentation of remote sensing images using the landslide region segmentation algorithm, the number of negative samples (non-landslide images) is usually much higher than the number of positive samples (landslide images), and if the false positive rate of the algorithm is excessively high, a large number of false positive events will be generated in the practical application, resulting in a decrease in the practicality of the algorithm. Therefore, we add the non-landslide image dataset as a test data to calculate the false positive rate of the algorithm. The calculation of the false positive rate can be expressed as Equation (11).

$$\mathrm{FPR}=\frac{\mathrm{FP}}{\mathrm{TN}+\mathrm{FP}}$$

(11)

4.2. Evaluation Results

We evaluated the precision, recall, and mIOU with the CAM-based method, cycleGAN-based method, and the combined CAM and cycleGAN methods. The test results are shown in

Table 1. Comparison of test results between weakly supervised learning and supervised learning.

Method		Precision	Recall	mIOU	FPR
weakly supervised learning	CAM	0.692	0.593	0.159	0.054
	cycleGAN	0.845	0.404	0.184	0.042
	CAM + cycleGAN	0.924	0.383	0.237	0.004
supervised learning	U-Net	0.955	0.555	0.408	0.011

.

4.2.1. CAM Results Analysis

For the CAM-based method, from the overall test results, the precision was 0.692, the recall was 0.593, and the false positive rate was 0.054. Analyzing the test results of a single sample, the weakly supervised learning of the CAM-based method can roughly locate the location of the landslide area (as shown in Figure 5).

We can also see from the prediction results that the CAM-based method will predict as many positive samples as possible, and the actual landslide area is almost completely included in the predicted area, so a higher recall (0.593) can be obtained using the CAM-based method. However, the method can hardly determine the boundary of the landslide area, so it cannot be used to directly segment the contour of the landslide area.

On the other hand, because the CAM-based method is too sensitive to abnormal regions in the image, there are more false detections, for example, column B of the non-landslides in Figure 5, the algorithm incorrectly recognized the house as landslide, and the correct rate is also the lowest among all methods (0.692). Comparing several other methods, the CAM-based method has the highest false positive rate of 0.054, indicating that it is more likely to produce false detections when predicting images without landslides compared to several other methods.

4.2.2. cycleGAN Results Analysis

We choose the optimal threshold T = 32 with Equation (7). The test results using the cycleGAN-based method are shown in the second column of Table 1, with a correct rate of 0.845, a recall rate of 0.404, and a false positive rate of 0.042, all of which are intermediate values for the three weakly supervised algorithms.

The segmentation results of the cycleGAN-based method are shown in Figure 6. As can be seen from the results generated by the model in the second column, comparing the original landslide remote sensing images (e.g., A and B in Figure 6), the fake images generated by cycleGAN not only change the color, but can also generate the texture of the landslide area similar to the non-landslide area.

For the non-landslide remote sensing images (such as C and D in Figure 6), the fake images generated by cycleGAN can almost be consistent with the original images, especially when comparing the original and generated images of sample C in Figure 6, we can find that, although there is a yellow non-landslide area similar to the landslide in the original image, the image generated by cycleGAN do not process this area, which indicates that the model does not only judge whether an area is a landslide area from the color, but also has the ability to distinguish the real landslide area.

After obtaining the fake image generated by cycleGAN, we will use the difference method to judge each pixel of the image, so compared with the CAM-based method, this method can obtain a more accurate segmentation boundary. On the other hand, due to reasons such as the lesser number of learning samples, the model will cause the false detection of regions such as the road in the samples (sample A in Figure 6), which is the most important reason for the existence of false detection in the cycleGAN-based method.

4.2.3. Analysis of Combined Method Results

As can be seen in the third column of Table 1, the precision of the combined CAM-based and cycleGAN-based methods is 0.924, the recall is 0.383, and the false positive rate is 0.004.

By comparing the shortcomings of the CAM-based and cycleGAN-based methods, we found that the segmentation results of both methods contain the correct detection results, and the reasons for the existence of misdetection are not the same. The misdetection of the CAM-based method is due to the inability of the model to correctly partition the boundary of the landslide area. However, the misdetection of the cycleGAN-based method was due to the fact that the model misidentifies areas such as roads as landslide areas. Therefore, we combined the above two methods and took the intersection of the two results to make the prediction results for positive samples more reliable, and finally, obtained the highest accuracy (0.924) among the three weakly supervised algorithms.

Because the combined method result is obtained by taking the intersection of the results from CAM and cycleGAN, it is the most stringent compared to the single algorithm. This method sacrifices recall while improving precision, resulting in the lowest recall (0.383) among the three methods.

In addition, the combined method has the lowest false positive rate of 0.004 among all methods, which is even lower than the fully supervised method, indicating that this method has the lowest false alarm rate in the prediction of non-landslide images.

In order to evaluate the performance of the three methods together, we tested the test set using mIOU. The test results show that the combined method could obtain an mIOU of 0.237, which is better than the mIOU of 0.159 for the CAM-based method and 0.184 for cycleGAN-based method.

4.2.4. Comparison of Weakly Supervised Method and Supervised Method

Today, there are a few open source landslide datasets. As the first open remote sensing landslide dataset [12], the Bijie dataset is the only landslide dataset that we have access to. In order to compare the weakly supervised learning approach with the supervised learning approach, we trained the Bijie data with supervised learning using the U-Net network [24], and the segmentation results using the U-Net network are shown in Figure 7.

From the comparison test results in Table 1, we can see that the weakly supervised algorithm combining the CAM-based and cycleGAN-based methods is already very close to the supervised algorithm in terms of model accuracy, and the main gap is reflected in the recall rate of the model. Compared with the mIOU of 0.408 obtained by the supervised method, the mIOU of our weakly supervised algorithm is 0.237, which still has a certain gap.

We also need to note that the cost of labeling the dataset is low due to the weakly supervised algorithm compared to the supervised learning algorithm. For this dataset, the average workload of labeling a supervised training datum is 29 coordinate points, while labeling a weakly supervised training datum requires only 1 label selection. Since the workload of labeling coordinate points is greater than the workload of labeling a selection, the workload of labeling a supervised training datum exceeds the workload of labeling a weakly supervised training data by a factor of 29. With continuously increasing data, we can easily improve the algorithm accuracy using the weakly supervised learning algorithm. While improving the supervised algorithm accuracy, we need to invest a greater labeling cost.

5. Conclusions

We studied a weakly supervised learning approach combining the CAM-based method and cycleGAN-based method for the semantic segmentation task of remote sensing landslide images, and compared the test results with the supervised learning approach.

The experimental results show that the mIOU of the combined method is higher compared with the single weakly supervised method, and the method can be used for remote sensing landslide area segmentation. Specifically, the accuracy of our model is 0.924, which is higher than 0.692 and 0.845 of CAM and cycleGAN, respectively, indicating that our model is more accurate in recognizing landslides. The FPR of our model is 0.004, which is lower than the 0.054 and 0.042 of CAM and cycleGAN, respectively, indicating that our model has a low probability of incorrect recognition. With the weakly supervised remote sensing landslide image segmentation algorithm, we can obtain a landslide semantic segmentation model based on image-level annotated training data. Compared with other models that must require pixel-level annotated training data, our model can greatly reduce the workload of annotators.

On the other hand, by comparing the weakly supervised learning method with the supervised learning method, we found that the experimental results of the method are still some distance away compared with the supervised learning method. However, since the weakly supervised learning method has the advantage of a low labeling cost, we can increase the training data to improve the accuracy of the algorithm. Another existing problem is that since the training data do not contain town buildings or rivers, this means that our model cannot be applied to remote sensing images containing town buildings or rivers. In the future, we can consider adding town buildings and rivers as training data in both the landslide dataset and non-landslide dataset to improve the accuracy of the model in remote sensing images of towns and rivers. For the model structure, using an end-to-end model for training is a worthwhile direction of investigation.