Open-Pit Mining Area Extraction from High-Resolution Remote Sensing Images Based on EMANet and FC-CRF

Abstract

Rapid and accurate identification of open-pit mining areas is essential for guiding production planning and assessing environmental impact. Remote sensing technology provides an effective means for open-pit mine boundary identification. In this study, an effective method for delineating an open-pit mining area from remote sensing images is proposed, which is based on the deep learning model of the Expectation-Maximizing Attention Network (EMANet) and the fully connected conditional random field (FC-CRF) algorithm. First, ResNet-34 was applied as the backbone network to obtain preliminary features. Second, the EMA mechanism was used to enhance the learning of important information and details in the image. Finally, a postprocessing program based on FC-CRF was introduced to optimize the initial prediction results. Meanwhile, the extraction effect of MobileNetV3, U-Net, fully convolutional network (FCN), and our method were compared on the same data set for the open-pit mining areas. The advantage of the model is verified by the visual graph results, and the accuracy evaluation index based on the confusion matrix calculation. pixel accuracy (PA), mean intersection over union (MIoU), and kappa were 98.09%, 89.48%, and 88.48%, respectively. The evaluation results show that this method effectively identifies open-pit mining areas. It is of practical significance to complete the extraction task of open-pit mining areas accurately and comprehensively, which can be used for production management and environmental protection of open-pit mines.

1. Introduction

Open-pit mining has become the mainstay of the mining industry. It occupies a significant economic and social development position due to its large production scale and high efficiency. In open-pit mining, the mining areas’ information must be continuously obtained. On the one hand, the information is used to devise and revise the production plan [1]; on the other hand, it is used to evaluate the impact of mining on the environment [2]. The traditional methods of obtaining boundary information of open-pit mines require people to survey and measure on-site, which is time-consuming and laborious. This task can be completed efficiently and accurately with modern science and technology. With the rapid development of remote sensing technology, high-resolution remote sensing images of open-pit mines provide data support for this task [3]. Automatic and accurate extraction of open-pit mines from remote sensing images can prevent over-exploitation. It can also provide more reliable information for specific practical applications, such as geographic information updating, mine environmental planning, and rapid assessment by relevant regulatory agencies [4].

In the early use of remote sensing images, the visual interpretation method was used to complete the classification of ground features [5]. Using traditional visual interpretation methods, it is expensive, laborious, and highly subjective to perform the extraction task of open-pit mining areas [6]. Visual interpretation has been used in remote sensing as an auxiliary function. For example, Mas et al. [7] proposed a classification method combining image segmentation, GIS analysis, and visual interpretation to analyze remote sensing data for assessing land use and cover change. The overall accuracy was 83.3% ± 3.1%.

Later, with the development of machine learning, scholars applied machine learning to remote sensing images of open-pit mines [8]. They improved the classification accuracy of remote sensing images by optimizing machine learning algorithms or combining them with other algorithms. For instance, Chen et al. [9] used three optimization methods to optimize the support vector machine (SVM) model. The optimized SVM model improves the precision of fine land cover classification in open-pit mining areas based on high-resolution remote sensing images. To obtain the land occupation around the open-pit mine, a multiscale segmentation algorithm has been proposed to extract large-scale primary feature information by Yu et al. [10] Combined with the random forest (RF) model, the remaining feature elements were classified and extracted. The overall extraction accuracy and kappa coefficient of the method reached 86% and 78%, respectively. Machine learning methods have improved efficiency and accuracy, but many problems still need to be solved. Machine learning methods mainly use spectral information from different bands of remote sensing images. This learning method leads to an unsatisfactory boundary extraction effect of open-pit mines, information redundancy, and “salt and pepper” noise [11].

In recent years, deep learning has developed rapidly. Deep learning adopts an end-to-end hierarchical learning approach with powerful feature extraction capabilities [12]. It autonomously learns practical feature information through training data sets. Researchers have used convolutional neural networks to analyze high-resolution remote sensing data and obtain land cover information from open-pit mines [13]. For instance, Chen et al. [14] proposed an improved deep learning network, U-Net+, with multilayer feature associations. They aimed to obtain the surface coverage of the open-pit mining area, and the experimental results indicated that the proposed framework outperforms the original U-Net by 0.02% overall. Furthermore, Ullo et al. [15] introduced an image-based landslide detection method by combining a pretrained Mask R-CNN model with transfer learning. The experimental results were quite pleasing, achieving a precision equal to 1.00. The above studies showed that the deep learning model has good accuracy, generalization, and robustness for information extraction from high-resolution remote sensing images. This is an effective way to promote the development of extraction of open-pit mines from remote sensing images [16].

Remote sensing images of open-pit mines have strong particularity in color and texture features, which provides convenience for scientific research. Some scholars have started research on the acquisition of open-pit boundary information. For example, Du et al. [17] proposed an improved DeepLabv3+ network for open-pit boundary information extraction. This model’s attention mechanism is embedded after atrous spatial pyramid pooling (ASPP) to enhance meaningful information learning. A low-level feature multiscale fusion module is added to the decoder to reduce the loss of information. Moreover, a postprocessing method, including object-based conditional random field (CRF) optimization, is proposed to refine the extraction results of open-pit mines. The F1-score at the pixel level reaches 91.2%. Inspired by SegNet, U-Net, and D-LinkNet, Xie et al. [18] applied a novel deep convolutional neural network for pixel-level semantic segmentation of optical remote sensing images termed DUSegNet. DUSegNet is an effective method for the segmentation of mining areas in remote sensing images of open-pit mines. At the end of the experiment, the average precision and F1-score of the model reached 94% and 67%, respectively. Zhang et al. [19] introduced an intelligent extraction method for open-pit mining areas from multisource remote sensing images based on an improved DenseNet network, including 10 densely connected blocks and 48 convolutional layers in the encoder and decoder. The method has good accuracy, with precision of 97.7% and IoU value of 72.1%. To extract open-pit mines from large-scale high-resolution remote sensing images, Wang et al. [4] presented an open-pit mine extraction model based on improved Mask R-CNN and transfer learning. This model can realize the autonomous identification and dynamic monitoring of open-pit mines. This model has the best performance in pixel accuracy and kappa, with values of 97.18% and 82.51%, respectively.

The comprehensive research shows that the neural network model has good robustness, generalization, and accuracy in the target recognition of open pits in remote sensing data. Therefore, methods based on deep learning and remote sensing images can effectively realize the task of boundary acquisition [20]. The rapid development of computer vision, image processing, semantic segmentation, and other technologies has created conditions for completing extraction tasks. Due to the complex composition and high heterogeneity of image data of open-pit mines, many methods cannot accurately and comprehensively divide mining areas. There are still many problems to be solved in this field.

Based on the above research, this paper proposes a method that combines the EMANet model with the FC-CRF algorithm for precisely extracting mining areas from remote sensing images of open-pit mines. First, ResNet-34 (pretrained on ImageNet) was adopted as the backbone network for preliminary feature extraction. Then, the EMA mechanism (a self-focused mechanism) is introduced and combined with a neural network to form the EMA unit (EMAU). Compared to other self-attentive mechanisms, it abandons the process of computing the attention map on the full graph and instead iterates through the EM algorithm to produce a compact set of bases on which to run the attention mechanism, thus significantly reducing time, space, and computational complexity. Optimal models can still be obtained in less time and with less arithmetical power. Finally, a postprocessor based on the FC-CRF is proposed to optimize the prediction results and improve extraction accuracy. This method can accurately, efficiently, and comprehensively complete the extraction task of open-pit mining areas. The data obtained through the experiment have practical significance for mine production safety management, mineral resource development, and surrounding environmental protection.

2. Material, Methods and Training Process

Our approach has two main components. The remote sensing data of the open-pit mine are extracted by the EMANet network [21]. Then, the postprocessing program FC-CRF is used to optimize the preliminary extraction results to obtain the final prediction results of the open-pit mining area with higher accuracy [22]. This section first introduces the data sets used in the experiment. Then, the model used in this paper is broken down into four parts for the introduction. Finally, the training process of the experiments is described.

2.1. Data Set

The data for this experiment came from Google Earth satellite remote sensing image data. The image data used in the experiment summarize high-resolution remote sensing images from open-pit mines worldwide. High-resolution remote sensing observes the Earth at a fine spatial resolution at the meter or even submeter level. The data used in this experiment are all from Google Earth and possess meter-level resolution. The acquired high-resolution remote sensing images can represent the structural and surface texture features of the open-pit mine, making it easier to distinguish its internal composition and edge information. Remote sensing images of the same size and resolution were selected for this experiment, and 1001 images were saved in JPG format.

The remote sensing image of an open-pit mine has unique color distribution and texture characteristics. Therefore, the LabelMe tool was used to outline the open-pit mining boundary line to form a closed area [23]. The conversion script was used to obtain the mask image and store it in PNG format. Ground truth is the result of manual annotation data. Each remote sensing image of the open-pit mine corresponds to a labeled image, i.e., ground truth. Before starting the experiment, data preprocessing is performed on the remote sensing images of open-pit mines and ground truth. The Albumentations tool is used for data preprocessing [24]. The preprocessing operation consists of two parts: pixel-level transformation and spatial-level transformation. Pixel-level transformation only changes the remote sensing image and the annotated image remains unchanged. The operations used include adding noise, blurring the image, etc. Spatial-level transformations perform consistent transformations on images and masks, such as rotate, crop, flip, etc. All the data used in the experiment are 1001 remote sensing images of open-pit mines and the corresponding 1001 labeled images. The ratio of the training set and test set is 80% and 20%, respectively. The training set contains 800 images and 800 masks and the test set contains 201 images and 201 masks. In the training and testing process of the experiment, both remote sensing images and their ground truth are input together and are used to optimize the model as well as to judge the model’s strengths and weaknesses. Figure 1 shows four randomly selected sample images from the experimental data. At the top is the original high-resolution image. Below are the corresponding images of masks that have been manually annotated, i.e., ground truth. In Figure 1, the red line on the first row of images is the boundary of the open-pit mining area, which corresponds to the ground truth in the second row.

2.2. EMANet Based on the Expectation-Maximization Attention

The EMA adopted by EMANet is an augmented version of the self-attention mechanism. The self-attention mechanism is a variant of the attention mechanism, one of the most widely used. It reduces the dependence on external information and better captures the internal correlation of data or features. EMA consists of three parts: responsibility estimation (A_E), likelihood maximization (A_M), and data reestimation (A_R).

Responsibility estimation (A_E) functions as the E step in the EM algorithm. Given the input feature graph $\mathbf{X}\in {ℝ}^{N\times C}$, the initial value of the basis is $\mathit{\mu }\in {ℝ}^{K\times C}$, and the hidden variable is estimated by the A_E (i.e., the responsibility of each basis to the pixel) $\mathbf{Z}\in {ℝ}^{N\times K}$. The responsibility of the k-th basis for the n-th pixel can be calculated as:

$$z_{nk}=\frac{\mathcal{K}\left({\mathbf{x}}_n,{\mathit{\mu }}_k\right)}{{\displaystyle {\sum }_{j=1}^K\mathcal{K}\left({\mathbf{x}}_n,{\mathit{\mu }}_j\right)}},$$

(1)

where $\mathcal{K}$ represents the general kernel function. The kernel $\mathcal{K}\left(\mathbf{a},\mathbf{b}\right)$ takes the exponential inner dot $\exp \left({\mathbf{a}}^{\top }\cdot \mathbf{b}\right)$. The operation of the A_E at the t-th iteration is formulated as:

$${\mathit{Z}}^{(t)}=\mathrm{softmax}\left(\lambda \mathit{X}{\left({\mathit{\mu }}^{\left(t-1\right)}\right)}^{\top }\right),$$

(2)

where λ is a hyperparameter to control the distribution of Z.

Likelihood maximization (A_M) works as the EM algorithm’s M step. With the estimated Z, A_M updates μ by maximizing the complete data likelihood. Update the basis μ using a weighted sum of X. In the t-th iteration of A_M, ${\mathit{\mu }}_k$ is updated as:

$${\mathit{\mu }}_k^{(t)}=\frac{z_{nk}^{(t)}{\mathbf{x}}_n}{{\displaystyle {\sum }_{n=1}^N{z}_{nk}^{(t)}}}.$$

(3)

Data reestimation (A_R) works as the EM algorithm’s R step. In EMA, the A_R step is run only once. EMA runs A_E and A_M alternately for T times. After that, A_R reestimates the X using the final generated μ^(T) and Z^(T). The following formula is used to compute the new X, namely X′:

$${\mathbf{X}}^{\prime }={\mathbf{Z}}^{(T)}{\mathit{\mu }}^{(T)}.$$

(4)

As X′ is constructed from a compact basis set, it is very compact in the feature space and the intra-object feature variance is smaller than the input X₁ feature variance. In other words, its intra-class variance is reduced while maintaining inter-class variance. Thus X’ has a low-rank property compared to the input X₁.

Many self-attention mechanisms proposed in recent years require the generation of a huge attention map, which has huge computational and time complexity [25]. In contrast to non-local modules (a self-attentive mechanism), EMA employs the EM algorithm to find a compact set of base μ for the input image [26]. This compactness is very important. This operation reduces the time and space complexity from O (N²) to O (NKT), where T is the number of iterations of A_E and A_M. In our experiments, EMA requires only three iterations to obtain good results. Therefore, T can be considered a small constant, which means that the complexity of EMA is only O (NK), so EMA has much less time, space, and computational complexity compared to most self-attentive models.

To better incorporate the proposed EMA with deep neural networks, the EMA unit (EMAU) was constructed for boundary information extraction of the open-pit mine. The overall structure of the EMAU is shown in Figure 2. At first glance, the EMAU looks as though it is the bottleneck of ResNet, except it replaces the heavy 3 × 3 convolution with the EMA operations. In EMAU, the first convolution without the ReLU activation is prepended to transform the value range of the input from (0, +∞) to (−∞, +∞). The last 1 × 1 convolution is inserted to transform the reestimated X′ into the residual space of X.

The initial values of bases before iterations are of great importance. Kaiming’s initialization is used to initialize μ⁽⁰⁾ for the first mini-batch. For the following mini-batches, moving averaging is adopted to update μ⁽⁰⁾ in the training process. After iterating over an image, the generated μ^(T) can be regarded as a biased update of μ⁽⁰⁾, where the bias comes from the image sampling process. To make it less biased, first average μ^(T) over a mini-batch and obtain ${\stackrel{-}{\mathit{\mu }}}^{\left(T\right)}$. Then update μ⁽⁰⁾ as follows:

$${\mathit{\mu }}^{(0)}\leftarrow \alpha {\mathit{\mu }}^{(0)}+(1-\alpha ){\overline{\mathit{\mu }}}^{(T)},$$

(5)

where α ∈ [0, 1] is the momentum and μ^(t) (1 ≤ t ≤ T) is normalized using Euclidean normalization (L2Norm). This solves the problem where significant differences exist and keeps the direction of each basis the same.

This paper proposes an EMANet model based on a deep learning method to extract mining areas from high-resolution remote sensing images of open-pit mines. The specific structure of EMANet is shown in Figure 2. The model consists of four parts. The data used for training are high-resolution remote sensing images of the three-channel color open-pit mine, with a uniform size of 1080 × 1920. That is, the size of the original image is 1080 × 1920 × 3. The size of the following feature figure is uniformly expressed in the format of H × W × C, where H, W, and C are the height, width, and number of channels of the feature figure, respectively.

First, the input open-pit remote sensing image is extracted by ResNet-34, the backbone network [27]. The imported original picture undergoes five convolution operations of stride size two to generate an initial feature map X₁ of size 34 × 60 × 512. Then, the EMAU calculates the initial feature map X₁ to capture the internal correlation of the data, resulting in a feature map X₂ of size 34 × 60 × 512. EMAU is a module that combines the EMA self-attention mechanism with neural networks. Using the EM algorithm, a compact set of bases is formed after several iterations, overcoming the complex problem of self-attention mechanism computation. Then, after two convolution operations, the number of channels of the X₂ is reduced from 512 to 2, and a feature map X₃ with a size of 34 × 60 × 2 is generated. At the end of the network, the feature map X₃ is restored to the original size by the upsampling operation using the interpolation method to ensure the integrity of the data. The result predicted by the EMANet model is a binary black-and-white image of the original size, with dimensions of 1080 × 1920 × 2.

2.3. Postprocessing Based on FC-CRF

The trained EMANet network is used to extract the open-pit mining area in remote sensing images, but there are some problems in the initial prediction results. First, due to the operation of downsampling in the neural network, the boundary information in the initial extraction results of the open-pit mine is fuzzy. Compared to the ground truth, the predicted edge is too smooth, which makes it impossible to divide the mining area and the surrounding environment accurately. Second, in the original extraction results, some small holes appear in the closed mining area, which does not align with the actual situation. This is due to the high heterogeneity of the open-pit mining area, and there are huge differences between mines with different mining resources and mining methods. The postprocessing procedure can effectively suture the cavity area, conform to reality, and improve accuracy. Third, isolated misclassification regions with small areas appeared in the extraction results. In the remote sensing image, when the mining area is small and scattered, the results predicted by the model are different from the ground truth, and it is easy to miss and make mistakes.

To solve these problems, an image postprocessing program is introduced to optimize the prediction results of the EMANet neural network model. The fully connected conditional random field (FC-CRF) is an image postprocessing method commonly used in deep learning semantic segmentation [22]. It is an improved model of conditional random field (CRF) [28]. By combining the relationship between all pixels in the original image, FC-CRF processes the predicted results obtained by the EMANet model to obtain more accurate and detailed extraction results. Experiments have shown that the FC-CRF can effectively solve the problems in the initial extraction results.

FC-CRF conforms to the Gibbs distribution:

$$P(\mathbf{X}\mid \mathbf{I})=\frac{1}{Z(\mathbf{I})}{\exp }^{-E(\mathbf{x}\mid \mathbf{I})},$$

(6)

where E(x|I) is the Gibbs energy. The energy function consists of a unary potential function and a binary potential function:

$$E(x)={\displaystyle \sum _i{\psi }_u}\left(x_i\right)+{\displaystyle \sum _{i<j}{\psi }_p}\left(x_i,y_j\right),$$

(7)

where the unary potential function represents the probability that the predicted value is the true value. The binary potential function explains how each pixel relates to all other pixels. This energy term separates our extraction results as much as possible from the edges of the open-pit mining area image. Therefore, it can be adopted to compensate for the fuzzy boundary problem of the extraction results mentioned above.

The unary potential energy is the probability distribution diagram in the actual operation of FC-CRF processing high-definition remote sensing images. It is the result of the prediction results of the EMANet model output and softmax function operation. The original image provides the position and color information in binary potential energy. When the energy $E(x)$ is lower, the predicted category label is more accurate and iteratively minimizes the energy function to obtain the final postprocessing result.

EMANet’s classification of open-pit mining and non-mining areas in each remotely sensed image is unlikely to be entirely correct. To cope with this phenomenon, the FC-CRF algorithm was used to refine the initial prediction results of EMANet. When using the FC-CRF algorithm, the original open-pit mine remote sensing image and the initial extraction results predicted by the EMANet model are taken together as input. Concerning the initial extraction results, FC-CRF assigns labels to the pixels in the original image based on position and color. In the specific operation process, two pixels with similar location and color characteristics, which have a high probability of being assigned the same label, have a low probability of being segmented. In addition, FC-CRF can perform global normalization of all features, find the global optimal solution, and obtain the energy minimum. Therefore, by applying FC-CRF and obtaining the energy minimum, the optimized open-mine mining area segmentation result can be obtained.

2.4. Training Process

Configure the Intel Core i7 CPU and the NVIDIA GeForce RTX 2060 graphics card. The experiment was conducted on a desktop computer with a Windows 64-bit operating system and 32 GB of RAM. The experimental model is run on Python 3.8, and the framework is PyTorch.

The ResNet-34 model pretrained on ImageNet is added to the subsequent model training to help improve the convergence of the whole model. Due to the memory limitation in our model training, the original image was scaled from 2160 × 3840 to 1080 × 1920. Set the batch size to 5. The initial learning rate is set as 9 × 10⁻³, and the attenuation function of the learning rate is shown in the following equation. At the i-th iteration, the learning rate is:

$$\mathrm{lr}=0.009\times {\left(1-\frac{i}{5000}\right)}^{0.9}.$$

(8)

The total data used in the experiment are 1001 remote sensing images of open-pit mines and the corresponding 1001 labeled images. The training and test sets are divided into proportions of 80% and 20%. The training set contained 800 images and 800 masks, and the test set contained 201 images and 201 masks. A total of 5000 iterations of the experiment were carried out to obtain the final mining area extraction prediction model. In the experimental training phase, the cross-entropy loss function is presented to calculate the loss function value between the predicted results and the labeled real situation [29]. In the experimental testing stage, the accuracy evaluation index based on confusion matrix calculation is adopted to realize the mathematical analysis of the model prediction results. At the same time, the effectiveness of the model is verified by visual analysis of the prediction results. The choice of these hyperparameters in the EMANet model was selected on the basis of a large number of experiments. For example, for the selection of the backbone network ResNet-34, is selected after comparison with ResNet-18 and ResNet-50. As another example, the input remote sensing image size 1080 × 1920 in the experiment is selected after comparison with sizes such as 2160 × 3840 and 540 × 960. The sizes of different remote sensing images also directly affect the choice of batch size, and the largest possible batch size is chosen without exceeding the memory of the computer.

Figure 3 shows the loss of the EMANet model and the change in the kappa coefficient of the accuracy index in the training process. Generally, the accuracy gradually increases during the training process, while the loss gradually decreases and converges. The analysis shows that in the middle of the 0–2000 iteration process, while the value of the training loss function decreases continuously, the model’s accuracy in extracting the open-pit mining areas also increases constantly. In 2000–5000 iterations, the loss value of the prediction results did not fluctuate much and tended to be stable. Meanwhile, the value of the kappa coefficient, the precision evaluation index used to analyze the prediction results of the model, also changed little.

3. Results

3.1. Test Results and Accuracy Evaluation

The EMANet and FC-CRF proposed in this paper are used to conduct many experiments on remote sensing image data. A model with a good effect is obtained by the training data set learning. The test data set is applied to check the merits and demerits of the model. Meanwhile, the extraction results of open-pit mine remote sensing images provide visualization analysis and accuracy evaluation.

(1)

Test results;

Figure 4 shows the results of extracting the open-pit mining area on the test data set. Each row represents a set of test data. The first column is the test data set. The second column shows the manual annotation of the open-pit mining boundary. The yellow line in the images in the first column is the boundary of the open-pit mining area, which corresponds to the ground truth in the second column. The third column shows the segmentation and recognition results predicted by the EMANet. The fourth column is the final extraction result after the FC-CRF processing. The visualization of the predicted results demonstrates that the EMANet model is capable of extracting open-pit mining areas from remote sensing images. Although some areas different from the annotated images (e.g., areas 1–4) appear in the initial extraction results, they can be resolved in subsequent optimization.

After the postprocessing of the extraction results, the accuracy is further improved. It can be observed that the final open-pit mining area extraction results are closer to the ground truth. As seen from the visualized results in Figure 4, the FC-CRF module mainly accomplishes four aspects of the work. First, it fills the holes in the initial prediction results (e.g., region 1). Second, the mining boundary is detailed. The initial extraction results are smooth and flat (e.g., region 2), which is inconsistent with the actual situation. After the FC-CRF, the details of the edge of the mining area are more apparent and more in line with the reality of open-pit mining. To see the details of the edges, the test sample in the first row of Figure 4 is used as an example. Figure 5 uses a local enlargement of the edges of the mining area to illustrate the significant optimization effect of the FC-CRF algorithm on the initial extraction results. The yellow line in the first column of images in Figure 5 is the manually labeled boundary, i.e., ground truth. Third, the parts that do not belong to the mining area (e.g., region 3) are excluded, with characteristics of isolation and small area. Fourth, there are discontinuous areas (e.g., region 4) with very close distances in the initial prediction results, and the FC-CRF program can suture the two parts well.

(2)

Accuracy evaluation;

To analyze and predict the extraction results of open-pit mines more intuitively, accuracy evaluation is introduced to evaluate the semantic segmentation results. A pixel-based evaluation method is used. It analyzes the quality of the model by reflecting the consistency of the extracted results in terms of geometric accuracy.

Each open-pit remote sensing image corresponds to a binary map with two categories. The number 1 represents the open-pit mining area, the number 0 represents the non-mining area, the letter T represents the true label, and the letter P represents the predicted label. TP represents the number of correctly classified background pixels. TN represents the number of pixels of the mining area correctly classified. FP is expressed as the number of pixels in the mined area misclassified as background. FN represents the number of background pixels misclassified as mining areas. Therefore, any pixel in the remote sensing image mask belongs to one of these four categories, thus forming a confusion matrix.

The accuracy evaluation indexes used in this experiment include pixel accuracy (PA), mean intersection over union (MIoU), and kappa coefficient. These accuracy evaluation indexes are calculated based on the confusion matrix. Pixel accuracy represents the proportion of correct prediction results in the total predicted value. The higher the pixel accuracy, the higher the degree of matching between predicted and real values.

$$\mathrm{PA}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(9)

The intersection over union (IoU) is applied to compare the similarity between the predicted output of the model and the ground truth. It is measured using the ratio between the intersection and union of the predicted results and the true values of a model for a category. The IoU of the background in the remote sensing image is expressed as:

$${\mathrm{IoU}}_{\left(\mathrm{background}\right)}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}.$$

(10)

The IoU of the open-pit mining area in the remote sensing image is expressed as:

$$\mathrm{Io}{\mathrm{U}}_{\left(\mathrm{mining} \mathrm{area}\right)}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}.$$

(11)

The MIoU is the average of the two categories of IoU. The kappa coefficient is proposed to test whether the model prediction results are consistent with the ground truth, and can be used to measure the effect of extraction [30]. If the kappa value is greater than 0.81 and less than 1, the predicted results are in close agreement with the true values. The kappa is calculated by the following equation:

$$\mathrm{kappa}=\frac{{\mathrm{p}}_0-{\mathrm{p}}_{\mathrm{e}}}{1-{\mathrm{p}}_{\mathrm{e}}},$$

(12)

where ${\mathrm{p}}_0$ represents the accuracy of the prediction and ${\mathrm{p}}_{\mathrm{e}}$ represents the coincidence agreement.

Evaluation results verify the effectiveness of the proposed method. The accuracy evaluation index results are listed in

Table 1. Evaluation of open-pit mining area extraction results.

	EMANet	EMANet and FC-CRF
PA	97.77%	98.09%
MIoU	87.67%	89.48%
kappa	86.28%	88.48%

. The initial prediction results of the test data in the experiment obtained by the already trained EMANet model perform very well. PA, MIoU, and kappa values were 97.77%, 87.67%, and 86.28%, respectively. The FC-CRF module plays a vital role in the accuracy and practicability of the method. After adding the FC-CRF, the prediction results are further optimized. Postprocessing optimization improved the MIoU and kappa coefficients by approximately 2%.

In addition, to evaluate the stability of the model, the extraction accuracy of 20 groups of test data is shown in Figure 6 below. The values of the three evaluation indicators in the 20 groups of data are all greater than 80%. The PA value remained stable, between 96% and 100%. The MIoU and kappa coefficient oscillations are consistent, and the minimum value will not exceed 84%. The two indicators fluctuated between 84% and 96% without a dramatic rise or fall. Through the above analysis, the stability and practicability of the model are verified.

The method’s accuracy, robustness, and generalization are verified based on the visualization analysis and the evaluation of the predicted results. This is of great significance for future applications in actual production.

3.2. Comparison

EMANet and FC-CRF have achieved excellent results in extracting open-pit mining areas, and the identification accuracy rate is about 90%. Three deep learning networks are selected for comparison to further illustrate the advantages of our method in extracting open-pit mining areas. Some experiments on MobileNetV3, U-Net, fully convolutional network (FCN), and our method are conducted using the same training and test data. At the same time, the visualization analysis and accuracy evaluation of the final prediction results under different models are carried out. The results show that EMANet and FC-CRF outperform other models in various respects. The following compares the extraction effect of open-pit mining areas between different models. Figure 7 shows the results of mining area identification on the test set. Each row represents a set of test data. The first column is a randomly selected sample of the open-pit mine’s high-resolution remote sensing image data. The second column is the manual marking of open-pit mining area information. The red line on the images in the first column is the boundary of the open-pit mining area, which corresponds to the ground truth in the second column. Columns three through six are the results of the extraction of MobileNetV3 [31], U-Net [32], FCN [33], and EMANet and FC-CRF, respectively. As shown in Figure 7, EMANet and FC-CRF, the model introduced in this paper, ensure that the original mining area’s shape and the boundary’s continuity are the best. At the same time, the FCN and U-Net models failed to identify the parts that did not belong to mining areas and were wrongly divided. The method adopted in this paper still obtains a good effect, and there is no messy wrong segmentation area. In addition, the method used in this paper is more detailed and accurate to extract the mining boundary, which is more in line with the real situation of the ground. Proper and comprehensive mining area extraction results are essential for the future application of this study to industrial scenarios [34]. The model adopted in this paper can solve the problem of extracting open-pit mine boundaries from remote sensing images through the above visualization results analysis of mining area prediction results of different models. Therefore, compared to some current models, this model has a strong and comprehensive performance when dealing with this task.

Table 2. Accuracy evaluation of open-pit mining area extraction results for different models.

	MobileNetV3	U-Net	FCN	EMANet and FC-CRF
PA	94.83%	93.18%	95.74%	98.09%
MIoU	74.36%	67.07%	78.04%	89.48%
kappa	67.53%	54.87%	80.56%	88.48%

shows the comparison of evaluation indexes among different models on remote sensing image test data sets. Three evaluation indicators are used to analyze the performance of different models in completing the open-pit mining area segmentation task. EMANet and FC-CRF have the best effect, followed by the FCN model and the MobileNetV3 neural network, and the U-Net model has the worst performance in boundary extraction.

Regarding the values of PA, MIoU, and kappa, EMANet and FC-CRF obtained the highest scores, all about 90%. The FCN performance effect is good, and the accuracy evaluation is more than 80%. The performance of the MobileNetV3 and U-Net network models is not good, and there is a certain gap between the predicted results and the real situation on the ground. They are greatly affected by the strong heterogeneity of open-pit remote sensing images and are prone to misclassification and missing mining areas. The accuracy evaluation value of the prediction result is not high, at approximately 60%. Through the analysis of the prediction results of different models, it can be concluded that the model adopted in this paper has the best effect in extracting open-pit mining areas.

4. Discussion

This section introduces the signal-to-noise ratio (SNR) concept to verify the model’s performance further. By comparing the extraction effect of remote sensing images with different SNRs under the proposed method, the superiority of the model is proved. The results show that the extraction effect of the deep learning model proposed in this paper is not affected by the SNR, and the extraction effect is still excellent for images of different scales. Therefore, the model’s accuracy, generalization, and robustness are proven. At the same time, the limitations of this paper are considered, and the direction of future research that needs to be improved is proposed.

(1)

Effect of signal-to-noise ratio

The SNR refers to the ratio of relevant and irrelevant information displayed (useful and useless information) in the actual task [35]. For this study, useful information refers to the part of the open-pit mining area. In contrast, useless information refers to the part outside the open-pit mining boundary in remote sensing images. Affected by the size and shape of the open-pit mine, the scale of the open-pit mining area in different remote sensing images is not the same [36]. The SNR measures the proportion of productive information in different remote sensing images. The SNR is calculated using a pixel-based approach. It is the ratio of the pixel sum of the mined area to the pixel sum of the non-mined area, where S represents the pixel sum of the mined area, and N represents the pixel sum of the non-mined area. The pixel sum of the mined area and the pixel sum of the non-mined area is calculated based on the ground truth corresponding to the remote sensing image of the open-pit mine. Since ground truth is a black-and-white binary image, its pixel values contain only two numbers—0 and 1. The value of these two-pixel sums can be obtained by using Python language script. The SNR of each open-pit mine remote sensing image can be obtained by substituting it into the formula:

$$\mathrm{SNR}=10\times \lg \left(200\times \frac{\mathrm{S}}{\mathrm{N}}\right).$$

(13)

The higher the SNR, the larger the mining area in the remote sensing image and the smaller the proportion of the noise area. In contrast, the lower the SNR, the smaller the mining area and the higher the proportion of noise area in remote sensing images. The SNR is introduced to evaluate the effectiveness of the EMANet and FC-CRF model in extracting images with different proportions of the mining area. For each image, its SNR is determined. We classify them and summarize the accuracy evaluation metrics of the final prediction results under different SNR intervals.

Figure 8 shows the frequency distribution histogram of the SNR of 200 groups of test data. As seen from Figure 8, it is divided into 10 sections. The data volume of images with high or low SNR is small, and the data with an SNR between 25 and 45 are predominant.

By conducting experiments on the data with different SNRs in these 10 intervals, the results of the accuracy evaluation indexes shown in Figure 9 are obtained. On a macro level, the numerical trend of the three evaluation indexes tends to be stable overall, which indicates that the SNR of remote sensing images has little influence on the application of this model to complete the extraction task. The lowest value of the evaluation index is approximately 80%, the highest value is close to 100%, and the rest are between the two. Experimental results show that the model adopted in this paper can achieve good results for extracting mining areas of remote sensing images with different SNRs. The accuracy of the model will not fluctuate greatly because the SNR is too large or too small. Therefore, the method used in this paper for open-pit mine boundary extraction has strong accuracy, generalization, and robustness.

(2)

Limitations and Potential Improvements

Open-pit mining areas are scattered and have different shapes, and the surrounding environment greatly affects the results [37]. Water bodies, buildings, trees, and so on will cause incorrect identification results [38]. Therefore, extracting the open-pit mining boundary from remote sensing images is a great challenge. The method based on the EMANet model and FC-CRF proposed in this paper has obtained good evaluation results in the extraction work, but there is still space for further progress. The following three limitations can be improved.

First, the mining area is annotated manually in this paper and the personnel’s ability and experience limit the annotation’s accuracy [39]. In addition, the use of manual annotation is time-consuming and laborious. Future research needs to be undertaken in the direction of semisupervised or unsupervised work [40].

Second, the amount of image data adopted in the experiment is not large. The data are from a single source and type [41]. Therefore, acquiring various types of remote sensing images from different satellites is necessary to expand data sources and volume [42]. Using a large amount of diverse data to train and optimize the deep learning network model can enhance the model’s practicality.

Third, the quality and quantity of acquired image data are subject to many limitations, such as clarity, shooting angle, and copyright issues. The parameters of neural network models are more complex, leading to higher hardware requirements. In addition, deep learning models are not highly interpretable, which is also a challenge.

5. Conclusions

Obtaining accurate and timely mining area information on open-pit mines is vital in mine production planning and protecting the surrounding ecological environment. There are problems of low efficiency and poor accuracy by traditional methods. An extraction method is proposed based on deep learning and high-resolution remote sensing images to complete the task more efficiently and accurately. Remote sensing images of the open-pit mine are used as experimental data support. The experimental framework combines the expectation-maximization attention network (EMANet) and the fully connected conditional random field (FC-CRF) algorithm. The EMANet model applies the EM algorithm to construct the EMA mechanism and combines it with the neural network to form an EMA unit (EMAU). The network can capture the correlation within the data, but does not calculate the attention diagram on the whole graph, greatly reducing computational complexity. A postprocessor based on the FC-CRF was adopted to eliminate the misclassified isolated small regions and further improve the extraction accuracy.

The visualization of the prediction results and the accuracy evaluation index of the experiment were used to analyze the final prediction results of the model. PA, MIoU, and kappa values were 98.09%, 89.48%, and 88.48%, respectively. At the same time, a comparison experiment with the MobileNetV3, U-Net, and FCN models was carried out. Compared to the other models, EMANet and FC-CRF retain the area of the mine as much as possible, and the extraction of the detail part is very good. The accuracy evaluation index value of our method is about 90%, which is much higher than the accuracy value of about 70% of the other three models. In addition, the data in different interval signal-to-noise ratios (SNRs) are tested. The accuracy index value of different interval test samples fluctuates in a small range between 80% and 100%. The advantages of the model are proven through experiments and discussion. The model has strong accuracy, generalization, and robustness in extracting open-pit mining areas.

Three aspects need to be addressed in follow-up work. The first is to work towards unsupervised learning to reduce manual labeling labor. The next is the need to improve data quality and enrich data sources. Finally, how to achieve efficient and accurate extraction tasks under limited hardware conditions is a major challenge.