An Automated Approach for Mapping Mining-Induced Fissures Using CNNs and UAS Photogrammetry

Abstract

Understanding the distribution and development patterns of mining-induced fissures is crucial for environmental protection and geological hazard prevention. To address labor-intensive manual inspection, an automated approach leveraging Convolutional Neural Networks (CNNs) and Unmanned Aerial System Photogrammetry (UASP) is proposed for fissure identification and mapping. Initially, the ResNet-50 network was employed for the binary classification of the cropped UASP orthophoto images. A comparative analysis was conducted to determine the optimal model between DeepLabv3+ and U-Net. Subsequently, the identified fissures were mosaicked and spatially projected onto the original orthophoto image, incorporating precise projection data, thereby furnishing a spatial reference for environmental governance. The results indicate a classification accuracy of 93% for the ResNet-50 model, with the U-Net model demonstrating a superior identification performance. Fissure orientation and distribution patterns are influenced by the mining direction, ground position of the mining workface, and topographic undulations. Enhancing the CNN performance can be achieved by incorporating variables such as slope indices, vegetation density, and mining workface locations. Lastly, a remote unmanned approach is proposed for the automated mapping of mining-induced fissures, integrated with UAS automated charging station technology. This study contributes to the advancement of intelligent, labor-saving, and unmanned management approaches advocated by the mining industry, with potential for broad applications in mining environmental protection efforts.

1. Introduction

Coal constitutes a pivotal element within the energy matrices of numerous developing nations, serving as a critical cornerstone for their energy sovereignty [1]. China emerges as a dominant force in the worldwide sphere of coal production and consumption, with a production of 4.56 billion tons of raw coal in 2022, accounted for 56.2% of the total energy consumption [2]. The western provinces of the country, notably Shanxi, Shaanxi, Inner Mongolia, and Xinjiang, possess abundant coal reserves, contributing approximately 80% of the country’s coal production. However, the region is characterized by superficial coal seams, thin bedrock strata, and a fragile ecological framework. The prevalent practice of extensive longwall mining in this area inflicts significant environmental damage, including ground subsidence and fissures, due to the inherent instability and vulnerability of the overlying strata [3]. Mining-induced ground fissures can lead to the rapid depletion of shallow water resources, methane releases, subterranean coal combustion, desertification, and environmental degradation, significantly increasing landslide risks [4,5,6,7]. These fissures present a significant risk to the safety and assets of local communities, adversely affecting the stability of surrounding infrastructure and the productivity of agricultural and grazing lands [8,9]. Typical mining-induced ground fissures are illustrated in Figure 1a,b. Therefore, the timely identification and precise mapping of mining-induced ground fissures are crucial for preventing geological hazards and preserving the ecological environment. Currently, conventional fissure identification in mining sites involves manual field inspections using total stations, Real-Time Kinematic (RTK) devices, and traditional mapping techniques to accurately locate and record fissure dimensions, as shown in Figure 1c. However, the affected area may extend over several kilometers, characterized by complex and varied terrains, with the subsidence zones presenting potential safety risks. These manual inspections face challenges such as prolonged durations, low efficiency, increased costs, and terrain-imposed constraints, resulting in coverage gaps [10,11]. Moreover, most fissures are less than 1 m wide, presenting challenges in their identification through satellite imagery, including delayed detection and the failure to recognize nascent fissures. Consequently, this impedes timely intervention and the implementation of measures to mitigate fissure progression.

In recent decades, the mining industry has witnessed a significant expansion in the applications of Unmanned Aerial System (UAS) technology [12,13,14]. UASs offer several advantages including maneuverability, high precision, extensive coverage, autonomous operation, and the capability to traverse diverse terrains [15,16]. Currently, techniques for analyzing UAS-captured fissure imagery encompass manual interpretation, thresholding, edge detection, and machine learning approaches [17]. Manual interpretation relies on engineering experience to draw fissure distribution maps, with results significantly influenced by subjective factors, which complicates large-scale data analysis. Thresholding is a straightforward image processing technique that segments images into fissure and non-fissure regions by setting a threshold. Despite its ease of use and rapid computation, it remains prone to inaccuracies due to variable lighting and complex backgrounds. Edge detection techniques detect fissures by identifying areas with significant changes in brightness, with common algorithms including Sobel and Canny’s [18]. Edge detection techniques can effectively identify the boundaries of fissures, but their effectiveness is compromised in noisy environments or when the edges of fissures are not clearly defined. Machine learning methods such as Support Vector Machines (SVMs) and Random Forests operate by learning the characteristics of fissures from a large volume of labeled data for fissure detection. Compared to traditional image processing techniques, these methods offer better adaptability and accuracy but require extensive labeled data for training and suffer from limited model generalization capabilities. Convolutional Neural Networks (CNNs) have become the cutting-edge approaches for fissure detection [19,20,21]. CNNs demonstrate superior accuracy and robustness, making them particularly suitable for processing large-scale and complex UAS imagery data [22,23,24]. This approach demands substantial computational resources and extensive training datasets, where the former can be addressed with high-performance computing, and the latter can be fulfilled through the periodic acquisition of high-resolution UAS photogrammetric Digital Orthomosaic Maps (DOMs) [25]. Ji et al. [26] used the DeepLabv3+ semantic segmentation model for detecting road cracks. The real size of the crack is calculated using the equation for the pixel–physical conversion coefficient, based on the height and pixel statistics of the detected crack. This approach facilitates the assessment of road risks and the planning of road maintenance. Jiang et al. [27] introduced MFPA-Net (Multi-scale Feature Pyramids and Attention Network) model for the detection of ground fissures in UAS images of coal mining areas. The model demonstrated a commendable performance, achieving a high precision rate of 69.4%, recall rate of 70.7%, F1-score of 70.0%, and a mean intersection over union (MIoU) of 75.1%.

Previous studies primarily concentrated on detecting fissures in isolated UAS images or video footage, posing significant challenges in comprehensively capturing all fissure-related data across mining areas. Additionally, the fissure identifications often lack detailed and precise information regarding their geographical locations, widths, and lengths. Here, we propose a novel approach utilizing CNNs to identify mining-induced ground fissures from UAS photogrammetric DOMs, enabling the efficient identification and marking of the entire mining impact area. This method provides crucial information on fissure locations, widths, lengths, and orientations within a real coordinate system, thus offering a viable strategy for operators to swiftly and accurately map fissures and implement remedial measures. A case study was conducted at the 40102 workface of the Gaojialiang Coal Mine in the Wanli Mining Area, Inner Mongolia, China.

In summary, this paper introduces a procedure for mapping mining-induced ground fissures from UAS photogrammetric DOMs using CNN models. Demonstrated through a case study, the core processes and novel contributions of this work include the following:

(1)

The initial cropping of a DOM of 40102 mining workface ground surface into nine medium-sized regional images of 288 m × 276 m, which are then further divided into 19,872 sub-regional images of 6 m × 6 m for detailed fissure identification. Each image was meticulously numbered to maintain geographic coordinate information.

(2)

The preliminary classification of cropped sub-regional images into two categories was performed using the ResNet-50 classification network, thereby enhancing computational efficiency for subsequent fissure detection and delineation steps. The evaluation of DeepLabv3+ and U-Net determined U-Net as the optimal model for fissure identifying.

(3)

The reconstruction of the study area and conducting fissure mapping through image mosaic and projection. The analysis of the relationship between the fissure distribution patterns and the mining activities and topographical features.

(4)

The proposal of a novel approach for the remote, unmanned monitoring of mining-induced ground fissures, integrating UAS automatic charging technology.

2. Methods

2.1. UAS Photogrammetry

UAS Photogrammetry (UASP) is a technology that involves a visible-light camera-equipped UAS to capture high-frequency and high-resolution aerial images of surface targets. These images are processed and analyzed using image processing and measurement techniques to produce remote sensing data, such as Digital Surface Models (DSMs) and Digital Orthophoto Maps (DOMs). The fundamental algorithms and steps of this technology include the following:

(1)

Structure from Motion (SfM): This algorithm recovers the motion and structure of UAS-captured images, accurately estimating the aerial position and attitude, resulting in high-quality sparse point cloud data.

(2)

Dense Matching: Using Semi-Global Matching (SGM) algorithms, dense matching is performed, which generates depth maps and fuses them to create dense point clouds. This step serves as the foundation for subsequent DSM generation and DOM correction.

(3)

DSM and DOM Production: DSM and DOM products are generated through point cloud interpolation and texture mapping techniques.

2.2. ResNet-50 Network for Image Classification

ResNet-50 was proposed by Microsoft Research Asia in 2015 and belongs to the ResNet (Residual Network) family. It was specifically designed to address the issue of vanishing and exploding gradients that arise when using deeper traditional neural networks. It excels at extracting image features even when the model has a substantial depth [28].

ResNet-50 is composed of 50 convolutional layers, comprising 49 convolutional layers and 1 fully connected layer. Its structure can be divided into seven parts, as depicted in Figure 2. In Part I, the input image undergoes convolution, regularization, a ReLU activation function, and max pooling operations. These computations yield the feature information of the input image, double the number of feature channels, and halve the size of the input image. Parts II, III, IV, and V consist of residual blocks comprising Convolutional (Conv) Blocks and Identity Blocks. Conv Blocks alter the dimension of the residual block, while preserving its size. Identity Blocks deepen the network and enable it to learn additional features. Starting with an image input of size 224 × 224 × 3, the first five parts of the convolutional calculations yield an output of 7 × 7 × 2048. The pooling layer in Part VI converts this output into a feature vector, and finally, Part VII’s classifier computes the feature vector and outputs the class probability. In the ResNet-50 network structure, each residual block comprises 3 convolutional layers. Thus, the network is comprised of 1 + 3 × (3 + 4 + 6 + 3) = 49 convolutional layers. Including the final fully connected layer, the ResNet-50 architecture consists of a total of 50 layers.

2.3. DeepLabv3+ Model

Introduced by Google researchers in 2018 [29], DeepLabv3+ is a state-of-the-art CNN model specifically designed for semantic segmentation tasks and represents the latest upgraded version of the DeepLab series models. It has achieved remarkable performances and found wide application in the domain of semantic segmentation [30,31,32,33]. As depicted in Figure 3, DeepLabv3+ comprises two primary components, an encoder and decoder [34]. On the network side, the encoder employs a MobileNet model as its backbone network and combines it with the ASPP (Atrous Spatial Pyramid Pooling) module for effective feature extraction. The feature maps from the MobileNet backbone network are split into two branches. One branch provides the feature map output from the backbone network’s last layer, which is sent to the ASPP module. The other branch offers the shallow feature maps from the backbone network’s middle layer, which are forwarded to the decoder module. In the decoder module, the shallow feature maps are integrated with the high-level semantic features generated by the ASPP module. The convolutional convolutions or up-sampling operations ensure that the two input feature maps have the same size. These maps are then concatenated and passed through a series of 3 × 3 convolutional blocks for further processing. Finally, the output segmentation map, with the same resolution as the original image, is obtained through up-sampling using linear interpolation. The DeepLabv3+ model, equipped with a decoder module and optimized along the image boundary, excels in producing superior segmentation results. This characteristic makes it particularly well suited for the identification of mining-induced fissures.

2.4. U-Net Model

U-Net [35] is a popular semantic segmentation model based on a full CNN network. It was initially widely used in the field of biomedical image segmentation and later applied to remote sensing image segmentation. The U-Net architecture employs an encoder–decoder framework to reconstruct the original image. As illustrated in Figure 4, the left contracting path of the U-shaped network involves stacking, convolving, and pooling the input images to increase the number of output feature channels and capture essential features from the original image. On the right side of the expansive path, deconvolution operations are utilized to gradually restore the image size. Additionally, after each deconvolution operation, the resulting image is combined with the feature map extracted from the left contracting path to restore image features. This allows the entire network to leverage the encoder and generate an image consistent with the original one. Biomedical images often have limited background information, and mining-induced fissure images usually require only binary classification based on the presence or absence of fissures. The semantic information in both domains is relatively straightforward. The reason to use the U-Net model is rooted in its suitability for handling simple semantic information and its effectiveness in biomedical and fissure detection tasks.

2.5. Model Dataset Establishment and Configuration

This study utilized the PyTorch deep learning framework on a Windows 11 system for conducting the experiments. The training and testing of the models were performed within the PyCharm integrated development environment. The computational resources consisted of an AMD R7-6800H CPU (Advanced Micro Devices, Inc., Santa Clara, CA, USA) and an RTX 3050 Ti GPU (NVIDIA Corporation, Santa Clara, CA, USA). To train and test the DeepLabv3+ and U-Net semantic segmentation models and achieve optimal results, it was necessary to establish a dataset. For this study, a total of 500 fissure images were selected. These images were then annotated using the Labelme image annotation software. Following manual visual annotation, the images were converted into binary mask label files to facilitate the training process. Next, the objective was to obtain the ideal model parameters. The CNNs training was divided into two stages: the freezing stage and the unfreezing stage, following the principles of transfer learning. During the freezing stage, the feature extraction backbone network of the model was kept frozen, allowing for resource efficiency and space conservation. Only the network parameters required fine-tuning. This strategy not only accelerated the training process but also preserved the weight parameters of the backbone network. Conversely, in the unfreezing stage, the feature extraction backbone network was unfrozen and actively utilized during training, resulting in adjustments to the entire model’s parameters. This approach enhanced the model’s suitability for the specific task at hand. To optimize the training accuracy and align with computational capabilities, comparative tests were conducted to determine the optimal settings. During the freezing stage, the training process consisted of 50 epochs, with a batch size of 8. The maximum learning rate was set to 0.007, and the minimum learning rate was 0.007 multiplied by 0.01. In the subsequent unfreezing stage, the model’s parameters were adjusted over a span of 360 epochs, with a batch size of 4. The maximum and minimum learning rate remained constant.

2.6. Research Procedure

The research procedure, as illustrated in Figure 5, consisted of the following five steps:

(1)

An aerial survey of the mining workface ground was conducted using a UAS equipped with a visible-light camera to obtain workface ground surface images. The images were then processed to generate DOMs.

(2)

A study area was selected, and its DOM was cropped into uniformly sized images. Each image was meticulously numbered to retain its original positional coordinate information.

(3)

A CNN classification network was utilized to categorize the cropped images into fissure and non-fissure images.

(4)

A training dataset was created to obtain the optimal model parameters. The CNN models were trained on fissure images and then compared to determine the better-performing model in mapping mining-induced fissures.

(5)

The optimal CNN model was employed to map fissures in multi-temporal data. The study area was reconstructed through an image mosaic and projection, enabling fissure mapping and development pattern analyses.

3. Engineering Background and Data Processing

3.1. Engineering Background

The study area was located in the southern part of the Wanli Mining Area, Ordos, Inner Mongolia, Western China, specifically in the Gaojialiang coal mine 40102 mining workface, as illustrated in Figure 6a. The workface extends for a total strike length of 3286 m and a total dip length of 291 m. The coal seam has an average depth of 234.25 m, and the average thickness is 3.77 m. The extraction of the 40102 workface occurred from April 2021 to February 2022. On the eastern side, adjacent to the 40102 workface, is the previously extracted 40101 workface, with a strike length of 2543 m and a dip length of 301 m, mined from April 2020 to February 2021. Above the 40102 workface, there is a goaf area from the 20117 workface, with a strike length of 894 m and a dip length of 301 m, mined from February 2017 to July 2017. The interlayer spacing between these two coal seams is approximately 87 m (with a coal seam average depth of 147.25 m). A rectangular area measuring 864 m × 828 m was selected as the study area for fissure analysis, as shown in Figure 6b,c, representing a typical region influenced by the three workfaces of 40101, 40102, and 20117.

The region is characterized by a semi-arid continental climate typical of the temperate zone. The topography of the region is undulating, featuring numerous loess gullies and sparse vegetation, making it ecologically fragile. Unfortunately, the extensive coal mining activities carried out in this area have exacerbated the degradation of the ecological environment, leading to diverse geological and environmental issues. Notably, ground subsidence and fissures have emerged as significant geological hazards. To address these risks, previous approaches employed manual driving cars to inspect mining-induced fissures, along with RTK devices to determine their precise locations. In analyzing the formation patterns of the fissures and their characteristics, appropriate treatment measures could be implemented.

3.2. UAS Data Acquisition

The DJI Matrice 300 RTK quadcopter UAS, equipped with a DJI Zenmuse P1 full-frame visible-light camera, was employed for UASP. The camera possesses an effective pixel count of 45 million, a pixel size of 4.4 µm, a fixed focal length of 35 mm, and a field of view of 84°. The UAS was operated at a flight altitude of 120 m, with an 80% forward overlap rate and a 70% side overlap rate, resulting in an average ground sampling distance (GSD) of approximately 1.5 cm/pixel. Subsequent to the image capture, the photos were reconstructed to generate DOMs and DSMs.

UAS surveying for the ground surface of 40102 workface was conducted on four different occasions: 17 August 2021 (referred to as Stage I), 25 October 2021 (referred to as Stage II), 11 January 2022 (referred to as Stage III), and 4 March 2023 (referred to as Stage IV). The specific dates and mining locations associated with the UAS data collection are provided in

Table 1. UAS data acquisition information.

Name	Acquisition Date	Interval Days	Distance to the Stopping Line	Number of Images	Number of GCPs
Stage I	17 August 2021	-	1895 m	1675	20
Stage II	25 October 2021	69	1173.3 m	2921	6
Stage III	11 January 2022	78	412.5 m	6597	30
Stage IV	4 March 2023	417	-	5367	30

. This study comprised four distinct phases, with each pair of phases corresponding to a specific stage of mining progress. The mining distances and their spatial relationship to the research area are illustrated in Figure 6b. The mining operations in the 401012 workface were completed on 23 February 2022. The Stage IV data represent an aerial survey conducted approximately 13 months after the cessation of mining, when the ground had achieved a state of sufficient extraction. The timeline of the workfaces mining and UAS surveying is illustrated in Figure 7.

3.3. DOM Cropping

Cropping the acquired high-resolution DOM can effectively reduce the complexity of background information in the UAS images. The resultant cropped fissure images can then be extracted and assembled to enhance the fissure detection accuracy [36], while concurrently reducing the time and hardware costs associated with fissure detection. This strategy prevents computational challenges in image recognition caused by the large dimensions of high-resolution DOM images. Initially, the DOM was processed using ArcGIS 10.6 software to crop out the fissure research area. Subsequently, the fissure research area was partitioned into nine medium-sized regions measuring 288 m × 276 m each. The further division of each medium-sized region yielded 2208 sub-regions, resulting in a total of 19,872 sub-regions that were systematically numbered for detection purposes. The resolution of each sub-region image was set at 400 × 400, corresponding to an actual size of 6 m × 6 m.

3.4. Image Mosaics

The identified fissures, obtained through an optimal CNN model, were mosaicked in sequence following the order of image cropping. This mosaic process was carried out using the paste function in the PIL (Python Imaging Library) within the Python platform. Each medium-sized image was divided into a grid consisting of 46 rows and 48 columns.

During the mosaicking, the starting point was defined as the bottom-left corner, while the endpoint was set as the top-right corner of the grid. The pixel values of each small image remained unchanged, maintaining a resolution of $400\times 400$ and consisting of 3 spectral bands with a data type of uint8. Through this process, the final medium-sized region image was generated.

However, after the semantic segmentation, the projection information of the fissure images was lost. To address this issue, the fissures were spatially matched with the original images using ArcGIS software. This matching process retrieves the projection information and enables the completion of the image concatenation accurately within the research area.

4. Results and Discussion

4.1. Image Classification Results

The ResNet-50 classification network was trained using a dataset of 700 training samples. Once trained, the network was utilized to perform binary classification on fissure images within the research area, determining whether fissures were present or absent. To evaluate the model’s classification performance, the visual interpretation of the images was conducted based on field surveys. After validating the model using 20 validation samples, the ResNet-50 classification network achieved a respectable accuracy of 93%. Following manual identification, it was observed that a few misclassified images resulted from interference caused by ground vegetation or gullies (as depicted in Figure 8), while some images containing small-scale fissures (width less than 0.1 m) were missed. Nevertheless, the low error rates of misidentification and omission do not compromise the primary objective of identifying large- and medium-sized fissures in engineering practice. Hence, this level of accuracy satisfies the requirements for fissure image classification. Subsequently, the trained ResNet-50 classification network was applied to classify a total of 19,872 small images obtained through the fissure cropping. The classification yielded 2082 fissure images and 17,790 non-fissure images based on the model’s predictions. The results indicate that the ResNet-50 classification network exhibits a relatively low number of misclassifications and omissions on fissure images. Analyzing the distinctive features of misclassified and missed images can contribute to further enhancing the accuracy of the classification network. Figure 8 illustrates several examples of misclassified images. The classification errors can be attributed to the complex nature of surface attachments and the non-uniform surface illumination resulting from weather conditions. Specifically, misclassification occurred when features such as withered tree branches and collapsed areas possessed grayscale values and a morphology similar to mining-induced fissures. Additionally, factors such as tree vegetation, mountains, and shadows generated by lighting angles can influence misclassification. To mitigate these issues, several strategies can be implemented. Firstly, flight missions should be scheduled during appropriate time intervals, such as two hours before or after noon, to ensure favorable lighting and weather conditions. Secondly, the DSM elevation information can be utilized to alleviate the impact of shadows in regions with significant elevation variations, such as gullies. Lastly, the manual selection of misclassified images through visual interpretation can be employed.

4.2. Comparison of Fissure Identification Results

Six representative fissure samples were selected from the training dataset, including single fissures, as shown in Figure 9(a1,b1); parallel fissure clusters, as shown in Figure 9(c1,d1); and intersecting fissure groups, as shown in Figure 9(e1,f1). The performances of the two CNN models underwent separate evaluations. The results show that the U-Net network model exhibited strong recognition capabilities for mining-induced fissures, accurately discerning small fissures wider than 0.05 m and intersecting fissure clusters, as shown in Figure 9(a3–f3). However, the DeepLabv3+ network model exhibited issues with fractured fissure identification, evident from multiple interruptions and missing segments (Figure 9(a4–f4)), particularly struggling with fissures narrower than 0.1 m. The vast research area and complex background introduced instances of omission or erroneous extraction in the recognition process for both CNN models.

To quantitatively evaluate the accuracy of the two groups of CNN models in fissure recognition and detection, the accuracy evaluation metrics used included the Mean Pixel Accuracy (MPA), MIoU, and F1-score. The MPA represents the average probability of correctly classifying pixels within the fissure images. The MIoU is the average ratio of the intersection over the union between the predicted results and the actual value for each class. The F1-score is the weighted average of precision and recall, where a higher value indicates a better model performance.

$$MPA=\frac{1}{k+1}\sum _{i=0}^k\frac{P_{ii}}{{\displaystyle \sum _{j=0}^k}{p}_{ij}}$$

(1)

$$MIOU=\frac{1}{k+1}\sum _{i=0}^k\frac{P_{ii}}{{\displaystyle \sum _{j=0}^k}{p}_{ij}+{\displaystyle \sum _{j=0}^k}{p}_{ji}-p_{ii}}$$

(2)

$$Precision=\frac{TP}{TP+FP}$$

(3)

$$Recall=\frac{TP}{TP+FN}$$

(4)

$$F1=\frac{2Precision\times Recall}{Precision+Recall}$$

(5)

In the above formulas, i represents the actual value, j represents the prediction, and $k+1$ represents the number of categories. $TP$ stands for the number of “fissures” correctly recognized by the model, $TN$ represents the number of correctly recognized “non-fissures”, $FP$ is the count of “non-fissures” incorrectly identified as “fissures”, and $FN$ is the count of “fissures” incorrectly identified as “non-fissures”. When i is predicted as i, it is counted as a $TP$. When i is predicted as j, it is counted as an $FN$. When j is predicted as i, it is counted as an $FP$. In the same dataset, a comparison was made between the DeepLabv3+ and U-Net algorithms, and a comparison of various metrics is shown in

Table 2. Results of different CNN algorithms.

Algorithms	MPA/%	MIoU/%	F1/%
DeepLabv3+	89.01	82.76	89.83
U-Net	90.73	83.98	90.67

.

Based on the proposed evaluation metrics, U-Net outperforms DeepLabv3+ in all three metrics. In terms of the MPA, U-Net shows a 1.72% improvement over DeepLabv3+. In the MIoU metric, U-Net and DeepLabv3+ differ by 1.22%. In the F1-score, U-Net exhibits a 0.84% improvement over DeepLabv3+.

In comparison to the edge detection and machine learning methods, CNNs have emerged as a superior approach in fissure identification [20,37,38]. Edge detection methods have been widely used in the identification of mining-induced fissures [17,18]. These methods rely on image pixel intensity ranges to extract fissures and depend on the manual selection and adjustment of edge detection operators, which can lead to an unstable recognition performance, especially with complex ground fissure images. In our case study, due to the spectral similarity between surface fissures and dry vegetation, these methods resulted in misidentification and a low accuracy. Machine learning methods have also demonstrated promising performances in fissure extraction from UAS images [36]. However, these methods often require the manual selection of hyperparameters, such as penalty parameters and kernel functions. In contrast, Convolutional Neural Networks (CNNs) can automatically learn and optimize parameters during training, simplifying the model development process and reducing the need for extensive manual parameter tuning. CNNs can be customized and optimized for specific tasks through adjustments in network architecture, activation functions, and regularization techniques, enhancing the classification accuracy and robustness. Furthermore, CNNs extract rich features from raw image data through multiple convolutional and pooling operations, eliminating the need for manual feature engineering, and improving the model’s generalization ability to unseen data.

Overall, regarding the recognition effectiveness of mining-induced fissures in mining areas, the U-Net network model demonstrated a superior performance compared to DeepLabv3+, with higher values for the MPA, MIoU, and F1, indicating a better segmentation accuracy. Therefore, the proposed method of mining-induced fissure recognition using the U-Net network model holds potential for application in similar ground terrain and landform conditions.

4.3. Analysis of Fissure Distribution and Evolutionary Patterns

To characterize the distribution pattern of fissures, the mining process from Stage II to Stage III of the 40102 workface, which had a significant impact on the selected study area, was selected for analysis, as depicted in Figure 10. As observed from the figure, the fissures are primarily distributed within the ground projection range along the boundaries of the workfaces of 40101, 40102, and 20117. Figure 10 illustrates seven densely concentrated fissure areas, designated as F1 to F7.

4.3.1. Fissure Distribution Patterns on the Mining Workface Boundary

The F1 fissure group is located above the goaf on the west side of the 20117 workface. As depicted in Figure 11, before the Stage IV phase, the subsidence caused by the mining of the 40102 workface had not affected this area; hence, no fissures were observed. The mining of the 20117 workface was completed in July 2017, and it is plausible that fissures in the F1 area, induced by the influence of the old goaf of the 20117 workface before Stage IV, may have already closed. During Stage IV, influenced by the extensive subsidence basin formed by the mining of the 40102 workface and the subsidence on the west side of the 40103 workface, the F1 area experienced subsidence-induced fissures parallel to the direction of the mining face retreat. The fissures predominantly exhibited a linear form, extending parallel to the direction of the mining face retreat, with lengths reaching several hundred meters and an average spacing of about 5 m. The timing of fissure appearance is closely related to the formation of the subsidence basin caused by mining. When monitoring mining-induced fissures, it is suggested to focus on the progress of mining and the timing patterns of ground subsidence. As shown in Figure 11, during Stage I and Stage II, no fissures were observed in the F2 fissure group area. In Stage III, approximately five new fissures appeared in the F2 area, with the longest fissure reaching up to a hundred meters and a width of around 50 cm. These fissures exhibited a parallel distribution along the outer side of the goaf on the west side of the 40102 workface. The F2 area is situated on the outer edge of the subsidence basin during Stage III mining, and the ground continuously underwent stretching deformation with intense fissure development. During Stage IV, a noticeable closure phenomenon was observed in the fissure group. It is speculated that fissure groups above the workface boundary exhibit closure phenomena, suggesting minimal long-term environmental impact. Furthermore, both the F1 and F2 fissure groups, along with the western parts of the F6 and F7 fissure groups, exhibit a parallel and spaced distribution. They develop parallel to the direction of mining face retreat, positioned on both sides of the goaf boundary. The observed phenomena align with the conclusions of previous studies [39], indicating that fissures with a parallel distribution to the mining face are found on the outer sides of goafs on both sides of the workface goaf boundary.

4.3.2. Fissure Distribution Patterns on the Workface Center and Old Goaf Areas

The F3, F4, and F5 fissure groups are located within the ground projection range of the 40102 workface center. The orientations of these fissures are generally perpendicular to the direction of the workface retreat, exhibiting a slightly arcuate and parallel space distribution. The arc centers are all directed toward the downhill side of the slope, i.e., in the direction of significant elevation decrease, as illustrated in Figure 12. The maximum elevation differences within the F3, F4, and F5 fissure group areas were approximately 18 m, 36 m, and 18 m, respectively. It is noteworthy that in the northeast direction of the F3 and F4 fissure groups and the south direction of the F5 fissure group, the elevation change in the ground is within the range of 3–9 m, and fissures are sparsely distributed in these areas. Consequently, it can be inferred that the occurrence or duration of fissures is closely related to the ground topography [4]. Mining in areas with significant elevation differences and slopes facing toward the center of the subsidence basin can lead to the formation of extensive spaced fissure groups.

The workface advancement can be divided into the upslope stage, crest advancement stage, and downslope stage, based on the characteristics of the ground valley slope and the direction of workface advancement [40]. During the upslope stage, the ground slope undergoes stretching failure zones, and the roof rock layer fracture failure zones are prone to penetration, making it easier to form dynamic ground pressure and ground fissures [41]. Yan [42] pointed out that the upslope positions at the workface boundary and in-face locations (slope direction pointing toward the center of the subsidence basin) exhibit more pronounced ground deformation and subsidence characteristics.

Regarding the direction of fissure distribution, Hou et al. [39] suggested that internal fissures within the workface exhibit a parallel combination distribution, forming an O-shaped fissure pattern with boundary fissures (open-off cut fissures, goaf fissures, and stop-line fissures). They believe that the development characteristics are positively correlated with mining height and terrain topography. The F5 fissure group represents boundary fissures at the open-off cut of the 20117 workface, with the arc center facing toward the goaf of the 20117 workface. The secondary mining of coal in the 40102 workface under the old goaf area increased the fissure widths, while the direction of the fissure distribution remains unchanged. The eastern part of the F7 fissure group is similar to the situation with F5, with the arc center facing toward the old goaf of the 40101 workface. The F6 fissure group is presumed to be fissures in the goaf roadway; however, due to the significant and fluctuating ground topography (elevation difference greater than 30 m), there is a phenomenon where the overall arc center direction faces toward the downslope of the slope.

The eastern part of the F6 and F7 fissure groups exhibits an arcuate distribution, with the overall arc center direction facing toward the downslope of the slope, as illustrated in Figure 12, where the elevation significantly decreases according to the DSM. The elevation differences within the fissure group areas are as high as 32 m and 17 m, respectively. Due to the combined effects of the goaf areas of the 40101, 40102, and 20117 workfaces, the F6 and F7 fissure groups are highly developed. During Stage III, the width of the fissures exceeds 3.3 m, which may lead to air leakage and coal self-ignition hazards, posing a potential impact on grazing activities and the movement of wildlife. In an ideal scenario for the sustainable coal development, necessary measures, such as backfill mining, should be implemented during multi-seam high-intensity mining to prevent the occurrence of such large fissures.

In conclusion, the orientation and distribution pattern of fissures are not only influenced by the mining direction and ground position but are also closely associated with the undulations in ground topography. Mine operators should delineate key locations for the occurrence of fissure groups based on these influencing factors, particularly where there are significant changes in ground surface topography, at boundary roadways, and areas affected by repetitive mining in multiple coal seams. This proactive approach helps in avoiding geological and environmental hazards. Additionally, future research could enhance the depth of the learning models by incorporating variables such as changes in DSM and terrain slope indices. This can contribute to reducing errors in fissure identification in aerial imagery and improve the overall applicability of the methodology.

4.3.3. Fissure Evolutionary Patterns on the Workface Central Area

The fissure areas of the F3, F4, and F5 fissure groups at the corresponding locations on the ground surface of the central part of the 40102 workface were calculated separately. The results are shown in Figure 13. In Stage I, influenced by the old goaf formed by the 20117 workface directly above the 40102 workface, the F3, F4, and F5 fissure groups had already appeared, with fissure areas of 122 m², 157 m², and 36 m², respectively. The maximum elevation difference within the F4 fissure group area is larger than that of F3, indicating more intense terrain variations. Therefore, the fissure width and area of F4 are larger than those of F3. In Stage II, the 40102 workface had not yet reached the study area, and the variation in fissure areas was less than 10%. This suggests that the expansion of the already formed ground fissure areas may be attributed to factors such as precipitation and weathering. In Stage III, the advancement of the 40102 workface exceeded the study area, causing a significant expansion of the subsidence basin. This resulted in considerable increases in the fissure areas by 58%, 75%, and 45% for the F3, F4, and F5 fissure groups, respectively. The F4 fissure group, characterized by the most intense terrain changes (maximum elevation difference of 36 m), exhibited the highest rate of area growth. In Stage IV, which occurred 13 months after the completion of mining in this workface, the development of ground fissures reached a stable stage. At this point, the variation in fissure areas was 12.8%, 15%, and 6%, respectively. This could be attributed to the combined effects of minor subsidence due to the concurrent mining activities of the adjacent 40103 workface to the west, as well as influences from precipitation and weathering.

4.3.4. Preventive and Remedial Measures for Mining-Induced Fissures

Researchers suggest that slope instability, landslides, and soil erosion are related to the activity of ground fissures [43,44,45]. Reducing and controlling mining-induced fissures is crucial for protecting the mining site environment, especially in the gully terrains [46]. In the design process of mining workfaces, it is imperative to avoid terrains with significant disparities and valley configurations in the direction of mining workface advancement. Palchik [47] determined, through in situ borehole measurements, that fissure width is associated with fissure location, the thickness of the extracted coal seam, key strata, and the rock layers above and below the fracture zone. To control subsidence rates, it is advisable to minimize upslope mining, accelerate the pace speed of workface advancement in valley terrains, and implement technological measures such as height restriction mining, pillarless mining, the coordinated mining of coal seam groups, and backfill mining. These measures are expected to contribute to reducing the occurrence of fissure groups [48,49]. Hou et al. [39] proposed an integrated approach focusing on fissure zoning, soil reconstruction, and overall fissure filling for different terrain types. They suggested methods such as laying cushion layers, impermeable layers, and sealing layers to ensure fissure filling, waterproofing, and strengthening effects. Additionally, techniques like micro-terrain modification were employed for fissure remediation. These approaches aim to restore vegetation, reduce soil erosion, and have shown positive results in practical applications. In the practice of the case study mine site, the crawler excavators are used to excavate soil in areas with dense fissures. The excavated soil is then used to fill large-scale ground fissures with a width greater than 0.4 m, and grass seeds are spread after smoothing to achieve fissure remedial and prevent geological disasters to a certain extent, as shown in Figure 14.

4.4. Prospect of a Remote Unmanned Approach for Mapping Mining-Induced Fissures

By introducing the unmanned UAS charging and data upload station technology, the advantages of employing UASs for the remote and unmanned mapping of mining-induced fissures become increasingly promising. A schematic of this remote unmanned approach is illustrated in Figure 15. UASs follow pre-established flight paths to conduct inspections of active workface ground. Upon returning, UASs utilize charging and data upload stations installed in nearby stable ground surface to recharge their batteries and upload imagery data. The remote data processing center then receives the transmitted data, and the CNN model developed in this study would then be employed to swiftly annotate the orientation of the fissures. Ultimately, information such as the location, length, and width of the fissures is visualized and pushed to mining operators to facilitate decision making and implement necessary measures. In the future, we can enhance the CNN model’s accuracy from two perspectives: (1) reducing the probability of identifying fissures in areas with steep terrain or dense vegetation and (2) increasing the probability of identifying fissures in upslopes, downslopes, or workface boundary areas. This procedure will guide mining operators in adjusting excavation plans and mitigating geological and environmental hazards such as air leakage, landslides, and soil erosion caused by penetrating fissures. Compared to traditional manual inspection methods, the anticipated benefits include avoiding potential safety risks, overcoming blind spots, labor-saving, and improving operational efficiency.

5. Conclusions

This study proposes an approach for mapping mining-induced fissures from orthophoto imagery using UASP and CNNs. In employing a case study in the Gaojialiang coal mine in Ordos, Inner Mongolia, Western China, UASP was used to collect and reconstruct orthophoto imagery. The ResNet-50 classification network was then employed to perform binary classification on the cropped UASP orthophoto imagery within the study area, enabling the mapping of fissure images. Subsequently, the effectiveness and accuracy of the DeepLabv3+ and U-Net algorithms were compared, and the distribution patterns of mining-induced fissures were investigated. The main conclusions are as follows:

(1)

The results from the case study demonstrate that the ResNet-50 network achieves a classification accuracy of 93%, meeting image classification requirements. Among the two CNN models, the U-Net exhibits a superior overall recognition, with segmentation evaluation metrics superior to DeepLabv3+. The MPA reaches 90.73%, the MIoU reaches 83.98%, and the F1-score reaches 90.67%. The U-Net network model is deemed more suitable for mining-induced fissure recognition in conditions similar to those of the mining area.

(2)

Observations indicate that fissures parallel to the mining direction are distributed on both sides of the workface, outside the goaf. Fissure orientation and distribution patterns are influenced not only by mining direction and ground position but also closely associated with topographical undulations. Mining operators are advised to identify key locations for fissure occurrence based on these influencing factors, particularly where significant changes in ground topography, boundary roadways, and areas affected by repetitive mining in multiple coal seams are evident. This proactive approach helps in avoiding geological and environmental hazards. Future research endeavors could enhance the CNN performance by incorporating variables such as terrain slope indices, vegetation density, and the corresponding location of mining workfaces, thereby reducing mapping errors in fissures and enhancing overall CNN applicability.

(3)

A remote unmanned approach for mapping mining-induced fissures is proposed, integrated with UAS automated charging and data uploading station technology. The anticipated benefits include avoiding potential safety risks, overcoming inspection blind spots, and improving operational efficiency compared to manual inspection. This approach enables mining operators to visually understand ground damage during the mining process, adjust excavation plans, and implement effective measures to address fissures. It aims to prevent geological and environmental hazards such as air leakage through penetrating fissures, landslides, and soil erosion. The findings can contribute to realizing the mining industry’s current advocacy for intelligent, labor-saving, and unmanned management approaches. It is recommended to implement this approach in mines with comparable conditions.