Using Improved Edge Detection Method to Detect Mining-Induced Ground Fissures Identified by Unmanned Aerial Vehicle Remote Sensing

Abstract

Information on the ground fissures induced by coal mining is important to the safety of coal mine production and the management of environment in the mining area. In order to identify these fissures timely and accurately, a new method was proposed in the present paper, which is based on an unmanned aerial vehicle (UAV) equipped with a visible light camera and an infrared camera. According to such equipment, edge detection technology was used to detect mining-induced ground fissures. Field experiments show high efficiency of the UAV in monitoring the mining-induced ground fissures. Furthermore, a reasonable time period between 3:00 am and 5:00 am under the studied conditions helps UAV infrared remote sensing identify fissures preferably. The Roberts operator, Sobel operator, Prewitt operator, Canny operator and Laplacian operator were tested to detect the fissures in the visible image, infrared image and fused image. An improved edge detection method was proposed which based on the Laplacian of Gaussian, Canny and mathematical morphology operators. The peak signal-to-noise rate, effective edge rate, Pratt’s figure of merit and F-measure indicated that the proposed method was superior to the other methods. In addition, the fissures in infrared images at different times can be accurately detected by the proposed method except at 7:00 am, 1:00 pm and 3:00 pm.

1. Introduction

It’s important to collect the information on mining-induced ground fissures quickly, timely and accurately for the safety of coal mine production and comprehensive management of the environment in the mining area. In recent years, new technologies employed in investigating and monitoring ground fissures are various, such as interferometric synthetic aperture radar (InSAR)/synthetic aperture radar (SAR) [1,2], unmanned aerial vehicles (UAVs) [3,4] and 3D laser scanning [5,6]. While it is notable that, InSAR/SAR and 3D laser scanning have high operating costs and long data acquisition cycles. Compared to these techniques, the monitoring technology of a UAV equipped with a digital compact camera has higher image resolution, more flexible maneuverability, higher efficiency, and lower operating costs. But these images can be easily influenced by severe conditions, such as bad weather and night environment, unfortunately, limited studies have been carried out to monitor mining-induced ground fissures by UAV equipped infrared camera.

The image edge detection method has been used in numerous fields by many researchers [7,8,9,10,11,12]. The typical image edge detection methods include the Roberts operator [13], Sobel operator [14], Prewitt operator [15], Krisch operator [16], Laplacian operator [17], Laplacian of Gaussian (LoG) operator [18], Canny operator [19], edge detection based on mathematical morphology [20,21,22], wavelet transform edge detection methods [23,24]. The detection and extraction of linear features is a fundamental operation in digital image processing [25,26]. Cracks typically have linear edges. At present, crack detection based on the edge detection method is mainly used for pavement crack detection and concrete crack detection. For example, Ikhlas, Abdel-Qader et al. used edge detection technology to identify and detect cracks in bridges [8]. Qin Zou et al. developed CrackTree, which was a fully-automatic method to detect cracks from pavement images, and demonstrated the method had a better performance [27]. Oliveira, H. et al. proposed an automatic system for crack detection and characterization following the guidelines defined by the Portuguese Distress Catalog with promising results [28]. However, the detection of ground fissures caused by mining collapse, in particular, ground fissure detection from infrared images collected by UAVs, has seldom been investigated with the edge detection method.

Coal mining is conducted through the day and night. Thus, 24 h of uninterrupted observation are needed to observe the surface fissures. The existing methods have difficulty achieving the continuous observation of ground fissures. The visible images collected by UAVs can be restricted by bad weather, and the infrared images are less affected by the external environment [29]. In this research, for the first time, a UAV equipped with a visible light camera and an infrared camera was used to collect images used to monitor mining-induced ground fissures. Various edge detection methods have been applied to detect mining-induced ground fissures. Furthermore, to improve the fissure detection effect, an improved edge detection method of detecting mining-induced ground fissures was proposed in this study.

2. Materials and Methods

2.1. Mining-Induced Ground Fissure Monitoring

In this study, a UAV (M600Pro, Dajiang, China) equipped with an infrared camera (Tau2-640R high-definition infrared camera, FLIR, America) and a visible light camera (ZENMUSE Z3 visible light camera, Dajiang, China) was chosen to collect ground fissure information above working face No. 12401 of the Shangwan Coal Mine, which is a typical shallow coal seam mine with large mining height. The targeted mining height of working face No. 12401 is 8.6 m. The elevation range of the ground surface is between 1188 and 1300 m. And the elevation of the coal seam floor is between 1043 and 1066 m. The Tau2-640R infrared camera was used to record the ground temperature. The ZENMUSE Z3 visible light camera was used to collect high-resolution images of the ground fissures. In order to obtain infrared images and visible images of mining-induced ground fissures, the UAV equipped with an infrared camera and a visible light camera was used to observe the fissures in spring. The images were captured at different times in one day (1:00 am, 3:00 am, 5:00 am, 7:00 am, 9:00 am, 11:00 am, 1:00 pm, 3:00 pm, 5:00 pm, 7:00 pm, 9:00 pm, and 11:00 pm). A total of 12 groups of infrared images and 6 groups of visible images of the region of interest (ROI) at different times were obtained. The ROI was selected on the surface directly above the central position of the working face, where the advancing distance is 238 m. The ROI with its boundaries drawn with lime is an approximately square area of 5.9 m × 6 m. And in one corner of the ROI, a square object with a size of 45 cm × 45 cm was placed as a size reference, as shown in Figure 1. Within the ROI, there are two groups of fissures, which are 40 m ahead from the longwall. Furthermore, the fissures appeared after 6 days of advancing the working face. A schematic diagram of UAV monitoring of mining-induced ground fissures is shown in Figure 1.

2.2. Infrared and Visible Image Fusion Method

Visible images are suitable for human visual perception because of their high spatial resolutions, rich scene details, and clear textures [29,30]. In contrast, infrared images highlight targeted areas effectively and exhibit less texture, contrast and resolution [30]. Infrared and visible images can complement each other. Therefore, fusion method was selected to fuse the collected infrared images and visible images in this study. In recent years, various infrared and visible image fusion methods have been explored. The hybrid methods based on multiscale transform and sparse representation are more suitable to fuse the detailed information of visible images and the target regions of infrared images. The fusion performance of the hybrid methods based on multiscale transform and sparse representation are better than others [31,32]. To obtain content-rich fusion image, the hybrid methods curvelet transform and sparse representation (CVT-SR) was chosen to fuse the infrared image and visible image. The code of the fusion method is publicly available, and the parameters are set according to the parameters in the original study.

2.3. Edge Detection Methods

Edge detection methods were used to detect mining-induced ground fissures in the images collected by UAV. In this paper, 6 edge detection algorithms were implemented in MATLAB and used to process the same infrared image, visible image and the fused image.

2.3.1. Classical Edge Detection Methods

The classical edge detection methods used in this study include first-order differential operator and second-order differential operator. The first-order differential operator such as Sobel, Roberts and Prewitt operator, take the pixel with maximum value of gray gradient as edge information, whose first derivative is zero. The second-order differential operator, such as Laplacian operator, detect the edge pixels with maximum change rate of gray scale by calculating the second derivative is zero.

Canny operator can obtain an optimal compromise between noise suppress and edge detection. Canny operator is based on three criteria. The basic idea is to use Gaussian function to smooth the image, and then the maximum value of the first derivative also corresponds to the minimum value of the first derivative. In other words, the point with sharp gray change (strong edge) and the point with slight gray change (weak edge) correspond to the zero-crossing point of the second derivative. Thus, these two thresholds are used to detect strong edges and weak edges. The fact that Canny algorithm is not disturbed by noise which makes it possible to detect real weak edges.

2.3.2. Improved Edge Detection Method

In this study, we proposed an improved edge detection method, which based on the Laplacian of Gaussian, Canny and mathematical morphology operators. The proposed method combined the merits of the Laplacian of Gaussian (LoG) filter, Canny operator and mathematical morphology, and overcame the difficulty of traditional edge detectors for handling the line structures in cluttered background. The diagram of edge detection process with the proposed method is shown in Figure 2. It consists of three main processes, and the image-processing steps of the proposed method are as follows.

• Step 1: The LoG filter is used to perform preliminary fissure detection in the image. LoG arithmetic is the convolution of Laplacian arithmetic and Gaussian arithmetic. To reduce noise, all acquired images are smoothed with a two-dimensional Gaussian filter. it may be desirable to first smooth the image by a convolution with a Gaussian kernel of width σ,

  $$G_{\sigma }\left(x,y\right) = \frac{1}{\sqrt{2\pi {\sigma }^2}}·\exp \left(-\frac{x^2+y^2}{2{\sigma }^2}\right)$$

  (1)

Then, Laplacian filter is applied to make the edges of the image appear more distinct and emphasize the contrast. Therefore, LoG filter not only enhances the edges and details of the image, but also suppresses noises. And the LoG arithmetic is more accurate than other operator in fissure edge detection [33]. For simplicity, we have omitted the normalizing coefficient $1/\sqrt{2\pi {\sigma }^2}$. LoG arithmetic is defined as follows [18].

$$L\mathrm{o}G\left(x,y\right)\triangleq {\nabla }^{2}G\left(x,y\right)=\frac{x^2+y^2-2{\sigma }^2}{{\sigma }^4}·\exp \left(-\frac{x^2+y^2}{2{\sigma }^2}\right)$$

(2)

where $\sigma$ denotes the parameter of LoG operator, which represents the size of the filter and how smooth this operator filtering an input image, while x and y denote spasial coordinates of any pixels of input image.

2.

Step 2: Canny operator is selected to detect fissures in the initial detection image. Due to the use of multi-level algorithm, each step can refine the results, so it has a good performance. Canny operator uses Gaussian function to calculate gradient, which works at multi threshold level based on primary edge and secondary edge, and has a good signal-to-noise ratio.

3.

Step 3: Finally, the closed operation of mathematical morphology is used to postprocess the fissure detection image of step 2. Mathematical morphology has two basic operations, namely dilation and erosion, are defined as follows [34]:

• Dilation:

  $$A\oplus B =\left\{x,y{|\left(B\right)}_{xy}\cap A\ne \varnothing \right\}$$

  (3)
• Erosion:

  $$A\ominus B =\left\{x,y{|\left(B\right)}_{xy}\subseteq A\right\}$$

  (4)

  where A is the input image, B is the structuring element.

Morphological closing, denoted $A·B$, is a dilation followed by an erosion:

$$A·B = \left(A\oplus B\right)\ominus B$$

(5)

Generally, closing operations join narrow breaks and fill holes smaller than the structuring element.

2.4. Quantitative Approaches to Edge Detection Evaluation

The objective and quantitative evaluation of solutions is indispensable in edge detection. In this paper, we selected four evaluating methods: the peak signal-to-noise rate (PSNR) [35], effective edge rate [36], Pratt’s figure of merit (PFoM) [37], and F-measure [38], respectively.

2.4.1. Peak Signal-to-Noise Rate

The peak signal-to-noise rate (PSNR) can be used to measure whether the result of the source image processed by a certain algorithm is satisfactory, is calculated by following equation [35]:

$$PSNR = 10 \times \lg \frac{{\left(2^b-1\right)}^2}{MSE}$$

(6)

where b is bits number of an image, and in this paper, b = 8. MSE denotes the mean square error between source image ($I_O$) and end edge image ($I_R$), can be estimated as follows [35]:

$$MSE = \frac{1}{m\times n}{{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n\left[I_R\left(i,j\right)-I_O\left(i,j\right)\right]}}}^2$$

(7)

where m, n denotes the number of pixels rows and columns, respectively. $I_O\left(i,j\right)$ and $I_R\left(i,j\right)$ represents the gray value of the pixel at i-th row and j-th column in $I_O$ and $I_R$, respectively.

2.4.2. Effective Edge Rate

In this paper, the effective edge is defined as the edge of fissures. The number of pixels on the edges of fissures and the total number of pixels on all edges were named $N_{\mathrm{f}}$ and $N$, respectively. By erasing the irrelevant edges in the edge image manually, the image with only edges of fissures was obtained. Then, these two numbers can be obtained by counting the pixels with gray value is 0 or 255 using MATLAB. $N_{\mathrm{f}}/N$ shows the effective edge rate of edge image, and the higher the $N_{\mathrm{f}}/N$ is, the better the detection effect is.

2.4.3. Pratt’s Figure of Merit

Pratt’s figure of merit (PFoM) reported by Pratt [37] was utilized to evaluate the edge detection performance. PFoM illustrates the deviation of the calculated edge from the ground truth, given by

$$PF\mathrm{o}M = \frac{1}{\max \left\{I_I,I_A\right\}}{\displaystyle {\sum }_{i=1}^{I_A}\frac{1}{1+\alpha d^2\left(i\right)}}$$

(8)

where $I_I$ and $I_A$ are the number of ideal and actual edge points, respectively. $d\left(i\right)$ denotes the distance from the i-th actual edge to the corresponding detected edge, and $\alpha$ is a scaling constant which is set to be 1/9 in Pratt’s work. PFoM returns a number between 0 and 1 depends the quality of the edge detection, with 1 being the best.

2.4.4. F-Measure

The F-measure is a measure of the accuracy and it can be interpreted as a weighted average of the precision and recall. It can be computed as follows [38]:

$$F = \frac{2P·R}{P+R}$$

(9)

where Precision $P = N_0/N_1$, Recall $R = N_0/N_2$, and $N_0$ is the number of the pixels in an edge image which can be matched in its ground truth image, $N_1$ and $N_2$ are the whole number of the pixels in ground truth image and in edge image, respectively. F-measure reaches its best value at 1 and worst value at 0 [38].

According to the calculation and comparison of PSNR, $N_{\mathrm{f}}/N$, PFoM and F-measure, the optimal algorithm for image data in this paper was determined. Then, fissure extraction was conducted on 12 infrared images at different times with the optimal algorithm.

2.5. Calculation of the Length of Fissure in Images

In order to calculate the length of the fissure in the images, a method was given in this paper. As shown in Figure 3, the fissure is regarded as the overlap of the heads and ends of multiple line segments, and the length of each line segment is calculated by Pythagorean Theorem and then accumulated to obtain the length of the fissure expressed in pixels. The actual length of the fissure can be calculated according to the resolution of the image. The length of fissure can be calculated as following equation:

$$L = \frac{R}{100}{\displaystyle \sum _{i=1}^n{l}_i = \frac{R}{100}{\displaystyle \sum _{i=1}^n\sqrt{r_i{}^2+c_i{}^2}}}$$

(10)

where $r_i$ represents the pixel number of the fissure in the i-th row and $c_i$ represents the pixel number of the fissure in the i-th column. R in units of ‘cm/pixel’ is the image resolution, L in units of ‘m’ denotes the cumulative length from the head of the fissure to the i-th row representing the different positions along the fissure.

In accordance with the above method, the length of Fissure I₁ in fissure detection results of 12 infrared images with the optimal algorithm was calculated. And the length of Fissure I₁ in the infrared images was also calculated as the reference for fissure detection results with the optimal algorithm. By comparing and analyzing the results of fissure detection and the length of Fissure I₁ at different times, the optimal time for identifying the ground fissures with the optimal algorithm under the conditions studied in this paper was given.

3. Results

3.1. Mining-Induced Ground Fissure Identification Result

12 infrared images and 6 visible images with approximately the same resolution and in the same direction were selected for analysis from the captured images at different times. The visible images are shown in Figure 4. The infrared images are shown in Figure 5. The images in Figure 4 and Figure 5 were clipped for convenience to study the ROI.

The visible light camera cannot collect information on mining-induced ground fissures at night. Only the visible images collected during the daytime are shown (although the 9:00 am and 3:00 pm visible images were lost) in Figure 4. The results showed that the visible images have a higher spatial resolution and more clearly show the fissures than the infrared images. But the visible images can be easily influenced by bad weather, ground vegetation and night environment.

As shown in Figure 5a–c,j–l, from 7:00 pm to 5:00 am, the mining-induced ground fissures appear as white, indicating high temperatures. From 9:00 am to 3:00 pm, the fissures appear black, indicating lower temperatures, as shown in Figure 5e–h. Interestingly, the identification effect of mining-induced ground fissures in the infrared images between 3:00 am and 5:00 am is the best result by comparing the photos collected among other periods. Compared with the images at alternative time points, the fissures in the infrared images from 3:00 am to 5:00 am are clearer and more complete. The results show that the fissure identification effect of infrared image is worst at 7:00 am and 5:00 pm. The color of the fissures is very close to sand, which makes the fissures difficult to distinguish from the surrounding objects (Figure 5d,i). In conclusion, the mining-induced ground fissures can be identified, and the fissures identification effect of infrared images collected at night are better than that of images collected during the day.

3.2. Mining-Induced Ground Fissure Detection

Figure 6a–c shows the infrared image, visible image and fused by the CVT-SR, which was collected at 5:00 am. The three images are selected to analyze the mining-induced ground fissures marked by the red squares in Figure 6a–c. The fissures are divided into two parts: fissure groups I and II. And the ground truth image of fissures is obtained by hand drawing, which is shown in Figure 6d.

The results of ground fissure detection using multiple edge detection methods (Roberts operator, Sobel operator, Prewitt operator, Canny operator, Laplacian operator and proposed method) for the visible image, infrared image and fused image are shown in Figure 7, Figure 8 and Figure 9. To evaluate a fissure detection result, we compute six measures: PSNR, $N_{\mathrm{f}}/N$, PFoM, F-measure, Precision and Recall by comparing the detected fissure against the human annotated ground truth fissure.

3.2.1. Fissure Detection Results for the Visible Image

Figure 7 shows the detailed qualitative edge detection results of the multiple edge detection algorithms for the visible image. The evaluation indexes scores of multiple edge detection methods for the visible image are shown in

Table 1. The evaluation indexes scores of multiple edge detection methods for the visible image.

	Evaluation Indexes	PSNR	$\raisebox{1ex}{$N_{\mathrm{f}}$}\!\left/ \!\raisebox{-1ex}{$N$}\right.$	PFoM	F-Measure	Precision	Recall
Methods
Roberts		18.8059	0.0119	0.0558	0.6309	0.4649	0.9814
Sobel		18.8059	0.0299	0.0809	0.6316	0.4650	0.9840
Prewitt		18.8059	0.0336	0.0803	0.6316	0.4650	0.9839
Canny		18.8076	0.0828	0.0827	0.6297	0.4648	0.9756
Laplacian		10.1105	0.0767	0.0087	0.0000	0.4000	0.0000
Proposed		18.8102	0.0819	0.1297	0.6255	0.4734	0.9217

. Figure 7a–c are the results of the fissure detection using Roberts operator, Sobel operator and Prewitt operator. The detected fissure groups I and II have been marked in red squares. Overall, the results are quite similar across the three methods. As shown in Figure 7a–c, the edges of fissure groups I and II can be basically detected. However, the most severe fissure loss is observed in edge detection results, particularly loss of fissure group II. The background noise in the visible image is relatively less, except white boxes marked areas. According to the results of four evaluation indexes in Table 1, there are no significant disparity between Sobel operator and Prewitt operator. While the $N_{\mathrm{f}}/N$, PFoM of Roberts operator are lower than that of Sobel operator and Prewitt operator. Figure 7d is the result of the fissure detection using Canny algorithm. As can be seen from the detection result, fissure groups I and II can be detected by Canny operator. But the edge of fissure is incomplete, and there is a certain amount of noise. In Table 1, the Canny operator has the highest $N_{\mathrm{f}}/N$ of 0.0828, and its PSNR, PFoM are larger than that of Roberts operator, Sobel operator, Prewitt operator and Laplacian operator. But F-measure ranks fourthly among all algorithms. Figure 7e shows the edge detection result of the Laplacian operator. The main edges of fissure groups I and II are retained. However, the detected edges are discontinuous. The Laplacian operator is second-order operator method, which is relatively sensitive to noise [39]. There is more noise in the fissure detection results, indicating that the ability to filter false edges is insufficient. In addition, edge detection using the Laplacian operator alone often results in a double pixel wide edge. As a result, except for $N_{\mathrm{f}}/N$, the PSNR, PFoM and F-measure of Laplacian operator significantly lower than others. For these reasons, Laplacian operator is not considered a very good operator for detection of edges. Figure 7f is the result of the fissure detection using the improved edge detection method for the visible image. In Figure 7f, fissure groups I and II can be detected, but some edges of fissures are lost. The background noise is relatively less, except the edges of white boxes. As shown in Table 1, the proposed method has the highest PSNR of 18.8102, PFoM of 0.1297 than other five. In particular, the PFoM value of the improved method is significantly larger than that of other methods. Compared with the ground truth shown in Figure 6d, the fissure loss observed in Figure 7a–d is more serious than that of the fissure detection result using the improved method in Figure 7f. And more noise is observed, although the fissure groups I and II are relatively complete in Figure 7e. Therefore, the PFoM value of the improved method is the maximum. The $N_{\mathrm{f}}/N$ of the proposed method is slightly smaller than that of Canny operator, significantly larger than other four. However, the F-measure of the proposed method for the visible image is only greater than that of Laplacian operator, ranks fifth among all algorithms.

3.2.2. Fissure Detection Results for the Infrared Image

The results of the fissure detection using the edge detection algorithms for the infrared image are shown in Figure 8. Figure 8a–c are the results of the fissure detection using Roberts operator, Sobel operator and Prewitt operator. The detected edges of ground fissure in the infrared image are more informative and can identify two fissure groups. However, the infrared image displays higher background noise, which leads to relatively more false edges. Consistent with the other results, the Roberts operator’s detection result is relatively low in noise. In

Table 2. The evaluation indexes scores of multiple edge detection methods for the infrared image.

	Evaluation Indexes	PSNR	$\raisebox{1ex}{$N_{\mathrm{f}}$}\!\left/ \!\raisebox{-1ex}{$N$}\right.$	PFoM	F-Measure	Precision	Recall
Methods
Roberts		18.8096	0.2763	0.3540	0.6321	0.4643	0.9896
Sobel		18.8104	0.1714	0.2434	0.6294	0.4639	0.9787
Prewitt		18.8104	0.1712	0.2448	0.6295	0.4639	0.9790
Canny		18.8086	0.5513	0.4448	0.6342	0.4650	0.9974
Laplacian		16.5738	0.1974	0.0660	0.0001	0.4340	0.0000
Proposed		18.8209	0.6259	0.5708	0.6331	0.4659	0.9876

, PSNR of Roberts operator, Sobel operator and Prewitt operator are almost the same, but $N_{\mathrm{f}}/N$, PFoM and F-measure of Roberts operator are higher than that of Sobel operator and Prewitt operator. Figure 8d is the result of the fissure detection using Canny algorithm. Only fissure group I was detected. However, the background noise suppression effect of Canny operator is better than Roberts operator, Sobel operator and Prewitt operator. The number of false edges is fewer, and the edges of fissure are continuous. Except for PSNR, the $N_{\mathrm{f}}/N$ and PFoM of Canny operator are obviously larger than that of Roberts operator Sobel operator and Prewitt operator. And F-measure value of Canny operator is 0.6342, which is highest than others. Figure 8e shows the edge detection result of the Laplacian operator. Fissure groups I and II can be detected. The detected edges are discontinuous. Except for $N_{\mathrm{f}}/N$, the PSNR, PFoM and F-measure of Laplacian operator are still the smallest. As shown in Table 1 and Table 2, the four evaluation indexes scores are larger than that of the visible image, since the background noise is relatively less than that of the visible image. Figure 8f is the result of the fissure detection using the proposed method for the infrared image. The edges of fissure groups I and II can be detected completely, and the noise is less. The fissure detection result of the infrared image is obviously better than that of other methods in this paper. The results in Table 2 show that PSNR, $N_{\mathrm{f}}/N$ and PFoM of the proposed method is 18.8209, 0.6259, 0.5708, respectively, which are obviously larger than that of other edge detection methods. And F-measure is 0.6331, which is slightly smaller than that of Canny operator.

3.2.3. Fissure Detection Results for the Fused Image

Figure 9 is the results of the fissure detection using multiple edge detection algorithms for the fused image. As shown in Figure 9a–e, the edges of ground fissure can be identified in the fused image which is similar to the infrared image. Compared with the fissure detection results of the infrared image, since the white boxes used to mark the ground fissure is incorporated into the fused image, more false edges are observed in the detection results. As can be seen in

Table 3. The evaluation indexes scores of multiple edge detection methods for the fused image.

	Evaluation Indexes	PSNR	$\raisebox{1ex}{$N_{\mathrm{f}}$}\!\left/ \!\raisebox{-1ex}{$N$}\right.$	PFoM	F-Measure	Precision	Recall
Methods
Roberts		18.8095	0.2030	0.2678	0.0401	0.3726	0.0212
Sobel		18.8106	0.1510	0.2065	0.3039	0.4298	0.2351
Prewitt		18.8107	0.1553	0.2086	0.3064	0.3401	0.2787
Canny		18.8091	0.2848	0.2892	0.6321	0.4647	0.9877
Laplacian		18.0569	0.1630	0.0910	0.3569	0.2894	0.4655
Proposed		18.8230	0.3444	0.3337	0.6321	0.4693	0.9679

, $N_{\mathrm{f}}/N$ and PFoM of Roberts operator are higher than that of Sobel operator and Prewitt operator. However, F-measure of Roberts operator are smaller than that of Sobel operator and Prewitt operator. PSNR of Roberts operator, Sobel operator and Prewitt operator are almost the same. And the $N_{\mathrm{f}}/N$, PFoM and F-measure values of the algorithms are all smaller than that of the infrared image. For the infrared image and fused image, the Roberts operator, Sobel operator and Prewitt operator still have shortcomings in filtering out false edges, resulting in false edge features in edge detection images. Therefore, the edge detection effect of images with complex noise is not good. In the case of a large number of background noises, many false edges can be detected in such images if the threshold is improperly set. In Table 2 and Table 3, except the PSNR values are nearly the same, the $N_{\mathrm{f}}/N$, PFoM and F-measure of Canny algorithm for the infrared image and fused image are obviously higher than that of Roberts Operator, Sobel operator and Prewitt operator, indicating that the effect of fissure detection using the Canny operator is better than Roberts Operator, Sobel operator and Prewitt operator. Canny operator has good edge detection performance [19,40]. The four evaluation indexes scores of Prewitt operator and Sobel operator for the three images are nearly equal. This shows that the performance of the two methods is roughly similar [41]. As shown in Figure 9f, the edges of ground fissure can be identified completely by the proposed method. However, compared with the fissure detection result of the infrared image, there is more background noise, which are the edges of white boxes used to mark the ground fissure. Thus, except for PSNR, the $N_{\mathrm{f}}/N$ of 0.3444, PFoM of 0.3337 and F-measure of 0.6321 are smaller than that of the infrared image, but still larger than that of the other edge detection methods.

From a comparison of the results shown in Table 1, Table 2 and Table 3, the PSNR, $N_{\mathrm{f}}/N$, and PFoM of the proposed method for the three images are all at a relatively high scores when compared with all the other methods in this paper, indicating that the proposed method outperforms the others. Moreover, the proposed method is effective in noise suppression and false edge filtering, resulting in making the fissure features visually significant. These results thereby imply the reliability of the improved edge detection approaches. Therefore, the proposed method in this study can detect the ground fissures in the three images accurately. As shown in Figure 10, by analyzing the four evaluation indexes scores of the proposed method for the visible image, infrared images and fused image, it can be found that the effect of fissure detection in infrared image is clearly superior to that of the fused image and also to that of the visible image.

3.3. Fissure Detection and Length Calculation of Infrared Images

3.3.1. Fissure Detection for Infrared Images at Different Times

As described in the Section 3.2, the proposed method has good edge detection performance. And the effect of fissure detection in infrared image outperforms that of the fused image and the visible image. Thus, the proposed method was used to detect the fissures in 12 infrared images showed in Figure 5 and the results of fissure detection are shown in Figure 11. The detected fissures are marked by the red squares in Figure 11.

According to the edge detection results shown in Figure 11, the fissures in infrared images at different times can be detected except 5:00 pm. Because the fissures in the infrared images collected at 5:00 pm could not be identified. However, the effect of fissure detection varies from each other, particularly that of infrared images taken during the day and night. From 7:00 pm to 5:00 am, the edges of fissures group I and II can be detected completely with less noise in the results. Contrast with the detection results of infrared images taken at night, the fissure loss is relatively serious in results of 7:00 am to 3:00 pm, and that display higher background noise, which are mostly the edges of plants. In particular, the length loss of Fissure I₁ detected from 1:00 pm to 3:00 pm is the most.

3.3.2. Fissure Length in Infrared Images and Detection Results

The visible image of the target observation area at 7:00 am is shown in Figure 12, and the resolution is 0.30 cm/pixel. The measured width of the Fissure I₁ location marked by the red rectangle in Figure 12 is 12.50 cm. The number of horizontal pixels at the marked Fissure I₁ location is counted as 41, and the calculated length is 12.38 cm. The error with the actual value is −0.96%. Therefore, the recognition accuracy of visible image is high enough, and the calculated length of Fissure I₁ in visible image can be taken as the real length to evaluate the fissure detection effect.

Using the method shown in Section 2.5, the calculated length of Fissure I₁ in fissure detection results and that in visible image captured at 7:00 am are shown in

Table 4. The Fissure I₁ length in fissure detection results and that in visible image captured at 7:00 am.

Time	Fissure I1 Length (m)		Difference (m)	Error (%)
	Visible Image	Fissure Detection Results
1:00 am	3.01	3.02	0.01	0.33
3:00 am		3.07	0.06	1.99
5:00 am		3.16	0.15	4.98
7:00 am		0.40	−2.61	−86.71
9:00 am		2.75	−0.26	−8.64
11:00 am		2.72	−0.29	−9.63
1:00 pm		1.23	−1.78	−59.14
3:00 pm		0.40	−2.61	−86.71
5:00 pm		-	-	-
7:00 pm		2.72	−0.29	−9.63
9:00 pm		3.01	0.00	0.00
11:00 pm		2.87	−0.14	−4.65

. The Fissure I₁ length in fissure detection results from 9:00 pm to 5:00 am is longer than that for the rest of the time. Moreover, the Fissure I₁ length is larger than that in the visible image between 1:00 am and 5:00 am. The maximum length of Fissure I₁ is 3.16 m at 5:00 am, and the minimum length is 0.40 m at 7:00 am, 3:00 pm. According to Table 4, the differences and errors in Fissure I₁ length between fissure detection results and visible image are small from 9:00 pm to 5:00 am. In these times, the absolute differences and errors are less than or equal to 0.15 m, 4.98%, respectively. The average absolute difference and error is 0.07 m, 2.39%. As shown in Table 4, the differences are −0.26 m, −0.29 m and −0.29 m at 9:00 am, 11:00 am, 7:00 pm, and the absolute errors are 8.64%, 9.63% and 9.63%, respectively, which are higher than those of 9:00 pm to 5:00 am. As shown in Table 4 and Figure 13, the absolute differences and errors at 7:00 am, 3:00 pm and 1:00 pm are obviously larger than other times. At 7:00 am, 3:00 pm and 1:00 pm, the differences are −2.61 m, −2.61 m and −1.78 m, which results in 86.71%, 59.14%, 86.71% error, respectively. Since the fissure cannot be identified in the infrared image at 5:00 pm, the Fissure I₁ length cannot be calculated. To sum up, the differences and errors in Fissure I₁ length between the detection results at night and that of visible image are small. The differences and errors are relatively large at day, especially at 7:00 am, 3:00 pm and 1:00 pm.

As shown in

Table 5. The Fissure I₁ length in infrared images and that in fissure detection results.

Time	Fissure I1 Length (m)		Difference (m)	Error (%)
	Infrared Images	Fissure Detection Results
1:00 am	3.00	3.02	0.02	0.75
3:00 am	3.19	3.07	−0.12	−3.65
5:00 am	3.04	3.16	0.12	4.03
7:00 am	0.71	0.40	−0.31	−43.44
9:00 am	2.86	2.75	−0.11	−3.72
11:00 am	2.88	2.72	−0.16	−5.59
1:00 pm	2.73	1.23	−1.50	−54.95
3:00 pm	2.76	0.40	−2.36	−85.58
5:00 pm	-	-	-	-
7:00 pm	2.94	2.72	−0.22	−7.35
9:00 pm	2.99	3.01	0.02	0.73
11:00 pm	3.04	2.87	−0.17	−5.60

, the maximum length of Fissure I₁ is 3.19 m at 3:00 am, and the minimum length of Fissure I₁ is 0.71 m at 7:00 am. The Fissure I₁ length in infrared images from 9:00 pm to 5:00 am is longer than that for the rest of the time. According to Table 5, the differences of Fissure I₁ length between in fissure detection results and that in the infrared images are relatively small, the average absolute difference is 0.10 m except at 7:00 am, 1:00 pm and 3:00 pm. The absolute errors in the Fissure I₁ length are approximately 8% or even smaller most of the time, and the average absolute error is 3.49%, except at 7:00 am, 1:00 pm and 3:00 pm. In contrast, Table 5 and Figure 14 show that the differences and errors are higher at 7:00 am, 1:00 pm and 3:00 pm. At these times, the length of Fissure I₁ in fissure detection results is 0.31 m, 1.50 m and 2.36 m shorter than that in the infrared image, which results in 43.44%, 54.95%, 85.58% error, respectively. As shown in Table 5 and Figure 14, the Fissure I₁ length in fissure detection results is longer than that in the infrared image at 1:00 am, 5:00 am and 9:00 pm, when the average difference and error in length are 0.06 m, 1.84%, respectively. The error is considered acceptable. The error may be caused by the closed operation in mathematical morphology. In conclusion, the identification accuracy of the proposed method is high enough to be acceptable except at 7:00 am, 1:00 pm and 3:00 pm.

4. Discussion

Figure 5 showed that the identification effect of mining-induced ground fissures in the infrared images was better at night relatively, and worst at 7:00 am, 5:00 pm. Objects radiate electromagnetic waves with different frequencies, named thermal radiation [29,42]. Infrared images are obtained by infrared sensor to record the thermal radiation of different objects [29]. Sandy soil, vegetation, and mining-induced ground fissures have distinct thermal behaviors, as shown in Figure 5. The infrared camera collects information on ground fissures by detecting the different temperatures of the object surface. Due to the small specific heat capacity of sandy soil, the temperature variation range is large, and the temperature drops fastly at night [43]. However, the decrease in temperature of the fissures is slower than that of the vegetation and sandy soil because they extend deep into the ground, resulting in a large temperature difference. Due to this difference, the ground fissures can be effectively monitored. The temperature of the sandy soil rose rapidly at 7:00 am and dropped rapidly at 5:00 pm. It is difficult to monitor the fissures when the temperature of the sandy soil is close to the temperature inside the fissures. Therefore, collecting images during large temperature differences can obtain the best fissure monitoring effect. However, some shallow ground fissures, which have little or no difference in temperature from the surrounding environment, cannot be recorded by the infrared camera. To solve this problem, three solutions were proposed, improving the thermal sensitivity of infrared camera, adjusting the flying height of the UAV to get higher resolution, being used with visible images to make them complementary.

Fissure detection results showed that the proposed method had an excellent fissures detection capacity. Although this method had a good effect on noise suppression, there were still some false fissure edges or isolated points induced by noise in the resulting image. For the image de-noise is very important to improve the image edge detection performance, image noise reduction pre-processing should be further enhanced in the next step. From the experiment of ground fissure detection with various edge detection methods, it is found that a larger threshold value can filter out false edges well. However, the information of the fissure edge is lost, and not all fissures can be detected completely [44]. For example, the fissure detection results of the fused image using the improved edge detection method at different thresholds are shown in Figure 15. With larger threshold values, the false edges (marked with red squares) are fewer, and some fissure edges (marked with blue squares) are missing. Therefore, there is a contradiction between the noise suppression ability and edge detection accuracy of an edge detection algorithm. This contradiction is prominent in the infrared image and fused image considered in this research. A suitable threshold can not only suppress noise effectively but also acquire a better edge detection effect. However, manually choosing a threshold is time-consuming, and it is difficult to find the optimal value. Therefore, the selection of a threshold is a difficult problem. An automatic method to determine the optimal threshold is urgently needed.

In addition, the fissure images collected in this study are few and only targeted at fissure groups I and II in ROI. Therefore, the excellent performance and applicability of the proposed method need to be further verified in the later stage, such as more fissure images with complex background and different fissure widths. And we find that the ground fissures induced by the collapse of the mined-out area have a certain direction and exhibit linear characteristics. According to the directional and geometric characteristics of ground fissures, appropriate algorithms should be developed by applying mathematical theory and artificial intelligence to remove false edges effectively and detect ground fissures accurately. Furthermore, the edge detection methods based on deep learning are a key research direction in the future study. It has good edge detection performance [45]. However, there are too few fissure images in this paper, which is not convenient for deep learning method. In the next research, more different types of mining-induced ground fissure images should be collected to train the deep learning model, so as to explore a better edge detection method.

In this study, image processing is carried out offline. Firstly, the UAV equipped with cameras is used to collect the images, and then the images are exported to the work computer. Finally, the fissures in the images are detected by MATLAB. The time between images being taken and the analysis being completed can be very quickly, but not real-time. Therefore, in order to make it more convenient to be applied in practical engineering, we plan to convert the proposed method to run on an embedded system on the UAV to realize real-time automatic fissure detection in the next step.

5. Conclusions

In this research, UAV remote sensing and edge detection technology were used to study mining-induced ground fissures. Some main conclusions and contributions of this paper are briefly summarized as follows.

We proposed that a UAV equipped with an infrared camera and a visible light camera can be used to monitor mining-induced ground fissures effectively. This approach was more efficient and less cost than other approaches. The infrared images collected during 3:00 am and 5:00 am were sharper and had more complete fissure information than the images collected at other times under the conditions studied in this paper. A variety of edge detection methods were selected to detect mining-induced ground fissures for the visible image, infrared image and fused image. However, the results of classical edge detection methods were not satisfactory. The improved edge detection method given by the author was effective in noise suppression and could detect mining-induced ground fissures in the three images accurately. Compared with the other edge detection methods, this method showed good ground fissures detection performance according to the PSNR, $N_{\mathrm{f}}/N$, PFoM and F-measure. And the effect of fissure detection in infrared image was clearly superior to that of the fused image and also to that of the visible image. The fissures in infrared images at different times can be detected by the proposed method except 5:00 pm. Contrast with the detection results of infrared images taken at daytime, the edges of fissures can be detected completely with less noise in the results from 7:00 pm to 5:00 am. The identification accuracy of the proposed method is high enough except at 7:00 am, 1:00 pm and 3:00 pm in this season. The average absolute difference and error of Fissure I₁ length between in fissure detection results and that in the infrared images is 0.10 m, 3.49%, except at 7:00 am, 1:00 pm and 3:00 pm. Therefore, the improved edge detection method is not suitable for indentifying the fissures in the images collected in the daytime. Because the color of fissure in the daytime is not prominent compared with that of the surrounding ground objects, which will produce more background noise in the fissure detection results. In addition, the improved edge detection method is only applied to mining-induced fissures images collected in this study, and its detection effect on other fissures of different types and scales needs to be further verified.