Towards a Long-Term Unmanned Aerial Vehicle (UAV) Monitoring Framework for Post-Mining Effects: Prosper-Haniel Case

Abstract

Direct and indirect effects after mine operations cease operating must ideally be subjected to perpetual monitoring routines in order to detect possible risks or avoid adverse effects on the surrounding ecosystems at an early stage. In this contribution, mining subsidence lakes created inside the nature reserve Kirchheller Heide and Hilsfeld Forest are subjected to analysis for a long-term monitoring scheme. For this purpose, we employ high-resolution unmanned aerial system (UAS)-based multispectral and thermal mapping tools to provide a fast, non-invasive and multitemporal environmental monitoring method. Specifically, we propose to monitor vegetation evolution through multispectral analysis, biotypes identification using machine learning algorithms, and water surface extent detection, together with their thermal behavior. The aim of this contribution is to present the proposed workflow and first results to establish a baseline for future analyses and subsequent surveys for long-term multi-temporal monitoring.

1. Introduction

Post-mining effects could present long-lasting impacts on the environment [1,2], directly and indirectly affecting soil, water, or air quality [3]. One of these legacies related to underground mining includes land subsidence, resulting from the collapse of underground voids left by mining, which might lead to sinkholes and subsidence basins deforming the surrounding surface area. Subsidence can result in damage to various crucial elements such as roads, telecommunications, gas and water pipelines, sewage systems, electrical networks, and buildings (Figure 1). Numerous authors have focused on predicting such impacts and described this phenomenon, including Keinhorst [4], Bals [5], Lehmann et al. [6], Awiershin [7], Sann [8], Knothe [9], Litwiszyn [10], Berry [11], Sashurin [12], Bush [13,14,15], Suchowerska Iwaniec et al. [16], and Xia et al. [17,18].

Land subsidence might occur in combination with other processes occurring after mine closure. Melchers et al. [20] and Westermann et al. [21] describe the rise in groundwater levels after the cessation of mining operations, which has a significant impact on water management and might lead to floodplains through land subsidence [22]. Initially, waterlogging occurs, and the groundwater level does not change over a given period of time but, in combination with a long-term subsidence effect, a water reservoir can originate (Figure 2) [23].

The complexity of post-mining processes has been addressed through the application of innovative and cutting-edge research methods and instruments in recent decades. One such tool to study Earth surface movements is radar interferometry. The European Ground Motion Service [24] has played a pivotal role in advancing the accessibility of Sentinel-1 data, thereby integrating radar images into standard monitoring routines for land subsidence. Specifically, it has facilitated the examination and analysis of land subsidence processes across various mining regions worldwide, including Belgium [25,26], Colombia [27], China [28,29], the Czech Republic [30], the United Kingdom [27,31], the Netherlands [32], Germany [19,20,27,31,33,34,35], Poland [36,37,38], and the Republic of South Africa [31].

The use of UAVs (unmanned aerial vehicles) combined with evolving sensor technology has become increasingly useful for monitoring and evaluating risks at a closer distance to the earth’s surface. In fact, some results from UAV-acquired data have made it possible to study soil conditions [39,40,41,42] and assess moisture content using multispectral analyses and vegetation indices [43]. Most recently, UAV spectral data have been used to monitor water acidity and different post-mining effects, providing high-resolution and reliable assessments [44]. Pawlik et al. [45] and Rudolph et al. [46] present various methods for the geomonitoring of post-mining processes in use: 3D geological structure modeling, multispectral satellite imagery, multispectral drone imagery, earth movement data, mobile GIS, and weather data. A thermal imaging camera can be used to search for fires [47,48,49,50], monitor post-mining dumps [51], and inspect the oil, gas, and mining industry [52,53,54,55,56].

In Germany, land subsidence occurs in various regions where coal-mining activities have already concluded. In particular, the closure of the Prosper-Haniel coal mine (North Rhine-Westphalia region) in 2018 marked the end of an era in Germany’s coal mining history and brought challenges related to land subsidence. Efforts to mitigate subsidence effects have included monitoring systems and structural reinforcement; however, subsidence has affected surrounding infrastructure and communities, necessitating continuous monitoring and mitigation efforts to address safety concerns and minimize environmental impact.

Consequently, we propose the application of multispectral and thermal drone-based sensing as a means to monitor the long-term impacts of land subsidence by examining different indirectly affected elements. We present a methodology that is iterative and combines multispectral data, thermal imaging, and ground validation GPS measurements to provide high-resolution maps. The aim is to periodically acquire UAV data and create continuous monitoring of the effects on vegetation, water, and structural changes occurring in such post-mining areas. By taking advantage of multispectral drone cameras, which have similar characteristics to the sensors placed on space mission satellites like Sentinel-2, comparative vegetation indices can be derived [57].

2. Materials and Methods

2.1. Research Area

The research area is located in the Ruhr region of western Germany (Figure 3), on the site of the closed Prosper-Haniel coal mine (Figure 4). Coordinates of the extent are as follows: upper left—6.8543345 E 51.5880290 N, and down right—6.8613088 E 51.5847676 N. Mining operations under the Kirchheller Heide area began in 1976 [58]. However, the main mining activity was in the 1990s and lasted until 2018 [59]. Operations at the Prosper-Haniel mine ceased in December 2018 [58]. There are forms of nature conservation in the study area; for example, protected natural areas and also bird protection areas. It is a forested area, with the Schwarzbach and Elsbach streams running through the central part of the area, as well as the established subsidence lakes Weihnachtssee and Pfingstsee. Pawlik et al. [60] observed, on the basis of a spatiotemporal analysis of satellite images based on vegetation indices, that vegetation changes occurred between 2002 and 2012.

2.2. Materials

The following subsections will present the research instruments used.

2.2.1. DJI Phantom 4 Multispectral

The drone produced by DJI enables a drone flight lasting up to 27 min on one battery [61]. It has six camera sensors: one RGB camera and five lenses dedicated to different spectral bands (

Table 1. Characteristics of DJI Phantom 4 Multispectral.

Spectral Bands	Wavelength (nm)
Blue	434–466
Green	544–576
Red	634–666
Red-Edge	714–746
Near-Infrared	814–866

) (Figure 5). Due to its weight of 1.5 kg, a remote pilot license of competency is required to fly the drone in the European Union [62,63,64]. The built-in solar sensor allows for the detection of solar radiation, “maximizing the accuracy and consistency of data collection at different times of the day” [61]. With the implemented RTK (Real Time Kinematic) module, centimeter-accuracy images can be obtained because this module acquires location corrections from satellite positioning in real time.

Multispectral analysis of images obtained by drone flights is based on spectral band analysis. Before performing a multispectral analysis, the characteristics of the multispectral camera sensor in question should be known. Unmanned aerial vehicles (UAVs), commonly known as drones or copters, have evolved into unmanned aerial vehicles equipped with various sensors (e.g., GNSS receivers and cameras). By using UAVs, it is possible to flexibly and cost-effectively capture image data of small to medium size (up to 10 ha) [65].

2.2.2. DJI Mavic 2 Enterprise Advance Thermal

DJI Mavic 2 Enterprise Advance Thermal allows you to fly the drone for about 28 min [66]. Its weight is 909 g [Ibidem], which means that drone licenses are required to fly the drone [62,63,64]. In this version, the drone has two cameras: RGB and thermal. The RGB camera is characterized by a ½″ CMOS sensor with a resolution of 48 MP. The accuracy of the thermal camera is ±2 °C [66]. Also, as in the case of the DJI P4 MS drone, an RTK module is implemented.

2.3. Methodology

This subchapter will present and describe the steps that are performed during drone flight (Figure 6).

2.3.1. Pre-Planning of the Flights

Since UAV surveying is, nowadays, an accepted and much used method, the pre-planning will not be overly discussed in this article. After reviewing the platforms, sensors, resolutions and accuracies used for the desired monitoring results must be checked against the conditions on site. This includes, for example, local topography, aviation obstructions, and possible restrictions on flight operations, such as airfields or residential development. The applicable regulations of the respective country apply here; Germany follows the EU Rules for Unmanned Aircraft Systems [67] in this regard. The legal basis at national level (LuftVG (Luftverkehrsgesetz), LuftVO (Luftverkehrs-Ordnung), LuftVZO (Luftverkehrs-Zulassungs-Ordnung)) has been adapted by the “Act on the Adaptation of National Regulations to the Commission Implementing Regulation (EU) 2019/947 of 24 May 2019 on the Rules and Procedures for the Operation of Unmanned Aerial Vehicles” [68].

2.3.2. Preparation of Drone Flights

Directly before the flight, it is still necessary to check whether any short-term NOTAMs (NOtice To AirMan) are available [69]. These are notices to airspace users informing them of special features in the region, such as a music festival or the holding of military exercises, which, in turn, may result in an airspace closure, also for UAVs. In addition to the legal framework conditions, flight weather must also be checked before each flight. For this purpose, there are, for example, various apps and online offers (e.g., UAV Forecast) [70] in which you can directly insert the parameters of the UAV manufacturer (e.g., maximum wind strength). The actual conditions on site must be checked again directly before the flight (portable anemometer, thermometer, also barometer). To ensure the correct georeferencing of UAV images, there are two methods [71]: direct georeferencing (via a GNSS module linked to the camera) and indirect georeferencing via pre-measured and marked ground control points (GCP). The use of a Positioning RTK [72] module can improve the accuracy of the GNSS chips typically installed in all UAVs, from several meters to a few centimeters. Most professionally used drones now have RTK either on board as a standard (e.g., the DJI Phantom 4 Multispectral used here) or available as an optional module (e.g., the DJI Mavic 2 Enterprise Advanced used here). Although both methods lead to good georeferencing, the particular combination of RTK and GCP use produces the best results [73]. Due to the remote location of the study area and the resulting lack of mobile Internet access (Figure 7), it was not possible to use the RTK network “Satellite Positioning Service SAPOS”, which is freely available in North-Rhine Westphalia [74]. The wide distribution of GCP was rejected due to the large area, poor visibility from the air, as well as lengthy surveying traverses, attributed to the impossibility of RTK-GNSS measurements. As a solution, a dedicated RTK base station (Figure 7) was established on a previously highly accurate surveyed point at the edge of the survey area. A Trimble R12 GNSS rover was reconfigured accordingly for this purpose and was able to send its own correction data via a Wi-Fi connection as an NTRIP (Networked Transport of RTCM via Internet Protocol) caster to the controller of the UAVs, which, in turn, passed it on directly to the drones. A few previously measured GCP then served to verify the georeferencing of the later results. The accuracy in the centimeter range could be confirmed with this solution and costly corrections in post-processing could be avoided.

2.3.3. Drone Flights

The drone flight begins with selecting the planned area of the flight, then the flight parameters are important (

Table 2. Drone flight parameters.

Drone Flight Parameters	Description	References
Altitude of flight	The maximum flight height is 120 m above the earth’s surface. It depends on whether the National Aviation Authority imposes a geographical zone with a lower limit in the area where you are flying.	[76]
Frontal and Side overlap	“The amount of overlap between frames in the forward and lateral direction from the perspective of the platform’s direction of movement—must be properly handled to create seamless mosaics that represent the location of the features in the image. To produce accurate terrain models, a minimum forward overlap of 80 percent and a minimum side overlap of 75 percent are recommended to maximize the number of observations of landscape features.”	[77]
Waypoints	Number of images taken.
Estimated time	Time required to carry out a drone raid. The value is needed to estimate the number of inter-landings and take-offs.

).

Figure 8 shows the drone flight planning in the DJI GSP application. All listed flight parameters enable the assumed accuracy of the final results to be achieved. The flight plan was selected in such a way that it meets the external conditions and the resolution requirements. The minimum flight altitude in nature reserves in Germany is 100 m, while the maximum flight altitude in the open category is 120 m. As the Ground Sample Distance (GSD) is antiproportional to the flight altitude (e.g., $GSD=Height÷18.9in\mathrm{c}\mathrm{m}/\mathrm{P}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}$ for the Phantom 4 Multispectral), a flight altitude of 100 m was selected. The overlap corresponds to the parameters for the best possible photogrammetric processing. The software automatically adjusts the course angle and margin to the selected flight plan. During the flight, the operator is obliged to maintain visual contact with the drone and monitor the weather (e.g., wind speed). Of course, the drone operator should monitor the situation on an ongoing basis and, in the event of an emergency, land it immediately. All flight plans are stored in the KML exchange format. This ensures the reproducibility and comparability of the results, even when changing sensors or drones. The flights are repeated at suitable intervals to ensure long-term monitoring through these iterations.

2.3.4. Post-Processing

For the processing of the raw data from the UAVs, the software Agisoft Metashape was used. However, the procedure of the photogrammetric analysis (Figure 9) used here is basically the same for most comparable commercial (e.g., ArcGIS Drone2Map) or free (e.g., OpenDroneMap) software products. Usually, only the names, parameters and implementations of the different work steps change. As this is a standard process that is implemented differently but very similarly by different software products, the interested reader is referred to the relevant specialist literature on this point [78].

The advantage of Metashape over comparable products is that the work steps can be carried out individually one after the other. This makes it easy to optimize the parameters in the intermediate steps and gives you much better control over the overall result. Thus, for the creation of a 3D model, it makes sense to generate a high-resolution 3D mesh, while for an orthophoto, a lower-resolution, 2.5D DEM (Digital Elevation Modell) is also sufficient.

Multispectral

For the processing of the multispectral data, these were assessed according to the workflow with regard to their basic quality, sorted and loaded into the program. As an additional step, the data were then radiometrically calibrated to obtain a result that is as independent as possible of external conditions (position of the Sun, cloud cover). For this purpose, the images were reflectance-calibrated using the sun sensor permanently installed in the DJI Phantom 4 Multispectral used and an additional calibration panel with which images were taken before and after [79]. The reflectance values of the panel were determined in the laboratory for all wavelengths from 250 to 950 nm, so that a radiometric correction value for the spectral ranges used (Red, Green, Blue, RedEdge, NIR) could be calculated and applied under the given lighting conditions on the basis of these images [80]:

$$F_i=\frac{{\rho }_i}{avg\left(Li\right)}$$

(1)

where F_i is the reflection calibration factor for band I, ρ_i is the average reflectance of the calibration panel for the i-th band (from the provided calibration data of the panel) and avg(Li) is the average value of radiance for the pixels within the panel for band i.

With the calibrated data, the subsequent processes up to the creation of the orthophoto could be carried out. Since no 3D model was needed, the images were only aligned, a rough, georeferenced mesh was created, and, from this, a DEM and finally a multispectral orthophoto were derived. Using these products as a foundation, additional derivatives can be generated in ArcGIS Pro (e.g., DTM—Digital Terrain Model, vegetation indices), and a selection of these is presented in Section 3.

Thermal Infrared and RGB

The used DJI Mavic 2 Enterprise Advanced has two cameras, capturing a high-resolution RGB image and a slightly lower-resolution thermal infrared image at each trigger point. Both receive the same image coordinate via the modular RTK module. First, the RGB and thermal images must be sorted, as they are processed separately. The RGB images can be used without further pre-processing to create high-resolution RGB orthophotos, DEMs and DTMs. The thermal images are in gray values by default, where each gray value within the image represents an absolute temperature value. These can be analyzed with special software (DJI Thermal Analysis Tool) based on the environmental parameters (ambient temperature, humidity, shooting distance). Since the scaling of the gray values changes slightly from dark = cold to light = warm from shot to shot, the absolute temperature values are inevitably lost during orthophoto generation and the associated color balancing. However, the information about the relative temperature differences is retained and allows conclusions to be drawn about various monitoring aspects; for example, for the identification of water surfaces and coastline. After the processing steps described above, a gray-scale image with relative temperatures is obtained as an orthophoto, which can be colored differently depending on the purpose.

3. Results

After the flight has been carried out, all data and measurement points are implemented in the software to achieve the originally intended goals. The results of drone flight development are as follows:

-

individual images;

-

orthophoto map;

-

calculated vegetation indices on the basis of multispectral drone flights;

-

thermal orthophoto map;

-

digital terrain model (DTM);

-

digital surface model (DSM).

The first result obtained was an orthophoto map (Figure 10), which makes it possible to visualize the area with a better spatial resolution than satellite images, as well as a first visual assessment and interpretation.

The results presented can be used for analysis and data interpretation by implementing other data into the computational process. In this chapter, the authors aim to present the key possibilities of UAV use for the geomonitoring of post-mining processes.

3.1. Updating and Revising Land Classifications

For the state of North Rhine-Westphalia (NRW), a land use classification is available on the Open-NRW geoportal website via the Web Map Service (WMS). It is worth noting that the classification is based on the acquired satellite images of the Sentinel-2 mission and the study is valid 1 April 2022. In the study area, the following are the different classes:

-

Vegetation;

-

Grass,

-

Shrub,

-

Hardwood,

-

Street;

-

Water.

Thanks to the use of drone flights, better spatial resolution and interpretability of the results obtained from the drone flights can be achieved. Figure 11 shows a summary of the received orthophotomap dated 28 March 2023 and the WMS Landbedeckung service from the OpenNRW portal. The combination enables a detailed analysis of the vegetation condition and verification of its conformity with the class presented by the WMS service. For the classification, a support vector machine (SVM) has been applied over the multispectral dataset. A supervised machine learning algorithm is applied using a training dataset created from a small area of pixels used as training data. The SVM’s duty is to detect a separating hyperplane (defined as the class boundary) that segregates the features space into two classes with the largest margin for each class [81]. An optimization problem is solved by structural risk minimization for identifying the aforementioned hyperplane. Only the samples that are closest to the class boundaries are required to train the classifier, the so-called support vectors. Essentially, the localization of the hyperplane in the feature space is largely influenced by those training samples. Therefore, the classification results of the support vector machine can achieve high accuracy, even when only a small number of training samples are available [81,82]. Figure 11 (left side) demonstrates the classification result using 500 samples to define each class. Such a number is recommended when the inputs are non-segmented rasters, as for the multispectral dataset.

3.2. Identification of Water Surfaces and Coastline

In the geomonitoring of post-mining processes, the observation of water bodies and, also, of potential sites that could become floodplains plays an important role. In this study, a vegetation index was used to identify water surfaces; namely, the normalized difference vegetation index—NDVI, which was developed in 1973 by Rouse et al. [83]. This index is based on two spectral bands (Formula (2)):

$$NDVI=\frac{{\mathsf{\rho }}^{\mathrm{N}\mathrm{I}\mathrm{R}}-{\mathsf{\rho }}^{\mathrm{R}\mathrm{E}\mathrm{D}}}{{\mathsf{\rho }}^{\mathrm{N}\mathrm{I}\mathrm{R}}+{\mathsf{\rho }}^{\mathrm{R}\mathrm{E}\mathrm{D}}}$$

(2)

This indicator makes it possible to observe the state of the vegetation. Kuechly et al. [84], in their paper, presented a classification of the indicator, which is shown in

Table 3. Classification of NDVI. Source: after [84].

Values of the NDVI	Land Cover Types	Color
<0.1	Waters, soils, rocks, sand or snow	Red
0.2 to 0.3	Vegetation of low vitality	Yellow
0.3 to 0.6	Medium to dense vegetation cover	Light green
>0.6	Very dense vegetation of high vitality	Dark green

.

Using the above classification, the water surface, according to the NDVI indicator, has values from −1 to 0.1. It is worth noting that soil, sand and snow also show similar spectral band characteristics to water, so they can also be found in the values. Figure 12b shows the results of the search, presenting the water surfaces and shorelines of the Weihnachtssee and Pfingstsee reservoirs. Due to the fact that the forest remains in the subsidence lakes concerned, there are also tree stumps, which make the surface area of the entire lake not uniform (Figure 12a).

3.3. Identification of Flowing Water on the Basis of Thermal Orthophoto

An important aspect of post-mining process monitoring is the study of water quality [20,21,86]. Sedano-Cibrián et al. [87] and Koparan et al. [88,89] indicate that water quality can be assessed by, among other things, temperature. Iezzi and Todisco [90] note that water temperature can be influenced by natural and anthropological factors. Various measurement techniques can be used to measure water temperature, primarily based on probing and measuring points [Ibidem], as well as modern methods based on the use of thermal infrared techniques [91] and also unmanned aerial vehicles [87]. A thermal orthophotomap makes it possible to know the temperature of the land surface and water, as any object that has a temperature greater than absolute zero is a source of infrared radiation [92]. Figure 13 shows a thermal orthophotomap of the surface of the Weihnachtssee reservoir.

4. Discussion

The use of drones provides a detailed and reliable source of data for the study of post-mining processes, making it possible to analyze and interpret post-operation phenomena. UAV platforms allow for the incorporation of various sensors (e.g., multispectral, hyperspectral, thermal, LiDAR—light detection and ranging), which significantly increases the possibilities of UAVs. The software used in the controllers allows for the planning and execution of flight mission parameters, so that results can be obtained with the quality and accuracy assumed before the flight. This paper presents the results of using multispectral and thermal cameras. Drone aircraft with multispectral cameras enable the acquisition of certain spectral bands associated with vegetation health. By interpreting the processed aerial data, land-use classes can be verified, which are generally available on the Open NRW website. The use of machine learning algorithms makes it possible to classify land use (Figure 11). Therefore, it is possible to identify vegetation biotypes. One of the research methods implemented in this paper was the use of vegetation indicators. These indicators are based on mathematical formulas that take into account at least two spectral bands in their calculation. This makes it possible to study the state of the vegetation in a given area, but also, as shown in Figure 12, to identify water surfaces and coastlines. Monitoring aquatic environments using optical remote sensing has some limitations due to the need for direct visibility. Vegetation in the vicinity of water bodies and tree crowns can be an obstacle to correctly interpreting the shoreline of water bodies, and it is possible to take images from different angles to solve the problem [93]. However, scheduling a drone flight outside the growing season, as was done in this study, allows the previously mentioned problem to be solved. The resulting imagery can be used to calculate a numerical terrain model (Figure 14), enabling visualization of the terrain contours and derived products, such as the following:

-

Slope map, which illustrates the terrain’s slope, can be utilized to calculate rainfall-runoff, aiding in the development of flood control programs;

-

Exposure map, facilitating the examination of sunlight impact in specific areas;

-

Visibility map, applicable in constructing observation towers for forestry and tourism, serving as viewpoints.

The result of the use of an unmanned aerial vehicle equipped with a thermal camera is shown in Figure 13. It allows us to present the temperature of water reservoirs in the form of a map. When flying a drone with a thermal camera, the position of the sun during the flight must be taken into consideration, as sunlight can cause shadows, which can distort the result. Figure 13 enables us to see how the gradient in temperature characterized the SE-NW course. In the case of lakes Weihnachtsee and Pfingstsee, higher temperatures were observed in the southern parts of the reservoirs, which may be caused by the rivers Schwarzbach and Elsbach flowing through them. The conducted research showed that monitoring with the use of UAVs makes it possible to provide reliable spatial information regarding a given research area. The analysis of the natural environment’s condition involves observing changes occurring within it and studying the range and temperature of water surfaces. UAV monitoring is one of the many methods of geomonitoring and, in order to fully understand the post-mining processes, the obtained results should be verified using other sources and research methods.

5. Conclusions

The aim of this article was to present the possibility of using drone flights as a method of geomonitoring post-mining processes, using the example of the closed Prosper-Haniel mine in the Ruhr region. Monitoring the environment using unmanned aerial vehicles makes it possible to obtain data with high spatial resolution. The use of various sensors placed on drones, in this case a multispectral camera (DJI Phantom 4 Multispectral) and a thermal camera (DJI Mavic 2 Enterprise Advance Thermal), enables a comprehensive study of the state of the natural environment:

-

State of vitality and changes of vegetation, using vegetation indicators based on data obtained from a multispectral camera;

-

Verification and identification of vegetation types using machine learning algorithms—supervised classification;

-

Identification of water surfaces and detection coastline of water reservoirs;

-

Identification of the temperature of water surfaces and terrain using a thermal camera;

-

Creating digital terrain models to visualize the research area.

The research methods presented above can be used in the long-term monitoring of post-mining processes, which will enable the performance of time–spatial analyses documenting the course of changes taking place in the environment. It is worth noting that only the integration of all possible data on the research area allows for the understanding of the phenomena and processes taking place in it.