Mine Closure Surveillance and Feasibility of UAV–AI–MR Technology: A Review Study

Abstract

In recent years, mine site closure and rehabilitation have emerged as significant global challenges. The escalating number of abandoned mines, exemplified by over 60,000 in Australia in 2017, underscores the urgency. Growing public concerns and governmental focus on environmental issues are now jeopardising sustainable mining practices. This paper assesses the role of unmanned aerial vehicles (UAVs) in mine closure, exploring sensor technology, artificial intelligence (AI), and mixed reality (MR) applications. Prior research validates UAV efficacy in mining, introducing various deployable sensors. Some studies delve into AI’s use for UAV data analysis, but a comprehensive review integrating AI algorithms with MR methods for mine rehabilitation is lacking. The paper discusses data acquisition methods, repeatability, and barriers toward fully autonomous monitoring systems for mine closure projects. While UAVs prove adaptable with various sensors, constraints such as battery life and payload capacity impact effectiveness. Although UAVs hold potential for AI testing in mine closure studies, these applications have been overlooked. AI algorithms are pivotal for creating autonomous systems, reducing operator intervention. Moreover, MR’s significance in mine closure is evident, emphasising its application in the mining industry. Ultimately, a hybrid UAV–AI–MR technology is not only viable but essential for achieving successful mine closure and sustainable mining practices in the future.

1. Introduction

Mine site closure and abandonment are becoming more severe and profound challenges for most stakeholders worldwide. Due to the substantial disruption caused by mining, post-mine closure differs significantly from non-mining restoration programs [1,2]. In this regard, closure is defined as restoring damaged sites to their initial states of productivity and self-sufficiency while recognising the site’s and the area’s beneficial uses [3]. Restoration aims to create a self-sufficient ecosystem that can sustain itself without outside intervention [4]. Numerous countries have enacted legislation mandating mining companies adhere to strict criteria for restored topsoil, vegetation, and water quality following mine rehabilitation [5]. To accomplish these tasks, comprehensive management is required. However, management is impossible without monitoring and measurement. Monitoring and measuring the progress of a mine closure program is one of the critical challenges for all stakeholders engaged in the project [6]. The type of information needed changes as the closure area develops, and stakeholders should be aware that the system is moving toward a stable, sustainable state and that its constituent parts are evolving correctly [6]. Scholars have reported [7,8] that within the life of a mine (LoM), especially in its closure and rehabilitation phases, keeping track of updates, progressively planning and managing operations, determining and managing risks, and achieving a social licence to operate are significant challenges for mining companies [7,8]. Hence, for the mining, administrative, and consulting sectors, having monitoring and interpreting tools that offer information on many of the difficulties encountered during mine closure would be invaluable [9]. Given the complexity of the systems being controlled, Tongway and Ludwig [9] refer to an integrated monitoring approach as a “toolbox” that employs several monitoring techniques during the project. This “toolbox” of methods can be used to demonstrate the evidence of achieving a predetermined completion requirement. Currently, neither regulators nor operators have an integrated software package to manage and continuously regulate crucial inputs and the process of generating a mine closure framework.

In recent decades, several data collection techniques have been introduced by researchers. For acquiring precise terrain surfaces, survey methods like the Real-Time Kinematic Global Positioning System (RTK-GPS) and electronic distance measuring (Total Station) are frequently used [10]. However, because of the size of the field of interest, the time-consuming nature of terrestrial surveying, the risks associated with direct monitoring, and the difficulty of accessing some areas, terrestrial surveying is usually not an appropriate data collection method. Typically, three remote sensing methods—satellites, human-crewed aircraft, and unmanned aerial vehicles (UAVs)—can capture high-resolution photographs of vast areas [11]. Satellites may be constrained by insufficient spatial resolution, cloud cover, the running cycle, and the height of an orbit. Piloted planes may be constrained by airspace restrictions, sun angle and weather [12]. UAV-based remote sensing systems feature both common airborne characteristics and distinctive characteristics. The technology could well be utilised in high-risk regions or on windy days and can be swiftly and routinely operated. Despite this, the system’s limited coverage area results from its short battery life. UAVs can also operate close to their intended targets while collecting high-quality photos [13]. Therefore, UAV remote sensing is a versatile and effective method of getting high-resolution photographs. It offers precise information from low altitudes and is less expensive than satellites and human-crewed aircraft [14].

UAVs are being used more frequently nowadays because of advancements in sensor and control technologies. They can use a range of single-band, multispectral, and hyperspectral sensors, including those that function in visible, infrared, and microwave spectrums [15]. Their application in heritage and archaeological site mapping [16], transportation [17], civil engineering [18], agriculture [19], landscape mapping [20], wildlife conservation [21], infrastructure inspection [22], and search and rescue [23] have been investigated in several studies. According to McKenna, et al. [5], who conducted a study on the use of remote sensing in mine rehabilitation over more than 50 years, drones have become increasingly popular in recent years. The study found that although 50% of the studies used data from Landsat sensors throughout the 50-year period, other types of sensors became more prevalent over time, particularly between 2010 and 2019. In fact, during the years 2017–2019, drones were used in 25% of the studies, while aerial hyperspectral, aerial optical, high-resolution earth observation Satellite Pour l’Observation de la Terre (SPOT), and WorldView were used in 8%, 4%, 5%, and 3% of the studies, respectively. Figure 1 shows a notable rise in the incorporation of drones in research studies over the last ten years.

Due to advances in science and technology, the development and application of fully autonomous and remotely controlled systems are required for mine closure/rehabilitation projects. The data collected using UAVs can be leveraged to create mixed reality (MR) models. MR combines physical and digital worlds to create a seamless, natural, intuitive 3D human, computer, and environmental interaction [24]. This emerging reality is made possible by progress in computer vision, graphic processing, display technologies, input systems, and cloud computing [25]. MR utilises advanced sensing and imaging technologies to allow users to interact with and manipulate real and virtual objects and environments. MR’s intuitive and user-friendly interface makes it a practical solution for people of all ages and technical abilities to engage with related topics or issues [26]. The potential applications of MR are vast, for planning, training, and management purposes. MR models can be used to visualise and plan construction projects, such as bridges, roads, and buildings [27]. Furthermore, in mining engineering, UAVs powered by artificial intelligence (AI) technology can be used to collect and analyse data on mine sites for different applications. These data can then be used to generate MR models to visualise and plan mining operations, such as identifying the optimal location for a mine, assessing the impact of a new mine on the environment, and developing plans for mine closure and reclamation. For example, a mining engineer could use an MR model to simulate the excavation of a new mine and determine the most efficient and safe way to extract resources. Hence, MR technology has the potential to revolutionise the planning and management of rehabilitation projects. Its unique ability to merge digital content with the real-world environment allows for a more immersive and interactive experience, which can enhance the effectiveness of rehabilitation interventions. Furthermore, MR’s user-friendly interface enables individuals without technical expertise to participate in the process, making it a valuable tool for stakeholders at all levels of the project. MR can provide valuable insights and improve communication and collaboration among team members.

This paper examines the latest UAVs, AI, and MR applications in mine rehabilitation to determine the existing UAV setups and sensors used for monitoring mine closure. Furthermore, the AI algorithms employed for assessing the UAV data and obstacles to automating data collection are discussed. Finally, the feasibility of hybridising UAVs, AI, and MR for developing a fully autonomous mine closure surveillance technology (i.e., UAV–AI–MR technology) is assessed critically.

2. Application of UAVs in Mine Closure/Rehabilitation Projects

UAVs are used in mining to map sites, inspect the equipment, and gather data [28]. This information is useful in planning mining operations and monitoring environmental impacts. UAVs enable the faster and more efficient maintenance and repair of equipment [29,30,31]. Studies also aim to quantify mineral resources and characterise geomorphic properties [32,33]. Therefore, as seen in Figure 2, the applications of UAVs in the mining industry can be categorised into four groups: Exploration, Exploitation, Reclamation, and Miscellaneous. Figure 2 illustrates the present and possible uses in each category.

Mine rehabilitation monitoring is a new branch of remote sensing applications in the mining industry. As stated by Padró, et al. [34], UAV imagery was not used specifically for monitoring opencast mine restorations in a “standardised” way before 2019. However, some studies in recent years tried to fill the gaps and establish a foundation for more comprehensive studies [34,35,36,37]. Some studies tried to extract the changes in the ground surface by the differentiation of extraction and collection sites in exploited areas [38,39]. Xiang, et al. [38] evaluated the geomorphic changes associated with mining in northeast Beijing. Digital elevation models (DEMs) were generated from high-resolution photos collected with a UAV in 2014 and 2016. A geomorphic change assessment was conducted using two methodologies. As a first step, the differences between the two DEMs were quantitatively estimated on a cell-by-cell basis to determine the detectable area, volumetric changes, and the mined tonnage. The volumetric changes in these accumulation areas primarily represent the increase in tailings over the two years that were taken into consideration. An example of the map of differences between the two DEMs can be seen in Section 4.

In addition, the slope local length of autocorrelation (SLLAC) was used to retrieve the extent of the open pit using an empirical model to determine the surface covered by open-pit mining. Based on the DoD analysis, real changes and volumetric changes could be estimated, while the SLLAC analysis and its derived parameters (SPC, Surface Peak Curvature) have allowed for detailed depictions of terraces and open pit mine extent. Mining companies could use this information to plan production and environmental recovery carefully. By applying DoD and SLLAC, a substantial time series dataset can be derived, which might assist in planning for sustainable environmental management or minimise the effects of anthropogenic modifications caused by mining.

Padró, et al. [34] conducted a study on the restoration of a medium-sized open-pit mine (<10 ha) using a small, lightweight rotary-wing UAV equipped with sensors with a spatial resolution of 9 cm. The study used highly automated batch processes to reduce fieldwork time significantly compared to traditional inspection methods. The fieldwork involved geometric stacks maintenance, field Spectro Radiometric measurements, and UAV flight. With just two hours of ground-truth data collection and UAV flight, the restored area of 10 ha was thoroughly inspected. An operator took two working days to process the collected data. The research utilised the k-Nearest Neighbour (kNN) classification algorithm [40] to classify land cover automatically. The kNN algorithm is known for its resilience when distinct spectral responses are linked to a single informational class [41]. The study used multispectral sensors, low-cost radiometric reference panels, and a field Spectro Radiometer to produce detailed maps, including orthophoto maps, Digital Elevation Models, Digital Surface Models, vegetation indices, and thematic land cover maps. The maps were categorised into six categories—Mine waste, Topsoil, Tree cover, Shrublands, Grasslands, and Remaining shadows—and yielded an accuracy rate of over 94%.

Moudrý, et al. [42] evaluated the effectiveness of UAV-derived RGB and airborne-derived Light Detection and Ranging (LiDAR) point clouds in characterising a post-mining site. The researchers took images under two conditions, leaf-off and leaf-on. They found that point densities in photogrammetrically acquired point clouds under leaf-off conditions exceeded those acquired under leaf-on conditions, and provided complete coverage of the study area. The study also revealed that the accuracy of the digital terrain model (DTM) derived from images (UAV-based) acquired under leaf-off conditions was comparable to that of the LiDAR-derived DTM. However, the accuracy of UAV-based DTMs decreased in the following order: forest, steppes, and aquatic vegetation. Moudrý, et al. [42] suggested that leaf-off UAV imagery could be an effective and cost-efficient alternative for building DTMs for assessing risks associated with the instability of spoil bank terrain. Additionally, Moudrý, et al. [42] proposed that a combination of leaf-off and leaf-on acquisitions has the potential to replace expensive airborne LiDAR surveys for applications that require information on vegetation cover or height. Moreover, the usage of UAV imagery during both leafless and leafy seasons has the potential to enable the estimation of vegetation’s structural characteristics, which could assist in examining the susceptibility for slope collapse or determining the effectiveness of restoration efforts. Nonetheless, more investigation is necessary to quantitatively assess the images’ sensitivity and examine this technique’s possibilities [42].

To generate cartographic outputs for a quarry restoration using UAV, Carabassa, et al. [37] developed a unified process by combining various remote sensing applications. This involved using image processing techniques and terrain analysis algorithms for supervised land cover classification. They used a Scale-Invariant Feature Transform (SIFT) algorithm to create a database of singular points. The outputs included ortho imagery, Digital Surface Models (DSM), land cover maps, volume variation calculations, dust deposition and erosion identification, and drainage network evaluation maps. The land cover classification was conducted using both multispectral and visible RGB sensors, with the latter yielding the Modified Green Red Vegetation Index (MGRVI), Brightness Index (BI), and Triangular Greenness Index (TGI), while the Normalized Difference Vegetation Index (NDVI) was derived from multispectral imagery. To derive new products like vegetation, change or soil loss from erosion, time series data would be useful for comparison. Such extensive data could lead to increased monitoring and scientific interest, enabling the study of ecological and hydrological processes. Other products, such as those related to geotechnical risks or pest infestations, could be included to enhance the workflow. Adopting this approach can facilitate the large-scale monitoring of land restoration, optimise resources, promote sustainable practices, and foster collaboration among companies, environmental scientists, and public authorities.

Open-pit mining activity often results in the direct emission of particulates and gases into the atmosphere. To identify potential hazards, Nguyen, et al. [43] proposed an air quality monitoring system and spatial analysis using UAVs. They integrated gas and dust sensors into the UAV and used different spatial interpolation methods, including trilinear interpolation, nearest neighbour, and natural neighbour, to derive air concentration levels of CO, SO₂, PM2.5, and CO₂ in the atmosphere of open-pit coal mines (see Figure 3). The examined hazards included SO₂, CO₂, CO, and PM2.5. The results indicate that the UAV-based air quality monitoring system performed safely and efficiently in deep open-pit coal mines. The proposed approach was considered an additional option with simple procedures and an acceptable price to estimate the accurate 3D distribution of air quality in open-pit mines with flexibility.

Drone-acquired imagery is an important tool for the morphometric characterisation of terrains in open-pit mining, particularly for studying erosion processes in restored areas. Padró, et al. [35] have put forward a novel approach for estimating the Digital Elevation Model (DEM) in advance of the erosion process using data acquired by UAVs and Geographic Information System (GIS) techniques. Specifically, hydrologic analysis, such as channel network and watershed detection, was employed to estimate the DEMT0. The difference between the estimated DEM T₀ and the current DEM T₁ was calculated to obtain a DoD. The DoD was used to determine the extent of the estimated eroded area and volume of soil between T₀ and T₁. A case study was conducted to demonstrate the practical application of this approach. A workflow for detecting and calculating erosion was implemented, yielding valuable information for the management of quarry restoration. The approach proved to be especially effective in identifying areas with high erosion rates and corresponding soil accumulation. Additionally, the control of slope and erosion rates facilitates identifying and monitoring areas with active erosion, thereby aiding in the decision-making process for implementing stabilisation measures in highly susceptible landslide areas. Overall, the proposed method has demonstrated its suitability for mining operations and other similar geomorphological scenarios. Its successful application highlights its potential for widespread use in environmental monitoring and management, offering valuable insights into terrain characterisation and erosion processes. With the ability to accurately estimate and monitor changes in terrain, this approach provides a powerful tool for enhancing restoration efforts and promoting sustainable mining practices.

Russell, et al. [44] suggested a novel approach for determining the necessary filling material for restoration by utilising data from drones and GIS processing. However, the main contribution of the method was in extending the analysis beyond the excavation area to include the surrounding terrain in the relief restoration model. The study was conducted in Montsià, Masdenverge municipality, Tarragona province, Catalonia, located in Northeast Spain. The approach involved referencing the DEM obtained from the drone with control points taken from the official agency’s model, which improved the precision and comprehensiveness of the analysis. A digital height difference model (DHdM) was obtained by dividing the DEMs and used in volume change calculations. The method improved the accuracy of the restored relief model and provided information on the surrounding terrain. Additionally, the water flow behaviour confirmed that the modelled relief was well integrated with the environment. Users evaluated the tool, which enabled the easy and precise estimation of the required filling material for restoring the excavation pit morphology and predicting its environmental impact. Due to its high spatial resolution and accurate mathematical operations, the method provided useful criteria for decision-making in the treatment and restoration of excavation sites. Russell, et al. [44] indicated that future research should focus on incorporating additional variables to increase the effectiveness and reliability of the models.

3. Sensor Technology for UAV-Based Mine Closure Surveillance

UAVs come in various shapes and sizes, ranging from small handheld units to larger aircraft that can carry heavy payloads. They have many advantages, such as being cost-effective, manoeuvrable, and able to access hard-to-reach locations. However, they also have some disadvantages, including limited battery life, susceptibility to interference, and potential safety concerns. It is essential to understand the characteristics of UAVs to fully appreciate their capabilities and limitations and determine their appropriate applications. Moreover, in some of the studies, the authors mentioned the term Unmanned Aerial System (UAS) instead of UAVs, which refers to a UAV, a ground station, a communications link, and often a sensor to capture data [35]. However, when other articles use the term UAV, they also refer to all these systems; hence in this paper, the term UAV is used instead of UAS. Aerospace experts describe UAVs as autonomous, small, versatile aircraft that carry sensors and payloads, have long flight endurance, can be remotely controlled, cost less than manned aircraft, integrate with other systems, and can be rapidly deployed for emergencies. Ongoing technological advancements are expanding their potential uses and capabilities.

When it comes to using UAVs in mine closure and rehabilitation, the most important component is their sensors. UAVs can transport various sensors, which can be selected based on their intended purpose. These sensors can include acoustic (Figure 4a), visual (Figure 4a), chemical (Figure 4b), thermal (Figure 4c), and LiDAR (Figure 4d), which are designed for environmental remote sensing applications [35]. Meanwhile, some sensors that belong to the previously mentioned categories can also be classified as biological sensors. These sensors can be used to collect data related to living organisms, including plants and animals. Examples of such sensors include multispectral, hyperspectral, thermal, and acoustic sensors.

The use of acoustic sensors in mine rehabilitation, usable for identifying wildlife populations and finding gas leaks, has not been explored in studies in recent years. Nonetheless, visual and biological sensors have been extensively employed in most of the research. The different types of sensors used in mine closure/rehabilitation studies are discussed below.

Red, green, blue (RGB) sensors: Padró, et al. [35] utilised a DJI Zenmuse X5S (FC6520) RGB sensor with 20.8 megapixels (5280 × 3956 pixels) in their study (Figure 5). The sensor is capable of three-axis stabilisation through precise motor rotation, controlled with a dedicated processor that allows for an accuracy of ±0.01 degree. Additionally, the sensor was designed to reduce vibration using three shock-absorbing balls.

Two UAV surveys (specifically a Skywalker X5) were conducted to survey the Miyun Iron Mine during the study of Xiang, et al. [38]. The UAV was fixed-wing, with a wingspan of 1.2 m and a length of 0.6 m. They equipped the UAV with a Sony QX100 camera featuring a Carl Zeiss 10.4–37.1 mm lens, capable of capturing images with a resolution of 20.9 million pixels.

Multispectral sensors: Padró, et al. [34] employed an optical sensor known as Parrot Sequoia. This type of multispectral sensor is lightweight and can be easily installed on platforms with a maximum take-off weight of less than 2 kg. The sensor is equipped with four spectral bands—green (GRE), red (RED), red-edge (REG), and near-infrared (NIR)—arranged to enhance the detection of vegetation and its characteristics.

Olsson, et al. [45] conducted a study comparing multispectral sensors with handheld spectroradiometers, and found that the normalised difference vegetation index (NDVI) showed the highest correlation (R² = 0.99). They observed that the camera temperature and atmospheric conditions affected the sensor’s sensitivity, suggesting that the camera should be allowed to warm up before image collection, and reflectance calibration panels should be captured at a high elevation to account for atmospheric influence. The study also revealed that the orientation of the sensors had a significant impact on the images, introducing noise that limited the use of raw sensor data to adjust for differences in light conditions. To mitigate this issue, smoothing functions were applied to the sensor data before the irradiance normalisation of the images. These findings have implications for using multispectral sensors in remote sensing applications, particularly in relation to accounting for environmental factors that could influence the accuracy of the data collected.

Carabassa, et al. [37] conducted a comparison between RGB sensors and multispectral sensors. They utilised low-cost, multi-engine UAVs manufactured by DJI (Phantom 3 Pro and Phantom 4 Pro) for the image acquisition process. The optical sensor employed for collecting RGB data was the FC300X (for Phantom 3 Pro) and CMOS ’1 (for Phantom 4 Pro). Meanwhile, the Parrot Sequoia sensor was used to collect multispectral data, which operates in the G (green), R (red), Reg (red margin), and NIR (near-infrared) bands (Figure 6). According to Carabassa, et al. [37], because RGB sensors have a ten-fold greater resolution than multispectral sensors at the same altitude, multispectral sensors cannot provide the same spatial resolution as RGB sensors at the same altitude. A multispectral camera may also not be able to fly at the low altitudes necessary when flying over areas with steep slopes or vertical walls, such as quarries, due to safety concerns and time constraints. This results in lower spatial resolution for sensors with infrared bands.

Chemical sensors: Nguyen, et al. [43] utilised a DJI Inspire 2 drone equipped with air sensors to conduct air quality monitoring in their study. This allowed for the safe flight of the drone while collecting data on four main pollutants: PM2.5, CO, CO₂, and SO₂. The monitoring system consisted of both electrochemistry sensors and laser dust sensors. The laser dust sensor had a measurement range of 0–500 $\mathsf{\mu }\mathrm{g}/{\mathrm{m}}^3$, precision of ±10%, and a resolution of 0.3 $\mathsf{\mu }\mathrm{g}/{\mathrm{m}}^3$. The electrochemistry sensors for CO₂ and CO had a measurement range of 0–750 ppm, precision of 5% ppm, and a resolution of 1 ppm, while the SO₂ sensor had a measurement range of 0–20 ppm, precision of 5% ppm, and a resolution of 0.1 ppm. The sensors were mounted onto the drone, creating a UAV multi-sensor collector (UAV-MSC) system that could monitor multiple air components simultaneously. Figure 7 shows a chemical sensor box mounted on a UAV.

Light Detection and Ranging (LiDAR): According to the authors’ research, no recent studies have utilized LiDAR sensors for mine rehabilitation. McKenna, et al. [5] and Kim, et al. [47] have acknowledged this and suggested its importance. Therefore, it is necessary to evaluate the accuracy and quality of UAV data collection using LiDAR sensors to capture images and create 3D spatial data. This analysis will enable the identification of changes in mine rehabilitation, errors in estimation, and types of vegetation.

4. Data Acquisition Methods and Standards

Data acquisition through UAV/drone technology has become increasingly important in many industries. However, it is essential to make several considerations to ensure the successful and safe use of this technology. One important consideration is complying with national drone use regulations, as failure to do so can lead to legal and safety implications [24]. Furthermore, the sensors used in drones must be carefully selected and calibrated to ensure accurate data collection. Another consideration is the drone’s battery life, as it can significantly impact the duration and effectiveness of data collection missions. Technical aspects such as flight planning, control, and data management are also critical considerations, as they enable the drone to collect data autonomously and efficiently. Also, good programming is essential to enable the drone to perform the intended tasks while minimising the risk of accidents or malfunctions. As per information available on the Australian Civil Aviation Safety Authority website, certain regulations govern UAV flights. These include restrictions on operating at altitudes under 120 m, maintaining a minimum distance of 30 m from individuals, and the mandatory registration and licensing of UAVs by the authority. Overall, the careful consideration and planning of these factors are crucial for effective and safe data acquisition through drones.

Typically, UAV imagery uses two georeferencing techniques: direct and indirect [48]. Inertial Measurement Units (IMUs) and Global Navigation Satellite Systems (GNSS) are used to locate the camera’s viewpoint centre and approximate its orientation parameters during acquisition using direct georeferencing. Direct georeferencing is impacted by the quality and accuracy of onboard GNSS and IMU (Figure 8a). Indirect georeferencing, on the other hand, uses Ground Control Points (GCP) or locations with well-known coordinates in the scene. UAV imagery typically displays a chessboard target surrounded by a GNSS-based static post-processing technique for identifying GCPs [49]. Indirect georeferencing requires more time and effort than direct georeferencing. Furthermore, the accuracy of indirect georeferencing depends on the technician’s ability to locate the references in the imagery, the correct measurement of GCP positions, and the number of GCPs used (Figure 8b) [50].

The impact of meteorological conditions on UAV data acquisition cannot be underestimated [35]. Weather conditions such as wind, rain, cloud, and temperature can significantly affect the UAV’s performance and data quality. Strong winds, for instance, can cause instability and affect the positioning accuracy of the UAV, resulting in blurry images or poor data. In addition, high winds can lead to a shorter battery life of the UAV, thereby reducing the time available for data collection [51]. Rain is another significant weather factor that can affect data acquisition. The drops can interfere with the camera lens and cause image distortion, making obtaining accurate data difficult. Additionally, if the UAV is not waterproofed, rain can cause damage to the UAV’s electronic components [52]. Cloud cover can also limit the amount of sunlight reaching the UAV’s camera, affecting image quality and reducing the resolution of the data collected [34]. To some extent, it is also possible to apply shadow reduction techniques [53]. Furthermore, temperature changes can affect the UAV’s battery performance, resulting in shorter flight times and affecting the stability of the UAV. According to Olsson, et al. [45], the camera’s sensitivity is affected by its temperature, and the atmosphere has an impact on the images it captures. They emphasised the importance of allowing the camera to warm up before commencing data collection.

5. Algorithms and AI Methods for Data Post-Processing

One of the most important aspects of UAVs and their application for mine closure surveillance is the post-processing of the high volume of information compiled by UAVs and their sensors. The analysis of such a volume of recorded information using conventional manual techniques or simple statistical analysis methods is time-consuming, biased, and inappropriate for semi- and fully autonomous monitoring systems. This is where artificial intelligence (AI) and machine learning (ML) algorithms can play a crucial role. AI techniques can be used to recognise patterns and relationships within large datasets, allowing for the faster and more accurate analysis of UAV-derived data. They can help to identify changes or anomalies in the data that may be difficult for humans to detect, and can even provide predictive insights for future events based on historical data. Moreover, creating a successful Mixed Reality (MR) model necessitates the accurate development of a 3D model. In the context of a UAV–AI–MR model, where tools are essential for planning and monitoring the status of different aspects of the project, such as volumetric land changes over time, landslide control, water management, and vegetation density control, precise algorithms are crucial for analysing data obtained from UAVs. While numerous tools are available for generating 3D models from UAV imagery, scepticism persist about their accuracy [54,55,56]. This section introduces the algorithms in this domain and provides an explanation of their applications.

Structure from Motion (SfM): In most UAV-derived models, Structure from Motion (SfM) was the main method applied for generating 3D models. The SfM is a photogrammetric technique used to create 3D models from a set of overlapping 2D images captured by a UAV. The process involves identifying features in each image, such as corners or edges, and then finding matching features in other images of the same scene. These matching features are then used to triangulate the position of each feature in 3D space. Once the position of each feature is determined, a 3D model can be reconstructed by connecting the features together (Figure 9).

This process is known as “structure from motion” because the 3D structure is derived from the motion of the UAV and features within the images. The SfM method differs from traditional photogrammetry because it does not require precise camera calibration or ground control points. Instead, it relies on the overlapping images and the position and orientation of the UAV to create a 3D model. SfM can also produce dense point clouds, which provide more detailed information about the 3D structure of the scene. SfM software packages (e.g., VisualSFM [58], OpenMVG [59], and COLMAP [60] as open access versions and Agisoft Metashape, Pix4Dmapper, and RealityCapture as commercial versions) have become widespread over the last decade [61,62,63]. It is now possible to obtain products with centimetre resolutions using SfM techniques, including orthophoto images, Digital Terrain Models (DTM), and Digital Surface Models (DSM) [64].

DEM of Difference (DoD): DEM of Difference (DoD) is a method used in 3D modelling to compare two DEMs of the same area taken at different times [65]. The DoD technique calculates the difference between the elevations of the two DEMs on a cell-by-cell basis. This difference represents the changes in the area between the two data collections, such as the removal of soil or rock due to mining activities or the deposition of sediments due to natural processes like erosion and deposition (see Figure 10) [66]. The DoD method is a simple but powerful technique for detecting and quantifying changes in the terrain. It can be used with various sensors, including airborne and satellite-borne remote sensing platforms. By comparing DEMs acquired at different times, the DoD method provides a cost-effective and efficient way to monitor and analyse changes in the landscape and support decision-making in various fields. As part of the DoD’s method, an earlier terrain elevation is subtracted from a later one while considering measurement errors, sampling bias, and interpolation errors associated with the DEMs [66]. Therefore, because the error statistic is averaged over the entire surface, it may overestimate or underestimate elevation changes in some areas of the DEM [67]. This limitation is acceptable when changes are subtle, but unsuitable when there is a small signal-to-noise ratio. Most studies in mining research have high signal-to-noise ratios, which are used to differentiate target features that are tens of meters in height and thousands of square meters in width, with an error of approximately 0.5 m. As a result of the acquisition parameters and DEM generation approach, uniform uncertainty seems reasonable. The level of detection (Lod) enables the DoD to distinguish between real and noise geomorphic changes to refine its uncertainty [68]. The Lod determines whether an elevation change is real, and those below it are typically discarded [69].

Digital Height Difference Model (DHdM): Similar to DoD, a DHdM is obtained by computing the variation between the surface and elevation models (see Figure 11) [34,71]. It provides data on the vertical positions of various features like trees, vegetation, and structures in the environment. This method is commonly used to observe elevation pattern alterations caused by erosion, deposition, or land sinking over time [44].

Slope local length of autocorrelation (SLLAC): Based on high-resolution digital elevation models (DEMs), Sofia, et al. [72] proposed the slope local length of autocorrelation (SLLAC) as a methodology for assessing open-pit mining’s geomorphic changes in 2014. The SLLAC calculates the extent of open-pit mining and estimates the surface covered by open-pit mining. It uses autocorrelation to determine slope values within a moving window of a specified length, called the slope local length of autocorrelation. In autocorrelation, a variable’s spatial relationship with itself at different lags is described. The SLLAC method estimates the length of terraces in open-pit mines by measuring the point when the autocorrelation of slope values decreases below a specified threshold [72]. Compared to other methods that rely solely on the difference between DEMs, this method accurately depicts changes within an open-pit mine [73]. In addition to estimating slopes and defining terraces, SLLAC provides a detailed depiction of the open-pit mine’s extent and surface peak curvature (Spc).

The Normalised Difference Vegetation Index (NDVI): NDVI is a widely used remote sensing index that provides information about the greenness or density of vegetation in a specific location [74]. It is the most frequently used vegetation index among more than 100 types, with its formula introduced by Rouse, et al. [75]. This formula involves dividing the difference between the amount of near-infrared radiation reflected by vegetation (NIR) and the amount of visible red light reflected by vegetation (Red) by their sum, resulting in a value ranging from −1 to 1, with higher values indicating more green vegetation [76]. NDVI has several benefits over other vegetation indices. First, the ratio processing of NDVI decreases errors caused by sun elevation angle, terrain slope, and observation angle. Second, NDVI reduces the effect of weather conditions to some extent. Lastly, NDVI eliminates the impact of water and bare soil, enhancing the ability to determine the growth state of vegetation. Overall, NDVI can provide a macroscopic view of vegetation growth and distribution, making it ideal for researching vegetation classification and monitoring, climate change monitoring, and monitoring ecological environment changes [77]. McKenna, et al. [5] research found that over the last four decades, remote sensing techniques have frequently employed the NDVI index to evaluate vegetation growth on restored landscapes.

The Flow Accumulation Model (FAM): FAM is a helpful and interesting technique for detecting preferred water circulation areas and streams in a terrain [37]. Jenson and Domingue [78] developed this model, which is commonly executed using the study area’s digital elevation model (DEM). The DEM is a grid of cells that represent the elevation of the ground surface, with each cell containing a value for its height above sea level. The FAM model employs the DEM to calculate each cell’s flow direction and accumulation within the grid [37].

Scale-Invariant Feature Transform (SIFT): The SIFT algorithm is a popular computer vision technique used for image feature recognition and detection. It was created by Lowe [79] and is widely applied in image processing tasks such as robotic vision, 3D modelling, and object recognition. SIFT detects and describes key features in an image that can be utilised to match and align images taken under various lighting conditions or from different viewpoints [37]. The algorithm comprises multiple steps: scale-space extrema detection, low-contrast point elimination, key point orientation assignment, and descriptor generation. SIFT’s primary advantage is its ability to detect and match features across various scales and orientations, which makes it resistant to alterations in viewpoint, illumination, and other factors. It has been used in several fields, including fingerprint matching in forensic science, automatic panorama stitching in photography, and object recognition in robotics.

The latest research on employing UAVs for mine reclamation reveals a significant discrepancy in implementing machine learning and AI-based algorithms in UAV flight control and image processing. While these techniques are frequently utilised in other fields, such as satellite image processing, they are not widely used in mine reclamation research using UAVs. As a result, incorporating these methods in future studies can significantly enhance the outcome of monitoring efforts for mine reclamation.

6. Mixed Reality Role in Mine Closure Surveillance

The process of restoring a mine is closely connected to decision-making, which involves not only experts but also stakeholders who lack specific engineering expertise. For this reason, it is crucial to effectively communicate the rehabilitation plans and progression to all stakeholders, such as local communities, government officials, and investors. By doing so, all parties can make well-informed decisions and contribute to successful mine restoration (see Figure 12).

In the previous sections, the remote sensing and artificial intelligence applications used in generating the data were discussed in detail. These technologies are being used more often to monitor the environment by scientists across many different fields as they become more well-known and accessible. However, significant expertise and knowledge are needed to evaluate and make logical sense of remote sensing evaluations of mine site rehabilitation [5]. Therefore, an integrated and automated program is required to transform the UAV-and-AI-generated data into a model to provide detailed data and analysis from the mining field for engineers, officials, stakeholders, and employees, in order to enhance and facilitate the rehabilitation and closure procedure in mining projects. Mixed Reality (MR) technology has the potential to be a viable solution due to its user-friendly and intuitive interface, which enables people of all ages and technical backgrounds to engage easily with related topics or issues.

In the 1960s, initial ideas and conceptualisations for Virtual Reality (VR) emerged, and advancements in technology during the 1990s led to the further development of VR applications [80,81]. The Virtuality Continuum, created by Milgram and Kishino [82], is widely used to classify VR and explain the various levels of immersion between real and virtual environments. Augmented Reality (AR) was introduced in the 1990s and involved digitally adding virtual objects to a real-world environment, while Augmented Virtuality refers to adding real-world elements to a fully virtual environment [82]. Azuma [83] distinguished the main difference between VR and AR as being that AR supplements reality rather than replacing it entirely. Mixed Reality (MR) is a technology that surpasses AR in its ability to merge the physical and digital worlds. Using advanced sensing and imaging technologies, MR allows users to interact with and manipulate real and virtual objects and environments [84]. Essentially, MR provides a more immersive experience where virtual objects can interact with the real world more authentically. This enables users to experience the presence of virtual objects in their actual environment, enhancing their perception of reality [85].

Generally, MR has multiple applications in engineering, such as design visualisation, training and simulation, maintenance and repair, collaboration and communication, and marketing and sales [86]. Design visualisation involves using MR to view and interact with 3D models and CAD designs in real-time, leading to more intuitive and immersive design decisions [87,88]. MR can create realistic simulations of real-world scenarios for training and simulation, providing a safe and controlled environment for engineers to practice and improve their skills [89,90]. In maintenance and repair, MR can offer real-time visual guidance to technicians, leading to reduced downtime and improved efficiency [91]. MR can also facilitate remote collaboration and communication between engineers and teams, enhancing work effectiveness and efficiency [91]. Finally, MR can be used for marketing and sales by providing interactive product demonstrations and presentations, resulting in a more engaging and immersive way to showcase products and designs [92].

While there has been considerable research on the applications of VR and AR in mining engineering, there is a lack of extensive studies exploring MR in this field. Nonetheless, it is possible to extend the fields of studies used in VR and AR to MR applications, potentially leading to more practical and meaningful results. Stothard, et al. [26] suggested that advanced mining methods could benefit from the application of MR technology, as it could enhance the sustainability of mining operations. MR can potentially improve the understanding of diverse data sets, models, and situations, ultimately leading to more informed operator decision-making. This could result in more sustainable mining practices [26]. In the Engineering Geology and Resource Geotechnics Research Group at Simon Fraser University, MR has been used to study landslides and open pits [93]. Isleyen and Duzgun [94] utilised VR to produce training simulations that focus on evaluating the danger of roof falls in tunnelling projects. The main goal of these simulations was to improve people’s decision-making abilities, particularly in scenarios where quick decisions are required to ensure the safety of the operation.

Edelbro, et al. [95] conducted 20 qualitative interviews with rock mechanics engineers and other professionals in related roles across different industries and roles. Their research revealed that there are notable benefits to utilising AR technology. The most important advantage identified by the authors for the future use of AR technology would be the ability to visualise data gathered from field investigations and to perform real-time analyses, such as wedge and stress analysis, to create solutions tailored to the specific needs of rock reinforcements. For training and education in virtual reality, UNSW’s School of Minerals and Energy Resources Engineering in Sydney created a longwall mine and a transport roadway [96]. In a project aimed at the environmental reclamation of an Alberta oil sands mine site, the BGC Engineering company attempted to develop MR models for communication purposes (Figure 13). Their objective was to assist clients in improving shareholder value, obtaining the social license for a project, and lowering project expenses. This project started in 2017, and the present status is unknown [97].

7. Hybridising UAV, AI and MR for a Novel Surveillance Technology

This study has discussed the latest remote sensing techniques for measuring mining damage rehabilitation. However, remote sensing has failed to produce useful results for ecological monitoring and restoration science over the past 50 years [5,98]. A minimal amount of consideration was given to restoration, ecosystem growth, and the ultimate objectives of rehabilitation, such as the closure of mines [5]. According to Lechner, et al. [99], a remote sensing tool with the ability to “look back in time” can be used to assess mining land cover, better understand the physical drivers of contemporary complexity, and combine techniques and viewpoints from various disciplines to provide a comprehensive overview of the socio-environmental effects of mining operations. To regularly monitor mine closure from these perspectives, studies should go beyond conventional methods. When compared to satellite imaging, McKenna, et al. [5] assert that UAV imagery is deficient in historical archive images. Stothard and Shirani Faradonbeh [24] highlighted the potential benefits of integrating autonomous UAV survey data, AI capability, and MR technology to create a comprehensive and practical tool for monitoring mine site rehabilitation projects. The study suggests that ensuring positive mine site sustainability outcomes and maintaining stakeholder engagement over the long term of mining projects is crucial. To achieve this, the study proposes establishing a baseline of mutually agreed milestones and parameters that must be met to achieve final landform rehabilitation to the satisfaction of all stakeholders.

According to Stothard and Shirani Faradonbeh [24], combining UAV data, images, and AI technology with MR simulation and visualisation could be a viable solution to achieve these goals. AI technology can be utilised at different stages of UAV surveys, including before, during, and after the flight. AI algorithms can assist with path planning issues and software development in the pre-flight phase. Typically, the analysis and interpretation of UAV images occur after the flight, which is a slow process and unsuitable for real-time feedback for MR simulations. Nonetheless, cloud-based platforms and edge computing can enable UAVs to perform onboard data processing using deep-learning techniques on the CPU and GPU [24,100]. This integration could provide an advanced compliance monitoring tool that helps to maintain the positive sustainability outcomes, engage stakeholders, and achieve mutually agreed rehabilitation goals. Figure 14 illustrates the flowchart of the UAV–AI–MR system.

As seen in Figure 14, the comprehensive system is initiated by introducing the rehabilitation project’s field of interest. It comprises four main components: parallel computing unit, UAV, AI, and MR. Since the UAV component is the slowest, the system’s speed is regulated by the UAV mission speed. The core unit splits the field according to the UAV’s data acquisition capacity and configuration. After partitioning the site, the core unit orders the UAV to start the data acquisition mission while increasing the AI-MR computing speed by assigning computation to parallel units. As data acquisition for a particular segment is completed, the AI-MR data processing for that section commences. Within the AI section, various algorithms are employed to verify the collected data’s accuracy. Should any errors be identified within the data, the drone will receive instructions to redo the data acquisition process. Once it is confirmed that the data are accurate, other artificial intelligence algorithms are utilised to examine and generate models from the data. These resultant models are subsequently employed to develop MR. The core unit awaits data for other parts to distribute to other processing units. Finally, the processing procedure concludes with the MR’s development for the last section, validated for accuracy by the core unit.

8. Conclusions

This study aimed to explore the feasibility of incorporating UAV–AI–MR as an all-encompassing system for effectively managing a mine closure project. The components of the proposed technology were thoroughly examined, and their respective applications in mine closure, as well as their benefits, drawbacks, and gaps, were discussed in detail. It was determined that UAV might be a more effective data collection method than traditional approaches. However, their limited battery life and payload capacity could hinder projects with large areas of interest. These technical issues can be addressed using AI sub-programs within the comprehensive system, which can identify the most efficient data collection routes, calculate flight times, and ensure the return to base for recharging before continuing the mission. Furthermore, certain sensors, such as LiDAR, have yet to be tested in a closure project on a UAV, and should be explored in future research. The utilisation of algorithms and AI techniques has seen a broad range of applications in recent years, including 3D modelling, ground surface change tracking, and vegetation indicators. However, there has been a lack of research focused on AI methods that can automatically detect and control changes in surface components. Additionally, the computational time required by recently developed algorithms has not been adequately discussed, despite its importance in developing comprehensive tools for the future. Furthermore, real-time compliance checker tools have not been developed, and the benefits of high-performance computing (HPC) and cloud computing in developing a comprehensive system have not been investigated in the literature. To create a comprehensive system, a multitude of AI-based sub-systems need to be developed and integrated into a system that can work simultaneously using HPC and cloud computing techniques. Integrating AI methods with MR engines can create a fully autonomous UAV–AI–MR program.

In conclusion, the development of a UAV–AI–MR system for mine closure does not seem far too far off, and may be achieved in the near future. Most of the necessary components of the system have been tested or developed in various studies in recent years. However, some gaps in data acquisition, data processing and visualisation, which are mentioned in the paper, should be addressed in future studies and research.