An AI-Based Approach for Developing a Recommendation System for Underground Mining Methods Pre-Selection

Abstract

Selecting the most appropriate mining method to recover mineral resources is a critical decision-making task in mining project development. This study introduces an artificial intelligence-based mining methods recommendation system (AI-MMRS) for the pre-selection of underground mining methods. The study integrates and evaluates the capability of two approaches for mining methods selection (MMS): the memory-based collaborative filtering (CF) approach aided by the UBC-MMS system to predict the top-3 relevant mining methods and supervised machine learning (ML) classification algorithms to enhance the effectiveness and novelty of the AI-MMRS, addressing the limitations of the CF approach. The results reveal that the memory-based CF approach achieves an accuracy ranging from 81.8% to 87.9%. Among the classification algorithms, artificial neural network (ANN) and k-nearest neighbors (KNN) classifiers perform the best, with accuracy levels of 66.7% and 63.6%, respectively. These findings demonstrate the effectiveness and viability of both approaches in MMS, acknowledging their limitations and the need for continuous training and optimization. The proposed AI-MMRS for the pre-selection stage supplemented by the direct involvement of mining professionals in later stages of MMS, has the potential to significantly aid in the MMS decision-making, providing data-driven and experience-based recommendations following the ongoing evolution of mining practices.

1. Introduction

The mining industry plays a crucial role in human civilization, providing raw materials, critical minerals, and metals vital for development, technological advancements, and economic growth. Central to the success of any mining project is the careful selection of the most feasible mining methods to extract mineral resources from the earth. These methods typically fall into two most common categories: surface and underground mining. Underground mining methods are more complex; they are applied to recover deposits that are not economically (socially or politically, technically, and economically) feasible to be recovered using surface methods. Underground mining methods are usually classified into three groups based on a geotechnical viewpoint depending on the extent of support required to sustain underground infrastructure and ensure the stability of the openings: unsupported, supported, and caving [1]. Each mining method presents its own set of advantages and drawbacks, and the selection of the most appropriate method(s) is influenced by multiple interconnected influencing factors. Key factors in selecting surface and underground mining methods include the orebody geometry, geology and hydrology, geotechnical properties, technological factors, economic factors, and environmental considerations [1,2]. The primary objective of mining methods selection (MMS) is to select method(s) that maximize profitability, operational efficiency, safety, and cost-effectiveness while minimizing negative environmental impacts. Typically, the task of MMS is performed by experienced mining professionals (or decision-makers) who usually conduct comprehensive evaluations of the various influencing factors to achieve the primary goal of MMS. Over the years, the subject of MMS has undergone extensive study, leading to the development of different methodologies aimed at addressing this complex decision-making process.

1.1. Previous Work on Mining Methods Selection

Research on mining methods selection (MMS) has been ongoing for several years, yielding various methodologies/systems aimed at aiding decision-makers in this complex decision-making process. This study categorizes MMS systems into four groups: qualitative, quantitative, MCDM-based, and machine-learning-based.

Qualitative and quantitative systems were introduced in the 1970s, 1980s and 1990s. Qualitative systems were flowcharts used to evaluate the suitability of mining methods for particular deposits, mostly based on orebody geometry, geology, and geotechnical properties [1,3]. The need for improvement and numerical methods led to the introduction of the first quantitative system in 1981: the Nicholas approach [4]. In 1995, the University of British Columbia (UBC) developed the UBC-MMS system [5]: a modified version of the Nicholas approach to adjust the Nicholas approach to Canadian mining operations. Quantitative systems are based on a numerical decision matrix with ranks/weights assigned to evaluate the feasibility of the mining methods based on orebody characteristics, including orebody geometry, geology, and geotechnical properties. The advantage of the quantitative system is providing solutions for the first phase of MMS or preliminary selection providing a set of safe or suitable mining methods that will be submitted for further detailed evaluation by the decision-making team. Perhaps that is one of the reasons quantitative systems are still the most practically implemented in industries and studies/research. However, in these systems, the relative importance of the influencing factors is not considered, implying that all factors have the same weight (i.e., level of importance) during the evaluation and selection process; plus, the systems may offer obsolete solutions for the current situation in the mining practices. Since the 2000s, attention to multi-criteria decision-making (MCDM) techniques has gained traction in addressing the shortcomings of quantitative systems, introducing MCDM-based systems. Over the years, crisp and fuzzy MCDM techniques, commonly the analytical hierarchy process (AHP), order of preference by similarity to ideal solution (TOPSIS), and preference-ranking organization method for enrichment evaluations (PROMETHEE), have been implemented for evaluating and selecting mining methods for different case studies [6,7,8,9]. The advantage of the MCDM-based systems is the consideration of the weights or the relative importance of the influencing factors and the possibility of including a wide range of factors and methods customized to a case study. In most MCDM-based systems, decision-makers (mining professionals) are directly involved in the task, which can be of great advantage since the team can customize the evaluations according to the objectives of a particular case study. Usually, the decision-making team is assigned to select relevant methods and factors to consider in the evaluations and to determine the weights or the relative importance of the factors based on their subjective judgment or opinions, which reflect their experience in the field. However, using subjective judgment may introduce a certain level of bias, which can inherently affect the eligibility and accuracy of the evaluations [10,11]. Plus, the subjective judgment from the decision-makers is mostly customized to a particular case study or mining project; that is, a decision made for a particular case study/project may not be transferrable to another case study/project [11]. In a study by Manjate et al. [10], the Entropy method (Shannon’s Entropy) was proposed as a method to determine the weights of the factors objectively without the direct involvement of decision-makers in an attempt to avoid bias and the customization of the decision-making process and enable the implementation of the results when the direct judgment of the decision-makers is unavailable (totally or partially) or not required.

In recent years, data-driven decision-making powered by artificial intelligence (AI) systems has been heavily studied and gradually implemented for solving complex problems in different fields of science and engineering. The concept of Smart Mining or Mining 4.0 has emerged in the mining industry, which involves integrating digital technologies, AI, and advanced data analytics to address challenges in the mining value chain to optimize mining activities. Machine learning (ML) is a popular field of study in AI aimed at training predictive models for AI systems; ML-based systems usually rely on algorithms trained based on extensive historical data to learn the data pattern and predict future outcomes. Recently, a few studies [12,13,14,15,16] have investigated the applicability of artificial neural networks (ANN) in MMS introducing ML-based MMS systems. Lv and Zhihui [13] and Chen and Shixiang [14] investigated and showed the effectiveness of ANN in evaluating and selecting underground mining methods for thin and thick coal seams. In 2020, Ozyurt and Karadogan [12] further investigated the applicability of ANNs and game theory to develop a model for underground MMS for different ore types. Their study further demonstrated the effectiveness of ANNs in developing a robust system for detailed evaluations of underground mining conditions to select the most feasible mining method. According to the authors, their model can be applied even when there is a lack of information regarding the relevant factors, thereby addressing one of the shortcomings of MCDM-based systems. In 2022, Shohda et al. [16] developed a model for MMS by modifying the UBC-MMS system and integrating ANN. Their study compared the results of the ANN model with a commonly used MCDM TOPSIS technique, showing that the ANN model provided similar results to the TOPSIS more easily and accurately. The above study demonstrated ANN’s effectiveness in solving the complexity of the MMS process. Manjate et al. [11] introduced the recommendation system concept in MMS through one of the collaborative filtering algorithms: the nonnegative matrix factorization (NMF) algorithm. The authors investigated the applicability of NMF to predicting underground mining methods, which was shown to be effective with reasonable performance. Their NMF model was also proposed for predicting missing information about the required input variables for MMS to enable the implementation of the model even when information about some input variables is not accessible. The advantage of ML-based systems is the provision of data-driven decision-making insights and the possibility to continuously train and optimize the models with up-to-date information. Moreover, ML-based systems may enable the possible use of the system with or without the direct involvement of decision-makers, even with missing variables.

1.2. Purpose of the Study

Considering improving and expanding the application of AI in MMS, this study proposes an AI-based approach to develop a system to recommend underground mining methods in the preliminary stage of MMS (i.e., pre-selection). The proposed MMS system is termed the AI-based mining methods recommendation system (AI-MMRS). The proposed approach is to explore available databases of mining projects to develop a decision-making support system during mine planning mainly based on the recommendation systems approach. Recommendation systems are AI systems aimed at helping users deal with information overload by filtering information to make customized recommendations (of items that users might like), thus improving users’ decision-making capability [17]. The study’s main objective is to investigate effective approaches for developing the AI-MMRS integrating the memory-based collaborative filtering approach and classification machine learning algorithms for MMS. This study also integrates the results from previous studies where the Entropy method [10] and nonnegative matrix factorization (NMF) [11] were proposed as strategies to address challenges and gaps in the MMS, therefore, enhancing the effectiveness of the AI-MMRS. The integrated AI-MMRS is intended to recommend underground mining methods by evaluating the similarities among the projects regarding orebody characteristics, including ore strength, host-rock strength, orebody thickness, shape, and dip. By acknowledging the critical importance of MMS in mine planning and the limitations of AI-based approaches in such a decision-making task, this study emphasizes the need for the direct involvement of the decision-making team (i.e., mining professionals) to perform detailed evaluations and selection of the most feasible methods in the MMS process. As such, the proposed AI-MMRS is intended to serve as a decision-making support tool for the preliminary selection of underground mining methods (i.e., based on orebody characteristics), thereby recommending a set of methods that are suitable to the orebody characteristics, which must then be submitted for further detailed evaluations (i.e., environmental, economic, technological, and other relevant aspects for evaluations) by the decision-making team. The proposed AI-MMRS has the potential to enhance the efficiency and effectiveness of the decision-making process by providing data-driven and experience-based recommendations based on past or existing mining projects’ procedures, practices, and experiences. Furthermore, by leveraging the power of AI aided by the direct involvement of decision-makers (in later stages of MMS), this approach can help to identify and address key challenges in mining project development and support the ongoing evolution of mining practices, especially the MMS discipline.

The remainder of this paper is organized into four main sections. Section 2 describes and explains the methodology for developing the AI-MMRS. Section 3 presents the experimental results. Section 4 presents a discussion of the results. Finally, the concluding remarks are presented in Section 5.

2. Materials and Methods

The process of evaluating and selecting mining methods typically involves two or three main stages, with the initial stage focusing on the preliminary selection of methods suited to the deposit’s characteristics, which is the outcome given in most qualitative and quantitative mining methods selection (MMS) systems [1,3,4,5]. This preliminary selection phase usually assesses factors such as orebody geometry, geology, and geotechnical properties to identify a set of suitable mining methods and eliminate less suitable ones. Subsequently, the methods deemed most suitable undergo further feasibility evaluations in later stages, considering technological, economic, environmental, and other relevant factors to select the most sustainable method(s) to recover the ore deposit. This study emphasizes the need for the direct involvement of the decision-making team (i.e., mining professionals) to perform detailed evaluations and selection of the most sustainable mining methods in the MMS process, thereby proposing an AI-based mining methods recommendation system (AI-MMRS) for the preliminary selection of underground mining methods.

The methodology for developing the AI-MMRS integrates artificial intelligence (AI) techniques to leverage available mining project databases and facilitate the decision-making process in mine planning, particularly in the mining methods selection (MMS) task. Specifically, the study explores the effectiveness of the memory-based collaborative filtering (CF) approach [18] and classification machine learning (ML) algorithms [19] for predicting underground mining methods based on orebody characteristics (i.e., ore strength, host-rock strength, thickness, shape, and dip). Figure 1 illustrates the methodological strategy for developing the AI-MMRS, encompassing data preparation and ML model evaluation as the main phases. In the data preparation phase, this study incorporates findings from a previous study to determine the relative importance of factors influencing the MMS process [10]. Additionally, insights from a prior study where the nonnegative matrix factorization (NMF) algorithm was proposed to predict mining methods and address missing variables in sparse input data are integrated into the AI-MMRS [11].

2.1. Estimating the Relative Importance of Factors Influencing MMS

Naturally, mining methods selection (MMS) is a complex task given the need to consider many conflicting (and interconnected) factors to evaluate the performance of different mining methods and select the most feasible method(s). In this study, the Entropy method is adopted in the data preparation phase to address the complexity of the MMS caused by many influencing factors. As such, the Entropy method is proposed as a feature selection technique to assess the relative importance of factors influencing the MMS process and identify the most relevant factors to be used as main input variables in the input dataset. In ML, feature selection [20] is an essential step to improve the performance of the prediction models (and reduce computation time) and the quality of the outcomes by reducing the number of irrelevant (and redundant) features and selecting the most relevant ones for building prediction models.

The Entropy method is used in a multi-criteria decision-making (MCDM) problem to assess the relative importance of influencing factors in the MMS process [10]. In most MCDM-based studies, the relative importance of the influencing factors is determined based on the direct subjective opinions or judgment of the decision-makers involved in the evaluation process for particular projects or case studies [6,7,8,9]. While the use of a subjective approach offers the benefits of providing customized solutions to a specific case study, it also means that the decisions made may not be generalized, i.e., the decisions may not be easily applicable or transferable to different case studies. The proposed AI-based mining methods recommendation system (AI-MMRS) seeks to leverage data-driven decision-making drawn from diverse case studies, enabling its implementation across different case studies (or project types). Subjective decision-making methods, while effective in certain contexts, would introduce bias into the outputs of the AI-MMRS, potentially compromising their eligibility and accuracy. Recognizing the limitation of subjective methods and considering the primary aim of the study, which is to develop a system capable of supporting the decision-making process across various case studies, the Entropy method was found suitable due to its advantage of estimating the weights of the factors objectively. Unlike subjective decision-making, the Entropy method determines the weights of the factors objectively without the need for the direct involvement of decision-makers, thus, providing less biased and generalized outcomes suitable for diverse case studies. In this case, the use of the Entropy method enhances the effectiveness and applicability of the AI-MMRS across diverse mining projects.

In our previous study [10], the Entropy method, which is rooted in information theory by Shannon [21], was applied to assess the relative importance of the factors categorized into orebody geometry, geology and geotechnical properties, technological, economic, and environmental factors. The Entropy method estimated the objective weights of the factors based on a decision matrix created from literature sources, particularly methodologies proposed by Miller et al. [5] and Hartman and Mutmansky [1], renowned for their contributions to the MMS discipline. This decision matrix encompassed twenty influencing factors which are used to evaluate twelve mining methods (comprising both surface and underground methods). Using the decision matrix as input, the Entropy method determined the objective weights of the factors based on the following steps:

• Normalization of values in the decision matrix using Equation (1) to ensure uniform scaling across factors (or criteria) and mining methods (or alternatives):

$$r_{ij}={\displaystyle \frac{x_{ij}}{{\sum }_{j=1}^n{x}_{ij}}}, i=\mathrm{1,2}\dots ,m; j=\mathrm{1,2}\dots n$$

(1)

where x_ij is the criteria rate; r_ij is the normalized criteria rate; n is the number of criteria (or factors), and m is the number of alternatives (mining methods).

2.

Computation of Entropy (Ej) values in the matrix using Equation (2), measuring the uncertainty among mining methods in the absence of factor preference:

$$E_j=-h\sum _{i=1}^m{r}_{ij}\ln \left(r_{ij}\right),j=\mathrm{1,2},\dots ,n; h={\displaystyle \frac{1}{ln\left(m\right)}}$$

(2)

where r_ijln(r_ij) = 0 if r_ij = 0 and h is the Entropy constant.

3.

Determination of diversity (Dj) or the degree of diversification using Equation (3), which indicates the dispersion between rates of different mining methods for each factor:

$$D_j=1-E_j$$

(3)

4.

Calculation of objective weights of criteria based on Equation (4), which represents the relative importance of factors in the evaluation and selection process:

$$w_j={\displaystyle \frac{D_j}{{\sum }_{j=1}^n{D}_j}},j=\mathrm{1,2},\dots ,n$$

(4)

where w_j is the degree of importance or the objective weight of criterion j.

The results of the Entropy method, as shown in

Table 1. Results of the Entropy method: highlighting the five most relevant factors.

Influential Factor	Entropy	Diversity	Weights
Ore strength	0.895	0.105	0.132
Host-rock strength	0.909	0.091	0.115
Thickness	0.917	0.083	0.104
Shape	0.920	0.080	0.100
Dip	0.943	0.057	0.072
Ore uniformity	0.946	0.054	0.068
Mining cost	0.952	0.048	0.061
Dilution	0.955	0.045	0.057
Ore grade	0.961	0.039	0.048
Selectivity	0.968	0.032	0.040
Recovery	0.976	0.024	0.030
Productivity	0.977	0.023	0.029
Capital investment	0.981	0.019	0.024
Depth capacity	0.982	0.018	0.023
Flexibility	0.982	0.018	0.022
Depth	0.985	0.015	0.019
Production rate	0.985	0.015	0.019
Development rate	0.985	0.015	0.019
Health and Safety	0.991	0.009	0.011
Stability of openings	0.995	0.005	0.007

, provide entropy, diversity, and objective weight values for the evaluated factors. These objective weights indicate the relative importance of each factor in the mining methods selection (MMS) process. Notably, geotechnical properties such as ore strength and host-rock strength, along with orebody geometry factors like thickness, shape, and dip, were assigned the highest objective weights. This underscores their critical role in the MMS process, especially during the preliminary selection phase. On the other hand, environmental factors such as health and safety and the stability of openings were attributed to comparatively lower objective weights. This suggests their relatively lower impact on the decision-making process during the preliminary selection stage. Based on the objective weights obtained, the study identified the first five factors, ore strength, host-rock strength, orebody thickness, shape, and dip, as the most critical in the preliminary selection stage with objective weights of 0.132, 0.115, 0.104, 0.100 and 0.072, respectively. The identified factors are then used as the main variables in the input dataset used to train and evaluate the machine learning (ML) models for the AI-MMRS. The high objective of the five factors weights signifies their substantial influence on the MMS process, particularly in the preliminary evaluation stage. These results emphasize the critical role of the orebody characteristics, including the geotechnical properties, orebody geometry, and geology, in the preliminary evaluation and selection of mining methods, as is already shown in other MMS systems [1,3,4,5].

2.2. Study’s Database and Input Dataset

The proposed AI-based mining methods recommendation system (AI-MMRS) relies on a comprehensive database comprising historical data extracted from mining projects’ technical reports (and literature-based data). The primary data source for this study is the SEDAR database (The System for Electronic Document Analysis and Retrieval (SEDAR): https://www.sedar.com/homepage_en.htm: accessed on 11 July 2022), a publicly accessible repository of documents filed by Canadian public companies. Although there are other platforms like Mining Data Online (MDO) (https://miningdataonline.com/###, accessed on 11 July 2022) offering mining projects’ data, SEDAR’s open nature and global coverage make it a suitable choice for this study. The study’s database encompasses technical reports dating from 2000 to 2022, representing mining projects across various continents and regions (in Africa, Asia, Europe, Oceania, and Central, North and South America), all owned by Canadian companies. Around 1315 mining projects’ technical reports were collected from SEDAR and managed within a document management software (DMS). The database built into the DMS serves as a vital complementary resource for the AI-MMRS, providing additional relevant insights for mine project planning, such as local and regional geological, geotechnical conditions, labor costs, sociopolitical, environmental considerations, and infrastructure availability, among others. In instances where data from SEDAR was insufficient, literature sources such as papers and theses were utilized as secondary data sources. The study data from the secondary source were cross-checked with other literature, and the entire data (from primary and secondary sources) underwent a validation process using the UBC-MMS system [5], to ensure data integrity and reliability. Additionally, a thorough background check was conducted on the mining projects included in the database, thereby excluding projects with a history of failure or negative practices.

The process of data cleaning and validation plays a crucial role in enhancing the quality and reliability of the dataset used in this study. However, it is essential to acknowledge that this rigorous process has also resulted in a reduction in the size of the input dataset, which occurs as unreliable or inconsistent datapoints are filtered out to ensure that only high-quality and relevant information is retained for subsequent use. The limited number of datapoints (or samples) in the input dataset stands as one of the biggest limitations of this study, which can potentially affect the performance of the machine learning (ML) models because usually, ML models require large input datasets in order to have a good performance in the training and evaluation processes.

The input dataset for this study comprises historical information from thirty-three mining projects, characterized by the five input variables selected using the Entropy method [10] (i.e., ore strength, host-rock strength, orebody thickness, shape, and dip) along with the underground mining methods selected or considered for each deposit. The seven underground mining methods considered in this study are categorized into unsupported, supported, and caving methods [1], with sublevel stoping, shrinkage, and room and pillar representing the unsupported methods. Cut and fill is the only commonly applied supported mining method, while longwall, block caving, and sublevel caving represent the caving methods. While variations exist within these methods, this study focuses on the seven main underground mining methods mentioned earlier for the preliminary evaluation and selection. The proposed AI-MMRS aims to provide recommendations for the preliminary evaluation and selection of underground mining methods, thus recommending a set of the most suitable main underground methods, which can be submitted to further detailed evaluations by the decision-making team in subsequent stages to select the most sustainable methods (including different variants of the main methods) considering variety of factors like technical, economic, sociopolitical, and environmental, among other factors.

The study’s assessment of ore and host-rock strength factors is based on the widely used and most available in the mining project historical data, the rock mass rating (RMR) system, while the thickness, shape, and dip factors are described using the Nicholas approach [11] and the UBC MMS system [12].

Table 2. Characteristics and classification of the variables in the input dataset.

Variables	Description
Ore strength	Very weak, weak, moderate, strong, and very strong
Host rock-strength	Very weak, weak, moderate, strong, and very strong
Thickness	Very narrow, narrow, intermediate, thick, and very thick
Shape	Irregular, tabular, and massive
Dip	Flat, intermediate, and steep
Underground mining methods	Sublevel stoping: 16 projects Cut and fill: 10 projects Longwall: 2 projects Shrinkage: 2 projects Room and pillar: 1 project Block caving: 1 project Sublevel caving: 1 project

shows the characteristics of the input dataset utilized to evaluate machine learning algorithms for the AI-MMRS in subsequent sections, including the classification of variables and the distribution of the projects (or samples) in each mining method.

2.3. Memory-Based Collaborative Filtering Approach for Mining Methods Selection

Memory-based collaborative filtering (CF), also known as the neighborhood-based approach [22], is one of the earliest and simplest approaches for building recommendation systems. It relies on the principle of similarity, which predicts unknown ratings for an active user based on the ratings of its nearest neighbors or the most similar users. To illustrate the CF recommendation system framework, let us consider a scenario where we have a typical user–-item rating dataset X shown in Figure 2, where each row represents a user U, each column represents an item I, and the entries in the matrix represent the ratings given by users for each item. The task is to generate recommendations for the active user U5 who has watched movies 2, 3, and 6 (I2, I3, and I6) but has not yet watched movies 1, 4, and 5 (I1, I4, and I5).

There are two main steps to building a memory-based CF recommendation system: (1) measure similarity and (2) generate predictions and recommendations [23]:

Step 1: Measuring similarity

The first step in building a memory-based CF recommendation system is to measure the similarity between users or items. For example, we can use the cosine similarity Equation (5) to compute the similarity between users. In this case, we need to find users who are similar to U5 based on their ratings for movies 2, 3, and 6.

$$sim\left(a,b\right)=\cos \left(\theta \right)={\displaystyle \frac{r_a.r_b}{‖r_a‖‖r_b‖}}={\sum }_I{\displaystyle \frac{r_{aI}.r_{bI}}{\sqrt{{\sum }_I{r_{aI}}^2}\sqrt{{\sum }_I{r_{bI}}^2}}}$$

(5)

where, $r_{aI}$ is the rating of user a to an item I, and $sim\left(a,b\right)$ is the similarity between users a and b.

Step 2: Generate Predictions and Recommendations:

Once similarities are computed, we use the ratings from the most similar users to predict unknown ratings for the active user U5. The weighted sum method shown in Equation (6) is commonly used for prediction, where we compute the weighted average of the ratings given by the most similar users.

$$r_{uI}={\displaystyle \frac{{\sum }_{i\in users}sim\left(u,I\right)\ast r_{uI}}{{\sum }_{v\in users}\left|sim\left(u,I\right)\right|}}$$

(6)

Finally, based on the predicted ratings, we can recommend the top-N movies with the highest predicted ratings to user U5.

The k-nearest neighbors (KNN) algorithm is often employed to implement the memory-based CF approach. The KNN algorithm is a non-parametric supervised machine learning (ML) algorithm used as an unsupervised learner in the memory-based CF approach to search for nearest neighbors and compute similarities among users and items [24]. In this study, the KNN-cosine similarity algorithm is used to compute similarities among mining projects. Thereafter, the weighted sum method is used to predict the ratings of the mining methods for generating recommendations. The approach aims to build a model that predicts and recommends the top-N relevant underground mining methods by analyzing the similarities among the projects based on the orebody characteristics.

Data Pre-Processing for Memory-Based CF Approach

To incorporate the memory-based collaborative filtering (CF) approach for predicting and recommending the top-N underground mining methods, the input dataset (described in Section 2.2) needs to be prepared accordingly. The first step involves transforming the qualitative of the input variables into numerical values: the qualitative values of ore strength (OS), host-rock strength (RS), thickness (TH), shape (SH), and dip (DP) are transformed into numerical values based on their objective weights (shown in Table 1) obtained from the previous study [10]. Then, transform the mining methods (which are classes) into items to be recommended: the mining methods such as sublevel stoping (SS), room and pillar (RP), shrinkage (SH), cut and fill (CF), longwall (LW), block caving (BC), and sublevel caving (SC) are transformed from classes (or dependent variable) to items to be recommended to the projects/users according to their similarity across the input variables. Thereafter, assign relevant mining methods to each project: each project is assigned three relevant mining methods, one primary and two secondary methods. The project’s primary mining methods are obtained from the database (i.e., technical reports and literature sources), and the two secondary mining methods are obtained using the UBC-MMS tool [5]. Finally, ratings are assigned to the mining methods for each project using a 9-point rating scale: a rating of 9 is assigned to the primary methods of the projects, a rating of 5 is assigned to the two secondary methods, and the remaining methods are assigned a rating of 1 (indicating they are not in the top 3 most relevant methods). In this case, the task of the memory-based model is to provide an output of the top-3 most relevant mining methods for each project. The employed 9-point rating system ensures the diversity of the proposed AI-MMRS to recommend a set of top-N most relevant mining methods for a particular active project according to the orebody characteristics.

Figure 3 illustrates the data pre-processing steps, showing the transformation of the input dataset for experiments to evaluate the proposed memory-based collaborative filtering approach. The transformed input dataset consists of thirty-three mining projects (project IDs: PJ001, PJ002, … PJ033) characterized by the five input variables (OS, RS, TH, SH, and DP) and the corresponding mining methods (SS, RP, SH, Cf, LW, BC and SC) with the respective ratings. In the experiments, the transformed dataset is used to evaluate the effectiveness of the proposed memory-based CF model in predicting and recommending the top-3 most relevant mining methods.

2.4. The Need for Supervised Classification Machine Learning Algorithms

In this study, supervised machine learning (ML) algorithms are a useful addition to the memory-based collaborative filtering (CF) approach in addressing its limitations associated with the dependence of the approach on the UBC-MMS tool, thus enhancing the effectiveness of the proposed AI-MMRS for mining methods selection (MMS). Supervised ML algorithms are trained using labeled datasets and are capable of predicting labels or responses based on the learned relationship between input variables and labels of datapoints or samples [25]. Classification ML models are particularly relevant for MMS problems as they are trained to predict class labels of datapoints based on input variables [19]. In this study, five classifiers were selected for evaluation to identify the best models for MMS problems: decision trees, K-nearest neighbors (KNN) with cosine similarity, support vector machines (linear SVM), kernel approximation (kernel SVM) and artificial neural networks (ANN). The KNN algorithm is similar to the one implemented in the memory-based CF approach; however, this section implements it as a supervised classification algorithm [26]. Like KNN, the decision tree is a non-parametric algorithm that learns the relationship between input variables and class labels as a hierarchical tree structure. SVM finds a hyperplane that classifies datapoints in N-dimensional linear space, while the kernel version solves the problem in non-linear space. Lastly, ANN is a powerful algorithm inspired by the biological neural network of humans or animals [27], consisting of interconnected nodes that mimic the neurons in the biological brain, enabling it to solve complex tasks like classification and regression problems.

These algorithms have been extensively investigated in mining studies, including MMS studies [12,13,14,15,28,29,30,31], and their implementation can enhance the effectiveness and novelty of the proposed AI-MMRS. By leveraging these supervised ML algorithms, the AI-MMRS can overcome the limitations of the memory-based CF approach and generate more accurate, diverse, and novel recommendations.

3. Results

3.1. Memory-Based Collaborative Filtering Approach for Predicting and Recommending Mining Methods

The memory-based collaborative filtering (CF) evaluation strategy for predicting and recommending the top-N relevant mining methods is similar to the strategy used for the offline evaluation of recommendation systems [18]. The strategy involves simulating the practical implementation or real-world scenario of the AI-based mining methods recommendation system (AI-MMRS). The first step in the experiments involves creating sparse datasets from a dense input dataset by masking/hiding the known ratings of all seven mining methods for a target project (e.g., Figure 4: in step 1, PJ001 masked rating). Then, the proposed memory-based CF model is used to predict the masked ratings of the mining methods for the target project (i.e., step 2 in Figure 4: predicted ratings highlighted by thesquare). A list of the top-3 most relevant mining methods is extracted based on the predicted ratings (i.e., step 3: extract the list based on ratings predicted in step 2); priority is given to mining methods with higher predicted ratings. The hidden/masked ratings are kept as ground truth data and used for evaluating the accuracy and quality of the predictions and top-3 recommendations by comparing the predictions and the actual relevant mining methods for the project. In other words, the performance of the memory-based CF model in predicting the missing ratings and recommending the top-3 relevant mining methods is evaluated based on how well it matches the target project’s three relevant mining methods (one primary and two secondary methods).

Prediction Accuracy and Quality of Recommendations

The memory-based CF model is tasked with predicting and outputting a list of top-3 most relevant mining methods for the projects. The performance of the memory-based model is evaluated in two ways:

Evaluation 1: Evaluate the model’s capabilities to predict the primary mining methods as the top-1 most relevant method for each project because the primary mining methods’ prediction defines the overall model’s quality. In this case, the global accuracy rate (GAR) in Equation (7) is used as the performance evaluation metric:

$$GAR\left(\%\right)={\displaystyle \frac{number of correct classifications}{total number of classification}}\times 100,$$

(7)

Evaluation 2: Evaluate the quality of the top-3 predicted and recommended mining methods for each project. The evaluation is performed using decision support metrics such as F1-score for a fixed N (number of items in the list) based on the Precision (@N), Recall (@N) and, which can be computed as shown in Equations (8)–(10) [32].

$$\mathit{Precision}(@N)={\displaystyle \frac{TP}{TP+FP},}$$

(8)

$$Recall\left(@N\right)={\displaystyle \frac{TP}{TP+FN}},$$

(9)

$$F1-score=2\times {\displaystyle \frac{Recall\times Precision}{Recall+Precision}},$$

(10)

Figure 5 shows the model’s performance results regarding the global accuracy rate (GAR) and the F1-score. Using the KNN-cosine similarity algorithm to find the nearest projects, it is required to set the parameter k, which is the number of neighbors or projects used for predictions [33]. In the experiments, the model’s accuracy is evaluated for the number of projects (neighborhood size) ranging from 2 to 10. The results show that the model’s performance depends on the number of projects used for predictions (neighborhood size), where the highest accuracy is achieved when using the two nearest projects.

The GAR measures the model’s overall accuracy for predicting the primary methods as the top-1 within the top-3 list of recommendations. The GAR ranges from 45.5% to 63.6%, indicating that the model can correctly predict the primary mining method as a top-1 method between 45.5% and 63.6% of the time, depending on the number of projects used for predictions. The F1-score is used to further evaluate the accuracy and precision of a recommendation system in predicting the top-3 relevant mining methods for a given project, regardless of their exact rank position. The results in Figure 5 show the F1-score ranging from 81.2% to 87.9%; the highest accuracy of 87.9% was achieved when using the two nearest projects for prediction. These results suggest that the memory-based CF model performs better when using a smaller neighborhood size.

In summary, the results show that the memory-based CF model provides the best results when using a smaller neighborhood size (number of nearest projects for predictions) for predictions and recommendations, which may be directly related to the intrinsic characteristics of the input dataset (i.e., the size and class distribution of the datapoints). The model performs reasonably well in predicting the primary mining methods as top-1 and performs even better in predicting the top-3 relevant mining methods. Despite the limitations of the dataset size (dataset with thirty-three projects), the proposed memory-based CF approach performs quite well in generating recommendations of the top-3 most relevant underground mining methods with about 87.9% prediction accuracy.

3.2. Classification Machine Learning for Predicting (Classifying) Underground Mining Methods

This section presents the experimental results of classification machine learning for predicting (classifying) underground mining methods. The experiments were conducted in MATLAB R2022a, using the Statistics and Machine Learning Toolbox, with the classification learner application [34]. Five classifiers were evaluated on the same dataset: decision trees, linear support vector machine (SVM), k-nearest neighbors (KNN) with cosine similarity, kernel approximation (SVM kernel), and artificial neural networks (ANN). The 10-fold cross-validation method was used to evaluate and validate the five classifiers. The k-fold cross-validation method is known to perform better on small datasets (datasets with a limited number of datapoints) as it ensures that every datapoint in the dataset can appear in the training and testing sets [35]. Using the k-fold cross-validation results in models that learn the underlying patterns well and evenly, creating more generalized and less biased models [36].

Prediction (Classification) of Underground Mining Methods

The global accuracy rate (GAR), F1-score, and the confusion matrix are used to evaluate the performance of the classifiers.

Table 3. Performance evaluation results of the classifiers in terms of global accuracy rate (GAR) and average per-class F1-score.

Classifier	GAR (%)	Average Per-Class F1-Score (%)
ANN	66.7	20.6
KNN	63.6	20.3
SVM	63.6	21.2
Decision Tree	57.6	18.2
SVM Kernel	54.5	16.0

shows the results of the performance of the classifiers in terms of the GAR and average per class F1-score. The artificial neural network (ANN) model performed the best at 66.7% GAR, followed by k-nearest neighbors (KNN) with 63.3%. It is worth noting that while the GAR metric provides an overall evaluation of the classifier’s performance, it may not always reflect the per-class performance. Therefore, the average per class F1-score is used to depict how well the classifier performs for each class label (each mining method). In the context of mining methods prediction, this means understanding how well the classifier performs for each specific mining method, as each method may have different characteristics and patterns that need to be learned by the classifier. In this case, we can observe lower values of the F1-score than the GAR which can suggest that the classifiers are not performing well in identifying or classifying individual classes.

Figure 6 further depicts the confusion matrix of the ANN model, showing each class’s (mining methods) Recall or True Positive Rate (TPR) and the True Negative Rate (TNR). The TPR shows the proportion of correct predictions in each mining method. As for any other classification algorithm, the class-imbalanced dataset problem negatively affects the performance of the evaluated classifiers. Class imbalanced datasets happen when the distribution of datapoints among the classes is uneven, with majority and minority classes [37]. The study’s input dataset is class imbalanced, comprising thirty-three projects (datapoints), with sixteen projects in sublevel stoping, ten in cut and fill, two in longwall and shrinkage, and one in room and pillar and sublevel caving. Imbalanced datasets can cause the classifiers to be biased towards the majority class, resulting in a model that overlearns the patterns of the majority classes and ignores the minority classes [38]. The confusion matrix shows that the ANN model accurately predicts the majority classes, sublevel stoping (100%), and cut and fill (60%). However, it performs poorly predicting the minority classes (block caving, longwall, room and pillar, and sublevel caving), all with 0% TPR. This result indicates that the class imbalance problem in the dataset negatively affected the performance of the classifiers in learning the patterns of the seven underground mining methods. This suggests the need to improve the dataset’s quality to build less biased and generalized classifiers towards developing a robust AI-MMRS.

In summary, the results of evaluating the applicability of the five machine learning classification algorithms to predict underground mining methods suggest that the ANN model is more effective in solving the complexity of mining methods selection (MMS). This finding is consistent with the results of previous studies [12,13,14,16], which have also demonstrated the effectiveness of regression ANN in MMS. The results also show that the KNN classification model was effective for MMS. As a recap, this study used the same KNN algorithm for implementing the memory-based collaborative filtering approach for predicting and recommending top-N underground mining methods, producing reliable and effective results with good accuracy. These findings consistently demonstrate the effectiveness of classification algorithms, including ANN and KNN, in solving the complexity of MMS, which can potentially be further optimized and implemented interchangeably to provide accurate recommendations.

4. Discussion

This study incorporates artificial intelligence (AI) to explore the available database of mining projects and develop a system that can assist in mine project development decision-making, particularly in the mining methods selection (MMS) task. This study develops an integrated MMS system termed the AI-based mining methods recommendation system (AI-MMRS). The study investigated the applicability of the memory-based collaborative filtering (CF) approach and classification algorithms (i.e., KNN, ANN, SVM, decision trees, and SVM kernel) for predicting underground mining methods. The proposed models were evaluated based on an input dataset comprising data from thirty-three mining projects described by five input variables (ore strength, host-rock, thickness, shape, and dip) and seven underground mining methods (block caving, cut and fill, longwall, room and pillar, shrinkage, sublevel caving, and sublevel stoping). The input dataset is characterized as small for typical machine learning problems and imbalanced class issues, which stands as one of the limitations of the study.

The results of evaluating the performance of the proposed memory-based approach to predicting and recommending the top-3 most relevant underground mining methods have demonstrated the models’ effectiveness with moderately good accuracy. On the other hand, among the five evaluated classification models, artificial neural networks (ANN) and k-nearest neighbors (KNN) performed the best in predicting (classifying) underground mining methods, showing their effectiveness in solving the complexity of underground MMS with moderate accuracy. The proposed memory-based approach generates recommendations of the top-3 relevant underground mining methods aided by the UBC-MMS system [5] in the data preprocessing step. The classification models are independent of the UBC-MMS system; they are implemented to enhance the effectiveness and novelty of the proposed AI-MMRS, as they can address the limitations of the memory-based approach and generate more accurate and diverse recommendations. Furthermore, the AI-MMRS integrates results from the previous study [11] using the NMF model to predict possible missing information about the required input variables (orebody characteristics) to enable the implementation of the AI-MMRS in situations where the information (or data) about the necessary input variables is not available/missing, especially during early stages of mine project development.

Some of the findings from this study are consistent with previous studies which have shown the effectiveness of regression ANN in MMS [12,13,14,16]. Similar to the ANN model by Ozyurt and Karadogan [12], this study is developed considering different ore types (or commodities), which differs from the models by Lv and Zhang [13] and Chen and Shixiang [14], which were specific to coal seams. In contrast to the model by Ozyurt and Karadogan [12] and similar to the models by Lv and Zhang [13] and Chen and Shixiang [14], the proposed AI-MMRS is fully based on data from real case studies. For the study, the proposed AI-MMRS requires minimum and fairly easily accessible input factors, especially during the early stages of mining project development where there can be limited access to information about the deposit. The weights and selection of the five input variables in the dataset were determined objectively in a previous study using the Entropy method without the need for the direct involvement of decision-makers [10], unlike most MCDM-based MMS systems [6,7,8,9], which typically require some level of subjective judgement. The advantage of AI or ML-based systems is the possibility of continuous learning; as new data becomes available and mining practices evolve, the models can be updated to reflect the changes and provide up-to-date recommendations. This may help avoid the risk of obsolete recommendations, which may be the case for quantitative and some MCDM-based systems.

Understanding the limitations of AI-based approaches in decision-making as well as the need for careful evaluations in the mining methods selection (MMS) process, this study emphasizes the need for the direct involvement of the decision-making team (i.e., mining professionals) to perform detailed evaluations and selection of most feasible methods in the MMS process. Therefore, this study proposes an AI-based approach to develop a mining methods recommendation system (AI-MMRS) for the preliminary evaluation and selection of underground mining methods. As such, the proposed AI-MMRS is intended to serve as a decision-making support system for the preliminary evaluation and selection of underground mining methods, that is, to recommend options of suitable underground mining methods based on orebody characteristics (geometry, geology, and geotechnical properties). The recommended suitable methods by the AI-MMRS can then undergo further feasibility evaluations by the decision-making team (mining experts), who will ultimately select the most sustainable method(s) based on technological, economic, environmental, and other relevant factors for the project evaluation. The proposed system is meant to aid in the pre-selection stage (for new projects or ongoing projects that may need to select mining methods for new horizons in the mine) based on historical and evolving data. The system can help professionals focus their detailed evaluations on a narrower, more relevant set of mining methods, thus enhancing the efficiency and thoroughness of the decision-making process. In practical implementation, a target project should provide five input variables (ore strength, host-rock strength, thickness, shape and dip) which will be used to search for other projects with similar orebody characteristics in the database. Thereafter, the system recommends a set of most suitable underground mining methods for the deposit characteristics, to be submitted for further evaluations by the decision-making team. The system is designed with the understanding that, due to the complexity of orebody characteristics (such as geometry and geotechnical conditions), using a single mining method is often impractical; therefore, the system provides recommendations for multiple suitable methods. The study also highlights the importance of incorporating sociopolitical and environmental risk assessments in the early stages of mine project planning, including during the MMS process, which is demonstrated in a study by Barakos and Misho [39]. The proposed integrated system can assist with these assessments, using its database of mining project historical data to provide relevant information for decision-making, including details about local or regional sociopolitical aspects, environmental concerns, common mining methods, geology, geotechnical conditions, labor costs, infrastructure, and mineral processing procedures and other project-specific factors.

In summary, this study demonstrated the effectiveness and viability of the proposed approaches to develop the integrated AI-MMRS, which has significant potential to improve the efficiency and effectiveness of decision-making in the MMS task. Despite the limitations, the findings from this study are significant and can pave the way for future studies on the MMS discipline and Smart Mining (Mining 4.0) in general, providing insights for researchers and industry interested in leveraging the power of AI and data analytics to improve efficiency and quality of the decision-making process. By leveraging the power of AI and recommendation systems, this approach can help to address key challenges in mining project development and support the ongoing evolution of mining practices. As such, the system should be continually updated and improved with new data to ensure its recommendations remain relevant and accurate.

5. Limitations and Future Works

The use of AI-based approaches in MMS is promising as it can be more efficient and provide experience-based recommendations. However, a major limitation of the study is the size and quality of the input dataset, primarily due to the limited access to reliable mining projects’ historical data, which reflects on the quality and performance of the models, especially classification models. As for any ML-based system, the proposed AI-MMRS will only provide good predictions and better recommendations if continuous data collection and training are performed. Therefore, the system is designed to be flexible and open to the inclusion of new mining methods as they are developed or identified by making the model open for new additions. Future studies will focus on improving the quality of the input dataset, optimizing machine learning models, and validating the models considering various case studies. Additionally, actively include sociopolitical, environmental and other critical aspects deemed critical in mine planning in the model to enable active risk assessment analysis in decision-making, particularly in the evaluation selection of mining methods.

To address the major limitation of the study, thereby improving the quality of the training dataset, it is proposed to make the model open for new additions through an open-line platform where industry professionals, researchers, and mining experts can contribute to mining projects, which will continuously expand the training dataset. This approach not only increases the dataset but also promotes collaborative knowledge sharing within the mining community. By enriching the dataset over time, the AI model will become more robust, improving its accuracy and utility in future applications. This strategy ensures that the AI system evolves along with advancements in mining methods, emerging technologies, and changes in orebody conditions and relevant selection factors, making it a valuable tool for both current and future projects.

6. Conclusions

This study proposed an artificial intelligence (AI)-based approach to developing an AI-based mining methods recommendation system (AI-MMRS) to serve as a decision support system for the preliminary evaluation and selection of underground mining methods. The study investigated the most effective approaches to exploring the available database of mining projects to develop a decision-making system for mining project development (mine planning), the AI-MMRS. The proposed AI-MMRS was developed based on the integration of two main approaches to train the prediction models: memory-based collaborative filtering (CF) and classification models to provide efficient and diverse recommendations for underground mining methods. The study evaluated the applicability of the two approaches in predicting underground mining methods based on orebody characteristics (i.e., ore strength, host-rock strength, thickness, shape and dip). The results have demonstrated the effectiveness of the memory-based CF approach and classification models (i.e., ANN and KNN) with moderately good accuracy.

Despite the study’s limitations in the size and quality of the dataset, the findings demonstrated that the integrated approaches in the AI-MMRS can be viable and practical for MMS. Understanding that the mining methods selection (MMS) process is one of the most critical decision-making tasks in mine planning, this study emphasizes the need for the direct involvement of the decision-makers (mining professionals) to perform further detailed evaluations and selection of mining methods. Therefore, the proposed AI-MMRS is intended to serve as a decision-support tool for the preliminary evaluation and selection of underground mining methods based on historical and evolving data. The system can help professionals focus their detailed evaluations on a narrower, more relevant set of mining methods, thus enhancing the efficiency and thoroughness of the decision-making process.

The proposed AI-MMRS has the potential to aid in decision-making in the mining industry by providing efficient and experience-based recommendations for the preliminary evaluation and selection of underground mining methods. However, as with any AI or ML-based systems, the proposed AI-MMRS will only provide better recommendations if continuous data collection and training are performed. Therefore, the system is designed to be flexible and open to the inclusion of new mining methods as they are developed or identified. Future studies will focus on improving the quality of the input dataset, optimizing machine learning models, and validating the models on various case studies.

The study’s findings provide new insights into applying memory-based collaborative filtering and classification models in MMS, providing a valuable resource for researchers and industry professionals interested in data-driven approaches to mining project decision-making. The study thus demonstrates that the application of AI and data analytics in the mining industry has the potential to optimize the efficiency and effectiveness of decision-making, which is the concept of Smart Mining (Mining 4.0).