Predicting the Pillar Stability of Underground Mines with Random Trees and C4.5 Decision Trees

: Predicting pillar stability in underground mines is a critical problem because the instability of the pillar can cause large-scale collapse hazards. To predict the pillar stability for underground coal and stone mines, two new models (random tree and C4.5 decision tree algorithms) are proposed in this paper. Pillar stability depends on the parameters: width of the pillar ( W ), height of the pillar ( H ), W / H ratio, uniaxial compressive strength of the rock ( σ ucs ), and pillar stress ( σ p ). These parameters are taken as input variables, while underground mines pillar stability as output. Various performance indices, i.e., accuracy, precision, recall, F-measure, Matthews correlation coe ﬃ cient (MCC) were used to evaluate the performance of the models. The performance evaluation of the established models showed that both models were able to predict pillar stability with reasonable accuracy. Results of the random tree and C4.5 decision tree were also compared with available models of support vector machine (SVM) and ﬁshery discriminant analysis (FDA). The results show that the proposed random tree provides a reliable and feasible method of evaluating the pillar stability for underground mines.


Introduction
Pillars are essential structural units of most underground mines, used to ensure safety and economically maximize the extraction of the ore body. Coates [1] defined the pillars as "the in-situ rock between two or more underground openings". Their primary purpose is to provide safe access to the working area and to support the weight of overburden material between adjacent underground openings temporarily or permanently, and drilling room of ceilings during excavation and mining [2][3][4][5][6][7]. Instable pillars may significantly increase workers' safety hazards and sudden roof collapse [8]. Additionally, with the increase in mining depth, the failure of the pillar becomes more frequent and critical owing to increased ground stresses. The determination of pillar stability is, therefore, important for attaining efficient and safe mining.
Several researchers have developed numerous methods for understanding and predicting pillar failures in thee past several decades. Pillar stability is generally evaluated using a factor of safety (FoS), which is the ratio between the pillar strength and the load imposed on the pillar structure. Theoretically, when the FoS ratio is greater than 1, the pillar is considered stable. However, studies suggest that pillar with FoS above this threshold may still fail due to non-regular geometries, uncertain material properties, and differences in mining operations [7,9]. random tree and C4.5 decision tree models. The data recently referenced by Zhou et al. [18] comprised of a total of 46 pillar cases from USA underground stone mines and Indian coal mines which are shown in Table 1. The database contains pillar width (W), pillar height (H), the pillar W/H ratio, pillar stress (σ p ), and uniaxial compressive strength of the rock (σ ucs ). The range of values associated with each input variable is shown in Table 2. It has been widely accepted, among the researchers such as Goodman [29], that the input factors selected by Zhou et al. [18] is a complete and suitable set to predict the pillar stability. The same factors have been widely used as significant factors by other researchers (e.g., Liang et al. [30]). Table 2 provides the descriptive statistics of each input parameter. The distribution of pillar case histories database is stable (14 cases) and failed (32 cases) according to the failure process and the instability mechanism of pillars. Figure 1 shows three common failure mechanisms for a naturally fractured pillar [31]. These include: (a) failure with the lateral kinematic release of pre-formed blocks because of rising vertical load and lack of confinement, (b) failure resulting from the buildup of inclined shear fractures that transect the pillar, and is prevailing in pillar with relatively low W/H ratio, and (c) failure along with transgressive fractures where the fracture inclination angle with the principal loading axis of the pillar exceeds the angle of friction. In all of these mechanisms, a pillar's mechanical response is directly related to the geological structures of the ground and those effects are most notable for the slender pillars. Although, wider pillars are more likely to collapse by a combination of brittle and shearing processes. Geological discontinuities in rock strata are of fundamental importance for analyzing the strength, permeability, and deformability. Characterization of the discontinuity in geometry (i.e., spatial connectivity, persistence length, and aperture, etc.) is the first step towards better understanding of rock masses overall behavior [32]. It is generally agreed that discontinuities in rock profile significanrly affect their strength properties. Due to misjudgment of the effect of discontinuities on rock strength, several examples may be quoted from the relevant rock failure literature. Jessu and Spearing [33] reported that discontinuities have a substantial influence with an increase in pillar inclination even at larger W/H ratios. Shang et al. [34] attempted to address the problems of the tensile strength of incipient rock discontinuities and quantified this parameter based on laboratory test results. These indicators are considered the main parameters for quantitatively discovering the activities in the context of the pillar, but data collection is the main challenge faced for their applicability. Hence, the five parameters are considered in the current study.  Note: The cases with a '*' sign represent the testing dataset. Table 2. Descriptive statistics of the pillar stability data.

Methodology
There are several data mining and machine learning (ML) algorithms available, among which random tree and C4.5 DT are the most frequently used algorithms for classification [24]. They represent decision trees that use strategies to divide and conquer in form of induction learning. The For implementing the random tree and C4.5 DT in this research study, a total of 40 datasets were considered for the training dataset, and the other six datasets were considered as the testing datasets. The testing and training datasets are the same as used by Zhou et al. [18].

Methodology
There are several data mining and machine learning (ML) algorithms available, among which random tree and C4.5 DT are the most frequently used algorithms for classification [24]. They represent decision trees that use strategies to divide and conquer in form of induction learning. The tree representation in these models helps to classify patterns in datasets, which are organized hierarchically in a collection of inter-connected nodes. The internal nodes evaluate each input attribute/feature in relation to a decision Appl. Sci. 2020, 10, 6486 5 of 12 constant, thereby deciding what will be the next descending node. The nodes that are considered to be leaves are aimed to identify the instances that reach them as per the corresponding label [35].

Random Tree
Random trees were initially proposed by Leo Breiman and Adele Cutler. The algorithm can address both regression and classification tasks [36,37]. It is a decision tree, developed through a stochastic cycle. It makes use of a bagging concept to create a random dataset to construct a decision tree. Using the best split of all attributes, every node is split in the standard tree. In a random tree model, each node is split using the best at that node from the sub-set of randomly selected attributes. It also provides an option to estimate the class probabilities. A typical random tree structure is shown in Figure 2.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 5 of 12 tree representation in these models helps to classify patterns in datasets, which are organized hierarchically in a collection of inter-connected nodes. The internal nodes evaluate each input attribute/feature in relation to a decision constant, thereby deciding what will be the next descending node. The nodes that are considered to be leaves are aimed to identify the instances that reach them as per the corresponding label [35].

Random Tree
Random trees were initially proposed by Leo Breiman and Adele Cutler. The algorithm can address both regression and classification tasks [36,37]. It is a decision tree, developed through a stochastic cycle. It makes use of a bagging concept to create a random dataset to construct a decision tree. Using the best split of all attributes, every node is split in the standard tree. In a random tree model, each node is split using the best at that node from the sub-set of randomly selected attributes. It also provides an option to estimate the class probabilities. A typical random tree structure is shown in Figure 2.

C4.5 Decision Tree
Commercial version 4.5 (C4.5), was proposed by Quinlan [38], is a widely known machine learning algorithm that is primarily used to develop decision trees. The method uses an entropy-based measure as the selection criteria that yield a good classification accuracy in minimal time. It is is an extension of the ID3 algorithm that also operates the divide-and-conquer policy [38]. The method's main improvements include the pruning methodology and excellent processing of numeric attributes, missing values and noisy data. The C4.5 algorithm usually consists of two processes; preparation of thee decision tree, and making the rules (structure and design). Then measuring the entropy, and calculating the highest attribute for information gain.
The entropy formula is shown below, where S is entropy, and p in the output is a class proportion.

( )
In addition, the attribute with the highest gain value is used as the root attribute. Gain (S, A) is presented in Equation (2) where, S is a case set; A is a case attribute; |Si| is a number of cases to i and |S| is number of cases in the set.
C4.5 algorithm develops a DT using the recursive-splitting and top-down and technique. The tree structure of C4.5 consists of a root node, internal nodes, and leaf-nodes. The root node encompasses all the input data. An internal node usually has two or more branches and is

C4.5 Decision Tree
Commercial version 4.5 (C4.5), was proposed by Quinlan [38], is a widely known machine learning algorithm that is primarily used to develop decision trees. The method uses an entropy-based measure as the selection criteria that yield a good classification accuracy in minimal time. It is is an extension of the ID3 algorithm that also operates the divide-and-conquer policy [38]. The method's main improvements include the pruning methodology and excellent processing of numeric attributes, missing values and noisy data. The C4.5 algorithm usually consists of two processes; preparation of thee decision tree, and making the rules (structure and design). Then measuring the entropy, and calculating the highest attribute for information gain.
The entropy formula is shown below, where S is entropy, and p in the output is a class proportion.
In addition, the attribute with the highest gain value is used as the root attribute. Gain (S, A) is presented in Equation (2) where, S is a case set; A is a case attribute; |S i | is a number of cases to i and |S| is number of cases in the set.
C4.5 algorithm develops a DT using the recursive-splitting and top-down and technique. The tree structure of C4.5 consists of a root node, internal nodes, and leaf-nodes. The root node encompasses all the input data. An internal node usually has two or more branches and is accompanied by a decision function. While the leaf node denotes the output of given input parameters. J48 is an open-source Java implementation of the C4.5 DT algorithm on the WEKA data-mining platform. The simple tree structure of C4.5 DT is as shown in Figure 3. Table 3 outlines the key differences between the two data mining algorithms adopted for the current study.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 6 of 12 accompanied by a decision function. While the leaf node denotes the output of given input parameters.
J48 is an open-source Java implementation of the C4.5 DT algorithm on the WEKA datamining platform. The simple tree structure of C4.5 DT is as shown in Figure 3. Table 3 outlines the key differences between the two data mining algorithms adopted for the current study.

Development of Prediction Models
In this paper, the C4.5 decision tree and random tree algorithms were used to develop models for predicting underground mine pillar stability. The input variables were the width of the pillar (W), height of the pillar (H), W/H ratio, the pillar stress (σp), and uniaxial compressive strength of the rock (σucs). Pillar stability (PS) (dependent variable) was the sole output variable. The train-and-test approach, which is one of the most widely used techniques for establishing the predictive performance of ML algorithms for a given database, was used to develop the models. In this research study, the dataset used by Zhou et al [18] was used as the training and testing datase for the proposed models. The working procedure for pillar stability prediction using the proposed models is shown in Figure 4.    Table 3. Differences between a random tree and a C4.5 decision tree.

Properties
Random Tree (RT)

C4.5 Decision Tree (DT)
Attributes available at each decision node Random subset All Attributes selected at each decision node Best among a random subset Highest information gain among all Trees count One One Samples of data used for the training All All Classification final result Based on the leaf node reached Based on the leaf node reached

Development of Prediction Models
In this paper, the C4.5 decision tree and random tree algorithms were used to develop models for predicting underground mine pillar stability. The input variables were the width of the pillar (W), height of the pillar (H), W/H ratio, the pillar stress (σ p ), and uniaxial compressive strength of the rock (σ ucs ). Pillar stability (PS) (dependent variable) was the sole output variable. The train-and-test approach, which is one of the most widely used techniques for establishing the predictive performance of ML algorithms for a given database, was used to develop the models. In this research study, the dataset used by Zhou et al. [18] was used as the training and testing datase for the proposed models. The working procedure for pillar stability prediction using the proposed models is shown in Figure 4. accompanied by a decision function. While the leaf node denotes the output of given input parameters. J48 is an open-source Java implementation of the C4.5 DT algorithm on the WEKA datamining platform. The simple tree structure of C4.5 DT is as shown in Figure 3. Table 3 outlines the key differences between the two data mining algorithms adopted for the current study.

Development of Prediction Models
In this paper, the C4.5 decision tree and random tree algorithms were used to develop models for predicting underground mine pillar stability. The input variables were the width of the pillar (W), height of the pillar (H), W/H ratio, the pillar stress (σp), and uniaxial compressive strength of the rock (σucs). Pillar stability (PS) (dependent variable) was the sole output variable. The train-and-test approach, which is one of the most widely used techniques for establishing the predictive performance of ML algorithms for a given database, was used to develop the models. In this research study, the dataset used by Zhou et al [18] was used as the training and testing datase for the proposed models. The working procedure for pillar stability prediction using the proposed models is shown in Figure 4.   To develop the proposed models, C4.5 DT and random tree algorithms are implemented based on training data on the Waikato Environment for Knowledge Analysis (WEKA) tool. The training dataset in ARFF form (attribute relation file format) was loaded in WEKA to train the adopted models which are subsequently tested by using a testing dataset based on the given training dataset. The optimal value of the C4.5 DT model parameters, including the confidence factor and the minimum number of instances per leaf, were calculated by the trial-and-error method to achieve the highest classification accuracy. The C4.5 DT algorithm's optimal value for confidence factor was 0.25 and that of the minimum instances per leaf was found to be 2. Figures 5 and 6 represent the random tree and C4.5 DT models based on the training dataset. As shown in the figure, the size of the random tree is 11 including a root node, 4 internal nodes, and 6 leaves, while the size of the C4.5 DT was 5 including a root-node, 1 internal node, and 3 leaves. The leaf node shows the stable and the failed class of underground mines. The value in the parentheses with each leaf node denotes the number of cases at that leaf. It is evident that for leaves, few instances are misclassified, which are clearly specified after a slash (see Figures 5 and 6).
To develop the proposed models, C4.5 DT and random tree algorithms are implemented based on training data on the Waikato Environment for Knowledge Analysis (WEKA) tool. The training dataset in ARFF form (attribute relation file format) was loaded in WEKA to train the adopted models which are subsequently tested by using a testing dataset based on the given training dataset. The optimal value of the C4.5 DT model parameters, including the confidence factor and the minimum number of instances per leaf, were calculated by the trial-and-error method to achieve the highest classification accuracy. The C4.5 DT algorithm's optimal value for confidence factor was 0.25 and that of the minimum instances per leaf was found to be 2. Figures 5 and 6 represent the random tree and C4.5 DT models based on the training dataset. As shown in the figure, the size of the random tree is 11 including a root node, 4 internal nodes, and 6 leaves, while the size of the C4.5 DT was 5 including a root-node, 1 internal node, and 3 leaves. The leaf node shows the stable and the failed class of underground mines. The value in the parentheses with each leaf node denotes the number of cases at that leaf. It is evident that for leaves, few instances are misclassified, which are clearly specified after a slash (see Figures 5 and 6).  The performance of the trained pillar stability models was estimated using a number of statistical classification evaluation criteria such as counts of true-positive (TP), true-negative (TN), false-positive (FP), and false-negative (FN). These counts together form the confusion matrix shown in Table 4. To develop the proposed models, C4.5 DT and random tree algorithms are implemented based on training data on the Waikato Environment for Knowledge Analysis (WEKA) tool. The training dataset in ARFF form (attribute relation file format) was loaded in WEKA to train the adopted models which are subsequently tested by using a testing dataset based on the given training dataset. The optimal value of the C4.5 DT model parameters, including the confidence factor and the minimum number of instances per leaf, were calculated by the trial-and-error method to achieve the highest classification accuracy. The C4.5 DT algorithm's optimal value for confidence factor was 0.25 and that of the minimum instances per leaf was found to be 2. Figures 5 and 6 represent the random tree and C4.5 DT models based on the training dataset. As shown in the figure, the size of the random tree is 11 including a root node, 4 internal nodes, and 6 leaves, while the size of the C4.5 DT was 5 including a root-node, 1 internal node, and 3 leaves. The leaf node shows the stable and the failed class of underground mines. The value in the parentheses with each leaf node denotes the number of cases at that leaf. It is evident that for leaves, few instances are misclassified, which are clearly specified after a slash (see Figures 5 and 6).  The performance of the trained pillar stability models was estimated using a number of statistical classification evaluation criteria such as counts of true-positive (TP), true-negative (TN), false-positive (FP), and false-negative (FN). These counts together form the confusion matrix shown in Table 4. The performance of the trained pillar stability models was estimated using a number of statistical classification evaluation criteria such as counts of true-positive (TP), true-negative (TN), false-positive (FP), and false-negative (FN). These counts together form the confusion matrix shown in Table 4. Consider that the failure of the underground mine pillar is actually positive then TP and TN represent the number of failed and stable instances that were predicted correctly. FN and FP occur when the outputs are incorrectly predicted as negative and positive, respectively. From the confusion matrix outcomes, the following classification evaluation metrics were used for comparing the prediction performance of the models [39][40][41]: Accuracy (ACC) is the sum of TP and TN that shows the percentage of correctly classified instances among all of the data. This metric assesses the overall prediction effectiveness of the model. ACC may be deceptive, owing to the imbalanced dataset since it may produce a high value when the failed pillar samples in the majority class are favorably predicted. Therefore, in this study, other evaluation metrics including precision, recall, F-measure, and MCC were used to further assess the model's performance. Precision is referred to as positive predicted value; recall is referred to as the true positive rate; F-measure represents a broad index that measures the performance of both recall and precision that ranged from 0 (worst value) to 1 (best value). MCC [41] represents the extent of correlation between observed and predicted classes of failed and stable cases. It is one of the standard metrics used by statisticians that takes values from −1 to 1. An MCC value of "−1" shows complete disagreement (strong negative association), "1" indicate the complete agreement (strong positive association), and "0" represent that the prediction was not related with the ground truth (very weak or no correlation between dependent and independent variable).

Results and Discussion
Several performance measures based on a confusion matrix were made using training and testing datasets for underground mines in order to quantify the performance measures of the proposed models. The performance results of the random tree and C4.5 DT were obtained and compared with FDA and SVM models available in the literature based on the same training and testing datasets. The confusion matrix of each model was then determined, which is shown in Table 5. Table 5 also shows the pillar stability prediction results from a couple of previous studies. The values along the main diagonal show the number of correctly predicted cases. It may be noted from the results of Table 5 that most of the cases were accurately classified using the random tree and C4.5 DT. Based on Table 6, the overall accuracy, recall, precision, F-measure were calculated using Equations (3)- (7), which are given in Table 6. All models have good performance in training and testing the pillar stability of underground mines datasets, according to the result obtained. Furthermore, the overall accuracy, recall, precision, F-measure and MCC for stable and failed classes of the random tree and SVM models on training and testing datasets were found at par. Furthermore, a major drawback of SVM is the black-box nature of the model and the fact that correlation between input and output variables of the system are given in terms of a matrix of weight and biases which make it very challenging for users to access and understand the prediction process [19]. Decision tree algorithms i.e., random tree and C4.5, are quite transparent, and model and internal parameters need not be configured either. The SVM model is a black-box model that can show the input data and the results of the subsequent classification. However, because of its own characteristics, the black-box model cannot demonstrate the internal data mining mechanism, so the useful information relating to pillar stability is difficult to obtain for system operators [35,42]. Random tree and C4.5 DT are white-box models that are more intuitive and interpretable than ones with the black-box model i.e., the SVM model. Due to tree-like structures, the proposed models can not only obtain accurate classification results, but can also show the internal mechanism for classification results. Although, C4.5 DT and FDA possessed the same accuracy, MCC, precision, recall and F-measure for failed and stable pillars on the testing dataset, the training performances were different. Comparing their values of MCC, recall, precision, F-measure for failed and stable pillars, the C4.5 DT performed better than FDA in the training dataset. Therefore, based on their training and testing performances and bearing in mind the black-box nature of the SVM model, the random tree model is relatively better SVM and RBF models available in literature due to their easier application and of a clear graphical outcome. Although the proposed method produces desirable results for predictions, certain shortcomings should be discussed in the future. The dataset is relatively unbalanced and small. Generally, if the dataset is limited it will affect model generalization and reliability. While random tree and C4.5 algorithms perform well with limited data sets, the performance of predictions on a larger dataset could be better. However, the developed models can always be updated to yield better results, as new data becomes available. Moreover, the dataset is unbalanced, owing to the fact that there are more failed cases than stable cases. Therefore, the establishment of a larger and more balanced pillar stability database is important. Other variables could also affect the prediction results. Numerous factors influence the stability of the pillar, such as external environments and the inherent characteristics of the pillar. While the five factors used in the current study, to some extent, may characterize the fundamental conditions for pillar stability, few other factors like underground water, joints, and blasting disturbance may also influence pillar stability. Analyzing the influence of these factors on the results of predictions is important.

Conclusions and Recommendations for Future Work
In this paper, a random tree and C4.5 DT models were applied to predict underground mines' pillar stability. The models were trained and tested using the pillar stability database of underground mines collected from the USA and India. The developed models predict the pillar stability using the most important input variables such as pillar width (W), pillar height (H), pillar W/H ratio, uniaxial compressive strength, and average pillar stress. The main findings of this study are as follows: 1.
Unlike most soft computing techniques, the developed models are very simple to use and do not require exhaustive training. The proposed models present an explicit relationship between explanatory (input) and target (output) variables. Study results reveal that relationships obtained are compatible with engineering judgments and intuitions.

2.
The classification accuracy of the random tree and C4.5 DT is 100% and 95.65%, respectively during the training and testing phases, which indicates that both models are efficient and reliable for practical applications.

3.
Compared to the SVM model, the developed random tree model shows at par results and the random tree model application is easier due to a simple graphical outcome, while the information gained by the SVM algorithm during the training phase is implicitly processed that make it very difficult to explain well the overall structure of the network. So, it may be argued that the SVM model has little insight into the fundamental mechanism of the current problem. 4.
The comparison of the developed models' performances showed that the random tree model achieved more reliable predictions than the C4.5 DT model.

5.
It is evident that the proposed models are open to further development in the future and that the acquisition of more data will improve the capacity for prediction.
For future research, it is recommended to work with a larger and more balanced pillar stability database to further evaluate the reliability of these models for the prediction of pillar stability. These models can also be refined by considering additional parameters pertaining to ground type and structural geology.