Diagnosis of Problems in Truck Ore Transport Operations in Underground Mines Using Various Machine Learning Models and Data Collected by Internet of Things Systems

Abstract

This study proposes a method for diagnosing problems in truck ore transport operations in underground mines using four machine learning models (i.e., Gaussian naïve Bayes (GNB), k-nearest neighbor (kNN), support vector machine (SVM), and classification and regression tree (CART)) and data collected by an Internet of Things system. A limestone underground mine with an applied mine production management system (using a tablet computer and Bluetooth beacon) is selected as the research area, and log data related to the truck travel time are collected. The machine learning models are trained and verified using the collected data, and grid search through 5-fold cross-validation is performed to improve the prediction accuracy of the models. The accuracy of CART is highest when the parameters leaf and split are set to 1 and 4, respectively (94.1%). In the validation of the machine learning models performed using the validation dataset (1500), the accuracy of the CART was 94.6%, and the precision and recall were 93.5% and 95.7%, respectively. In addition, it is confirmed that the F1 score reaches values as high as 94.6%. Through field application and analysis, it is confirmed that the proposed CART model can be utilized as a tool for monitoring and diagnosing the status of truck ore transport operations.

1. Introduction

Because the productivity and profits of mines can vary greatly depending on the design and planning of the production process, optimal operation methods and equipment utilization strategies are needed to maximize productivity and equipment efficiency and minimize operating costs [1,2,3,4,5]. The cost of transporting ore and waste accounts for over 50% of the total mine operational cost, therefore, it is crucial to design and operate the transport system efficiently [6]. Methods to improve the productivity and efficiency of the mine transport system are divided broadly into two types: methods to properly establish an operational plan so that the mine can be operated effectively, and methods to monitor and manage the site to see whether the established plan is being well implemented.

Recently, various mathematical decisions and deterministic and probabilistic simulation models have been proposed by researchers to establish an operational plan, such as optimizing the operational method and equipment allocation plan of the mine transport system and minimizing material handling costs [4,7,8,9,10,11,12,13]. Since the first implementation of discrete event simulation by Rist to solve problems related to ore transport in mines, many researchers have conducted research on discrete event simulation [14]. Salama and Greberg [15] performed a simulation of a loading-haulage-dumping machine (LHD) and a truck to optimize the number of trucks used in haulage operation in an underground mine. Choi [16] developed a discrete event simulation program to simulate the shovel-truck transport system of an open-pit mine using the GPSS/H simulation language. Choi and Nieto [3] extended this to analyze the optimal transport path of a truck. Subsequently, they performed discrete event simulations of transport equipment and provided a function to visualize the simulation results. Park and Choi [17,18,19,20,21,22] developed GPSS/H-based programs and user-friendly programs to simulate truck-loader transport systems, considering various conditions such as fixed/real-time allocation, crusher capacity, and possibility of truck failure.

If the operational plan of the transport system of the mine has been properly established, it is also crucial to continuously monitor the operational status of the transport system and to verify whether the established plan is properly implemented at the site. Until now, research on monitoring and diagnosing the operating status of transport systems or equipment has been conducted by various researchers. Thompson et al. [23] provided the basis for mine maintenance management systems (MMS) by integrating data collected through onboard multi-sensors that were installed on trucks with existing mine communication and asset management systems. Park and Choi [24] developed a system that could collect truck travel time data using Bluetooth beacons and tablet computers. In addition, a method for analyzing and diagnosing the transport route status of underground mines was proposed using the collected data. Wodecki et al. [25] proposed a monitoring system that could identify major possible causes of machine failure events using the operational parameters of LHD in mines. Carvalho et al. [26] developed a system that could automatically identify the failure of a roller, one of the important components of a belt conveyor, by combining a thermal imaging camera with an unmanned aerial vehicle (UAV).

Recently, machine learning techniques have been actively utilized to monitor the transport systems and assets of mines, diagnose failures, and perform proper maintenance. Paduraru and Dimitrakopoulos [27] utilized neural networks and policy gradient reflection learning in data-driven decision-making processes to optimize material flow in large mining complexes. Ristovski et al. [28] used machine learning to predict the probability distributions of equipment activity durations used in mining operations. Xue et al. [29] and Sun et al. [30] used a machine learning model to predict truck travel time. Zhang et al. [31] used the support vector machine (SVM), a machine learning technique, to diagnose and classify the defects of the scraper conveyor in coal mine. D’Angelo et al. [32] proposed a method for real-time diagnosis of defects in rollers of belt conveyors using an object detection model based on a deep learning architecture.

Establishing operational plans, such as mine design, production forecasting, and equipment allocation, is important to ensure productivity and efficiency of mines. In addition, identifying in advance the section in which the truck travel time is expected to be abnormal is crucial because this makes it possible to prevent the occurrence of problems in the section and vehicle, as well as in the future. However, no research case has been reported thus far for monitoring and diagnosing the condition of a mine transport system using machine learning techniques. Therefore, we propose a method to evaluate the stability of transport routes and to diagnose the operational status by combining the mine production management system using a tablet computer and Bluetooth beacon with machine learning techniques. To this end, a limestone underground mine in Korea—to which a tablet computer and Bluetooth beacon was applied—was selected as a research area, and log data related to truck travel time were collected for a certain period. In addition, machine learning models were trained using the collected data. Thereafter, the stability of each section of the transport route in the study area was evaluated, diagnosed, and analyzed using the learned model.

2. Study Area and Data Collection

In this study, an underground mine (37°17′12″ N, 128°43′53″ E) owned by Seongshin Minefield in Korea was selected as the research area. Figure 1 depicts an aerial view of the study area and an underground tunnel. The mine uses the room and pillar mining method to produce 1 million tons of high-quality limestone annually. They drill with a V-Cut method using jumbo drills and crawler drills. It then produces approximately 4500 tons of limestone, with an average of 8–9 blasts per day using ammonium nitrate fuel oil (ANFO), emulite, and electric detonator (6 ms). The mined limestone is loaded into a 25–40 tons dump truck with a loader (3.0–5.6 m³) and transported to the crusher located outside the mine. The study area operates eight loading areas and three unloading points, and three loaders and ten trucks are used to produce limestone.

The underground mine selected as the study area is equipped with a tablet computer and Bluetooth beacon-based mine production management system. This system provides functions for navigation, equipment proximity warning, production log creation, and measurement of truck travel time for each section of the underground mine [33]. The operation of the system is performed in the following order: (1) Signals are received from Bluetooth beacons installed at major points along the transport route, crusher, and loaders using a tablet computer mounted on the truck. (2) The tablet computer records the time the signal was received and the location of the truck, and (3) transmits the data stored in the internal memory to the cloud server in the area where wireless communication is possible. (4) Finally, the cloud server continuously stores and manages data transmitted from multiple trucks with tablet computers installed. For details on the operation of the system, please refer to Park and Choi [33]. Tablet computers were installed in 10 trucks used for transport operations. Bluetooth beacons were installed at loading and unloading points (8 and 3, respectively) and at major points along the transport route (11). Figure 2 shows an example of a tablet computer and Bluetooth beacon installed in the study area. Figure 3 depicts a schematic diagram showing the locations of the loading and unloading points and the Bluetooth beacon installed on the main transport route.

The purpose of this study is to calculate the truck travel time for each section based on the main points where Bluetooth beacons are installed, evaluate the stability of each transport route using a machine learning model, and diagnose the status of the transport route. The system developed by Park and Choi [33] uses a tablet computer to record the time a truck passes through the point where a Bluetooth beacon is installed; however, it cannot record the travel time of a truck traveling between the two beacons. Therefore, in this study, the truck travel time for each section was calculated using the log data analysis program developed by Park and Choi [24]. The program calls the log data files uploaded to the cloud server at once, organizes the log data, and calculates the truck travel time for each section.

In this study, log data collected from 9 November 2020 to 21 February 2021 (15 weeks) were used to evaluate and diagnose the stability of each section of the transport route using machine learning techniques. During this period, 361 log data files were uploaded to the cloud server, and 33,435 truck travel time data by section were collected.

3. Methods

The purpose of this study is to evaluate and diagnose the stability of the transport route by using the truck travel time for each section of the transport route and machine learning techniques. To achieve the purpose of the study, the research was conducted in the order of data collection for learning and verification, data processing, machine learning model selection and application of the model.

3.1. Data Preprocessing for Machine Learning Model

Factors for diagnosing the status of each section of the transport route include physical factors (location and slope of section, presence or absence of surrounding workplaces, width of transport routes, whether or not ores are loaded, etc.) and environmental factors (weather, presence or absence of groundwater, etc.) [24].

Therefore, the training data of the machine learning model for diagnosing the state of the transport path was composed of six input features and a label that judges the status of the transport path as shown in

Table 1. Description and data type of data set for training machine model.

Dataset	Description	Data Type
Features	Origin beacon ID	Integer (1–22)
	Destination beacon ID	Integer (1–22)
	Transport time	Seconds (sec)
	Average daily temperature	Celsius temperature (°C)
	Daily precipitation	Millimeter (mm)
	Whether ores are loaded	0: Loaded, 1: Empty
Label	Truck transport time status on transport route	0: Normal, 1: Abnormal

. Data types can be divided into categorical data and continuous data. The categorical data include the origin and destination (consisting of beacon IDs) of the transport route section and whether ores are loaded. Continuous data include truck travel time, average daily temperature, and daily precipitation.

Coding of raw data to train the machine learning model was performed using log data related to truck travel time, which was collected from the mine production management system and weather data provided by the Korea Meteorological Administration. The status of the transport route was determined by the mine production management system installed in the research area. The truck driver judges whether the operation of the truck was normal or abnormal by considering whether any irregularity of operation occurs, such as natural causes, vehicle maintenance, tunnel closure, work interruption, accident, or excessive waiting. The truck drivers use the application of the mine production management system to input whether the operation was performed normally or abnormally when the loading, transporting, and unloading work is completed once. In this study, the case of normal operation was classified as 0, and the case of abnormal operation was classified as 1. Of the 33,435 truck travel time data collected by section, 3314 were classified as abnormal (1) by the truck driver.

The data types of input features used in this study consist of categorical data and continuous data. Because data of different dimensions are not normalized, features with small absolute values are ignored in the fault diagnosis system. Therefore, data were normalized using the min-max scaling-normalization method (Equation (1)):

$${x^{\prime }}_i=\frac{x_i-\min \left(x\right)}{\mathit{max}\left(x\right)-\min \left(x\right)}$$

(1)

this method can effectively prevent overfitting when training a machine learning model [34] and remove absolute differences between data items through data preprocessing, while maintaining relative differences in data within the same item. It can also improve the effectiveness of classification because it reduces the adjustment steps of parameters and improves the training speed of the model.

The dataset for training the machine learning model sets the ratio of the data classified as normal to the data classified as abnormal in the state of the transport path as 1:1, and consists of a set of 6000 data (normal: 3000, abnormal: 3000). To validate the model trained with the training dataset, the entire data was divided into a training dataset and a validation dataset. The training and validation datasets were set to 75% and 25% of the total dataset, respectively (i.e., training dataset: 4500, validation dataset: 1500).

3.2. Experimental Setup for Machine Learning Algorithms

In this study, the stability and status of each section in the underground mine was evaluated and diagnosed by using machine learning algorithms. For this, Gaussian naïve Bayes (GNB), k-nearest neighbor (kNN), support vector machine (SVM), and classification and regression tree (CART) were used.

Naïve Bayes (NB) is a set of supervised learning algorithms that apply Bayes’ theorem with the “naive” assumption of independence between every pair of features [35]. Naïve Bayes can be trained efficiently in a supervised learning environment. Parameter estimation for the naïve Bayes model uses the method of maximum likelihood. In many applications, it has been confirmed that training is possible without accepting Bayesian probability or Bayesian methods. In addition, there is an advantage in the quite small amount of training data for estimating the parameters required for classification. NB can be mainly divided into Gaussian naïve Bayes (GNB) and multinomial naïve Bayes according to the type of data (i.e., continuous or categorical). GNB is an algorithm that calculates the continuous values associated with each class, often assuming that they follow a Gaussian distribution. For example, after dividing the training data including the continuous attribute $x$ according to the class, the mean and variance of $x$ in each class are called μ_x and σ_k, respectively. Then, assuming that a certain observation value $v$ has been collected, the probability distribution of the values of a given class can be parameterized with μ_x and σ_k and calculated through the normal distribution equation (Equation (2)):

$$p\left(x=v|c\right)=\frac{1}{\sqrt{2\pi {\sigma }_c^2}}{e}^{-\frac{{\left(v-{\mu }_c\right)}^2}{2{\sigma }_c^2}}$$

(2)

The kNN model is one of the most intuitive and simple supervised learning models among machine learning models. The kNN does not learn in advance, but rather defers this step and then performs classification when a task request for new data is received. Therefore, it is also variously called instance-based learning, memory-based learning, or lazy learning. The idea in the kNN method is to assign new unclassified examples to the class to which the majority of its $k$ nearest neighbors belong. It is effective to reduce the error of misclassification when the number of samples in the training dataset is large; however, the classification accuracy depends on the value of $k$, the number of neighbors, and depends greatly on the distance used to calculate the closest distance to the value of $k$ [36]. In simple kNN, the search is based on the number of class data classified closer to the new data. Figure 4 shows the classification of the data according to different $k$ values. When the first data are found, as shown in Figure 4a, while expanding the virtual circle (in case of two-dimensional) focusing on the new data to be known, the group to which the data belong becomes the group to which the new data belong ($k$= 1). Similarly, a virtual circle is extended until three data ($k$= 3) are found, and the largest group of the three data found at this time determines the group to which the new data belong (Figure 4b).

SVM was introduced by Boser et al. [37] in 1992 and has been popular in the learning community since 1996. Recently, it has been successfully applied to various problems related to pattern recognition in bioinformatics and image recognition [38]. In addition, it is sufficiently powerful to be used for both linear and non-linear regression and classification and is widely used by the public. SVM is basically a model that classifies data linearly like linear logistic regression and classifies data in three stages as shown in Figure 5. Assuming that there are two-dimensional data composed of two classes as shown in Figure 5a, there can be an infinite number of straight lines separating these classes; however, using the decision boundary selection condition of SVM, only one straight line can be selected. The selection condition is to select a hyperplane that maximizes the distance between the data points of each class that are closest to each other. First, as shown in Figure 5b, the closest points between each class are selected, and when the margin between two parallel straight lines including these points is maximized, two straight lines including these points are selected. The points used to select two straight lines are called support vectors, and when these two straight lines are determined, the central straight line located at the same distance between the two straight lines becomes the decision boundary, as shown in Figure 5c.

The optimal hyperplane can be defined as the following equation [39]:

$$y_i\left(\omega ·x_i+b\right)\ge \mathrm{for} 1\le i\le \mathrm{n}, \omega \in R^d,b\in R$$

(3)

where $x_i$ is an instance with its corresponding label $y_i\in \left(-1, 1\right)$, and b is an intercept term; that is, a normal vector to the hyperplane’, d is the number of properties of each instance and the dimension of input vector, and n is the number of instances. A hyperplane is defined by the instances that lie nearest to it; such instances are called support vectors. By this definition, there should be no data points between the hyperplanes containing the support vectors (hard margin classification); however, this classification cannot occur in the real world. This is because real data often contain outliers that are significantly different from other instances of the same class, in addition to the possibility of errors in data entry, measurement errors, etc. Therefore, we used a definition (soft margin classification, Equation (4)) proposed by Tuba et al. [39] for the optimal hyperplane by overcoming this problem and relaxing the conditions to use SVM for real data classification:

$$y_i\left(\omega ·x_i+b\right)\ge 1-{ϵ}_i, {ϵ}_i\ge 0, 1\le i\le n$$

(4)

here, ${ϵ}_i$ is a slack variable that allows the corresponding instance to leave the margin. To find the optimal hyperplane, we must solve the quadric programming problem as follows:

$$\min \frac{1}{2}{\Vert \omega \Vert }^2+C{\displaystyle \sum }_{i=1}^n{ϵ}_i.$$

(5)

Here, C represents the parameter of the soft margin cost function, and the quality of the SVM model largely depends on the choice of this parameter; that is, the larger the value of C, the more similar the generated model obtained from the hard margin classification definition. However, because the soft margin classification method can only be applied to linearly separable data, a kernel function is used, rather than a dot product. The kernel function maps the instances into a higher-dimensional space to ensure that they can be linearly separated. There are various kernel functions, such as polynomial, Gauss (radial basis function or RBF), and sigmoid functions, but RBF is the most commonly used and can be defined as follows:

$$K\left(x_i, x_j\right)=\exp \left(-\gamma {\Vert x_i-x_j\Vert }^2\right)$$

(6)

where $\gamma$ is a free parameter that significantly affects classification accuracy and this parameter defines the impact of each training instance.

CART is a decision tree (DT)-based algorithm that can be used for both classification and regression problems [40]. The data are divided into uniform labels based on the answers (yes/no) to the predictor values through an iterative procedure, and finally a binomial tree is generated. If the dependent variable is qualitative, it is called a classification tree, and if it is quantitative, it is called a regression tree. The node containing the entire dataset is called the root node. Starting from the root node, it is divided into left and right, and this process is repeated until the estimation error related to the dependent variable is minimized to classify the data [41]. Because CART is inherently non-parametric, no assumptions are made regarding the underlying distribution of values of the predictor variables [42]. Therefore, CART can handle numerical data that are highly skewed or multi-modal, as well as categorical predictors with either an ordinal or a nonordinal structure. In addition, it identifies the “splitting” variable based on a thorough search for all possibilities. Because efficient algorithms are used, CART has the advantage of being able to search for all possible variables with splitters, despite the existence of hundreds of possible predictors. CART is a relatively automated machine learning method because the analyst’s input is less than the complexity of the analysis.

Grid search through 5-fold cross-validation was utilized to improve the performance of the machine learning model and the reliability of the performance evaluation on the validation dataset. In general, the performance of a machine learning model depends on parameters. Various parameters exist depending on the machine learning algorithm. Therefore, to design a model with high accuracy, it is important to set the optimal parameters. 5-fold cross-validation is a method in which a dataset is divided into 5 pieces that are used one by one as a validation dataset while the rest are combined and used as a training dataset. Using this method, 100% of the data we have can be used as a validation dataset. The grid search is originally an exhaustive search based on a defined subset of the hyper parameter space [43]. That is, when creating a model, it is a search method to find the variable with the highest performance after sequentially inputting the hyperparameters set by the user.

Table 2. Values used in grid search for parameter tuning.

	GNB	kNN	SVM	CART
Parameter	var_smoothing	neighbors	C/$\gamma$	min_samples_leaf/min_samples_split
Min	10−9	1	10/0.1	1/2
max	1	100	100/1	10/10
Step	($\times$) 1.232847	(+) 1	(+) 5/0.1	(+) 1/1

shows the parameters and parameter tuning used in each model. The GNB predicted the accuracy of the model by setting the range of variance (var) smoothing from 10^–9 to 1 and increasing the parameter values by approximately 1.23 times because the accuracy of the model varies depending on the var smoothing. The classification accuracy of the kNN depends on the $k$ value, which means the number of neighbors, and the accuracy was predicted by increasing the $k$ value by 1 from 1 to 100. Because the classification accuracy of the SVM model varies greatly depending on the parameters C and $\gamma$, the optimal pair of parameters (C: from 10 to 100, $\gamma$: from 0.1 to 1) was determined by increasing the values by 5 and 0.1, respectively. Finally, in the CART model, the accuracy of the model is determined by the minimum samples leaf (min_samples_leaf) and minimum samples split (min_samples_split). In this study, the optimal parameter was determined by setting the minimum samples leaf from 1 to 10 and increasing by 1, and for the minimum samples split, setting a range from 2 to 10 and increasing by 1.

3.3. Validation of Machine Learning Models

The parameter showing the highest learning accuracy of the machine learning model was determined using grid search through 5-fold cross-validation. Subsequently, the performance of the model was verified using the validation dataset (25% of the total data, 1500). Performance indicators that can evaluate the performance of a model generally depend on the type of supervised learning (regression or classification). In this study, the performance of the model was verified using the accuracy, precision, recall, and F1 score, which are typically used in classification problems. Accuracy refers to the number of correct predictions among all predictions, precision refers to the probability of the state actually being positive when a positive prediction is made, recall refers to the probability of correctly predicting an actual positive, and F1 score refers to the weighted average of precision and recall. The formula for each performance indicator is as follows:

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(7)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(8)

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(9)

$$\mathrm{F}1 \mathrm{score}=\frac{2\times \mathrm{Recall}\times \mathrm{Precision}}{\mathrm{Recall}+\mathrm{Precision}}$$

(10)

where, P (positive) and N (negative) denote whether the prediction of the model is positive (yes) or negative (no), and T (true) and F (false) imply whether the prediction is correct or wrong. When this is expressed as a matrix, it is called a confusion matrix and can be expressed as shown in

Table 3. Confusion matrix of a classifier.

		Predicted Data
		Negative (0)	Positive (1)
Actual data	Negative (0)	TN (True Negative)	FP (False Positive)
	Positive (1)	FN (False Negative)	TP (True Positive)

.

4. Results

4.1. Results of Data Preprocessing

For the learning and validation of machine models, 33,435 truck travel time data were collected for 15 weeks using the mine production management system installed in the study area. Except for the departure and arrival points for each section of the transport route and the transport time of the truck that can be acquired using the system, additional features such as daily average temperature, daily precipitation, ores loading of trucks, and labels to determine abnormal status of transport routes were entered. The decision to load ores can be divided into the case of empty truck or loaded truck. This was determined by judging whether the truck is headed for the loading points (empty truck) or the crusher (loaded truck) according to the sequence of beacon IDs for each section of the transport route. As a result of judging the transport route status of the data collected during the 15-week period, 3314 data out of 33,435 truck travel time data were found to be abnormally measured.

When all 33,345 data are used as training data, the normal case is much larger than the abnormal case, and a biased training result may appear. Therefore, the ratio of the data classified as normal to the data classified as abnormal was set to 1:1 to prepare the training dataset. A total of 6000 data were prepared by random sampling of 3000 data marked as abnormal and 3000 data marked as normal. Before normalizing the training dataset, the mean values, standard deviations, minimum and maximum values for truck travel time, daily average temperature, and daily precipitation were calculated (

Table 4. Feature of data set for training machine learning model.

	Truck Travel Time (s)	Average Daily Temperature (°C)	Daily Precipitation (mm)
Mean	95.55	−2.71	0.24
Standard deviation	74.12	5.03	0.85
Minimum value	1.00	−14.30	0.00
Maximum value	299.00	11.40	6.90

). Figure 6 shows the histogram for statistical values. The average truck travel time was 95.55 s and the standard deviation was 74.12 s. The average daily temperature was −2.71 °C and the standard deviation was 5.03 °C. The average daily precipitation was 0.24 mm and the standard deviation was 0.85 mm.

4.2. Results of Model Training and Application

In this study, GNB, kNN, SVM, and CART models were used to evaluate and diagnose the stability of truck transport routes. To design the most accurate predictive model, the parameter values related to the learning accuracy of each model were optimized. For this purpose, grid search through 5-fold cross-validation was used.

The classification accuracy of the GNB model depends on the parameter var smoothing. The optimal model was determined by setting the parameter value range from 10⁻⁹ to 1 and increasing the parameter value by approximately 1.23 times. Figure 7 is a graph showing the accuracy of the model according to the change of the var smoothing. The accuracy of the model decreases rapidly when the parameter value exceeds 10⁻². The GNB showed the highest learning accuracy (0.60) when the var smoothing value was 0.000188.

The accuracy of the kNN model depends on the value of k. Figure 8 shows the prediction of the accuracy of the model while increasing the $k$ value by 1 from 1 to 100. The accuracy of the kNN model was higher as the $k$ value was smaller ($k$ = 1, 0.85).

The SVM model was optimized by changing the values of C and $\gamma$ to determine the parameter value showing the highest training accuracy. The parameter C was set in the range from 10 to 100 and increased by 5, while $\gamma$ was increased by 0.1 from 0.1 to 0.9 to calculate the accuracy of the model. Figure 9 shows the training accuracy of the model according to the change in parameter C and $\gamma$ value. As the values of C and $\gamma$ increased, the accuracy of the model also tended to increase. In the SVM model, when the C value was set to 100 and the $\gamma$ value was set to 0.9, the model accuracy was the highest at 0.78.

The training accuracy of CART depends on the values of minimum samples leaf and minimum samples split. In this study, the values of two parameters were optimized by increasing min_samples_leaf by 1 from 1 to 10 and increasing min_samples_split by 1 from 2 to 10. Figure 10 shows the training accuracy of the CART model depending on the changes in the two parameter values. When min_samples_leaf is 3 or less, the accuracy of the model tends to decrease as min_samples_split increases; however, when min_samples_leaf was at least 4, the accuracy did not change significantly even if min_samples_split was increased. The training accuracy of CART showed the highest accuracy (0.94) when min_samples_leaf was set to 1 and min_samples_split was set to 4.

The previously determined parameters were applied to each model, and verification was performed. The validation of the machine learning models was performed using the validation dataset (25% of the total data, 1500 pieces).

Table 5. Confusion matrix classified using the GNB model.

Normalization $\mathsf{v}\mathsf{a}{\mathsf{r}}_{\mathsf{s}\mathsf{m}\mathsf{o}\mathsf{o}\mathsf{t}\mathsf{h}\mathsf{i}\mathsf{n}\mathsf{g}} = 0.000188$		Predicted Data
		Negative (0)	Positive (1)	Accuracy
Actual data	Negative (0)	687 (TN)	75 (FP)	0.90
	Positive (1)	496 (FN)	242 (TP)	0.33
	Accuracy	0.58	0.77	0.62
		Training accuracy		0.60

,

Table 6. Confusion matrix classified using the kNN model.

Normalization n Neighbors = 1		Predicted Data
		Negative (0)	Positive (1)	Accuracy
Actual data	Negative (0)	642 (TN)	120 (FP)	0.84
	Positive (1)	122 (FN)	616 (TP)	0.83
	Accuracy	0.84	0.84	0.84
		Training accuracy		0.85

,

Table 7. Confusion matrix classified using the support vector machine (SVM) model.

Normalization $\mathbf{C} = 100, \mathbf{\gamma } = 0.9$		Predicted Data
		Negative (0)	Positive (1)	Accuracy
Actual data	Negative (0)	655 (TN)	107 (FP)	0.86
	Positive (1)	196 (FN)	542 (TP)	0.73
	Accuracy	0.77	0.84	0.80
		Training accuracy		0.79

and

Table 8. Confusion matrix classified using the CART model.

Normalization Leaf = 1, Split = 4		Predicted Data
		Negative (0)	Positive (1)	Accuracy
Actual data	Negative (0)	713 (TN)	49 (FP)	0.94
	Positive (1)	32 (FN)	706 (TP)	0.96
	Accuracy	0.96	0.94	0.95
		Training accuracy		0.94

shows the model verification results as a confusion matrix.

Table 5 shows the verification results of the GNB as a confusion matrix. There were 687 cases (TN) where the section in which the truck travel time classified as normal was predicted to be normal. Conversely, there were 242 cases (TP) where the section classified as abnormal was predicted to be abnormal. In addition, it was found that there were 496 (FN) and 75 (FP) cases of predicting a section where the truck travel time was abnormal as normal and predicting a section where the truck was normal as abnormal, respectively. GNB’s verification accuracy was 0.62, and when predicting the data classified as normal as normal, it showed relatively high accuracy (0.90); however, the accuracy of predicting data classified as abnormal as abnormal was very low at 0.33.

In the case of the kNN model, TN and TP, which are cases of correct prediction of the actual data among 1500 verification data, appeared 642 times and 616 times, respectively. FN and FP, which were failed predictions, appeared 122 times and 120 times, respectively (Table 6). The validation accuracy of the kNN model was found to be 0.84, and it showed a similar level of accuracy in all cases.

Table 7 shows the verification results of the SVM model as a confusion matrix. There were 655 cases (TN) where the section in which the truck travel time was classified as normal was predicted to be normal. Conversely, there were 542 cases (TP) where the section classified as abnormal was predicted to be abnormal. In addition, FN and FP, which were failed predictions, appeared 196 times and 107 times, respectively. The verification accuracy of the SVM model was 0.80, and it showed high accuracy (0.86) in the data classification problem, which was actually abnormal; however, in the problem of classifying actually normal data, the accuracy (0.73) was relatively low.

The verification results of the CART model are shown in Table 8. In fact, 713 times (TN) were predicted to be normal where the section in which the truck travel time was classified as normal, and 706 times (TP) were predicted to be abnormal where the section classified as abnormal. In addition, FN and FP that failed prediction appeared 32 times and 49 times, respectively. The verification accuracy of the CART model was very high at 0.95, and both the problem of classifying the actual normal sections (0.96) and the problem of classifying the abnormal sections (0.94).

The performance of the model was evaluated based on the confusion matrix of each model analyzed using the validation dataset. The performance assessment of the model was conducted using accuracy, precision, recall, and F1 score.

Table 9. Performance assessment indicators of machine learning (ML) models.

Performance Assessment Indicators	GNB	kNN	SVM	CART
Accuracy (%)	61.9	83.9	79.8	94.6
Precision (%)	76.3	83.7	83.5	93.5
Recall (%)	32.8	83.5	73.4	95.7
F1 score (%)	45.9	83.6	78.2	94.6

shows the performance index of each model. The prediction accuracy of the machine learning model was the highest in CART (94.6%), followed by kNN (83.9%), SVM (79.8%), and GNB (61.9%). The CART model also exhibited high precision, recall, and F1 score. Therefore, it can be said that the CART model achieves the best performance in the problem of evaluating the stability of the transport route for each section in the underground mine.

5. Discussion

5.1. Analysis of Model Accuracy for Each Transport Route Section

The prediction accuracy of each section was calculated using 1500 pieces of data used in the verification process of the CART model. Figure 11 shows the accuracy of the model for each section when operating with empty or loaded trucks. The model achieved an accuracy of at least 57.1% for all sections (45 sections) and exhibited an average accuracy of 93.3%. In the case of the route (23 sections) operating with empty trucks, the prediction accuracy of the model was found to be very high with an average of 90.9%. Except for four sections (beacon ID: 1→3, 5→21, 13→15, 16→17), all were confirmed to show an accuracy of at least 80%. In the case of operating with loaded trucks (22 sections), the prediction accuracy of the model was found to be very high, with an average of 95.9%. In addition, 20 of the 22 sections showed over 80% accuracy. The accuracy of the model for each section of transport route tended to be generally higher as the amount of data for each section included in the dataset used for learning increased (

Table 10. Relationship between the prediction accuracy and the average number of data used in machine learning for each section.

Operation Type	Prediction Accuracy (%)	Number of Sections	Average of the Number of Data Used for Machine Learning for Each Section
Empty haul	91–100	14	105.9
	81–90	5	90.2
	71–80	3	59.3
	61–70	0	N/A
	57.1–60	1	26.0
Loaded haul	91–100	19	138.0
	81–90	1	90.0
	71–80	1	48.0
	66.7–70	1	27.0

). Therefore, in the case of the section where the prediction accuracy of the model is high, it is judged that the model can be used sufficiently to evaluate whether the truck was operated normally in the section; however, in the case of a section where the accuracy is low, it is judged that the machine learning model should be improved through additional data collection for training the model is necessary.

5.2. Further Verification of the CART Model Using Unused Data

The CART model was further verified using the remaining 27,435 data not used to train the model.

Table 11. Confusion matrix for further verification of the CART model on the remaining 27,435 data.

Normalization Leaf = 1, Split = 4		Predicted Data
		Negative (0)	Positive (1)	Accuracy
Actual data	Negative (0)	26,027 (TN)	1094 (FP)	0.96
	Positive (1)	3 (FN)	311 (TP)	0.99
	Accuracy	1.00	0.22	0.96

shows the verification results of the CART model as a confusion matrix. There were 26,027 cases (TN) where the section in which the truck travel time was classified as normal was predicted to be normal. Conversely, there were 311 cases (TP) where the section classified as abnormal was predicted to be abnormal. There were three (FN) and 1094 (FP) cases of predicting a section where the truck travel time was abnormal as normal and predicting a section where the truck was normal as abnormal, respectively. The verification accuracy of the CART model using the remaining 27,435 data was 0.96, which was similar to the result (0.95) of the CART model trained and verified with 6000 data in Table 8.

Table 12. Performance assessment indicators for further verification of CART model on remaining 27,435 data.

Performance Assessment Indicators	CART Model
Accuracy (%)	96.0
Precision (%)	22.1
Recall (%)	99.0
F1 score (%)	36.2

shows the performance index of the CART model verified using the remaining 27,435 data. The prediction accuracy of the model was 96%, and the precision, recall, and F1 score were 22.1%, 99%, and 36.2%, respectively. In general, in the case of a classification problem using data with less data imbalance, it can be said that the higher the performance index, the better the model performance [44]. However, when data imbalance exists, even if precision is low, the model can be trusted when recall is high [45]. In the case of the remaining 27,435 data, there was an imbalance in the data because the normal data takes up a much larger proportion than the abnormal data. Therefore, it can be determined that the CART model is reliable when considers the value of Recall appears as 99%.

5.3. Practical Use at the Underground Mine Site

The proposed machine learning model can diagnose the operation status of the section by determining whether the truck travel time for each section is normal or abnormal. In this study, during the validation of the CART model, one section with high prediction accuracy and one with low prediction accuracy were selected. Then, using the log data additionally collected from the mine production management system, evaluation was performed on whether the truck was operated normally in the relevant section. For this purpose, log data collected during the 16th week (22–27 February 2021) were used for analysis. For the section of the transport route, the section from beacon ID 11 to 6 and section from beacon ID 13 to 14 were selected. In these sections, the validation accuracy of the model when validating the machine learning model was 100% and 82%, respectively.

First, in the case of sections 11 to 6 of beacon IDs, three trucks operated the section a total of 41 times in a week. By truck, truck A drove 1 time, truck B drove 25 times, and truck C drove 15 times. Log data for the section showed that truck travel time was measured within the normal range in 37 operations, and within the abnormal range in four operations. This section is a transport route for empty trucks toward the loading point, and there is no loading or dumping near the route. Therefore, most trucks have the characteristic of moving without stopping in the relevant section. After converting the log data of the relevant section (beacon ID 11→6) into the input data of the CART model, prediction was performed on whether the truck travel time was measured normally or abnormally. As a result, it was predicted that the truck travel time was measured within the normal range in the case of actual normal operation. In addition, in the case of abnormal operation, it was predicted that the operation was performed abnormally. In other words, it was found that the actual data and the prediction results by the CART model were identical.

Table 13. Confusion matrix classified using the CART model for beacon IDs 11 to 6.

Normalization $\mathsf{L}\mathsf{e}\mathsf{a}\mathsf{f} = 1, \mathsf{S}\mathsf{p}\mathsf{l}\mathsf{i}\mathsf{t} = 4$			Predicted Data
			Negative (0)	Positive (1)
Actual data	Truck A	Negative (0)	1 (TN)	0 (FP)
		Positive (1)	0 (FN)	0 (TP)
	Truck B	Negative (0)	22 (TN)	0 (FP)
		Positive (1)	0 (FN)	3 (TP)
	Truck C	Negative (0)	14 (TN)	0 (FP)
		Positive (1)	0 (FN)	1 (TP)

shows the prediction results of the CART model for the 16-week data by classifying them by trucks that have operated the relevant section and is presented as a confusion matrix.

Table 14. Prediction result of the CART model by time/truck for beacon IDs 11 to 6.

Time	22 February 2021			23 February 2021			24 February 2021			25 February 2021			26 February 2021			27 February 2021
	Truck			Truck			Truck			Truck			Truck			Truck
	A	B	C	A	B	C	A	B	C	A	B	C	A	B	C	A	B	C
08:00					■						●	●			●		●
09:00		●			●	● ●									●		●
10:00		●			●	●						●			●			●
11:00								●			■						●
12:00 (Break time)
13:00		●			●				●		●			■	●
14:00		●			●							●					●
15:00		●	●							●	● ●						● ●
16:00					●	■			●								●

● TN Cases in which data that are actually normal are predicted to be normal. ■ TP: Cases in which data that are actually abnormal are predicted to be abnormal.

is a visualization of the confusion matrix divided by time period. This means that, during the period, trucks operated well reflecting the trend of the existing truck travel time. Furthermore, it means that there are no problems in the truck or in the transport section that will affect the truck travel time.

Next, in the section from beacon ID 13 to 14, two trucks operated a total of 58 times (Truck A: 34 times, Truck B: 24 times) during a week. In this section, 54 truck travel times were measured within the normal range, but four times were measured within the abnormal range. This section is a transport route where an empty truck goes to the loading point. However, because the loading point (Area D) is located around the route, it is a section where variations in truck travel time may occur. As a result of predicting the state of truck travel time using the CART model for the section, the prediction accuracy was found to be very low (86.2%). Normal data were predicted as normal 46 times (TN), and abnormal data were predicted as abnormal (TP) four times. In addition, it was found that there were eight (FP) cases of predicting a section where the truck was normal as abnormal (

Table 15. Confusion matrix classified using the CART model for beacon IDs 13 to 14.

Normalization $\mathsf{L}\mathsf{e}\mathsf{a}\mathsf{f} = 1, \mathsf{S}\mathsf{p}\mathsf{l}\mathsf{i}\mathsf{t} = 4$			Predicted Data
			Negative (0)	Positive (1)
Actual data	Truck A	Negative (0)	25 (TN)	7 (FP)
		Positive (1)	0 (FN)	2 (TP)
	Truck B	Negative (0)	21 (TN)	1 (FP)
		Positive (1)	0 (FN)	2 (TP)

). For this section, considering that the verification accuracy has already been shown to be low, it can be confirmed that the prediction accuracy appears low even in the prediction using the 16-week data.

Table 16. Prediction result of the CART model by time/truck for beacon IDs 13 to 14.

Time	22 February 2021		23 February 2021		24 February 2021		25 February 2021		26 February 2021		27 February 2021
	Truck		Truck		Truck		Truck		Truck		Truck
	A	B	A	B	A	B	A	B	A	B	A	B
08:00	●				●	●		●	●	●	●	●
09:00			● ●			● ■	● ●		●	●		●
10:00		● ●	■		● ●	●	● ●		●	●
11:00			●		●			●	●
12:00 (Break time)					●					●	●
13:00	●		● ●				●		●
14:00				●	●			● ●			■
15:00	●			●	●	●	●	■	●	●	● ●	●
16:00	●	●				●

● TN Cases in which data that are actually normal are predicted to be normal; ■ TP: Cases in which data that are actually abnormal are predicted to be abnormal; ● FP: Cases in which data that are actually normal are predicted to be abnormal.

is a visualization of the confusion matrix divided by time period. In this section, some prediction failures of the CART model occur. To improve the accuracy of the model, additional data collection is required for training the machine learning model, and the model needs to be improved. In addition, because some data show abnormal truck travel time, this section needs to be carefully monitored. To improve the overall productivity of the mine and the efficiency of the trucking operation, and to reduce the time required to transport the ores, it is necessary to monitor and respond to these sections in advance.

If a case in which the truck travel time is abnormally predicted is observed only in a specific truck, the possibility that the truck driver’s skill is insufficient, the truck’s maintenance is poor, or the maintenance period has arrived should be suspected. In addition, the manager of the mine should take appropriate action in this regard. According to what we have seen so far, the proposed CART model can predict the status of truck travel time and then monitor the problem or possibility of occurrence in the transport route or equipment in advance. In addition, it can help to analyze the cause and prepare countermeasures. Therefore, the CART model can be used as a tool for mine managers to improve the productivity and efficiency of transport operations.

5.4. Comparison between the Existing and Machine Learning-Based Methods

Various researchers are using machine learning techniques to monitor and diagnose mine operating systems, equipment, and facilities. However, hitherto, no research case has been reported on monitoring and diagnosing the condition of a mine transport system using machine learning techniques. As a similar research case related to diagnosing and predicting the status of transport routes using truck travel time data, Park and Choi [24] evaluated the stability and classified the types of transport routes using the statistics of the truck travel time for each section of the transport route. The method of collecting log data related to truck travel time is the same as that used in this study. However, Park and Choi [24] used percentiles (P10, P90) of truck travel time to evaluate the stability and condition of each section of the transport route. That is, if the newly collected truck travel time was measured in the range between percentiles P10 and P90, the status of the transport route was classified as normal; otherwise, it was classified as abnormal. The truck travel time of the mine may vary depending on the production plan, vehicle dispatch plan, tunnel maintenance and repair status, season (temperature), precipitation, and driver’s driving skill. In this study, the truck travel time for each section was evaluated by considering the beacon IDs (origin and destination), temperature, precipitation, and whether the truck was loaded in addition to the transport time, and then the status of the transport route was diagnosed. Therefore, the proposed method is considered to have a higher level of reliability than the existing method used to evaluate the stability and condition of each section of the transport route by considering the statistics of the truck travel time.

6. Conclusions

In this study, we proposed a method that can utilize log data related to truck travel time and machine learning model (GNB, kNN, SVM, CART) to evaluate the stability of the underground mine transport route and to diagnose the operation status. To this end, a limestone mine that collects truck travel time data in underground mines using Bluetooth beacons and tablet computers was selected as a study area, and truck travel time data were collected for a certain period of time. In addition, learning and validation of models were performed using the collected data, and the results of monitoring and diagnosis of the transport route status in the study area were presented. As a result of performing grid search through 5-fold cross-validation using the training dataset, the accuracy (94.1%) was highest when the parameters min_samples_leaf and min_samples_split of the CART model were set to 1 and 4, respectively. In the validation of the CART model performed using the validation dataset (1500 data), data with normal truck travel time were predicted as normal 713 times, and abnormal data were predicted as abnormal 706 times. The performance of the machine learning model was judged using accuracy, precision, recall, and F1 score. The accuracy of the CART model was 94.6%, and the precision and recall were 93.5% and 95.7%, respectively, and it was confirmed that the F1 score was also high at 94.6%.

The proposed CART model proposed can be used for monitoring and diagnosing the status of the transport route that constitutes the truck transport system in the underground mine. In addition, it is judged that it can be used sufficiently as a tool to improve the productivity and efficiency of mine transport operations. Because the truck travel time for each section has variability depending on the driver’s driving skill, tunnel maintenance and repair status, vehicle dispatch plan, etc., the truck travel time has a significant impact on the efficiency and productivity of truck transport operations. Therefore, it is crucial to know the section in which the truck transport operation is expected to be abnormal and to prevent problems occurring in the section, the vehicle, or the future. The proposed CART model showed an average prediction accuracy of 94.1% for all sections of the study area. This means that the stability of the transport route can be evaluated and diagnosed by judging whether the newly collected truck travel time data are measured within the normal range or within the abnormal range at a relatively high level of reliability. However, the prediction accuracy was relatively low in some sections. To improve the prediction accuracy of this section, additional collection of truck travel time data for training the machine learning model is required, and accordingly, the model will have to be improved.

In this study, it was confirmed that machine learning techniques can be used to diagnose and predict the condition of transport routes to maintain equipment and workplaces in underground mines. To that end, a method for diagnosing and predicting mine transport systems using machine learning techniques was proposed. We expect that the proposed method can be sufficiently applied not only in underground mines but also in open-pit mines from a methodological perspective.