Deep Neural Network for Predicting Ore Production by Truck-Haulage Systems in Open-Pit Mines

: This paper proposes a deep neural network (DNN)-based method for predicting ore production by truck-haulage systems in open-pit mines. The proposed method utilizes two DNN models that are designed to predict ore production during the morning and afternoon haulage sessions, respectively. The conﬁguration of the input nodes of the DNN models is based on truck-haulage conditions and corresponding operation times. To verify the e ﬃ cacy of the proposed method, training data for the DNN models were generated by processing packet data collected over the two-month period December 2018 to January 2019. Subsequently, following training under di ﬀ erent hidden-layer conditions, it was observed that the prediction accuracy of morning ore production was highest when the number of hidden layers and number of corresponding nodes were four and 50, respectively. The corresponding values of the determination coe ﬃ cient and mean absolute percentage error (MAPE) were 0.99% and 4.78%, respectively. Further, the prediction accuracy of afternoon ore production was highest when the number of hidden layers was four and the corresponding number of nodes was 50. This yielded determination coe ﬃ cient and MAPE values of 0.99% and 5.26%, respectively.


Introduction
In open-pit mines, ore loading and hauling material-handling operations account for approximately 50% of the total mine-operation cost [1]. Therefore, it is essential to design a truck-haulage system that not only maximizes mine productivity and equipment-management efficiency but also minimizes haulage cost [2]. Recently, several discrete-event simulation techniques [3][4][5][6][7][8][9][10][11] have been proposed for realizing effective truck-haulage systems designs. The allocation phase in the simulation of truck-haulage systems involves selecting the type, size, number, and payload of fleets suitable for use in haulage operations. This is followed by a dispatch phase that assigns trucks to a specific shovel by considering ore production and equipment utilization [12]. Several simulation algorithms based on mixed-integer linear programming [13][14][15], queuing theory [2,[16][17][18][19], linear programming [20][21][22], goal programming [23], and stochastic programming [24] have been proposed for fleet allocation. Simulation techniques based on the assignment-problem [25,26] and transportation-problem [27] approaches have also been developed for truck dispatching. Several other products for sophisticating truck dispatching, such as Modular Mining Systems' DISPATCH ® [28] and Caterpillar's CAT ® MINESTAR ™ [29], have also been commercialized.
Information communication technologies (ICTs), such as wireless communications, sensor networks, global positioning system (GPS), and cloud computing, have been implemented at open-pit-mining sites to facilitate real-time monitoring of the operating status and haulage information of equipment, as well as collection of equipment-tracking data on web servers [30][31][32][33][34][35][36][37][38]. Further, several In this study, an open-pit limestone mine installed with an ICT-based mine-safety-management system was used as the investigation location. Two DNN models were designed to predict morning and afternoon ore productions, respectively, with the input-layer nodes of each model configured with the truck-haulage operating conditions and haulage-operation time as variables. DNN-model training data were generated by processing packet data collected over the two-month period December 2018 to January 2019. The two models were trained by varying the number of hidden layers and their corresponding nodes, and their prediction accuracy was evaluated by calculating the determination coefficient and mean absolute percentage error (MAPE) for the training and validation data. By using the DNN model with the lowest prediction error, morning and afternoon ore productions on five days of the second week of February 2019 were predicted and compared against observed values.

Study Area
In this study, an open-pit mine owned by Hanil Cement Co., Ltd. in the Republic of Korea (position coordinates: 128° 19′ 58′′ E; 37° 1′ 59′′ N), was selected as the investigation location. The mine produces limestone, from which approximately 8.1 million tons of cement is manufactured each year.  The mine is equipped with 10 shovels, two loaders, and three dozers to facilitate ore production, along with 15 dump trucks with loading capacities of 45, 60, and 84 tons for ore haulage, which is accomplished via real-time dispatch. A production manager determines loading points by considering daily ore-production targets and target ore grades. Additionally, these managers inform The mine is equipped with 10 shovels, two loaders, and three dozers to facilitate ore production, along with 15 dump trucks with loading capacities of 45, 60, and 84 tons for ore haulage, which is accomplished via real-time dispatch. A production manager determines loading points by considering daily ore-production targets and target ore grades. Additionally, these managers inform truck drivers of their destinations. Truck drivers drive to loading points to load ore into their trucks, and subsequently, drive to the nearest dumping zones. If there is no truck waiting to dump the ore it is carrying, they immediately dump the ore into the shaft; otherwise, they wait for their turn. If the quantity of dumped ore exceeds the shaft capacity, the trucks dump the excess ore into the storage yard within the dumping zone. Next, the truck drivers drive back to loading points, and repeat the ore-hauling operation. The ore dumped into the shaft is subsequently crushed and transferred to the cement plant via use of a conveyor belt.
The open-pit mine selected in this study is installed with an ICT-based mine-safety-management system that tracks the locations of equipment and mineworkers, as well as monitors operating conditions. Four wireless access points (APs) are installed in the investigation location considered. These wireless APs confirm the equipment location by recognizing tags attached to the equipment and send this information to a web server in real time. An operation manager can check this equipment location and its operating status in real time by visualizing the information sent to the server on his office dashboard (refer Figure 2). Figure 3 depicts the structure of packet data sent to the web server upon identification of equipment tags by wireless APs. Packet data are categorized into tag-recognition date, tag-recognition time, IP address of the identifying wireless AP, and tag-location data. The tag-location data are further divided into start of text (STX), data type, emergency type, tag sequence, tag ID, latitude, longitude, CHECKSUM, and end of text (ETX). The tag sequence indicates packet-data continuity and comprises numbers between zero and 255. Truck locations are expressed in the form of latitude and longitude coordinates, as measured by GPS, recorded in the degree-minutes-seconds (DMS) notation. Approximately 700,000 packet data are sent to the web server per day.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 4 of 26 truck drivers of their destinations. Truck drivers drive to loading points to load ore into their trucks, and subsequently, drive to the nearest dumping zones. If there is no truck waiting to dump the ore it is carrying, they immediately dump the ore into the shaft; otherwise, they wait for their turn. If the quantity of dumped ore exceeds the shaft capacity, the trucks dump the excess ore into the storage yard within the dumping zone. Next, the truck drivers drive back to loading points, and repeat the ore-hauling operation. The ore dumped into the shaft is subsequently crushed and transferred to the cement plant via use of a conveyor belt. The open-pit mine selected in this study is installed with an ICT-based mine-safety-management system that tracks the locations of equipment and mineworkers, as well as monitors operating conditions. Four wireless access points (APs) are installed in the investigation location considered. These wireless APs confirm the equipment location by recognizing tags attached to the equipment and send this information to a web server in real time. An operation manager can check this equipment location and its operating status in real time by visualizing the information sent to the server on his office dashboard (refer Figure 2). Figure 3 depicts the structure of packet data sent to the web server upon identification of equipment tags by wireless APs. Packet data are categorized into tag-recognition date, tag-recognition time, IP address of the identifying wireless AP, and taglocation data. The tag-location data are further divided into start of text (STX), data type, emergency type, tag sequence, tag ID, latitude, longitude, CHECKSUM, and end of text (ETX). The tag sequence indicates packet-data continuity and comprises numbers between zero and 255. Truck locations are expressed in the form of latitude and longitude coordinates, as measured by GPS, recorded in the degree-minutes-seconds (DMS) notation. Approximately 700,000 packet data are sent to the web server per day.   Appl. Sci. 2020, 10, x FOR PEER REVIEW 4 of 26 truck drivers of their destinations. Truck drivers drive to loading points to load ore into their trucks, and subsequently, drive to the nearest dumping zones. If there is no truck waiting to dump the ore it is carrying, they immediately dump the ore into the shaft; otherwise, they wait for their turn. If the quantity of dumped ore exceeds the shaft capacity, the trucks dump the excess ore into the storage yard within the dumping zone. Next, the truck drivers drive back to loading points, and repeat the ore-hauling operation. The ore dumped into the shaft is subsequently crushed and transferred to the cement plant via use of a conveyor belt. The open-pit mine selected in this study is installed with an ICT-based mine-safety-management system that tracks the locations of equipment and mineworkers, as well as monitors operating conditions. Four wireless access points (APs) are installed in the investigation location considered. These wireless APs confirm the equipment location by recognizing tags attached to the equipment and send this information to a web server in real time. An operation manager can check this equipment location and its operating status in real time by visualizing the information sent to the server on his office dashboard (refer Figure 2). Figure 3 depicts the structure of packet data sent to the web server upon identification of equipment tags by wireless APs. Packet data are categorized into tag-recognition date, tag-recognition time, IP address of the identifying wireless AP, and taglocation data. The tag-location data are further divided into start of text (STX), data type, emergency type, tag sequence, tag ID, latitude, longitude, CHECKSUM, and end of text (ETX). The tag sequence indicates packet-data continuity and comprises numbers between zero and 255. Truck locations are expressed in the form of latitude and longitude coordinates, as measured by GPS, recorded in the degree-minutes-seconds (DMS) notation. Approximately 700,000 packet data are sent to the web server per day.

DNN Prediction
A DNN model comprises three layers: Input, hidden, and output. Figure 4a depicts the DNN-structure model. Each layer comprises multiple nodes connected hierarchically to all nodes in the subsequent layer. The input and output layers are, in general, single layers, whereas the hidden layer may comprise two or more layers. Data features are fed to the input layer, and prediction values are derived from the output layer after processing them in the hidden layers. Figure 4b describes the prediction principle of a DNN model. The weighted sum of nodes is calculated and predicted values are derived using an activation function that exists within each hidden-layer node that receives the weighted sum of nodes as input and converts them into valid values. The most commonly used activation function in regression analysis is the rectified linear unit (ReLU) [66], which produces a value equal to the input if the weighted sum of nodes exceeds or equals zero; otherwise, it yields a value of zero.

DNN Prediction
A DNN model comprises three layers: Input, hidden, and output. Figure 4a depicts the DNNstructure model. Each layer comprises multiple nodes connected hierarchically to all nodes in the subsequent layer. The input and output layers are, in general, single layers, whereas the hidden layer may comprise two or more layers. Data features are fed to the input layer, and prediction values are derived from the output layer after processing them in the hidden layers. Figure 4b describes the prediction principle of a DNN model. The weighted sum of nodes is calculated and predicted values are derived using an activation function that exists within each hidden-layer node that receives the weighted sum of nodes as input and converts them into valid values. The most commonly used activation function in regression analysis is the rectified linear unit (ReLU) [66], which produces a value equal to the input if the weighted sum of nodes exceeds or equals zero; otherwise, it yields a value of zero. ReLU predicts values by repeatedly modifying the weight during each DNN-model training [67]. This weight modification is performed in reverse from the output layer to the input layer via backpropagation [68] until the cost function is minimized. The cost function can be expressed as the sum of the squares of differences between the observed and predicted values. Equation (1) defines the relation to evaluate the cost function (E). Here, n refers to the number of output-layer nodes, and and denote the observed and predicted values of the kth output node, respectively. Equation (2) adjusts the weight in such a way that the difference between the previous weight and partial derivative of the error function can be assumed as the next weight. Here, denotes the node number in the previous layer, denotes the node number in the next layer, denotes the weight at time t, and denotes the learning rate.

Design of DNN Model
As already mentioned, two DNN models were designed in this study to independently predict morning and afternoon ore productions owing to variations in the truck-haulage operation characteristics during different periods of the day. Figure 5 depicts the structure of a DNN model comprising a single input layer, hidden layers, and one output layer. The input and output layers comprise 19 nodes and a single node, respectively. The hidden layer was designed to comprise m nodes in all layers. ReLU predicts values by repeatedly modifying the weight during each DNN-model training [67]. This weight modification is performed in reverse from the output layer to the input layer via backpropagation [68] until the cost function is minimized. The cost function can be expressed as the sum of the squares of differences between the observed and predicted values. Equation (1) defines the relation to evaluate the cost function (E). Here, n refers to the number of output-layer nodes, and y k andŷ k denote the observed and predicted values of the kth output node, respectively. Equation (2) adjusts the weight in such a way that the difference between the previous weight and partial derivative of the error function can be assumed as the next weight. Here, i denotes the node number in the previous layer, j denotes the node number in the next layer, w t ij denotes the weight at time t, and η denotes the learning rate.

Design of DNN Model
As already mentioned, two DNN models were designed in this study to independently predict morning and afternoon ore productions owing to variations in the truck-haulage operation characteristics during different periods of the day. Figure 5 depicts the structure of a DNN model comprising a single input layer, l hidden layers, and one output layer. The input and output layers comprise 19 nodes and a single node, respectively. The hidden layer was designed to comprise m nodes in all l layers.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 6 of 26 Figure 5. Proposed DNN-model structure to predict ore production during morning and afternoon haulage-operation periods.
In general, truck-haulage systems simulation algorithms analyze how often trucks transport ore from the loading zone to the dumping zone during unit working time to predict ore production [6]. For this purpose, simulation factors (daily working time, number of dispatched trucks, truck loading capacity, etc.) and temporal factors (truck-haulage operation time) are entered into the simulation algorithm [38]. Truck-haulage operation time consists of discrete events, such as ore loading, traveling, ore dumping, spotting, and waiting, and can be defined by the truck cycle time theory proposed by Subolesk [69].
In this study, the input nodes in the input layer were set to the input factors of the truck-haulage system simulation that were modified to fit the system of the study area. The input nodes in the input layer can be divided into truck-haulage operating conditions and haulage-operation time. The output-layer node was either set to ore production during the morning or afternoon haulageoperation period, conditionally. Table 1 lists the different input nodes based on their type and measurement unit. To control the scale of the input variables, operation times were expressed in the relative sense. For example, if morning operation starts at 8:30 a.m., the corresponding start time was set to 0 min. Thus, if the morning operation ends at 12:00 p.m., the corresponding end time equals 210 min. The same reasoning applies to afternoon start and end times. The interval between operation times refers to the difference between relative operation start and end times. The number of trucks dispatched by loading capacity equals the number of trucks observed at the operation start time. The average travel time of a truck equals the time required by a truck to travel to the loading point (from the dumping zone), perform ore loading, and return to the dumping zone. In general, truck-haulage systems simulation algorithms analyze how often trucks transport ore from the loading zone to the dumping zone during unit working time to predict ore production [6]. For this purpose, simulation factors (daily working time, number of dispatched trucks, truck loading capacity, etc.) and temporal factors (truck-haulage operation time) are entered into the simulation algorithm [38]. Truck-haulage operation time consists of discrete events, such as ore loading, traveling, ore dumping, spotting, and waiting, and can be defined by the truck cycle time theory proposed by Subolesk [69]. In the truck cycle time theory, truck cycle time (TCT) can be expressed as the sum of spotting time at the loader (STL), loading time (LT), travel time of the loaded truck (TL), stopping time at the dumping zone (STD), dumping time (DT), travel time of the empty truck (TE), and average delay time (AD), as defined by Equation (3).
In this study, the input nodes in the input layer were set to the input factors of the truck-haulage system simulation that were modified to fit the system of the study area. The input nodes in the input layer can be divided into truck-haulage operating conditions and haulage-operation time. The output-layer node was either set to ore production during the morning or afternoon haulage-operation period, conditionally. Table 1 lists the different input nodes based on their type and measurement unit. To control the scale of the input variables, operation times were expressed in the relative sense. For example, if morning operation starts at 8:30 a.m., the corresponding start time was set to 0 min. Thus, if the morning operation ends at 12:00 p.m., the corresponding end time equals 210 min. The same reasoning applies to afternoon start and end times. The interval between operation times refers to the difference between relative operation start and end times. The number of trucks dispatched by loading capacity equals the number of trucks observed at the operation start time. The average travel time of a truck equals the time required by a truck to travel to the loading point (from the dumping zone), perform ore loading, and return to the dumping zone. Loading capacity of 45 tons trucks Tons 8 Loading capacity of 60 tons trucks Tons 9 Loading capacity of 84 tons trucks Tons 10 Dumping zone A utilization of 45 tons trucks Ratio 11 Dumping zone B utilization of 45 tons trucks Ratio 12 Dumping zone A utilization of 60 tons trucks Ratio 13 Dumping zone B utilization of 60 tons trucks Ratio 14 Dumping zone A utilization of 84 tons trucks Ratio 15 Dumping zone B utilization of 84 tons trucks Ratio 16 Average stay time of trucks at dumping zone A Min 17 Average stay time of trucks at dumping zone B Min 18 Average travel time of trucks using dumping zone A Min 19 Average travel time of trucks using dumping zone B Min

Data Preparation for DNN Model
Training data for the proposed DNN model were generated by processing approximately 35.7 million data packets obtained from the mine-safety-management system. The processing was performed over two months-December 2018 and January 2019. The results of packet-data analysis revealed that the morning operation progressed from 8:30 a.m. (relative time: 0 min) to 12:00 p.m. (relative time: 210 min), whereas afternoon operations lasted from 1:00 p.m. (relative time: 0 min) to 4:30 p.m. (relative time: 210 min). The data-generation process employed in this study is explained as follows (refer to Figure 6):

1.
First, all incident packet data were preprocessed. Subsequently, packet data sent from the third wireless AP (AP3) were extracted along with truck-tag recognition data. All hexadecimal values were converted to decimal.

2.
Packet data recorded during valid operation time intervals were subsequently sampled. Operation time intervals were set to 30, 60, 90, 120, 150, 180, and 210 min with incremental shifts of 2-min each to consider probable cases covering the period from 0 to 210 min. For example, if the operation time interval equaled 30 min, 91 probable cases, such as 0-30, 2-32, 4-34, and 180-210 min, can be considered during the morning session.

3.
Extracted packet data were classified according to the truck-tag ID, and dumping-zone utilization of the trucks was calculated. The dumping-zone utilization of zone A by 45-ton trucks was calculated as the ratio of the number of dumping-zone-A visits to the sum of dumping-zone-A and dumping-zone-B visits (refer to Equation (4)). If trucks remained inside a dumping zone for more than 1 min (can be calculated by comparing the latitude and longitude coordinates of a truck recorded as packet data with dumping-zone coordinates), the number of dumping-zone visits made by a truck was increased by one. In Equation (4), U(DA) 45 and U(DB) 45 denote dumping zone A and B utilizations, respectively, by the 45-ton trucks, whereas NV(DA) 45 and NV(DB) 45 denote the number of visits made by 45-ton trucks to dumping zones A and B, respectively. 4.
The average stay and travel times of trucks inside and outside the dumping zone, respectively, were also calculated. Equation (5)  A. In this equation, TTT(DA) 45 , TTT(DA) 60 , and TTT(DA) 84 represent the sum of travel times corresponding to the 45-, 60-, and 84-ton trucks, respectively, using dumping zone A. 5.
The amount of ore produced during a given operation time was calculated by multiplying the loading capacity of each truck with the number of visits it made to the dumping zone (refer to Equation (7)).

6.
Finally, all calculated values were saved in a training-data format, the next operation-time interval was set, and the above process was repeated 2-5 times.
U(DA) 45  , and TTT(DA) represent the sum of travel times corresponding to the 45-, 60-, and 84-ton trucks, respectively, using dumping zone A. 5. The amount of ore produced during a given operation time was calculated by multiplying the loading capacity of each truck with the number of visits it made to the dumping zone (refer to Equation (7)). 6. Finally, all calculated values were saved in a training-data format, the next operation-time interval was set, and the above process was repeated 2-5 times.

Statistical Analysis of Training Data
Using the above-described process, 16,217 training datasets were generated for the proposed DNN model for prediction of morning ore production, and corresponding datasets concerning afternoon ore production equaled 16,005. Figure 7 depicts the distribution of morning and afternoon ore productions. As can be observed, the average morning ore production equaled 2987 tons, whereas the corresponding afternoon production equaled 2960 tons. Table 2 lists average values of the number of dispatched trucks, dumping zone utilizations, stay times, and travel times observed during morning and afternoon operations.

Statistical Analysis of Training Data
Using the above-described process, 16,217 training datasets were generated for the proposed DNN model for prediction of morning ore production, and corresponding datasets concerning afternoon ore production equaled 16,005. Figure 7 depicts the distribution of morning and afternoon ore productions. As can be observed, the average morning ore production equaled 2987 tons, whereas the corresponding afternoon production equaled 2960 tons. Table 2 lists average values of the number of dispatched trucks, dumping zone utilizations, stay times, and travel times observed during morning and afternoon operations.    The value of the Pearson correlation coefficient between input variables and ore production was calculated, and the correlation between them analyzed using training data (refer to Table 3). Equation (8) provides the relation for evaluating the Pearson correlation coefficient (r). In this equation, n denotes the number of samples; x i and y i denote values of the ith sample; x denotes the mean of x samples; and y denotes the mean of y samples. Values of the Pearson correlation coefficient between ore production and operation-time interval were observed to be 0.77 and 0.81 for the morning and afternoon operation times, respectively. The interval between operation times demonstrated the highest positive correlation with ore production, regardless of the operation time. Additionally, relative operation-end times and number of dispatched 60-and 80-ton trucks demonstrated a positive correlation with ore production, whereas relative operation-start time revealed a negative correlation with ore production.

Experimental Setup for DNN Model Training
To optimize the DNN models to predict the morning and afternoon ore productions efficiently, the DNN models were trained with different settings for the number of hidden layers and the number of nodes within them. Table 4 describes the number of hidden layers and the number of hidden layer nodes set for DNN model training by experience. The number of hidden layers was varied from three to five, and the number of hidden layer nodes varied from 30 to 50 in increments of 10. A 5-fold cross validation was conducted five times to evaluate prediction performance of DNN models with different hidden layer configurations. The 5-fold cross validation randomly shuffles the data set and divides the data set into five sets. One set is used for validation set and the other four sets are used for training set. Five experimental validations are executed at one round. After one round of cross-validation, random shuffling process of the whole data set is repeated again. That is, totally 25 experimental runs are used for the performance evaluation on the DNN model.
To evaluate the prediction error of the DNN models, the values of the coefficient of determination (R 2 ) and mean absolute percentage error (MAPE) between predicted and observed values were evaluated. Equation (9) defines the evaluation of the coefficient of determination, wherein n denotes the number of datasets; y i denotes the observed value of the ith training data;ŷ i denotes the predicted value of the ith training data; and y denotes the mean observed value. Equation (10) describes MAPE evaluation.
T-significance test was performed to determine if the performance differences of DNN models for validation data are statistically significant. T-test calculates the p-value considering the mean and variance of MAPE differences of two DNN models and determines if null hypothesis is rejected or accepted. Null hypothesis means there is no predictive performance difference between two DNN models. If p-value is equal or less than significance level, null hypothesis is rejected. Equation (11) refers to the variance of MAPE differences of two DNN models, wherein k denotes the number of folds, r denotes the number of total rounds of cross-validation, x denotes MAPE differences for validation data of two DNN models, m denotes the mean of MAPE differences of two DNN models. Equation (12) describes corrected repeated k-fold cross-validation t-test [70]. Here, n 1 denotes the number of training data, n 2 denotes the number of validation data. T-value is converted into the p-value through the student's t-distribution considering the degree of freedom. Significance level was set to 0.05, and two-tailed test was conducted.σ Park, CA, USA)-an open-source deep learning module. The structure of the proposed DNN model was first configured, and subsequently, prediction error evaluation was performed with 5 × 5-fold cross validation. After choosing the best hidden layer configuration of DNN model, final DNN model training was performed using whole training data. Optimal DNN model demonstrating the lowest prediction error was considered for subsequent utilization. One thousand iterations for DNN model training was performed, and ReLU was used as the activation function. The Adam optimizer was considered as a means to employ the gradient-descent method.

Inference Using DNN Model
Estimates of morning and afternoon ore-production amounts in February 2019 were obtained using optimized DNN models developed and trained as described above. The first week of February 2019 was a week of national holidays in the Republic of Korea, thereby resulting in reduced ore production. The daily average ore production in the first week of February was observed to have been reduced by approximately 10,000 tons compared to the previous month. The utility of the proposed DNN model with regard to accurate ore-production predictions at the identified mine location was assessed by applying the same to perform the said predictions during morning and afternoon operation times over five days: 9, 11, 12, 13, and 14, February 2019. Ore production in the morning accounted for the amount of ore mined in 210 min between 8:30 a.m. and 12:00 p.m. Accordingly, afternoon ore production accounted for mining operations performed between 1:00 p.m. and 4:30 p.m. Input data for DNN models were generated by processing approximately 280,000 packet-data samples recorded over the above-mentioned five days. Table 5 lists variable values provided as input to the two DNN models. Based on these input data, it was revealed that the number of 45-, 60-, and 84-ton trucks dispatched for operation on the morning of 9 February amounted to four, two, and four, respectively, and that all these trucks used dumping zone B. Additionally, it was revealed that the average stay time of the trucks in dumping zone B, the average travel time, and calculated ore-production were 2.83 min, 10.6 min, and 6819 tons, respectively. To account for the error-increase rate between the calculated and observed values, the inference error was evaluated by the percentage error (PE), as defined by Equation (13):

Inference Using DNN Model
Estimates of morning and afternoon ore-production amounts in February 2019 were obtained using optimized DNN models developed and trained as described above. The first week of February 2019 was a week of national holidays in the Republic of Korea, thereby resulting in reduced ore production. The daily average ore production in the first week of February was observed to have been reduced by approximately 10,000 tons compared to the previous month. The utility of the proposed DNN model with regard to accurate ore-production predictions at the identified mine location was assessed by applying the same to perform the said predictions during morning and afternoon operation times over five days: 9, 11, 12, 13, and 14, February 2019. Ore production in the morning accounted for the amount of ore mined in 210 min between 8:30 a.m. and 12:00 p.m. Accordingly, afternoon ore production accounted for mining operations performed between 1:00 p.m. and 4:30 p.m. Input data for DNN models were generated by processing approximately 280,000 packet-data samples recorded over the above-mentioned five days. Table 5 lists variable values provided as input to the two DNN models. Based on these input data, it was revealed that the number of 45-, 60-, and 84-ton trucks dispatched for operation on the morning of 9 February amounted to four, two, and four, respectively, and that all these trucks used dumping zone B. Additionally, it was revealed that the average stay time of the trucks in dumping zone B, the average travel time, and calculated ore-production were 2.83 min, 10.6 min, and 6819 tons, respectively. To account for the error-increase rate between the calculated and observed values, the inference error was evaluated by the percentage error (PE), as defined by Equation (13):  Figure 9 depicts trends observed in the calculated average values of R 2 and MAPE, obtained during 5 × 5-fold cross-validation of the morning and afternoon prediction DNN models, in accordance with changes in hidden-layer conditions. As can be seen, the average value of R 2 for validation data is equal to 0.99 with increase in the number of hidden-layer nodes, irrespective of the number of hidden layers. MAPE values for validation data decrease with increase in the number of hidden-layer nodes, irrespective of the number of hidden layers. The highest MAPE value for the validation data was 6.08% when the number of hidden layers was three and the number of corresponding nodes was 30, whereas the lowest MAPE value of 4.78% was observed for four hidden layers and 50 hidden-layer nodes.

Experimental Evaluation of Trained DNN Models
Appl. Sci. 2020, 10, x FOR PEER REVIEW 13 of 26 Figure 9 depicts trends observed in the calculated average values of R 2 and MAPE, obtained during 5 × 5-fold cross-validation of the morning and afternoon prediction DNN models, in accordance with changes in hidden-layer conditions. As can be seen, the average value of R 2 for validation data is equal to 0.99 with increase in the number of hidden-layer nodes, irrespective of the number of hidden layers. MAPE values for validation data decrease with increase in the number of hidden-layer nodes, irrespective of the number of hidden layers. The highest MAPE value for the validation data was 6.08% when the number of hidden layers was three and the number of corresponding nodes was 30, whereas the lowest MAPE value of 4.78% was observed for four hidden layers and 50 hidden-layer nodes. Figure 9. Results of 5 × 5-fold cross validation for predicting ore production during morning. Table 6 describes the mean and standard deviation of MAPE of DNN models for validation data, in accordance with the number of rounds of 5-fold cross validation. The lowest mean and standard deviation of MAPE were analyzed in (4, 50), (5,50), and (4, 40) of hidden layer configuration conditions. Table 7 lists the results of t-significance test for DNN models. In the table, 'p < 0.05' indicates that the difference of MAPE of two models was statistically significant. In contrast, 'NS' means the predictive performance of two models were both equal to each other. The results of the tsignificance test for MAPE of DNN models shows that DNN models with (4, 40), (5,50), and (4, 40) of hidden layer configuration follow with no statistically significant differences in the predictive performance. Figure 9. Results of 5 × 5-fold cross validation for predicting ore production during morning. Table 6 describes the mean and standard deviation of MAPE of DNN models for validation data, in accordance with the number of rounds of 5-fold cross validation. The lowest mean and standard deviation of MAPE were analyzed in (4, 50), (5,50), and (4, 40) of hidden layer configuration conditions. Table 7 lists the results of t-significance test for DNN models. In the table, 'p < 0.05' indicates that the difference of MAPE of two models was statistically significant. In contrast, 'NS' means the predictive performance of two models were both equal to each other. The results of the t-significance test for MAPE of DNN models shows that DNN models with (4, 40), (5,50), and (4, 40) of hidden layer configuration follow with no statistically significant differences in the predictive performance. Table 6. Statistical analysis results of mean absolute percentage error (MAPE) of DNN models for predicting morning ore production according to the number of rounds of 5-fold cross validation. In this study, the DNN model corresponding to the MAPE value for validation data was considered to realize optimum performance in predicting morning ore production. The optimum model comprised four hidden layers and 50 hidden-layer nodes. Similar to Figure 9, Figure 10 depicts the trends for R 2 and MAPE values corresponding to the DNN model for prediction of afternoon ore production. The average value of R 2 was equal to 0.99, regardless of the number of hidden layers and hidden-layer nodes. The average value of MAPE values, in general, decrease with increase in hidden-layer nodes. The highest MAPE value of 7.00% corresponds to three hidden layers and 30 hidden-layer nodes, whereas its lowest value of 5.22% corresponds to five hidden layers and 50 hidden-layer nodes. Appl. Sci. 2020, 10, x FOR PEER REVIEW 15 of 26 Figure 10. Results of 5 × 5-fold cross validation for predicting ore production during afternoon. Table 8 shows statistical analysis results of MAPE of DNN models for prediction afternoon ore production, in accordance with the number of rounds of 5-fold cross validation. It was observed that the prediction error of afternoon ore production was lowest when the number of hidden layers and number of corresponding nodes were (5,50), (4, 50), and (3, 50), respectively. As observed in Table 9, there was no statistically significant differences in predictive performances among the three DNN models. Table 8. Statistical analysis results of MAPE of DNN models for predicting afternoon ore production according to the number of rounds of 5-fold cross validation.  Figure 10. Results of 5 × 5-fold cross validation for predicting ore production during afternoon. Table 8 shows statistical analysis results of MAPE of DNN models for prediction afternoon ore production, in accordance with the number of rounds of 5-fold cross validation. It was observed that the prediction error of afternoon ore production was lowest when the number of hidden layers and number of corresponding nodes were (5,50), (4, 50), and (3, 50), respectively. As observed in Table 9, there was no statistically significant differences in predictive performances among the three DNN models. Table 8. Statistical analysis results of MAPE of DNN models for predicting afternoon ore production according to the number of rounds of 5-fold cross validation.  In this study, the optimum DNN model was chosen as the one corresponding to the lowest mean and standard deviation of MAPE value for the validation data (i.e., four hidden layers and 50 hidden-layer nodes) for accurate prediction of afternoon ore production.

Inference Drawn Using Optimum DNN Models
Using the optimum morning and afternoon DNN models, ore-production estimates were obtained for five days in the second week of February 2019; the results are illustrated in Figure 11. Table 10 lists prediction-error values for ore production by date. In regards to morning ore production, compared to actual values, ore-production predictions were found to be overestimated by 14.96%, 9.11%, 7.41%, and 4.10% on 9, 11, 12, and 14 February, 2019, respectively, whereas predictions underestimated by 21.42% were obtained on 13 February, 2019, respectively. Morning ore-production predictions for the five days demonstrated an average absolute error of 11.40%. In regards to afternoon ore production, compared to actual values, ore-production predictions underestimated by 22.92%, 4.41%, 12.23%, 2.68%, and 2.12% were obtained on 9, 11, 12, 13, and 14 February, 2019, respectively. The average absolute error in afternoon ore production for the five days equaled 8.87%, which is lower than corresponding to morning ore production by 2.53%. A comparison between the sum of predicted morning and afternoon ore-production value and the actual daily ore production demonstrated the two values to be nearly identical, and the resulting MAPE equaled 4.17%.
The main reason behind the observed difference in the actual ore production and that predicted by DNN models is that haulage operating conditions, such as the number of trucks dispatched, their dumping-zone stay times, and average travel time, vary in real time during the morning and afternoon hours of operation. Ore production can vary significantly if the number of trucks dispatched during operation is reduced, or a truck is assigned to loading points far away from dumping zones. However, in this study, since ore-production predictions were made by exclusively considering operating conditions at operation-start times, inference errors for validation data were observed to exceed prediction errors. February, 2019, respectively. The average absolute error in afternoon ore production for the five days equaled 8.87%, which is lower than corresponding to morning ore production by 2.53%. A comparison between the sum of predicted morning and afternoon ore-production value and the actual daily ore production demonstrated the two values to be nearly identical, and the resulting MAPE equaled 4.17%.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 17 of 26 Figure 11. Inference results obtained using optimum DNN models: (a) Ore-production prediction during morning operation (tons); (b) ore-production prediction during afternoon operation (tons); (c) total ore production in a day (tons). The main reason behind the observed difference in the actual ore production and that predicted by DNN models is that haulage operating conditions, such as the number of trucks dispatched, their dumping-zone stay times, and average travel time, vary in real time during the morning and afternoon hours of operation. Ore production can vary significantly if the number of trucks dispatched during operation is reduced, or a truck is assigned to loading points far away from dumping zones. However, in this study, since ore-production predictions were made by exclusively considering operating conditions at operation-start times, inference errors for validation data were observed to exceed prediction errors.

Real-Time Ore-Production Prediction Using Optimum DNN Models
In this study, ore-production predictions were predicted every 30 min during the morning and afternoon operation times on 9, 11, and 14 February, 2019 using optimum DNN models to account for changes in haulage-system conditions that occur in real time. Seven ore-production predictions were performed at start times . Inference results obtained using optimum DNN models: (a) Ore-production prediction during morning operation (tons); (b) ore-production prediction during afternoon operation (tons); (c) total ore production in a day (tons).

Real-Time Ore-Production Prediction Using Optimum DNN Models
In this study, ore-production predictions were predicted every 30 min during the morning and afternoon operation times on 9, 11, and 14 February, 2019 using optimum DNN models to account for changes in haulage-system conditions that occur in real time. Seven ore-production predictions were performed at start times of 8:  Figure 12 depicts prediction results obtained for 9 February, 2019. Predicted ore production values during the morning and afternoon operation times along with cumulative ore production up to the end of the last prediction are shown in the figure. Owing to a gradual increase in average stay time in dumping zone B between 8:30 a.m. and 10:00 a.m., corresponding ore production demonstrated a gradual decrease. At 10:30 a.m., one 45-ton truck was removed from dispatch; this illustrates the lowest ore production at the said time. However, the average travel time of trucks decreased from 11:00 a.m. onwards, and consequently, ore production tended to increase again. At 1:30 p.m., one 45-ton and 60-ton truck each were removed from dispatch, thereby resulting in a significant decrease in ore production. However, given the decrease in average truck-travel time from 2:00 p.m. onwards, ore production was observed to increase once again.
illustrates the lowest ore production at the said time. However, the average travel time of trucks decreased from 11:00 a.m. onwards, and consequently, ore production tended to increase again. At 1:30 p.m., one 45-ton and 60-ton truck each were removed from dispatch, thereby resulting in a significant decrease in ore production. However, given the decrease in average truck-travel time from 2:00 p.m. onwards, ore production was observed to increase once again.  was less compared to the corresponding predicted value (predicted at 8:30 a.m.) owing to removal of one 84-ton truck from dispatch at 9:00 a.m. Similarly, the accumulated actual ore production until 4:30 p.m. exceeded the afternoon ore production predicted at 1:00 p.m. because some trucks were assigned to loading points close to dumping zones, thereby resulting in shorter average travel times and more-than-expected ore production.   was less compared to the corresponding predicted value (predicted at 8:30 a.m.) owing to removal of one 84-ton truck from dispatch at 9:00 a.m. Similarly, the accumulated actual ore production until 4:30 p.m. exceeded the afternoon ore production predicted at 1:00 p.m. because some trucks were assigned to loading points close to dumping zones, thereby resulting in shorter average travel times and more-than-expected ore production.
illustrates the lowest ore production at the said time. However, the average travel time of trucks decreased from 11:00 a.m. onwards, and consequently, ore production tended to increase again. At 1:30 p.m., one 45-ton and 60-ton truck each were removed from dispatch, thereby resulting in a significant decrease in ore production. However, given the decrease in average truck-travel time from 2:00 p.m. onwards, ore production was observed to increase once again.  was less compared to the corresponding predicted value (predicted at 8:30 a.m.) owing to removal of one 84-ton truck from dispatch at 9:00 a.m. Similarly, the accumulated actual ore production until 4:30 p.m. exceeded the afternoon ore production predicted at 1:00 p.m. because some trucks were assigned to loading points close to dumping zones, thereby resulting in shorter average travel times and more-than-expected ore production.   :00 a.m. compared to that at 8:30 a.m. This led to prediction of increased ore production. At 11:30 a.m., both the average travel and stay times of trucks decreased, thereby resulting in prediction of even higher ore production. In contrast, during afternoon haulage operations, the average stay time of trucks increased at both dumping zones A and B, and therefore, the DNN model predicted a gradual decrease in ore production. trucks decreased at 9:00 a.m. compared to that at 8:30 a.m. This led to prediction of increased ore production. At 11:30 a.m., both the average travel and stay times of trucks decreased, thereby resulting in prediction of even higher ore production. In contrast, during afternoon haulage operations, the average stay time of trucks increased at both dumping zones A and B, and therefore, the DNN model predicted a gradual decrease in ore production. Performing multiple ore-production predictions per day affords several advantages over performing only a single prediction in the morning or afternoon at the start of the haulage operation. Haulage operating conditions prevalent at the start of operation may change as the operation progresses. On 9 February, a difference was observed between the predicted ore production at the start of the haulage operation and cumulative ore production at the end of the said operation owing to changes in the number of the trucks dispatched during operation hours. If we analyze haulagesystem conditions in real time by obtaining haulage data from mining sites in accordance with a specific operation period and provide them as input to DNN models, accurate ore-production predictions can be realized. Furthermore, task managers can make intuitive and effective decisions, such as dispatching additional trucks or changing the location and number of loading points to meet daily production targets by verifying the ore production predicted over a specific period.

Comparison of the DNN and the Multiple Regression Analysis
In order to compare the predictive performances of the DNN model with those of other learning methods, morning ore production were predicted through the multiple regression analysis. The 16,217 training datasets were used for multiple regression analysis model training, and 5 × 5-fold cross validation was performed to determine the degree of multiple regression equation. Table 11 shows cross validation results performed when the degree of multiple regression equation is set to 2. The average R 2 for validation data was 0.91, and the average MAPE was 23.54%. The average and standard deviation of MAPE for validation data was higher than that of DNN model. According to the t-significant test results, the predictive performance differences between the DNN model and multiple regression analysis model were considered statistically significant (t-value: −4.09, p-value < 0.005). Performing multiple ore-production predictions per day affords several advantages over performing only a single prediction in the morning or afternoon at the start of the haulage operation. Haulage operating conditions prevalent at the start of operation may change as the operation progresses. On 9 February, a difference was observed between the predicted ore production at the start of the haulage operation and cumulative ore production at the end of the said operation owing to changes in the number of the trucks dispatched during operation hours. If we analyze haulage-system conditions in real time by obtaining haulage data from mining sites in accordance with a specific operation period and provide them as input to DNN models, accurate ore-production predictions can be realized. Furthermore, task managers can make intuitive and effective decisions, such as dispatching additional trucks or changing the location and number of loading points to meet daily production targets by verifying the ore production predicted over a specific period.

Comparison of the DNN and the Multiple Regression Analysis
In order to compare the predictive performances of the DNN model with those of other learning methods, morning ore production were predicted through the multiple regression analysis. The 16,217 training datasets were used for multiple regression analysis model training, and 5 × 5-fold cross validation was performed to determine the degree of multiple regression equation. Table 11 shows cross validation results performed when the degree of multiple regression equation is set to 2. The average R 2 for validation data was 0.91, and the average MAPE was 23.54%. The average and standard deviation of MAPE for validation data was higher than that of DNN model. According to the t-significant test results, the predictive performance differences between the DNN model and multiple regression analysis model were considered statistically significant (t-value: −4.09, p-value < 0.005). Table 11. Results of accuracy analysis of multiple regression analysis model for predicting morning ore production using training and validation data.

Degree of Multiple Regression Equation Statistics
Round Morning ore production estimates were obtained for five days in the second week of February 2019 through multiple regression analysis. Figure 15 illustrates the results of morning ore production inference, and Table 12 shows inference error between observed and predicted values. MAPE for morning ore production inference over five days was analyzed to be about 16.37%, which was about 4.97% higher than the DNN model inference error. As the results of the inference, it was possible to predict morning ore production more accurately when using the DNN model rather than the multiple regression analysis. Morning ore production estimates were obtained for five days in the second week of February 2019 through multiple regression analysis. Figure 15 illustrates the results of morning ore production inference, and Table 12 shows inference error between observed and predicted values. MAPE for morning ore production inference over five days was analyzed to be about 16.37%, which was about 4.97% higher than the DNN model inference error. As the results of the inference, it was possible to predict morning ore production more accurately when using the DNN model rather than the multiple regression analysis. Figure 15. Ore-production inference during morning operation (tons) obtained through the multiple regression analysis.

Further Study
In recent years, prediction has become popular and is being applied in many fields. Many advanced predictive deep learning models, such as stacked auto encoders (SAE) [71], deep belief networks (DBN) [72], and Deep Boltzmann machine (DBM) [73], have been developed and proven to be effective. In addition, many machine learning algorithms (such as random forest regression [74] and support vector regression (SVR) [75]) also have been used to analyze and predict data properties. Moreover, advanced deep learning algorithms have been developed and compared with conventional algorithms to evaluate performances [76][77][78][79][80][81].
In this study, basic DNN models that were not combined with other advanced algorithms was used to predict morning and afternoon ore productions, because this study is a first attempt to predict Figure 15. Ore-production inference during morning operation (tons) obtained through the multiple regression analysis.

Further Study
In recent years, prediction has become popular and is being applied in many fields. Many advanced predictive deep learning models, such as stacked auto encoders (SAE) [71], deep belief networks (DBN) [72], and Deep Boltzmann machine (DBM) [73], have been developed and proven to be effective. In addition, many machine learning algorithms (such as random forest regression [74] and support vector regression (SVR) [75]) also have been used to analyze and predict data properties. Moreover, advanced deep learning algorithms have been developed and compared with conventional algorithms to evaluate performances [76][77][78][79][80][81].
In this study, basic DNN models that were not combined with other advanced algorithms was used to predict morning and afternoon ore productions, because this study is a first attempt to predict ore production using equipment-tracking data obtained from open-pit mines. It was possible to confirm that DNN models can predict the ore production of truck-haulage systems with low MAPE. However, it is difficult to affirm that basic DNN is particularly suitable and the best model to predict morning and afternoon ore productions. Therefore, it will be necessary to conduct performance comparisons between DNN models and other predictive models to explore the best case for ore production prediction.
In order to estimate the accurate predictive performance of the DNN model, bias error and variance error should be considered. Bias and variance error concepts were introduced by structure risk minimization (SRM), which is suggested by Vapnik [82]. Bias error and variance error indicate the predictive sensitivity caused by the parameters of the learning algorithm and the training dataset size, respectively. High bias causes overfitting of the DNN model on the training data, on the other hand, high variance causes underfitting.
Haykin [83] compared predictive performances of the DNN model considering the learning parameters (i.e., the learning rate, the momentum constant, and the number of training epochs) and the training dataset size to optimize the DNN model configuration. However, in this study, fixed training dataset size and learning parameters were used for optimizing hidden layer structure configurations. Therefore, further studies are required to analyze the predictive sensitivity of DNN models on various training dataset sizes and learning parameter conditions to optimize hidden layer structure configurations.

Conclusions
This paper proposed using DNN models to predict ore production by truck-haulage systems in open-pit mines. Training data for two DNN models (one each for morning and afternoon ore-production prediction) were generated by processing packet data obtained from a preselected mining site over a two-month period. Additionally, the DNN models were optimized by varying the number of hidden layers and their corresponding nodes. The results obtained in this study established that MAPE for morning and afternoon ore-production predictions equaled 11.40% and 8.87%, respectively, and the error between the actual and predicted ore productions in a given day was of the order of 4.17%.
This study aids analysis of truck-haulage operating conditions and corresponding operation times by using a large packet dataset collected over a two-month period. Additionally, the study aids comprehension of truck-haulage-system characteristics along with discrete haulage-operation sequences and supports prediction of ore production through training of DNN-based deep learning models without the need to develop additional algorithms. Therefore, it is expected that the proposed ore-production prediction method will be able to eliminate problems encountered by conventional truck-haulage-system simulation methods based on complex algorithms.
Mines are dynamic systems, and working environments therein change very frequently. If DNN models can be sufficiently trained using data collected over an extended period, most dynamic haulage-operation conditions can be accounted for, thereby resulting in highly accurate predictions. Additionally, DNN models must be frequently updated with the latest training data to account for the latest changes in haulage-operation conditions. There exists a need to undertake further research in this direction to determine the optimum period for collecting training data, as well as the intervals over which DNN models need to be updated.