Deep Q-Network for Optimal Decision for Top-Coal Caving

Abstract

In top-coal caving, the window control of hydraulic support is a key issue to the perfect economic benefit. The window is driven by the electro-hydraulic control system whose command is produced by the control model and the corresponding algorithm. However, the model of the window’s control is hard to establish, and the optimal policy of window action is impossible to calculate. This paper studies the issue theoretically and, based on the 3D simulation platform, proposes a deep reinforcement learning method to regulate the window action for top-coal caving. Then, the window control of top-coal caving is considered as the Markov decision process, for which the deep Q-network method of reinforcement learning is employed to regulate the window’s action effectively. In the deep Q-network, the reward of each step is set as the control criterion of the window action, and a four-layer fully connected neural network is used to approximate the optimal Q-value to get the optimal action of the window. The 3D simulation experiments validated the effectiveness of the proposed method that the reward of top-coal caving could increase to get a better economic benefit.

1. Introduction

Coal is one of the most important energy sources in the world [1]. Even though its consumption has decreased in the past years, coal will persist in the domination of primary energy for the next several decades [2,3,4]. Improving the coal mining technology sustainably to alleviate the damage to the environment is the preferred choice for countries that lack oil sources while rich in coal [5,6]. Underground mining the thick coal seam is the most economical and environmentally-friendly mode currently [7,8]. In the longwall workspace, the top-coal caving is the most effective technology to exploit the coal seam whose thickness is greater than 4 m [9,10,11]. Especially in China, more than 40% coal is in the thick coal seam; hence, top-coal caving is the development direction in the future years [12].

By this method, the coal seam is first cut by coal cutter at, then the rest of the top-coal falls by the combined action of gravity and pressure, and the falling coal is captured by pulling back the tail canopy of the hydraulic support. Therefore, the hundreds of hydraulic supports in the workspace are the key devices for the security and top-coal capturing, and controlling the tail canopy of hydraulic supports optimally is the critical issue for top-coal caving [13,14].

The tail-beam, named “window” for capturing the top-coal, opens to capture the falling coal and closes to avoid the falling rocks as much as possible [15]. The traditional control method of the window is manual manipulation based on the experience, and the most important criterion is to close the window if the rock emerges.

In recent years, top-coal caving automatically and intelligently is an important research topic [16] in which the optimal control of windows’ action is the key issue for the regulation of tail-beam. That means there should be an optimal control algorithm to regulate the electro-hydraulic control system that drives the hydraulic supports. In classical control theory, the optimal control algorithm is designed based on the model of the controlled plant. Hence, the model of the top-coal caving process should be established first. Unfortunately, the process is so complex that the dynamic equations about the top-coal caving are hard or even impossible to obtain. This is one of the key issues for automating the windows actions.

For this reason, at present, the research of top-coal caving focuses on optimizing the process technology and most of it usd 2D simulation based on discrete element method (DEM) [17]. In [18,19], the particle flow code is used to research the effect of top-coal thickness on top-coal recovery ratio under the caving mining technique. Wang et al. [20] employed the DEM to validate the boundary-body-ratio of top-coal caving. In [21], the relationship between mining heights and shield resistance in the longwall panel is researched by DME simulation. In [22], EDM is used to analyze the top-coal recovery.

Furthermore, to obtain a better performance, several different kinds of technological processes of top-coal caving are proposed to improve the coal recovery. The typical works are introduced in [15,17,23,24]. The core idea of the method is to control the boundary between the rock layer and the coal layer by operating the windows’ action in a multi-round. During the top-coal caving, the boundary should be as straight as possible so that the coal and rock can be separated easily by setting the open time of the windows [23].

The aforementioned references show that at present the research about the optimal strategy of top-coal caving focuses on the technological process. However, as a control system, the input of the windows operation decision is just the open time of the window, and the decision mechanism is “close the window if the rock emerges”. It is well known that the top-coal movement depends on a complex dynamic process [25,26] in which the geologic structure [14,27], roof pressure [10], hydraulic support state [28], etc. are coefficients to the performance of top-coal caving [29,30,31]. That means the huge coal may be on the top of some rock, especially if there is a rock layer in the coal seam. Hence, if we hope to capture the top-coal accurately, more information should be introduced into the decision system, and an optimal decision mechanism should be designed based on that information.

The window executing action over time is a typical Markov process [32]. The action decision is regulated by a given mechanism based on the environment state, while the Markov property of the action decision processes holds that the current action only depends on the last process. Reinforcement learning (RL) [33] is an effective methodology to deal with the optimal decision of the Markov process. It learns the regulation mechanism by the reward of the agent iteratively. Especially, the model-free method of RL named Q-learning [34,35] can learn the action–value function without needing a control model; then, the optimal action can be calculated based on the action–value function. Hence, the Q-learning method can avoid establishing the control model in the classical control theory.

Nevertheless, in top-coal caving, the state of the environment is a continuous parameter, such as the thickness of the coal seam, the pressure of the hydraulic support, and the ratio of the rock in the coal. Hence, the action–value function of the Q-learning is difficult to be depicted by a simple state–action table and hard to learn the value.

The rapid rise of artificial intelligence [36], especially the deep neural network [37,38,39], provides another powerful instrument to approximate the nonlinear system with a high dimension. As a matter of course, the deep neural network is employed to approximate the action–value function of Q-learning. It is a new branch of interdisciplinary research called deep reinforcement learning [32,40,41,42,43] and obtains great successes in autonomous driving [44], robot control [45], multi-task, multi-agent [46,47], etc.

In this paper, a new 3D DEM simulation test platform based on the open-source framework YADE [48,49,50] is developed to analyze the dynamic process of top-coal caving. To get the optimal decision of windows intelligently based on the simulation platform, this paper, along with our preliminary work [51], introduces more information of windows action during top-coal caving as the state of the control system and employs the deep Q-network method of reinforcement learning to approximate the windows optimal decision. The main contributions of this work include:

(1)

The optimal control of the window’s action of hydraulic support is transformed into a Markov decision process and a new method based on deep Q-network is proposed to regulate the optimal decision of the window’s action. In the method, the state of the environment, the loss function of the optimizer, and the reward of each step are given according to the process of top-coal caving.

(2)

A 3D discrete element method simulation platform is created to analyze the process of top-coal caving based on Yade. Based on the simulation platform, simulation experiments were carried out and the results theoretically validate an available way of applying the intelligent method to top-coal caving.

The rest of the article is organized as follows. In Section 2, the 3D simulation platform is introduced. The optimal decision of top-coal caving by deep reinforcement learning is presented in Section 3. In Section 4, the experiment and result analyses are given. The conclusion is shown in Section 5.

2. Top-Coal Caving 3D Simulation Platform

Most top-coal caving simulations about the optimal decision of windows action is based on DEM in two dimensions, as shown in Figure 1. For this simulation, the boundary between rock and coal can be shown clearly, and the effect of the drawing coal could be analyzed directly.

In most 2D simulations, the window executing the action often ignores the process, i.e., the window changes the state from close to open instantaneously. Hence, even though the regulation mechanism of the windows’ action is only based on “close the window if the rock emerges”, it can get a good result. However, in practice, it takes time to open and close the window. If the rock emerges near the window, the closing process would make the rock fall into the drag conveyor. Furthermore, 2D simulation fails to reflect the real scenario, especially when the particles are lying on a plane that the movement of the coal on shield-beam cannot depict, and the real boundary between coal and rock is not a line but a plane. Hence, the method to control the boundary should expand to adjust the flatness of the plane.

Yade is a well known open-source framework driven by python for DEM [49,52] in the Linux operation system. It is flexible to integrate the control algorithm of the window action with the complex calculation. In this paper, we focus on the windows optimal control from a control system perspective; hence, we only demonstrate five windows for the process of top-coal caving. The three scenarios are shown in Figure 2.

There are five hydraulic supports simplified to the top-beam, shield-beam, and tail-beam. The tail-beam can rotate around the bottom of shield-beam with the given speed, and the scope of swaying is restricted by lower and upper bounds. The parameters of the simulation platform are shown in

Table 1. Parameters of 3D top-coal caving simulation platform.

${\mathit{w}}_{\mathit{sp}}$	${\mathit{w}}_{\mathit{hy}}$	${\mathit{h}}_{\mathit{hy}}$	${\mathit{l}}_{\mathit{sh}}$	${\mathit{l}}_{\mathit{ta}}$	${\mathit{\theta }}_{\mathit{s}}$	${\mathit{\theta }}_{\mathit{u}}$	${\mathit{\theta }}_{\mathit{l}}$
6.8 m	1.5 m	3.8 m	3 m	2 m	50°	15°	45°

.

In Table 1, $w_{sp}$ is the width of workspace, $w_{hy}$ is the width of hydraulic support, $h_{hy}$ is the height of hydraulic support, $l_{sh}$ is the length of shield-beam, $l_{ta}$ is the length of tail-beam, ${\theta }_s$ is the angle of between shield-beam and top beam, ${\theta }_u$ is the upper angle between tail-beam and shield-beam, and ${\theta }_l$ is the lower angle between tail-beam and shield-beam. The height of the rock layer and coal layer can be set as the requirement. The other parameters such as height of the space, the boundary of environment, and the location of each hydraulic support can be calculated according to the geometric relationship.

It should be noted that the parameters of the simulation are set as close to the real situation as possible; however, some parameters such as the height of rock could not set the same as the real case because the huge calculation of DEM may take several days. Meanwhile, this scheme does not hinder the validation of the algorithm.

In this platform, the material properties of rock and coal are shown in

Table 2. Particle material properties of the simulation.

	Young	Cohesion	Density	Friction	Poisson	Tensile	Normal	Shear
	Modulus (Pa)	(Pa)	(kg/m3)	Angle (°)	Rate	Strength (Pa)	Stiffness (Pa)	Stiffness (Pa)
coal	2 × 108	2.06 × 106	1373	44.82	0.29	6.9 × 105	1.5 × 106	1.13 × 106
rock	4 × 108	2.11 × 106	2542	33.6	0.23	1.51 × 106	15.1 × 106	1.13 × 106

. They are set according to Tashan coal mine in China.

3. Optimal Decision of Top-Coal Caving with Deep Q-Network

3.1. Markov Process of Top-Coal Caving

The process of windows executing action is a time series and meets the Markov property. The window action decision is considered as a control system; the input is the environment state and the output is the window action. Hence, the optimal control of window action is a Markov decision process essentially.

Given the state space of the top-coal caving denoted by $\mathbb{S}=\{s_1,s_2,\dots ,s_n\}$, $s_i$ is the state of environment, $i=1,2,\dots ,n$; n is the dimension of state space.For top-coal caving; and $s_i$ is a continuous variable. The window action space is $mathbbA=\{a_1,a_2,\dots ,a_m\}$; $a_j$ is the discrete action value, $j=1,2,\dots ,m$; and m is the dimension of action. The Markov decision process is denoted by $\mathcal{M}=\{\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma \}$, where $\mathcal{S}\in \mathbb{S}$, $\mathcal{A}\in \mathbb{A}$, and $\mathcal{R}$ is the reward of action under the given state. $\mathcal{P}$ is the transition probability distribution of the current state to next state with the action. $\gamma$ is the discount factor, $\gamma \in (0,1)$.

The policy $\pi$ is the function of state s. There are two ways to get policy value. The first is deterministic policy, $a=\pi \left(s\right)$. It indicates that, if the state is s, the action a must be chosen as the deterministic value. The second is probabilistic policy, denoted by $\pi \left(a|s\right)$. That means the probability of executing action a under the state s.

According to dynamic programming [33,53], define the value function $v^{\pi }\left(s\right)$ at point t with the state s under the given policy $\pi$.

$$\begin{array}{c}\hfill v^{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|s_t=s\right)\end{array}$$

(1)

The purpose of reinforcement learning is to find an optimal policy that can get the maximum value of the $v^{\pi }\left(s\right)$ by Equation (1). More formally, the action–value function is defined based on $v^{\pi }\left(s\right)$, i.e., the cumulative rewards at point t if the action is chosen as a. Action–value is formalized as Equation (2).

$$\begin{array}{c}\hfill Q^{\pi }\left(s,a\right)={\mathbb{E}}_{\pi }\left(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|s_t=s,a_t=a\right)\end{array}$$

(2)

By Equations (1) and (2), the optimal policy can be calculated from the optimal action–value function shown as Equation (3).

$$\begin{array}{c}\hfill Q^{*}\left(s,a\right)=\underset{\pi }{max}{\mathbb{E}}_{\pi }\left(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|s_t=s,a_t=a\right)\end{array}$$

(3)

The deterministic policy to get optimal action is shown as Equation (4) if the state is s.

$$\begin{array}{c}\hfill a^{*}=\underset{a}{argmax}{Q}^{*}\left(s,a\right)\end{array}$$

(4)

Hence, to get the optimal action of the window, the action–value function $Q\left(s,a\right)$ should be trained. If the state is a discrete variable, $Q\left(s,a\right)$ could be considered as the table that depicts the relation between state space and action space. The Q-learning algorithm is a mechanism to train $Q\left(s,a\right)$. The $Q\left(s,a\right)$ iteration of Q-learning shown as (5).

$$\begin{array}{c}\hfill Q\left(s,a\right)\leftarrow \left(1-\alpha \right)Q\left(s,a\right)+\alpha \left(R\left(s,a\right)+\gamma \underset{a}{max}Q\left(s^{{}^{\prime }},a\right)\right)\end{array}$$

(5)

where $s^{{}^{\prime }}$ is the next state and $\alpha$ is the learning rate.

However, for the continuous state, it is hard or even impossible to show all the states in a table due to the dimension curse [33]. Fortunately, the deep neural network [36,37,39] provides an effective method to approximate the $Q\left(s,a\right)$, that is deep Q-network(DQN) [32,40,41].

3.2. Deep Q-Network for Top-Coal Caving

The framework of deep Q-network for top-coal caving is based on the work in [54]. Consider the state of top-coal caving system where $s_i$, $i=1,2,\dots ,n$ are continuous variables and the actions $a_j$, $j=1,2,\dots ,d$ are discrete variables. DQN employs a deep neural network to approach $Q\left(s,a\right)$ in Equation (2). The Q-network is formalized as $Q\left(s,a;\theta \right)$, where $\theta$ is the parameter of the neural network and it needs to be trained. Hence, there are two import issues for training $\theta$: the samples to train the neural network and the loss function to indicate the effect of neural network training.

During the top coal caving, the decision system gets the state s from the environment in each step, and the decision mechanism gives an action a to the window. The window executes the action, and then the decision system gets the next state $s^{{}^{\prime }}$ and calculates the reward R by the captured coal and rock. Hence, each step can produce a quaternion $\{s,a,R,s^{{}^{\prime }}\}$ that is a sample to train the Q-network. In the DQN, the experience store and replay are the key technologies to deal with the samples. For the top-coal caving, the samples of the quaternion $\{s,a,R,s^{{}^{\prime }}\}$ produced by each step are stored in the experience dataset, and then a batch of samples are selected randomly for replaying to train the parameters.

The loss function for training Q-network is based on Equation (5). Because dynamic programming [53] deduces the optimal decision from the destination inversely, the term $R\left(s,a\right)+\gamma {max}_{a}Q\left(s^{{}^{\prime }},a\right)$ is the related value of next state; hence, this term is regarded as the target the current $Q\left(s,a\right)$ should approach. That target network is shown in Equation (6):

$$\begin{array}{c}\hfill Q_{target}\left(s,a;{\theta }^{-}\right)=R\left(s,a\right)+\gamma \underset{a}{max}Q\left(s^{{}^{\prime }},a;{\theta }^{-}\right)\end{array}$$

(6)

Hence, the loss function is formalized as Equation (7).

$$\begin{array}{c}\hfill f_{loss}={\left(Q\left(s^{{}^{\prime }},a;\theta \right)-R\left(s,a\right)-\gamma \underset{a}{max}Q\left(s^{{}^{\prime }},a;{\theta }^{-}\right)\right)}^2\end{array}$$

(7)

Training the parameters means making the loss function close to zero. In practice, the target network uses the same architecture of Q-network and the parameter ${\theta }^{-}$ is not trained but gets the Q-network parameter $\theta$ to update periodically. The Q-network is trained during the top-coal caving, at the beginning of the process, in order to cover as much of the state and action as possible. The window action is chosen by $ϵ$ greedy algorithm [33], and the $ϵ$ value is set favorably for exploration. The details of DQN for top-coal caving are shown as Algorithm 1.

Algorithm 1: DQN for top-coal caving.

4. Experiment on Top-Coal Caving

4.1. DQN Model of Top-Coal Caving

In the 3D simulation platform introduced in Section 2, the state space of each window is $\mathbb{S}=\{s_1,s_2\}$, $s_1$ is total particle number of windows checking area, and $s_2$ is the percentage of coal particle in $s_1$. The action space of window is $\mathbb{A}=\{a_1,a_2\}$, and $a_1=0$, and $a_2=1$ mean close and open action, respectively. Hence, we establish the Q-network architecture as a four-layer fully connected neural network, and the parameters are set as

Table 3. Architecture of DQN.

	Input Layer	Hidden Layer 1	Hidden Layer 2	Output Layer
Number of neurons	2	56	128	2
Initial $\theta$	0.5	0.5	0.5	0.1

. The reward R of each step depends on the captured particles formalized Equation (8).

$$\begin{array}{c}\hfill R=n_c\times r_c+n_r\times r_r\end{array}$$

(8)

where $n_c$ and $n_r$ are, respectively, the captured particles of coal and rock, and $r_c$ and $r_r$ are, respectively, the reward of capturing a coal and a rock. In this paper, $r_c=1$ and $r_r=-3$.

To distinguish which falling particle belong to which window, four clapboards are added to the deposition area, as shown in Figure 3a. The area for checking the state is the stereo region for A to B, as shown in Figure 3b. In the simulation, if the particle location is lower than the bottom of the tail-beam, that means it is captured by the window. The reward of each window is calculated by Equation (8).

4.2. Experiment and Result Analysis

Two algorithms were used to control window, the proposed DQN and the “close window if rocks emergency”, which are denoted by “DQN” and “Cmethod”, respectively. In the experiment, the thickness of coal and rock were set as 2 m, because the thickness was not the determining factor to validate the proposed algorithm. The radius of the particles was 0.15 m. The other parameters of training DQN were $batchsize=100$, $N=10000$, $C=300$, $\alpha =0.001$, $\gamma =0.99$, and $ϵ=0.9$. For the DQN algorithm, the first step was to fill the experience dataset, and then train the Q-network. The experiences distribution shown in Figure 4 indicates that the states for the Q-network are closely related to the reward.

The training processing is shown in Figure 5. The parameters of Q-network can convergence to an optimal value obviously; hence, the optimal action–value function of $Q(s,a)$ can be approximated by a deep neural network.

After the parameters of Q-network were trained, 10 tests were carried out to validate the effectiveness of the proposed algorithm. The typical final scenario is shown in Figure 6. It is clear that, along the top of shield-beam, the coal is covered by rock, and the boundary is not a smooth line. Hence, in the next step, the hydraulic support moves forward; if the window control criterion were “close window if rock emergency”, the top-coal would be difficult to capture completely. DQN could avoid the problem since in Figure 6a the coal on the tail-beam is nearly captured completely. Furthermore, Figure 6b shows the boundary of the rock layer on the tail-beam obviously; the performance is the same as with 2D simulation.

The details of the experiment results are shown in

Table 4. Experiment result.

No.	Coal Number		Rock Number		Coal Rate		Rock Rate		Reward
	Cmethod	DQN	Cmethod	DQN	Cmethod	DQN	Cmethod	DQN	Cmethod	DQN
1	566	680	2	21	0.996	0.970	0.004	0.030	560	617
2	683	704	28	28	0.961	0.962	0.039	0.038	599	620
3	680	693	25	24	0.965	0.967	0.035	0.033	605	621
4	655	667	16	15	0.976	0.978	0.024	0.022	607	622
5	672	664	17	10	0.975	0.985	0.025	0.015	621	634
6	649	676	9	13	0.986	0.981	0.014	0.019	622	637
7	646	701	6	21	0.991	0.971	0.009	0.029	628	638
8	673	675	15	12	0.978	0.983	0.022	0.017	628	639
9	682	670	18	10	0.974	0.985	0.026	0.015	628	640
10	681	693	16	8	0.977	0.989	0.023	0.011	633	669
Average	658.7	682.3	15.2	16.2	0.978	0.977	0.022	0.023	613.1	633.7

. For the average value of each index, we can find that the captured coal particles and the reward of DQN are greater than for the Cmethod obviously with a high coal rate and low rock rate. That means the economic effect of top-coal by DQN would better than by Cmethod.

5. Conclusions

This paper aims to get the optimal decision of the window’s action of hydraulic support in top-coal caving by control theory. A DQN method based on the Markov optimal decision process is proposed according to the property of the top-coal caving. The effectiveness of the method was validated by the created 3D simulation platform. The experimental results show that:

(1)

The DQN method can get more coal particles obviously than the classical method with a very small price of increasing the rock rate. In the 10 tests, the average coal particles of the classical method and DQN are 658.7 and 682.3, respectively. Meanwhile, the rock rate of the DQN only rises 0.001.

(2)

The reward of the window’s action by the DQN is better than the classical method. In the 10 tests, the average reward of the DQN is 633.7 while the classical method is 613.1. That means the DQN can produce more benefits than the classical method.

This paper tries to resolve the key problem of top-coal caving by artificial intelligent methodology, and the simulation results validate one of the ways that deep learning is applied to the top-coal caving. That means the optimal decision of windows would be regulated adaptively and intelligently. Even though the simulation result is better than that of the classical method, there are several issues that should be researched deeply in future work.

(1)

The state of the DQN is selected as the total number of the particles and the coal rate. At present, our method is only used in the simulation; one of the obstacles for practice application is that the DQN needs the data of the state. However, the data are difficult to obtain in practice; hence, in future work, we will try to use a deep neural network to approximate the needed data from other geological information.

(2)

The DQN gets the optimal Q-value by training, therefore there should be as much experience as possible, while, in practice, the experience obtained from top-coal caving is not as convenient as simulation. Hence, in future work, the learning mechanism of the DQN will be researched to get a lightweight learning framework based on the state space.

At present, the top-coal caving is mainly applied in China, and the serviceability of this method would be limited to Chinese coal mines. If the above issue is resolved, this method could be applied in practice. It could obviously increase the benefit of the top-coal caving and decrease the number of workers. Because the issue of the optimal decision for the control equipment in the other mineral resources is similar, the proposed method could be useful for other equipment in coal mining, and even for the optimal decision of the equipment in the underground metal mine.