D-P-Transformer: A Distilling and Probsparse Self-Attention Rockburst Prediction Method

Abstract

Rockburst may cause damage to engineering equipment, disrupt construction progress, and endanger human life. To this day, the occurrence of rockburst remains complex and difficult to predict. This study proposes the D-P-Transformer algorithm to address this issue by improving the embedding structure of the Transformer for specific applications to rockburst data. To reduce the computational requirement, sparse self-attention is adopted to replace self-attention. A distilling operation and multiple layer replicas are simultaneously used to enhance the robustness and speed up the algorithm’s process. Taking all relevant rockburst factors into consideration, multiple experiments are conducted on seven large-scale rockburst datasets with different training ratios to verify the reliability of the proposed D-P-Transformer rockburst prediction algorithm. As compared to the original algorithm, the proposed algorithm shows average reductions of 24.45%, 46.56%, 17.32%, and 48.11% in the mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and mean absolute percentage error (MAPE), respectively. The results indicate that the novel D-P-Transformer rockburst prediction algorithm is superior to the Transformer prediction algorithm, and could be used for coal mine rockburst prediction analysis.

1. Introduction

Research has shown that rockburst disasters are occurring more frequently due to deeper work in underground mines [1]. It is believed that reliable rockburst prediction techniques can significantly reduce reconstruction risk, project costs, and human casualties [2]. Hence, accurate rockburst prediction has become a popular research topic in recent years.

Rockburst is a nonlinear dynamic phenomenon in which a rock mass instantaneously releases energy along the excavation face [3]. In recent years, the number of rockburst incidents in China reached 660, due to which more than 200 people have died [4]. Rockburst has occurred frequently in the United States in recent years, causing many injuries and even deaths. [5]. Rockburst occurs frequently and causes huge losses. Thus, rockburst research has received increasingly more attention from scholars.

After realizing the risk and harm of rockburst, numerous researchers have posited that accurate rockburst prediction is a powerful solution for the mitigation and solution of underground rockburst accidents and risk [6,7,8]. In the 1960s, Cook [9] first studied rockburst by conducting uniaxial compression experiments. More recently, He et al. [10] developed a true triaxial rockburst experimental system with three-way, six-sided loading and single-sided sudden unloading. They conducted multiple rockburst experiments, which successfully reproduced the rockburst phenomenon during the excavation of deep underground caverns. As the frequency, amplitude, and ringing counts of rockburst are important pieces of precursor information, they have been thoroughly explored by researchers [11,12,13,14]. Liu and Wang [15] took frequency as an early rockburst warning signal by analyzing the frequency change of the electromagnetic radiation (EMR) signal. Zhang et al. [16] discussed the magnitude of rockburst risk via microseismic (MS) monitoring techniques. Zhang et al. [17] further discussed the variation law of acoustic emission (AE) counts of different degrees of rock damage during rock destruction, concluding that a sudden increase in AE counts can be used as an indicator of rockburst. Via experimentation, Chu et al. [18] proved that the energy trend can be used as a rockburst indicator to prevent the occurrence of rockburst. Gao et al. [19] proposed an energy conservation index to evaluate rockburst potential by studying the capacity evolution characteristics of rocks.

Some scholars also believe that accurate rockburst prediction requires not only various key parameters, but also rigorous and appropriate algorithmic models [20,21,22,23]. Chen et al. [24] used the dynamic failure mechanism (DFM) to characterize the dynamic damage of rockburst. Jamei et al. [25] compared neural network algorithms with classical algorithms and demonstrated that the former is more suitable for rockburst prediction. Xue et al. [26] developed the particle swarm optimization-extreme learning machine (PSO-ELM) rockburst prediction model to predict typical rockburst cases in riverside hydropower plants in China. Wojtecki et al. [27] used machine learning algorithms to identify and assess rockburst. Moreover, researchers have started to develop and apply artificial intelligence (AI) models combining multiple factors for rockburst analysis and prediction.

Although previous studies have made some achievements, existing prediction techniques, especially AI techniques, require further improvements to more accurately predict the occurrence of rockburst. Therefore, the novel D-P-Transformer rockburst prediction model is proposed in this study. The model takes peak frequency, amplitude, ringing count, and absolute energy at the moment of rockburst as model input and predicts absolute energy using a novel embedding structure, probsparse self-attention, distilling operation, and multi-layer replicas. Seven large-scale rockburst datasets and four evaluation metrics are used to examine the reliability and validity of the proposed algorithm.

2. Methodology

As shown in Figure 1, the system framework consists of the following four main components.

(1)

Data sources: In this study, real-time experimental AE data were collected by using a rockburst physical simulation system;

(2)

Pre-processing: The experimental data were imported into the “Big Data AI Visualization and Analysis Platform” independently developed by the Big Data AI Lab with independent intellectual property rights for feature extraction. The data were then fed into the embedding structure of the D-P-Transformer rockburst prediction algorithm for pre-processing;

(3)

D-P-Transformer: The D-P-Transformer rockburst prediction algorithm model was constructed;

(4)

Prediction: The D-P-Transformer rockburst prediction algorithm was used to predict the absolute energy, and four evaluation metrics were used to evaluate the model.

2.1. Data Sources

Rockburst occurs frequently in mining sites such as tunnels and deep wells.

The strain rock explosion physical simulation test system, developed by the China University of Mining and Technology, was utilized in this study. This system consists of a main engine, a load control system, and a data collection system, and can simulate the occurrence of rockburst in the laboratory, as depicted in Figure 2. The experimental data can be collected by AE sensors, pressure sensors, and high-speed cameras, which record the complete changes of the rock sample during the rockburst in real-time. The experiment used two acoustic emission monitoring systems, including the PCI-2 acoustic emission sensor from the PAC company in the US and the PXWAE acoustic emission sensor from the Pengxiang company in China. Finally, based on comparative analysis and research, the data collected by the PCI-2 acoustic emission sensor is the most accurate, with the smallest data error. As a result, the experimental data are the results of the PCI-2 acoustic emission sensor. The PCI-2 acoustic emission sensor is a dual channel sensor, comprising a first channel and a second channel. The experimental data used is primarily from the first channel, and the experimental data from the second channel serves primarily to validate the data from the first channel.

The rockburst datasets used in this study were those of marble-1#, marble-3#, marble-4#, granite-61#, granite-64#, sandstone-60#, and slate-30# rock samples. These rock samples were cut to the specified scale size under conditions of good homogeneity and integrity. As illustrated in Figure 3, force was applied to the rock sample in three directions and on six sides to simulate the force on the rock before mining in the field, such as in the coal mine and deep shafts, after which rapid unloading was carried out along the advance direction of the excavation (direction of the hollow arrow) to simulate the force on the rock during construction up to the occurrence of rockburst.

2.2. Pre-Processing

Rockburst experiments generate a large amount of experimental data. The data generated by each rockburst experiment were stored in tens of thousands of text files, each of which contained 4096 sets of index data of multiple parameters at each moment. Then, a large amount of rockburst data collected in the previous step was imported into the “Big Data AI visualization and analysis platform” for rockburst feature extraction; the peak frequency, amplitude, ringing count, absolute energy, and other important rockburst features were extracted. Finally, a text file of the main feature parameters of the rockburst experiment was exported by the system. Parametric sensitivity analysis has been analyzed in our previous work, which shows that the frequency, amplitude, and absolute energy of the acoustic emission signal can be used for rockburst prediction [28].

To individually relate the extracted features to the experimental runs, the feature data inside the main parameter text file were used as the input of the embedding structure in the D-P-Transformer rockburst prediction algorithm, as presented in Figure 4. This structure consists of three main parts, namely, the scalar, relative time stamp, and absolute time stamp. This structure is obtained by Equation (1) [29]:

$$X_{\mathrm{en}}^t=\alpha u_i^t+P{E}_{(L_X(t-1)+i,)}+{\displaystyle \sum _p{[S{E}_{(L_X(t-1)+i)}]}_p}$$

(1)

where PE is the position and ordering information of the rockburst feature data in the long temporal sequence of rockburst, SE is the temporal information of the input sequence of rockburst feature data, $L_X$ is the input length of the rockburst feature data, $X_{en}^t$ is the input of the rockburst feature at time t in the encoder, which contains the temporal and positional information of the rockburst feature, and $\alpha u_i^t$ is the input rockburst feature data converted from a 1-dimensional to 512-dimensional feature vector by conv1D, which corresponds to the scalar. Moreover, PE(LX(t−1) + i,) is the integrated information of the position encoding, which is the same as the position embedding in the Transformer algorithm. It has the form of Equations (2) and (3) and corresponds to the relative time stamp [29].

$$P{E}_{(pos,2i)}=\sin (pos/{10000}^{2i/d_{model}})$$

(2)

$$P{E}_{(pos,2i+1)}=\cos (pos/{10000}^{2i/d_{model}})$$

(3)

where pos is the absolute coordinate of the entire time series and 𝑑𝑚𝑜𝑑𝑒𝑙 is the dimension of the transformed feature. Equation (2) is the PE calculation formula for even positions and Equation (3) is the PE calculation formula for odd positions. Moreover, ${\displaystyle \sum _p{[S{E}_{(L_X(t-1)+i)}]}_p}$ is the respective time of each rockburst feature throughout the rockburst experiment, which corresponds to the absolute time stamp.

2.3. D-P-Transformer

The various characteristics of rockburst are different in different time states; thus, rockburst occurrence is a long time-series problem. In recent years, numerous researchers have applied their knowledge of mathematics, computer science, and other fields to illustrate the great advantages of the Transformer model in long time-series prediction from both the theoretical, foundational and practical levels. The present work proposes the D-P-Transformer algorithm based on the rockburst prediction method to analyze and predict rockburst occurrence, based on the encoder and decoder structures of the Transformer model, as illustrated in Figure 5.

The structure of the encoder in the proposed prediction algorithm is designed to extract the long-term dependencies of long sequence inputs. The encoder, as a result of the probsparse self-attention mechanism, is subject to redundancy in its feature mapping. Therefore, the proposed D-P-Transformer rockburst prediction algorithm uses a distilling operation to assign higher weights to dominant rockburst features, and generates rockburst feature mappings at the next layer using Equation (4) for layers j to j + 1 [30]:

$$X_{j+1}^t=MaxPool\left(ELU\left(Convld\left({\left[X_j^t\right]}_{AB}\right)\right)\right)$$

(4)

where ${[X_j^t]}_{AB}$ is the key operation in the attention block, and the multi-head probsparse self-attention Convld denotes a one-dimensional convolution operation on a time series. Moreover, ELU is the activation function and MaxPool is maximum pooling. Equation (4) indicates that input $X_j^t$ goes through the AB layer, Convld mapping, ELU, and the MaxPool layer, in turn, to obtain the coded input $X_{j+1}^t$ at the subsequent moment.

Via the distilling operation step indicated by the blue trapezoids in Figure 5, the input sequence length of each layer of the decoder is shortened, the size of the network is drastically reduced, and the stacking of layers increases the robustness of the model. In addition, multiple layer replicas are used for simultaneous computation during algorithm operation to speed up the operation and drastically reduce the amount of computation.

The decoder of the proposed prediction algorithm uses two multi-head attention layers and provides the input vector via Equation (5) [30]:

$$X_{de }^t=\mathit{\text{Concat}}\left({\mathit{X}}_{token }^t,{\mathit{X}}_0^t\right)\in R^{\left(L_{token}+L_y\right)d_{model }}$$

(5)

where $X_{de}^t$ is the input of the decoder, ${\mathit{X}}_{token }^t$ is the known part of the encoder input, $L_{token }^t$ is the length of that part, ${\mathit{X}}_0^t$ is the masked part of the encoder input, and L_y is the length of the masked part. The decoder inputs a sequence $X_{de}^t$ for some time and the latter part is masked with 0. The purpose of the entire algorithm is to predict the masked part in the decoder. The hidden multi-head attention mechanism does not cause each position to pay attention to the next position, which avoids autoregression and creates a fully connected layer to obtain the final output.

The first step of the proposed prediction algorithm is to use probsparse self-attention instead of traditional self-attention to form the P-Transformer rockburst prediction algorithm. The probsparse self-attention first selects random queries from K to calculate the sparse score, and then selects the u queries with the highest sparse score to calculate the attention value. The remaining queries are not calculated but are sent directly to the input of the self-attention layer, and the average value is taken as the output. As a result, the algorithm only calculates the matrix of the Top-u q_i, and directly takes the average value of the remaining q_i; thus, the amount of computation is greatly reduced.

The traditional self-attention input is (Query, Key), and the scaled dot-product is obtained by Equation (6) [31]:

$$A(Q,K,V)=Softmax\left(\frac{Q{K}^T}{\sqrt{d}}\right)V$$

(6)

where $Q\in R^{L_Q\times d}$, $K\in R^{L_K\times d}$, and $V\in R^{L_V\times d}$. Moreover, d is the dimension of the input, and Q, K, and V are the active query vector, key vector, and value vector of the input, respectively. The weighted probability of the ith query is in the form of Equation (7) [31]:

$$A \left(q_i,K,V\right)=\sum _j\frac{k(q_i,k_i)}{{\sum }_{l}k(q_i,k_l)}{V}_j=E_{p\left(k_i\mid q_i\right)}\left[v_j\right]$$

(7)

where $p\left(k_i\mid q_i\right)=\frac{k(q_i,k_i)}{{\sum }_{l}k(q_i,k_l)}$ and $k\left(q_i,k_j\right)=\exp \left(\frac{q_i{k}_i^i}{\sqrt{d}}\right)$. Moreover, q_i, k_i, and v_i, respectively, denote the ith row of matrices Q, K, and V.

KL divergence can effectively evaluate the sparsity of the ith query and this method is an effective way to calculate the attention and probability distribution of the query with unity. The KL divergence is calculated by Equation (8) [31]:

$$KL(q\left|\right|p)=ln\sum _{l=1}^{L_K}{e}^{\frac{q_i{k}_l^T}{\sqrt{d}}}-\frac{1}{L_K}\sum _{j=1}^{L_K}\frac{q_i{k}_J^T}{\sqrt{d}}-ln{L}_K$$

(8)

Because $ln{L}_K$ is a constant, it does not affect the derivation. Therefore, the measurement equation for the sparsity of the ith query is Equation (9):

$$M\left(q_i,K\right)=ln\sum _{j=1}^{L_K}{e}^{\frac{q_i{k}_l^T}{\sqrt{d}}}-\frac{1}{L_K}\sum _{j=1}^{L_K}\frac{q_i{k}_J^T}{\sqrt{d}}$$

(9)

For all keys, $q_i$ is the Log-Sum-Exp (LSE), and to simplify the formula, $ln\sum _{j=1}^{L_K}{e}^{\frac{q_i{k}_l^T}{\sqrt{d}}}$ is replaced by the maximum value $\underset{j}{\text{max}}\left\{\frac{q_i{k}_j^T}{\sqrt{d}}\right\}$ and $\frac{1}{L_K}\sum _{j=1}^{L_K}\frac{q_i{k}_J^T}{\sqrt{d}}$ is the arithmetic mean. Therefore, after simplification, Equation (10) can be used instead [31]:

$$\overline{M}\left(q_i,K\right)=\underset{j}{\text{max}}\left\{\frac{q_i{k}_j^T}{\sqrt{d}}\right\}-\frac{1}{L_K}\sum _{j=1}^{L_K}\frac{q_i{k}_J^T}{\sqrt{d}}$$

(10)

The final formula for probsparse self-attention can be obtained as Equation (11) [31]:

$$A(Q,K,V)=Softmax\left(\frac{\overline{Q}{K}^T}{\sqrt{d}}\right)V$$

(11)

where $\overline{Q}$ is a sparse matrix with the same size as q, and $\overline{Q}$ is the active query after the linear transformation of the input $X_{en}^t$.

The P-Transformer rockburst prediction algorithm is presented in Algorithm 1.

Algorithm 1 The P-Transformer algorithm.

1:Input: Database $y_i\in \{y_1,y_2,\dots y_n\}$ feature map S, $i\in [1,N]$    Tensor $Q\in R^{L_Q\times d}$, $K\in R^{L_K\times d}$, $V\in R^{L_V\times d}$

2:Output: Predicted value ${\overline{y}}_i$

3:for $i=1,2,\dots N$ do

4:According to Equations (1)–(3), get $y_i$

5:end for

6:randomly select dot-product pairs from K as $\overline{K}$

7:compute the KL divergence

8:set the sparse score $\overline{S}=Q\overline{K}$

9:compute the measurement $\overline{M}=\max (\overline{S}\left)\text{–}\mathrm{mean}\right(\overline{S})$ by row

10:set Top-u queries under $\overline{M}$ as $\overline{Q}$

11:$S_1\leftarrow softmax(\overline{Q}{K}^T/\sqrt{d})\cdot V,S_0\leftarrow$ mean(V)

12:S ← {S1, S0}

13:for$i=1,2,\dots N$ do

14:According to Equation (5) of the Decoder, get ${\overline{y}}_i$

15:end for

Based on the P-Transformer rockburst prediction algorithm, the D-P-Transformer algorithm is formed by adding a distilling operation to the encoder part and simultaneously calculating multiple layer replicas. The specific algorithm is presented in Algorithm 2. When the rockburst feature map S is obtained, the rockburst features with larger weights are extracted by the convolution operation, activation function, and maximum pooling to significantly reduce the size of the network and are used as the code input for the next moment.

Algorithm 2 The D-P-Transformer algorithm.

1:Input: Dataset $y_i\in \{y_1,y_2,\dots y_n\}$ feature map S, $i\in [1,N]$,    Tensor $Q\in R^{L_Q\times d}$, $K\in R^{L_K\times d}$, $V\in R^{L_V\times d}$

2:Output: Predicted value ${\overline{y}}_i$

3:for$i=1,2,\dots N$ do

4:According to Equations (1)–(3), get $y_i$

5:end for

6:compute the measurement $\overline{M}=\max (\overline{S}\left)\text{–}\mathrm{mean}\right(\overline{S})$ by row

7:Set the Top-u queries under $\overline{M}$ as $\overline{Q}$

8:$S_1\leftarrow softmax(\overline{Q}{K}^T/\sqrt{d})\cdot V,S_0\leftarrow$ mean(V)

9:S ← {S1, S0}

10:for$i=1,2,\dots N$ do

11:According to Equations (4) and (5) of the Encoder and multiple layer replicas, get ${\overline{y}}_i$

12:end for

2.4. Prediction

In this study, seven large-scale rockburst datasets were used as the experimental data, namely, marble-1#, marble-3#, marble-4#, slate-30#, sandstone-60#, granite-61#, and granite-64#. These datasets were used to enhance the reliability of the experimental results after the prediction model was constructed. Moreover, various metrics were used to verify the performance of the algorithm model, namely, the mean absolute error (MAE), mean square error (MSE), root mean square error RMSE, and mean absolute percentage error (MAPE). The prediction experiments and results are detailed in the next section.

3. Results and Discussion

3.1. Evaluation Metrics

To assess the reliability and accuracy of the proposed model, several metrics were used to provide a comprehensive understanding of the rockburst datasets used and the proposed model. The formulas and meanings of the evaluation metrics are detailed in

Table 1. The evaluation metrics.

Metric	Formula	Significance
Mean absolute error (MAE)	$MAE=\frac{1}{n}{\displaystyle \sum _{i=1}^n|y_i-{\overline{y}}_i|}$	The ratio of the mean errors for all the predictions in the rockburst experiments
Mean square error (MSE)	$MSE=\frac{1}{n}{{\displaystyle \sum _{i=1}^n|y_i-{\overline{y}}_i|}}^2$	The ratio of the mean square errors for all the predictions in the rockburst experiments
Root mean square error (RMSE)	$RMSE=\sqrt{\frac{1}{n}{{\displaystyle \sum _{i=1}^n|y_i-{\overline{y}}_i|}}^2}$	The ratio of the root mean square errors for all the predictions in the rockburst experiments
Mean absolute percentage error (MAPE)	$MAPE=\frac{1}{n}{\displaystyle \sum _{i=1}^n\left|\frac{y_i-{\overline{y}}_i}{y_i}\right|}$	The ratio of the mean percentage errors for all the predictions in the rockburst experiments

. In the model evaluation, the smaller the value of these metrics, the better the performance of the tested model. Moreover, n is the number of rockburst feature data points in the experiment, and $y_i$ and ${\overline{y}}_i$ are the ith data points of rockburst feature measurement and prediction, respectively, in the experiment.

3.2. Results

To solve the problem of the huge amount of rockburst data, the P-Transformer rockburst prediction algorithm, formed by probsparse self-attention instead of traditional self-attention, was first used for experimentation, and the prediction results were compared with those of the Transformer algorithm model. Among the prediction results of different ratios of training and test sets for the same type of rock sample, three sets of optimal prediction results were respectively selected.

Table 2. The metrics of the P-Transformer and Transformer algorithms on different datasets.

Method Metric		MAE		MSE		RMSE		MAPE
		Transformer	P-Transformer	Transformer	P-Transformer	Transformer	P-Transformer	Transformer	P-Transformer
1#	Num.1	1.799	2.009	532.634	534.817	23.078	23.126	0.468	0.641
	Num.2	1.732	1.737	477.107	415.467	21.842	20.383	0.880	0.567
	Num.3	1.736	1.737	364.028	328.099	19.080	18.114	0.652	0.574
3#	Num.1	0.513	0.303	135.020	97.976	11.620	9.898	1.141	0.417
	Num.2	0.439	0.427	97.799	69.145	9.889	8.315	1.015	0.494
	Num.3	0.143	0.132	69.077	25.009	8.311	5.001	0.656	0.153
4#	Num.1	0.075	0.050	5.960	0.948	2.441	0.974	1.050	0.318
	Num.2	0.099	0.088	7.961	0.951	2.822	0.975	1.323	1.015
	Num.3	0.063	0.030	0.950	0.001	0.975	0.032	1.321	0.906
30#	Num.1	9.076	7.486	400.435	430.314	20.011	20.744	1.163	1.967
	Num.2	10.486	7.595	334.541	370.259	18.290	19.242	2.699	2.588
	Num.3	8.194	7.194	232.350	334.541	15.243	18.290	5.000	3.486
60#	Num.1	0.750	0.807	111.434	112.391	10.556	10.601	14.878	13.231
	Num.2	0.775	0.763	111.704	98.588	10.569	9.929	14.761	12.591
	Num.3	0.849	0.754	113.671	9.200	10.662	3.033	15.326	14.045
61#	Num.1	0.214	0.262	0.123	0.140	0.351	0.375	2.049	1.200
	Num.2	0.109	0.084	0.093	0.004	0.305	0.064	1.978	1.194
	Num.3	0.179	0.041	0.152	0.098	0.389	0.314	2.405	1.136
64#	Num.1	19.100	19.996	414.763	479.841	20.366	21.905	3.672	4.370
	Num.2	14.573	14.559	357.963	352.919	18.920	18.786	2.667	2.448
	Num.3	11.114	11.072	352.790	212.707	18.783	14.584	6.403	3.057

reports the results of a large number of experiments using multiple types of rockburst data and different training and test set ratios; the best results are highlighted in boldface. Most of the prediction results of the P-Transformer rockburst prediction algorithm were found to be better than those of the Transformer algorithm. P-Transformer selects the Top-u rockburst data to calculate the attention value, which not only greatly reduced the amount of calculation, but also improved the indicator results as compared to those of the original model.

In the proposed algorithm, a distilling operation and multiple layer replicas are used simultaneously, which greatly reduces the running time and enhances the robustness.

Table 3. The metrics of the D-P-Transformer and Transformer algorithms on different datasets.

Method Metric		MAE		MSE		RMSE		MAPE
		Transformer	D-P-Transformer	Transformer	D-P-Transformer	Transformer	D-P-Transformer	Transformer	D-P-Transformer
1#	Num.1	1.799	2.320	532.634	435.933	23.078	20.879	0.468	0.553
	Num.2	1.732	1.768	477.107	377.680	21.842	19.434	0.880	0.609
	Num.3	1.736	1.728	364.028	258.550	19.080	16.079	0.652	0.391
3#	Num.1	0.513	0.222	135.020	97.774	11.620	9.888	1.141	0.352
	Num.2	0.439	0.305	97.799	69.014	9.889	8.307	1.015	0.457
	Num.3	0.143	0.134	69.077	25.005	8.311	5.000	0.656	0.237
4#	Num.1	0.075	0.048	5.960	0.676	2.441	0.822	1.050	0.163
	Num.2	0.099	0.058	7.961	0.021	2.822	0.144	1.323	0.839
	Num.3	0.063	0.010	0.950	0.001	0.975	0.012	1.321	0.339
30#	Num.1	9.076	7.234	400.435	356.908	20.011	18.892	1.163	2.515
	Num.2	10.486	7.255	334.541	356.908	18.290	18.892	2.699	2.167
	Num.3	8.194	5.255	232.350	324.450	15.243	18.013	5.000	2.106
60#	Num.1	0.750	0.787	111.434	53.757	10.556	7.332	14.878	15.419
	Num.2	0.775	0.433	111.704	35.206	10.569	5.933	14.761	11.093
	Num.3	0.849	0.577	113.671	8.839	10.662	2.973	15.326	9.428
61#	Num.1	0.214	0.006	0.123	0.101	0.351	0.317	2.049	1.200
	Num.2	0.109	0.004	0.093	0.005	0.305	0.069	1.978	1.194
	Num.3	0.179	0.035	0.152	0.098	0.389	0.313	2.405	1.136
64#	Num.1	19.100	15.771	414.763	353.194	20.366	18.793	3.672	1.634
	Num.2	14.573	14.071	357.963	212.835	18.920	14.589	2.667	1.480
	Num.3	11.114	9.759	352.790	353.166	18.783	18.793	6.403	1.602

compares the prediction results of the D-P-Transformer rockburst prediction algorithm with those of the Transformer algorithm based on numerous experiments using multiple types of rockburst data and different training and test sets ratios; the best results are highlighted in boldface. Most of the results of the D-P-Transformer rockburst prediction algorithm were found to be better than those of the Transformer algorithm. The metrics were significantly better than those of the Transformer algorithm model.

3.3. Discussion

For model evaluation, three sets of optimal prediction results were selected for statistical analysis to ensure the performance of the proposed model. To prevent excessively large or excessively small errors, multiple sets of prediction results for the same type of rock were averaged by value. Figure 6 presents the comparison of the average values of multiple sets of prediction results of the P-Transformer rockburst prediction algorithm with those of the Transformer model. As shown in Figure 6a, the MAE values of the P-Transformer algorithm were found to be slightly lower than those of the Transformer model on the marble-3#, marble-4#, slate-30#, sandstone-60#, and granite-61# rock samples. The values for the marble-3#, marble-4 #, and granite-61# rock samples were close to 0, and are therefore equivalent to the best model. As shown in Figure 6b,c, the MSE and RMSE values of the P-Transformer algorithm were found to be slightly lower than those of the Transformer model on the marble-1#, marble-3#, marble-4#, sandstone-60#, and granite-64# rock samples. The best results, with almost no error, were obtained on the marble-4# and granite-61# rock samples, indicating that this model is the most suitable for these types of rocks. As shown in Figure 6d, the MAPE values of the P-Transformer algorithm were found to be significantly lower than those of the original model for all rock samples, indicating that the errors of this model were generally smaller than those of the Transformer model, and this model is therefore more suitable for rockburst prediction.

To further reduce the error and computation time, the D-P-Transformer rockburst prediction algorithm was formed by using multiple layer replicas for simultaneous computation, on the basis of the P-Transformer algorithm. Figure 7 exhibits the comparison of the average values of multiple sets of prediction results of the D-P-Transformer algorithm with those of the P-Transformer model. As shown in Figure 7a, the MAE values of the D-P-Transformer algorithm were lower than those of the P-Transformer model, except for the value for the marble-1# rock sample. The MAE values for the marble-3#, marble-4#, and granite-61# rock samples were close to zero, which is equivalent to no error and indicates high model prediction ability. As shown in Figure 7b,c, the MSE and RMSE values of the D-P-Transformer algorithm were found to be almost equal to those of the P-Transformer model for the slate-30# rock samples, and were lower than those of the original model for the rest of the rock samples. The best results, with almost no errors, for the marble-4# and granite-61# rock samples indicate that this model is most suitable for these types of rocks. As shown in Figure 7d, the MAPE values of the D-P-Transformer algorithm were found to be significantly lower than those of the original model for all the rock samples. This indicates that the errors of this model were all smaller than those of the P-Transformer model, and that this model is more suitable for rockburst prediction.

Based on the comparison with the original prediction algorithm, the average metric results of multiple groups of experiments were compared with the results of the LSTM prediction algorithm, as exhibited in Figure 8. The results indicate that the proposed D-P-Transformer rockburst prediction algorithm achieved the best overall performance. As presented in Figure 8a, it is obvious from the three rock sample experiments of marble-3#, slate-30#, and granite-64# that the prediction errors of the algorithms, from least to greatest, were those of the D-P-Transformer, LSTM, P-Transformer, and Transformer algorithms. In particular, for the three rock sample experiments of marble-3#, marble-4#, and granite-61#, the prediction errors of the D-P-Transformer algorithm were quite low, even close to 0, indicating that this algorithm is especially suitable for these three types of rocks. As presented in Figure 8b,c, it is evident from the four rock sample experiments of marble-1#, marble-3#, sandstone-4#, and granite-64# that the errors of the LSTM, Transformer, P-Transformer, and D-P-Transformer algorithms decreased sequentially, indicating the improvement of the model performance. For the marble-4# and granite-61# rock experiments in particular, the MSE and RMSE values of the D-P-Transformer rockburst prediction algorithm were almost 0, indicating that this algorithm is very suitable for these two types of rock samples. Figure 8d shows that the prediction errors of the D-P-Transformer algorithm were better than those of the other three algorithms for the marble-1#, marble-3#, marble-4#, and granite-64# rock samples, and were close to 0 for the marble-1#, marble-3#, and marble-4# samples.

On the basis that the proposed model is superior to the original model, the prediction data of each model were observed, the maximum and minimum values were removed, and the average value was then taken as the final model evaluation metric result and compared with the HGSO-ANFIS algorithm [32], as reported in

Table 4. The performance comparison of various models.

Methods/Metrics	MAE	MSE	RMSE	MAPE
HGSO-ANFIS	19.975	1020.100	31.939	1.197
LSTM	2.233	160.935	12.686	1.337
Transformer	2.466	183.095	11.964	2.303
P-Transformer	2.089	143.137	10.820	1.799
D-P-Transformer	1.863	97.852	9.892	1.195

. The values of all performance metrics of the proposed D-P-Transformer rockburst prediction algorithm were better than those of the other algorithms. Moreover, in terms of the MAE, MSE, RMSE, and MAPE values, the performance of the P-Transformer algorithm was improved by 15.29%, 21.82%, 9.56%, and 21.88%, respectively, as compared with the Transformer model. In terms of the MAE, MSE, RMSE, and MAPE values, the performance of the D-P-Transformer algorithm was improved by 24.45%, 46.56%, 17.32%, and 48.11%, respectively, as compared with the Transformer model. In summary, in terms of the MAE, MSE, RMSE, and MAPE metrics, the proposed D-P-Transformer algorithm is the most suitable for rockburst prediction, with a smaller error and better results than the other four compared algorithms.

4. Conclusions

This study proposed a novel D-P-Transformer rockburst prediction algorithm for the prediction of the absolute energy of rockburst experiments. First, the embedding structure was improved based on the traditional Transformer model to increase the applicability of the algorithm to rockburst data. Then, probsparse self-attention was used instead of traditional self-attention to significantly reduce the computational effort. Finally, a distilling operation and multiple layer replicas were employed simultaneously to improve the computational efficiency.

The results of prediction revealed that, compared with those of the Transformer algorithm, the MAE, MSE, RMSE, and MAPE values of the proposed D-P-Transformer rockburst prediction algorithm were reduced by averages of 24.45%, 46.56%, 17.32%, and 48.11%, respectively. The results of prediction revealed that, compared with the other algorithms, the MAE, MSE, RMSE, and MAPE values of the proposed D-P-Transformer rockburst prediction algorithm were reduced by a maximum of 90.67%, 90.41%, 69.03%, and 48.11%, respectively. In all aspects, the D-P-Transformer rockburst prediction algorithm outperforms HGSO-ANFIS and other prediction algorithms. The results indicate that the proposed D-P-Transformer algorithm not only produces less prediction error but is also more suitable for the rockburst prediction analysis of multiple types of rocks, as compared to the traditional Transformer model.

The proposed D-P-Transformer rockburst prediction algorithm can potentially reduce mining-related economic losses, casualties, and environmental impacts. Nevertheless, differences exist between rockburst simulated under experimental conditions and rockburst in the field. Therefore, in future research, it is suggested that the proposed method be applied to the Tayi tunnel in Yunnan Province, China to verify its performance in a real-life situation. Future studies could also benefit from the combination of the proposed D-P-Transformer rockburst prediction algorithm with multiple decision mechanisms to increase the accuracy of coal mine rockburst prediction. Multidisciplinary approaches are required to integrate mining’s technical, economic, environmental, and social aspects into sustainable development.