Application of a Multi-Teacher Distillation Regression Model Based on Clustering Integration and Adaptive Weighting in Dam Deformation Prediction

Abstract

Deformation is a key physical quantity that reflects the safety status of dams. Dam deformation is influenced by multiple factors and has seasonal and periodic patterns. Due to the challenges in accurately predicting dam deformation with traditional linear models, deep learning methods have been increasingly applied in recent years. In response to the problems such as an excessively long training time, too-high model complexity, and the limited generalization ability of a large number of complex hybrid models in the current research field, we propose an improved multi-teacher distillation network for regression tasks to improve the performance of the model. The multi-teacher network is constructed using a Transformer that considers global dependencies, while the student network is constructed using Temporal Convolutional Network (TCN). To improve distillation efficiency, we draw on the concept of clustering integration to reduce the number of teacher networks and propose a loss function for regression tasks. We incorporate an adaptive weight module into the loss function and assign more weight to the teachers with more accurate prediction results. Finally, knowledge information is formed based on the differences between the teacher networks and the student network. The model is applied to a concrete-faced rockfill dam located in Guizhou province, China, and the results demonstrate that, compared to other knowledge distillation methods, this approach exhibits higher accuracy and practicality.

1. Introduction

As the core structure of hydraulic engineering, dams play a critical role not only in flood control, power generation, and water supply but also in effectively regulating the water environment and aquatic ecosystems. They have a significant impact on the safety of life and property, economic development, and the ecological environment of downstream populations [1]. Over time, the structural performance of dams inevitably degrades, and, in the event of a failure, the consequences are often catastrophic, leading to substantial loss of life and property for downstream residents. In fact, most dam failures are not sudden events but rather a gradual evolutionary process [2]. Therefore, it is essential to adopt reasonable and effective methods to monitor and evaluate the operational conditions of dams, with deformation prediction being a crucial component of the monitoring process [3,4,5]. The comparison between predicted values and actual measurements can help identify abnormal conditions in a timely manner, enabling prompt response measures and ensuring the safe operation of dams [6,7].

Traditional dam deformation prediction methods are predominantly based on regression-based statistical models. These statistical approaches primarily establish functional relationships between effect quantities (such as deformation, seepage, and stress) and independent variables (such as water level, temperature, and time). Due to their simplicity and ease of implementation, they have been widely used in the field of dam safety monitoring [8,9,10]. However, during the actual operation of a dam, the relationship between the independent variables and the effect quantities is often extremely complex, exhibiting strong nonlinear characteristics. Consequently, traditional linear models are unable to accurately describe such nonlinear relationships, which further restricts the prediction accuracy. With the rapid development of machine learning and deep learning, techniques including but not limited to artificial neural networks, support vector machines, random forests, and genetic algorithms have been applied to dam deformation analysis and prediction. These algorithms exhibit significant advantages in addressing nonlinear problems and have demonstrated promising results [11,12,13]. Yang et al. employed long short-term memory (LSTM) networks to address the long-term dependencies present in dam deformation [14]. Building on this foundation, Yang et al. introduced Convolutional Neural Networks (CNNs) to identify and learn the spatial relationships within-time series data, developing a reliable dam prediction model [15]. Jiedeerbieke et al. proposed a dam deformation prediction model based on clustering partitioning and bidirectional long short-term memory (BiLSTM) networks. By considering the intrinsic correlations among monitoring points and integrating multiple feature information, the model achieves more comprehensive deformation monitoring [16]. Su et al. proposed a data processing framework that uses a long short-term memory (LSTM) model coupled with an attention mechanism to predict the deformation response of a dam structure in order to obtain more accurate nonlinear prediction results [17]. Zhou et al. established a dam displacement prediction model based on a multi-expert network. The model employs long short-term memory (LSTM) to extract temporal features and utilizes a multi-head attention mechanism to capture adjacency values, constructing a spatial graph to describe spatial relationships within the multi-expert network. Additionally, graph convolutional networks (GCNs) are applied to extract temporal relationships, enabling the prediction of multi-point dam displacements [18]. Wang et al. proposed a hybrid algorithm that combines backpropagation with a genetic algorithm (GA-BP) and a multiple population genetic algorithm (MPGA) to optimize the BP neural network. This hybrid algorithm has improved convergence speed and prediction accuracy [19]. The current research shows that hybrid neural networks often achieve better performance than single-structure models. However, this improvement in performance comes at a cost. As the model structure becomes more complex, the number of parameters in the neural network also increases significantly. This means that more computational resources are required, and longer training times are needed. The increase in computational overhead directly affects the application efficiency of the model, especially in real-world application scenarios with extremely high real-time requirements. Therefore, it is necessary to design a method that can significantly reduce the number of model parameters while improving the model’s accuracy, thus shortening the training and inference times.

The principle of knowledge distillation (KD) [20] is effectively optimizin the original model by developing a lightweight, efficient, and small model. A lightweight student network is used to mimic the behavior and output of a powerful teacher network. However, currently, knowledge distillation has mostly been applied to classification problems [21,22,23,24], where knowledge has been transferred through the soft labels generated by the teacher model. Nevertheless, this rule is not applicable in regression problems.

In response to the above-mentioned problems, this paper proposes a knowledge distillation method for regression problems, which is used for dam deformation prediction. This method consists of multiple modules, such as the pre-training of teacher networks, clustering integration based on information theory, the knowledge distillation process, and the prediction module. The main content is as follows: First, considering the robustness of teacher networks, we pre-trained teacher networks based on the original data to expand the teacher team. Subsequently, we adopted the clustering ensemble method to effectively reduce the number of redundant and low-precision teacher networks, thus avoiding the confusion in guidance caused by an excessive number of teacher networks with varying qualities during the subsequent distillation process. Then, we used the proposed weighted loss function for knowledge transfer to train a lightweight student network. Finally, the high-precision prediction of the student network was achieved. We used an actual concrete-faced rockfill dam as the experimental object, and the experimental results showed that the proposed model had higher accuracy and practicality in specific engineering cases.

Based on the above description, the contributions of this study are as follows:

$\left(1\right)$

A multi-teacher distillation framework for regression problems is proposed. Traditional knowledge distillation methods have certain limitations when dealing with regression problems. We provide a new paradigm for this field. With the help of our framework, it is possible to provide the student network with richer and more comprehensive knowledge information.

$\left(2\right)$

We adopted a clustering integration method based on information theory to reduce the number of redundant and low-precision teacher networks. This avoids knowledge redundancy and conflicts caused by too many teacher networks, further enhancing the performance and stability of the model.

$\left(3\right)$

We propose of a novel weighted loss function for regression to guide the student network. Different weights are assigned according to the performance of different teacher networks, enabling the student network to pay more attention to the knowledge content of high-quality teachers during the learning process.

$\left(4\right)$

Our method was applied to a concrete-faced rockfill dam in Guizhou province, China. The results show that, through our distillation algorithm, the prediction accuracy (MSE) of the student network can be effectively reduced by 40% to 73%. Compared with other knowledge distillation methods, our method has higher accuracy and practicality.

Next, an outline of the remaining parts of this paper is presented. Section 2 introduces the theoretical basis and traditional knowledge distillation (KD) methods. In Section 3, we present the proposed method. Section 4 reports the experimental results. In Section 5, the conclusions and future work are presented.

2. Related Works

2.1. HST Statistical Model

First, we briefly introduce the classic HST statistical model [25], According to the current knowledge system and relevant experience, the displacement of a dam is mainly related to three major influencing factors, namely, the upstream water level H, the temperature T, and the aging $\theta$. We further express this as the water pressure component $y_H$, the temperature $y_T$, the aging component $y_{\theta }$, and the constant term c as follows:

$$y=y_H+y_T+y_{\theta }+c$$

(1)

The water pressure component $y_H$ of the horizontal displacement y of a gravity dam can be expressed by the upstream water levels H as follows:

$$y_H=\sum _{i=1}^3{a}_i{H}^i$$

(2)

where $a_i(i=1, 2, 3)$ are the regression coefficients.

The actual situation of a dam is often complex. Since it is difficult to install a sufficient number of thermometers on the bedrock and the dam body, when the internal temperature field of the dam reaches a stable state, we usually use multi-cycle harmonic variations as the factors influencing the temperature component $y_T$ as follows:

$$y_H=\sum _{i=1}^3{a}_i{H}^i{y}_T=\sum _{i=1}^2\left(b_{1i}sin{\displaystyle \frac{2\pi it}{365}}+b_{2i}cos{\displaystyle \frac{2\pi it}{365}}\right)$$

(3)

where $b_{1}i$ and $b_{2}i$ represent regression coefficients, and t represents the number of days from the detection date to the initial monitoring date.

The aging component $y_{\theta }$ of dam deformation is irreversible. For most dams, the displacement caused by aging develops rapidly during the initial operation period but tends to stabilize in the later stages. Therefore, $y_{\theta }$ is usually generalized as a combination of a logarithmic function and a linear function as follows:

$$y_{\theta }=c_1\theta +c_{2}ln\theta$$

(4)

where $\theta$ = t/100; t is calculated as the cumulative number of days from the monitoring day to the start of the monitoring period. Combining Equations (1)–(4), the complete mathematical expression of the HST model is as follows:

$$y=\sum _{i=1}^3{a}_i{H}^i+\sum _{i=1}^2(b_{1i}sin{\displaystyle \frac{2\pi it}{365}}+b_{2i}cos{\displaystyle \frac{2\pi it}{365}})+c_1\theta +c_{2}ln\theta +c$$

(5)

where the other symbols have the same meanings as in the previous equation.

2.2. Transformer Model

This section provides a comprehensive introduction to the Transformer. The Transformer [26] model was initially designed to address the challenges in natural language processing (NLP). Traditional models perform poorly in handling long-range dependencies. In contrast, the Transformer, with its unique architecture and innovative mechanisms, demonstrates outstanding performance and has extended its influence to various fields, including time-series prediction [27,28,29,30,31]. In time-series data, accurately capturing long-term dependencies and complex patterns is crucial for improving prediction accuracy. The Transformer can precisely meet this requirement. By modeling the relationships between different time steps in the time series, it can make more accurate predictions. The Transformer mainly consists of two core components: the encoder and the decoder. The encoder module is used to capture the complex relationships and contextual information among the elements in the input sequence, laying the foundation for subsequent processing. The decoder module focuses on feature reconstruction. Figure 1 shows the detailed framework of the Transformer.

The encoder is composed of multiple identical layers, and its primary function is to capture the dependencies within the input sequence. Each encoding layer takes the output of the previous layer as its input, and adjacent layers are connected through residual connections. This structure can effectively transmit information and mitigate the vanishing gradient problem during the training process. A normalization operation is performed after each residual module. Each layer of the encoder contains two core sub-layers: multi-head attention and a fully-connected feed-forward network. Different from a single self-attention mechanism, multi-head attention compares each query with a set of key vectors from multiple representation sub-spaces. Similar to the encoder, the decoder is also composed of multiple identical layers, with each layer taking the output of the previous layer as its input. Each decoding layer consists of three core sub-layers: masked multi-head attention, a fully-connected feed-forward network, and multi-head attention. The outputs of these sub-layers are passed through residual links and combined with layer normalization to ensure the stability of the input for each layer. The computation processes of the fully connected feed-forward network and multi-head attention are the same as those in the encoder. Masked multi-head attention is the same as multi-head attention, except for the self-attention computation process.

Compared with Recurrent Neural Networks (RNNs) that process data sequentially, the Transformer can process all data simultaneously to improve efficiency and performance. However, when applied to complex tasks such as dam deformation prediction, the Transformer faces a series of challenges, which can be mainly summarized as high computational resource requirements, large memory usage, and limited processing speed.

2.3. Multi-Teacher Knowledge Distillation

Knowledge distillation (KD), as an important method for model compression, mainly aims to effectively transfer the “dark knowledge” from complex and large-scale models to concise and small-scale models, enabling the smaller models to achieve performance similar to that of large models. In 2014, reference [20] proposed a method that allows the student network to learn the output of the teacher network, defining the results of the teacher model as “soft labels”. Knowledge distillation works well in classification tasks because of its advantage in “dark knowledge”, which refers to the softened hidden layer representations (logits) output of the teacher model. The specific process of knowledge distillation is shown in Figure 2a,b.

The objective function of knowledge distillation for classification is as follows:

$$L_{\mathrm{KD}}=\alpha ·L_{\mathrm{CE}}(O_s^{\mathrm{T}},O_t^{\mathrm{T}})+(1-\alpha )\ast L_{\mathrm{CE}}\left(O_s^1,y_{label}\right)$$

(6)

where $\alpha$ is the balance coefficient of the loss function, which is used to adjust the balance between the soft loss and the hard loss. Its value ranges from [0, 1]; $L_{\mathrm{CE}}$ is the cross-entropy loss function, T is the temperature coefficient, $O_s^{\mathrm{T}}$ is the probability distribution of the output of the student network after passing through the Softmax layer with the temperature coefficient T, $O_t^{\mathrm{T}}$ is the probability distribution of the output of the teacher network after passing through the Softmax layer with the temperature coefficient T; $O_s^1$ is the probability distribution of the output of the student network after passing through the Softmax layer; $y_{label}$ is the label of the input data, which is defined as the hard label.

The objective function of knowledge distillation commonly used for regression is as follows:

$$L_{KD}=\alpha \ast L_{MSE}\left(O_s,O_t\right)+(1-\alpha )\ast L_{MSE}(O_s,y_{label})$$

(7)

where $O_s$ represents the direct output information of the student network; $O_t$ represents the soft-label information directly output by the teacher network; $L_{MSE}$ is the mean squared error loss function.

Multi-teacher distillation [33,34,35,36], as a variant of knowledge distillation, employs multiple teacher models to jointly guide the training of the student model. Compared with the traditional single-teacher knowledge distillation method, the multi-teacher distillation strategy provides more comprehensive and stable knowledge for the student model by aggregating the expertise from different teachers. Under the multi-teacher distillation framework, the learning process typically includes the following key steps:

$\left(1\right)$

Teacher model training: Multiple teacher models are trained independently. Each teacher model may be based on a different model architecture or a different training subset, ensuring that they possess unique knowledge and capabilities.

$\left(2\right)$

Teacher model integration: in multi-teacher distillation, weighted averaging of the prediction results is usually adopted to ensure more accurate knowledge transfer.

$\left(3\right)$

Student model training: The student model is trained to mimic the outputs of multiple teacher models as closely as possible. In classification tasks, the cross-entropy loss function is commonly used to measure the consistency between the outputs of the student model and the teacher models so as to achieve efficient knowledge transfer.

Compared with the traditional knowledge distillation that relies on a single teacher model, multi-teacher distillation can provide more diverse and robust knowledge, thereby significantly improving the performance and generalization ability of the student model [37,38,39]. When dealing with complex tasks and large-scale model training, multi-teacher distillation shows excellent results. It not only optimizes the efficiency of knowledge transfer but also enhances the adaptability and robustness of the model in the face of different data and tasks, demonstrating its wide application potential and important value in the field of deep learning.

2.4. The Basic Concept of Information Entropy

Here, we present some basic concepts of information entropy. Let v be a random variable with k possible values. $D\left(v\right)=\{v_1, v_2, \dots , v_k\}$ is the set containing all possible values of the variable. If v is a discrete variable, its information entropy is defined as

$$H\left(\nu \right)=-\sum _{i=1}^{k}P\left({\nu }_i\right)\log P\left({\nu }_i\right)$$

(8)

where $P\left({\nu }_i\right)$ is the probability of event $v=v_i$ occurring. If v is a continuous variable, then the information entropy is defined as

$$H(\nu )=-\int f(\nu )\log f(\nu )d\nu$$

(9)

where $f\left(v\right)$ is the probability density function of v; the smaller the value of $H\left(\nu \right)$, the more certain the value of this variable.

Conditional entropy quantifies the uncertainty of a random variable, assuming the value of another random variable is known. Let u be a random variable with $k^{\prime }$ possible values and $D\left(u\right)=\{u_1, u2, \dots , u_{k^{\prime }}\}$ be the set containing all possible values of this variable. At this time, the conditional entropy of v given u can be defined as

$$H\left(\nu \right|u)=\sum _{j=1}^{k^{\prime }}P\left(u_j\right)H\left(\nu \right|u_j)$$

(10)

If v is discrete variable, it is defined as

$$H\left(\nu |u_j\right)=-\sum _{i=1}^{k}P\left({\nu }_i|u_j\right)\log P\left({\nu }_i|u_j\right)$$

(11)

If v is continuous, it is defined as

$$H\left(v|u_j\right)=-\int f\left(v|u_j\right)\log f\left(v|u_j\right)dv$$

(12)

Therefore, information entropy can be used to evaluate the consistency of an object set over a given feature set. The more objects have the same or similar feature values, the more certain these feature values of the objects.

3. Regression Method Based on Multi-Teacher Distillation

3.1. Method Overview

Traditional single-teacher networks may encounter bottlenecks in generalization ability and expressiveness when guiding student networks [40]. However, when adopting the multi-teacher distillation strategy, we may face challenges such as knowledge conflicts and high resource consumption [41]. To overcome these limitations, we propose an innovative method: using the idea of clustering ensemble to accurately cluster and efficiently integrate multi-teacher networks, ensuring that teacher networks with similar structures can be clustered into the same category. This strategy helps reduce model redundancy and improve the efficiency of knowledge transfer to form a more powerful guiding model. To further optimize the application of the multi-teacher distillation method in regression problems, we introduce a new loss function, aiming to solve the knowledge transfer problems faced by multi-teacher network distillation in regression tasks and improve the accuracy and stability of the student model when dealing with complex regression problems.

In the entire knowledge distillation process, first, we pre-train N teacher networks. Then, we perform clustering ensemble on the pre-trained N teacher networks. The clustering method is explained in detail in Section 3.2. Through our method, the N teacher networks are clustered into k clusters. In each cluster, we select the top b teacher networks for weighted averaging, thereby integrating multiple teacher networks into a small number of high-quality teacher networks, which helps improve the subsequent distillation efficiency. Next, the predicted outputs of the aggregated k teacher networks are used to guide the learning process of the student network.

The detailed process is shown in Figure 3. The learning process of the student network consists of two parts. One part comes from the predicted outputs of the teacher networks, denoted as Limit, sand the other part comes from the real labels, Label, denoted as $L_{MSE}$. We use the balance factor $\alpha$ to adjust the weights of the student network for different knowledge sources. In addition, we provide

Table 1. Explanations of some symbols used in this paper.

Symbol	Description
${\lambda }_h$	The feature value of the h-th base clustering
D(v)	The domain of variable (feature) v
${\lambda }_{hl}$	The clustering label of $C_{hl}$
${\mathbb{C}}_h$	The h-th base clustering set
$C_{hl}$	The l-th cluster in ${\mathbb{C}}_h$
S ⊆ T	A cluster composed of multiple objects in T
$\mathbb{S}={\left\{S_l\right\}}_{l=1}^p$	A set composed of clusters S where $S_l$ is the l-th cluster
$\mathbb{C}$	$\mathbb{S}=\mathbb{C}$ The set composed of the objects in T, which is defined as a partition
${\mathbb{C}}_{*}$	The partition result formed by the final clusters
k	The number of clusters in partition ${\mathbb{C}}_{*}$
${\lambda }_{*}$	The final clustering feature vector

to summarize the main symbols used.

3.2. Multi-Teacher Network Clustering Ensemble Based on Information Theory

Our goal is to integrate the wisdom of multiple teacher networks to form a small number of elite teacher networks, thereby enhancing the learning efficiency of the student network. To make the clustering results more reliable, we draw on the idea of clustering ensemble, aggregating multiple clustering results to generate more robust and stable clustering outcomes.

In our approach, we draw inspiration from the information–theoretic clustering ensemble methodology [42], as illustrated in Figure 4. We consider each teacher network as embedded within two feature spaces: the original feature space and the basic clustering feature space.

First, we generate a pool of teacher networks based on the original data. Then, through similarity calculation, we form the original data feature set $T\left(A\right)$. Subsequently, using multiple clustering methods on the original data feature set, we construct the base clustering feature set $T\left(\mathrm{\Delta }\right)$. After that, we obtain the final clustering result through hierarchical clustering. Our requirement for the final clustering result is to have high consistency with the base clustering feature set. We use information entropy to evaluate the uncertainty between them to obtain high-quality clustering results. Finally, we obtain the final clustering result $T\left({\lambda }_{*}\right)$ for the subsequent weighted integration of teacher networks. The detailed definition and process of this method are presented in Figure 4.

$T\left(A\right)$ is an N × N matrix, where N is the number of teacher networks. We define it as the original data feature set. First, we present the construction processes of the original feature set and the basic clustering-feature set. For teacher networks $t_i$ and $t_j$, we flatten the weights of the linear layers in the last layer of the teacher networks into one-dimensional vectors $v_i$ and $v_j$. We use cosine similarity to evaluate the similarity between the two teacher networks:

$${cos}_{ij}=\frac{v_i·v_j}{\parallel v_i\parallel \parallel v_j\parallel }$$

(13)

In addition, we define $Di{s}_{ij}$ as representing the distance between the two teacher networks $T_i$ and $T_j$:

$$Di{s}_{ij}=1-{cos}_{ij}$$

(14)

We construct the original data feature set of teacher networks based on the distances between different teacher networks. The purpose of this is to fully consider the structural similarity among the different teacher networks when generating the basic clustering feature set. Subsequently, we select M clustering algorithms and generate M clustering results according to the original data feature set $T\left(A\right)$. These results form the base clustering feature set $T\left(\mathrm{\Delta }\right)$. $T\left(\mathrm{\Delta }\right)$ is an N × M matrix, where M represents the number of clustering methods. It denotes the base clustering feature set composed of all initial clustering results.

Next, we introduce how to define information entropy to measure the consistency between the clustering results and the base clustering feature set $\mathrm{\Delta }$. We define each ${\lambda }_h$ as a random variable, which represents the feature value formed by the h-th base clustering result. Let $D\left({\lambda }_h\right)={\left\{{\lambda }_{hl}\right\}}_{l=1}^{k_h}$, where ${\lambda }_{hl}$ is the label of the clustering result of $C_{hl}$, and $P\left({\lambda }_{hl}\right)$ is the probability that ${\lambda }_{hl}$, where ${\lambda }_h$ is the feature value of the h-th base clustering. Suppose S ⊆ T is a cluster that contains multiple objects and S is a subset of the set T. We calculate P(${\lambda }_{hl}$|S) by means of the relative frequency of the label value ${\lambda }_{hl}$ occurring in the cluster S:

$$P({\lambda }_{hl}\mid S)={\displaystyle \frac{\left|\right\{x_i\mid x_i\left({\lambda }_h\right)={\lambda }_{hl},x_i\in S\left\}\right|}{\left|S\right|}}$$

(15)

The information entropy of ${\lambda }_h$ given S is

$$H({\lambda }_h\mid S)=-\sum _{i=1}^{k_j}P({\lambda }_{hl}\mid S)\log P({\lambda }_{hl}\mid S)$$

(16)

In addition, we assume that each ${\lambda }_h$ is independent of the others. Then, $\mathrm{\Delta }$ can be regarded as a set composed of M random variables. Given cluster S, the information entropy of $\mathrm{\Delta }$ can be expressed as:

$$H(\mathrm{\Delta }\mid S)=\sum _{h=1}^{M}H({\lambda }_h\mid S)$$

(17)

Suppose $\mathbb{S}=\left\{{S_l}_{l=1}^p\right\}$ is the set of S, including p clusters, where $S_l$ is the l-th cluster, $1\le l\le p$. Therefore, according to the definition of conditional entropy, given S, the information entropy of $\mathrm{\Delta }$ is

$$H(\mathrm{\Delta }\mid \mathbb{S})=\sum _{l=1}^{p}P(S_l\mid \mathbb{S})H(\mathrm{\Delta }\mid S_l)$$

(18)

$P\left(S_l\right|\mathbb{S})$ is the probability of $S_l$ in $\mathbb{S}$, and the calculation process is as follows:

$$P(S_l\mid \mathbb{S})={\displaystyle \frac{\mid S_l\mid }{{\sum }_{l=1}^p\mid S_l\mid }}$$

(19)

$H\left(\mathrm{\Delta }\right|\mathbb{S})$ can be regarded as the sum of the consistencies of all clusters $\mathbb{S}$, which we define as the consensus sum. If the consistency of each cluster $S_l$ with respect to $\mathrm{\Delta }$ is very high, then the value of $H\left(\mathrm{\Delta }\right|\mathbb{S})$ is lower. Based on Formulas (17) and (18), we can define the consensus sum of $\mathbb{S}$ on $\mathrm{\Delta }$ as follows:

$$I\left(\mathrm{\Delta }\right|\mathbb{S})=H(\mathrm{\Delta }\left|S\right)-H\left(\mathrm{\Delta }\right|\mathbb{S})$$

(20)

We believe that if set $\mathbb{S}$ of clusters is a good set for cluster S, it can enhance the consensus of cluster S on $\mathrm{\Delta }$. The larger the $I\left(\mathrm{\Delta }\right|\mathbb{S})$, the higher the clustering consistency on $\mathrm{\Delta }$.

Given T, $H\left(\mathrm{\Delta }\right|T$) is determined. Then, the more objects in each cluster of the set $\mathbb{S}$ share the same labels in the basic clustering feature set $\mathrm{\Delta }$, the lower the value of $H\left(\mathrm{\Delta }\right|\mathbb{S}$). Therefore, clustering results with high consistency on $\mathrm{\Delta }$ are considered to have good performance. In addition, if a set of clusters $\mathbb{S}=\{S_i,S_j\}$ contains two clusters $S_i$ and $S_j$, then, according to Formula (20), we have

$$I\left(\mathrm{\Delta }\left|\right\{S_i,S_j\}\right)=H\left(\mathrm{\Delta }|S_i\cup S_j\right)-H\left(\mathrm{\Delta }\left|\right\{S_i,S_j\}\right)$$

(21)

Given $S_i$ and $S_j$, $H\left(\mathrm{\Delta }\right|S_i\cup S_j)$ is determined. If most of the objects in $S_i\cup S_j$ have the same labels on $\mathrm{\Delta }$, then $I\left(\mathrm{\Delta }\right|\left\{S_i,S_j\}\right)$ decreases significantly. At this time, we consider that these two clusters are similar on $\mathrm{\Delta }$.

The clustering-ensemble task can be regarded as a clustering operation performed on the basic clustering feature set. We use information entropy as the objective function to screen out the optimal final clustering results. Assuming $\mathbb{S}=\mathbb{C}$ is the set of clusters formed by T, where $\mathbb{C}$ is defined as a partition, we ultimately use $H\left(\mathrm{\Delta }\right|{\mathbb{C}}_{*})$ as the objective function of the clustering ensemble method and design the corresponding optimization model to perform the clustering ensemble task on the basic clustering feature set.

The formal description of the clustering ensemble optimization model is

$$\underset{{\mathbb{C}}_{*}}{min}H\left(\mathrm{\Delta }\right|{\mathbb{C}}_{*})$$

(22)

According to the above-mentioned content, we can conclude that, given T,

$$\underset{{\mathbb{C}}_{*}}{max}I(\mathrm{\Delta }\mid {\mathbb{C}}_{*})=H(\mathrm{\Delta }\mid T)-\underset{{\mathbb{C}}_{*}}{min}H(\mathrm{\Delta }\mid {\mathbb{C}}_{*})$$

(23)

According to Formula (23), we know that minimizing the objective function $H\left(\mathrm{\Delta }\right|{\mathbb{C}}_{*})$ is equivalent to maximizing the consensus sum between the final clustering results and the basic clustering feature set.

We adopt a hierarchical clustering strategy to generate the final clustering results. Define ${\mathbb{C}}^{\left(z\right)}$ as the partition generated by merging cluster $C_i^{(z-1)}$ and cluster $C_j^{(z-1)}$ at the z-th time. First, we construct an initial partition ${\mathbb{C}}^0$, where each teacher network in $T=\{t_1, t_2, \dots , t_N\}$ serves as a separate cluster; that is, ${\mathbb{C}}^{\left(0\right)}=\{C_1, C2, \dots , C_N\}$. According to Formula (18), at this time, $H\left(\mathrm{\Delta }\right|{\mathbb{C}}_0)=0$. Next, we continuously combine two partitions to form a new partition ${\mathbb{C}}^{(z+1)}$. Based on the properties of conditional entropy, we have

$$H\left(\mathrm{\Delta }\right|{\mathbb{C}}^{(z+\mathrm{l})})\ge H\left(\mathrm{\Delta }\right|{\mathbb{C}}^{\left(z\right)})$$

(24)

As the number of clusters decreases, the consensus sum of the clustering results also gradually decreases. Therefore, each time we merge partitions, we hope that the consensus sum decreases as little as possible. The optimization problem can be transformed into

$$\underset{{\mathbb{C}}^{(z+1)}}{min}\left[H(\mathrm{\Delta }\mid {\mathbb{C}}^{\left(z\right)})-H(\mathrm{\Delta }\mid {\mathbb{C}}^{(z+1)})\right]=\underset{C_i^{\left(z\right)},C_j^{\left(z\right)}\in {\mathbb{C}}^{\left(z\right)}}{min}I(\mathrm{\Delta }\mid \{C_i^{\left(z\right)},C_j^{\left(z\right)}\})$$

(25)

Therefore, each time we need to merge the two clusters $C_i^{\left(z\right)}$ and $C_j^{\left(z\right)}$ with the highest consistency in ${\mathbb{C}}^{\left(z\right)}$, the merging condition is

$$\underset{C_i^{\left(z\right)},C_j^{\left(z\right)}\in {\mathbb{C}}^{\left(z\right)}}{min}I\left(\mathrm{\Delta }\right|\{C_i^{\left(z\right)},C_j^{\left(z\right)}\})$$

(26)

When the number of clusters in ${\mathbb{C}}^{\left(z\right)}$ is k, the merging process terminates, and the final clustering result is ${\mathbb{C}}_{*}={\mathbb{C}}^{\left(z\right)}$.

The aforementioned steps effectively group multiple teacher networks, which are followed by our screening process to enhance the overall model’s performance and efficiency. We obtain the final clustering result $T\left({\lambda }_{*}\right)$ from ${\mathbb{C}}_{*}$, which ensures that the clustering result maximally reflects the similarities and differences among the teacher networks. Here, ${\lambda }_{*}$ denotes the final clustering result, where $1\le {\lambda }_{*}\le k$. Our objective is to further refine the team of teacher networks. To achieve this, we rank the teacher networks within each cluster based on their mean squared error (MSE), allowing us to clearly understand the performance of each teacher network within its cluster. Subsequently, we select the top b teacher networks from each cluster (where b is a predefined parameter, typically adjusted according to specific requirements and model characteristics). This selection aims to retain the best-performing teacher networks in each cluster while eliminating those with lower prediction accuracy, thereby ensuring the quality of the refined teacher network team.

After completing the screening of teacher networks, we determine the weighted average of the predictions from the selected teacher networks. This weighted average takes into account the performance differences among the teacher networks, assigning higher weights to those with superior performance. Based on the final clustering result, we rank the teacher networks in each cluster according to the MSE evaluation metric, aiming to further streamline the teacher network team. We sequentially select the top b teacher networks from each cluster and compute a weighted average of their predictions, ultimately generating k strong teacher networks. These strong teacher networks not only retain the advantages of multiple teacher networks but also eliminate redundant and low-performance networks. Consequently, this step provides the student network with more accurate and effective knowledge guidance, enhancing the learning efficiency and predictive performance of the student network. Our proposed method is formally described in Algorithm 1.

Algorithm 1 Multi-teacher clustering ensemble based on information theory.

Input: $T\left(A\right)$, $T\left(\mathrm{\Delta }\right)$, k Output: $D\left(O_t\right)$ Set z = 0 and produce an initial partition ${\mathbb{C}}^{\left(z\right)}={\left\{C_i^{\left(z\right)}\right\}}_{i=1}^N$ where $C_i^{\left(z\right)}=\left\{t_i\right\}$, $1\le i\le N$; while ${\mathbb{C}}^{\left(z\right)}| >k$ do    Select set two clusters $C_i^{\left(z\right)}$ and $C_j^{\left(z\right)}$ which satisfy ${min}_{ C_i^{\left(z\right)},C_j^{\left(z\right)}\in {\mathbb{C}}^{\left(\mathcal{z}\right)}}{I}_w\left(\mathrm{\Delta }\right|\{C_i^{\left(z\right)},C_j^{\left(z\right)}\})$;    Execute clustering merging $C_i^{\left(z\right)}=C_i^{\left(z\right)}\cup C_j^{\left(z\right)}$;    Update partition results ${\mathbb{C}}^{(z+1)}={\mathbb{C}}^{\left(z\right)}-C_j^{\left(z\right)}$;    The number of merges increases automatically z = z + 1; end while Get $T\left({\lambda }_{*}\right)$ corresponding to ${\mathbb{C}}^{\left(z\right)}$; Cluster the teacher networks into k clusters according to $T\left({\lambda }_{*}\right)$; while $i< k$ do    Select the top b teacher networks in the i-th cluster;    Calculate the weighted average result $O_{ti}$; end while Return $D\left(O_t\right)=\{O_{t1}, O_{t2}, \dots , O_{tk}\}$;

3.3. Knowledge Distillation

Under the framework of muti-teacher distillation, the design of the loss function is of vital importance. It not only determines the learning efficiency of the student network of the knowledge of the teacher networks but also directly affects the performance of the final network. Next, the carefully designed multi-teacher distillation loss function for the regression task is elaborated in detail.

Inspired by [43], we chose to use Limit to replace the mean squared error (MSE) used in traditional regression tasks. When the performance of the student network is better than that of the teacher network, we did not add the loss of the teacher network at this time as an additional loss of the student network. In this way, we could effectively guide the model learning.

At the same time, the loss function we propose fully considers the performance differences among multiple teacher models. We set parameter $\lambda$ to balance their impacts on the student model. This weighting strategy ensures that teacher models with excellent performance can play a greater role in the training process, thereby accelerating the learning efficiency of the student model and improving its generalization ability. The objective function we propose is as follows:

$$L_{KD}={\displaystyle \frac{1}{n}}\sum _i^n\sum _j^m\alpha {\left(O_s-y_{label}\right)}_i^2+(1-\alpha ){\lambda }_j{L}_{mt}{(O_s,O_{tj})}_i$$

(27)

$$L_{imit}\left(O_s,O_t\right)=\left\{\begin{array}{cc}{\left(O_s-O_t\right)}^2,\hfill & \mathrm{if}\left|O_s-y_{label}\right|>\mid O_t-y_{label}\mid \hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(28)

$${\lambda }_j=\left(1-{\displaystyle \frac{\left(ms{e}_j\right)}{sum\left(M\right)}}\right)$$

(29)

$$M=\{{\displaystyle \frac{1}{n}}\sum _i^n{(O_{it}-y_{label})}_j^2:j=1, \dots , k\}$$

(30)

where ${\lambda }_j$ is the set weight coefficient, $\alpha$ is the balance coefficient, n is the number of samples, k is the number of clustered teacher networks, and $M={\left\{ms{e}_j\right\}}_{j=1}^k$ is the set of mean squared errors of the clustered teacher networks.

The following presents its mathematical derivative-derivation process:

$$\begin{array}{cc}\hfill \nabla & ={\displaystyle \frac{\partial L_{KD}}{\partial W}}={\displaystyle \frac{{\displaystyle \frac{1}{n}\sum _i^N\alpha {(Wx+b-y_{label})}^2+\sum _j^k(1-\alpha ){\lambda }_j{(wx+b-O_{tj})}^2}}{\partial W}}\hfill \\ \hfill & ={\displaystyle \frac{2}{n}}\alpha x(Wx+b-y_{label})+2(1-\alpha )x\sum _j^k{\lambda }_j(wx+b-O_{tj})\hfill \end{array}$$

$$W=W_0-\eta ·\nabla L_{KD}\left(W_0\right)$$

where $W_0$ is the parameter of the independent variable, $\eta$ is the learning factor, and W is the updated $W_0$.

4. Results

4.1. Project Overview

This study focused on a concrete-faced rockfill dam located in Guizhou province, China. The dam had a crest elevation of 154 m, a crest width of 10 m, a crest length of 428.93 m, a maximum dam width of 454.085 m, an installed capacity of 90,000 kilowatts, and a total reservoir capacity of 1.325 billion cubic meters. To monitor the dam’s operational status, various sensors were installed internally to collect data and assess its performance. Horizontal displacement and settlement monitoring sensors were placed at the cross-section 0+005.000, with four monitoring lines. For a visual representation, refer to Figure 5. Monitoring data from four monitoring points (SG19, SG20, SG25, and SG26) from September 2021 to May 2024 were selected for modeling, with a data interval of 5 days. The dataset was divided into 80% for training and 20% for testing. During model training, the training set was further split into 0.8 for training and 0.2 for validation. The deformation data of the monitoring points are shown in Figure 6.

The dam deformation prediction model developed in this study used influencing factors such as water level, temperature, and time effects as inputs, with dam deformation as the output. A total of 10 influencing factors were selected, including water pressure factors $H^1$, $H^2$, $H^3$, and $H^4$; temperature factors $sin(2\pi t/365)$, $cos(2\pi t/365)$, $sin(2·2\pi t/365)$, and $cos(2·2\pi t/365)$; as well as time-effect factors $\theta$ and $ln\left(\theta \right)$.

4.2. Evaluation Metrics

To comprehensively evaluate the prediction accuracy of the model, various indicators are usually used. We adopted RMSE (root mean squared error), MAPE (mean absolute percentage error), MAE (mean absolute Eerror), and the coefficient of determination ($R^2$) to evaluate the model. The specific formulas are given in Equations (31)–(34):

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2}$$

(31)

$$MAPE={\displaystyle \frac{1}{n}}\sum _{i=1}^N\left|{\displaystyle \frac{y_i-{\widehat{y}}_i}{y_i}}\right|\times 100\%$$

(32)

$$MAE={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|y_i-{\widehat{y}}_i\right|$$

(33)

$$R^2=1-{\displaystyle \frac{{\displaystyle \sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2}}{{\displaystyle \sum _{i=1}^N{(y_i-{\overline{y}}_i)}^2}}}$$

(34)

where N is the total number of samples, $y_i$ is the predicted value of the i-th sample, ${\widehat{y}}_i$ is the observed value of the i-th sample, and ${\overline{y}}_i$ is the average value of the observed values of the i-th sample.

RMSE evaluates the prediction accuracy, MAPE evaluates the model’s accuracy, and MAE reflects the actual situation of the prediction error. The smaller the values of these three parameters, the higher the model’s accuracy. $R^2$ represents the fitting effect of the model, wehre the higher its value, the better the prediction performance.

4.3. Experimental Settings

We adopted the Transformer model as the teacher network, which included a position-encoding module, encoder layers, and a decoder composed of a simple linear layer. It was specifically designed for processing and predicting complex sequential data. We used a Temporal Convolutional Network (TCN) as the student network model. We selected the Temporal Convolutional Network (TCN) as the student network because it adopts a convolutional structure. Compared with traditional recurrent neural networks, convolutional operations can process data in parallel, thereby improving computational efficiency. Moreover, convolutional kernels can automatically extract local features from the data. Most importantly, the TCN has a relatively simple structure with a small number of parameters and thus has a relatively low demand for computing resources. Due to space limitations, we omit the description of the TCN and recommend readers refer to [44] for detailed information.In addition, we provide the algorithmic complexities of the Transformer and TCN in

Table 2. Complexity of teacher networks and student network.

Network	#Params	FLOPs
Transformer	87,007	8735
TCN	20.735 M	0.9928 M

. On the one hand, it served as a reference for selecting the teacher network and the student network. On the other hand, it was used to verify the degree of reduction in model complexity after distillation. We set the initial number N of teacher networks to 50, the number k of clustering categories of teacher networks to 3, and the basic clustering set M composed of basic clustering results to 15. We used three methods, namely, the classic k-means, spectral clustering, and Gaussian mixture clustering, to cluster the 50 teacher networks according to the original feature set, with the number of clusters being 5. Each method selected 5 different random seeds to obtain 5 different clustering results, and b was set to 6 during the clustering selection process. Both the teacher network and the student network used the Adam optimizer. We initialized the learning rate at 0.0001, set the batch size to 8, and set $\alpha$ to 0.8 during the knowledge distillation process.

4.4. Parameter Sensitivity Experiment

In the experiments presented in this paper, parameter $\alpha$ was incorporated into the objective function. Its purpose is to regulate the contribution of the knowledge extracted from the teacher network. The value of $\alpha$ has a significant and intuitive impact on the knowledge distillation process.Theoretically, a larger value of $\alpha$ means that the teacher network has more influence on the distillation process. Conversely, when $\alpha$ is smaller, the student network places more emphasis on its own learning ability during the distillation. In the experiments in this study, to comprehensively and thoroughly investigate the impact of parameter $\alpha$ on the model’s performance, we selected a set of representative values from the set {0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 0.9, 0.99}. To more intuitively demonstrate the relationship between the model’s performance and parameter $\alpha$, we plotted a trend graph showing the model’s $R_2$ (coefficient of determination; the closer the $R^2$ value to ond, the better the model fits the data) against $\alpha$, as shown in Figure 7.

We found that when $\alpha$ was set to 0.8, the model achieved the optimal accuracy. This result indicates that when $\alpha =0.8$, an ideal balance is reached between the teacher network and the student network. On the one hand, it can fully utilize the rich knowledge provided by the teacher network; on the other hand, it gives the student network enough room to exert its own learning ability, enabling the model to achieve the best performance during the knowledge distillation process. Based on this important experimental result, we made a reasonable and scientific decision. In the subsequent experiments, we fixed the value of $\alpha$ at 0.8.

4.5. Distillation Performance Evaluation

Table 3. Performance comparison among teacher networks after clustering, student network, and student network after distillation.

Dataset	Model	MSE (mm)	RMSE (mm)	MAE (mm)	MAPE	${\mathit{R}}^2$
	TeacherCL1	0.06049	0.24595	0.18483	2.36203	0.97679
	TeacherCL2	0.09914	0.31487	0.25275	3.35216	0.96196
SG19	TeacherCL3	0.10856	0.32949	0.25964	3.57259	0.95834
	Student	0.13955	0.37357	0.29399	3.88849	0.94644
	CLMTKD	0.07578	0.27527	0.21325	2.77232	0.97092
	TeacherCL1	0.00267	0.0513	0.04072	0.59478	0.96749
	TeacherCL2	0.00317	0.05634	0.04432	0.64194	0.96088
SG20	TeacherCL3	0.00507	0.07117	0.05386	0.78496	0.93758
	Student	0.00520	0.07210	0.05923	0.85757	0.93594
	CLMTKD	0.00308	0.05554	0.04265	0.62291	0.96199
	TeacherCL1	0.01446	0.12027	0.09045	1.14168	0.97692
	TeacherCL2	0.03564	0.18879	0.15237	1.93899	0.94313
SG25	TeacherCL3	0.03783	0.19449	0.15885	1.98612	0.93964
	Student	0.04463	0.21127	0.17393	2.18014	0.92878
	CLMTKD	0.01171	0.10819	0.07837	0.98788	0.98132
	TeacherCL1	0.01018	0.10090	0.08128	1.01031	0.96411
	TeacherCL2	0.01446	0.12026	0.09729	1.21837	0.94901
SG26	TeacherCL3	0.01482	0.12174	0.09554	1.17603	0.94775
	Student	0.02176	0.14753	0.12029	1.50459	0.92327
	CLMTKD	0.00766	0.08754	0.06781	0.84433	0.97298

shows the prediction results of the teacher networks after clustering, the student network, and CLMTKD on the test set. TeacherCL1,TeacherCL2 and TeacherCL3 are advanced teacher networks generated by weighted averaging the top six teacher networks after clustering. From SG19, SG20, SG25, and SG26, it can be seen that the proposed CLMTKD provides a significant improvement compared with the student network and is close to or even better than the clustering results of the advanced teacher networks at some measurement points.

Figure 8 presents a comparison between the student network and the student network after knowledge distillation. It is evident from the figure that the fitting performance of the student network significantly improved after knowledge distillation. Specifically, the distilled student network demonstrates superior fitting capability in the regression fitting curves, better capturing the distribution characteristics of the data. This indicates that the knowledge distillation process effectively enhances the performance of the student network, enabling it to learn and predict more accurately while maintaining lower complexity. This result further validates the effectiveness of the proposed CLMTKD algorithm in improving the learning efficiency and prediction accuracy of the student network.

4.6. Comparative Experiment

To compare the effectiveness of our method with traditional knowledge distillation methods for regression, we selected a total of 5 comparison methods, including single-teacher network distillation and multi-teacher network distillation. Methods such as AIL [45], TBR [46], and TOR [47] use a single teacher network for knowledge distillation. Since the research on multi-teacher distillation models for regression is still in its infancy, we conducted two comparative experiments. The first method was forming a strong teacher network by weighted averaging all teacher networks and then selecting the regression loss of the AIL method to guide the learning of the student network (MTKD). The second method involved using all teacher networks simultaneously and selecting the regression loss in Formula (28) to guide the learning of the student network (ALLMTKD).

According to the comparison of the experimental results in

Table 4. Comparison with other benchmark models.

Dataset	Model	MSE (mm)	RMSE (mm)	MAE (mm)	MAPE	${\mathit{R}}^2$
SG19	AIL	0.11798	0.34348	0.26597	3.25020	0.954724
	TBR	0.09040	0.30066	0.23679	3.07238	0.965308
	TOR	0.11135	0.33638	0.27738	3.53747	0.95658
	MTKD	0.08714	0.29520	0.22549	2.94286	0.96656
	ALLMTKD	0.29588	0.54394	0.47393	6.32527	0.88645
	CLMTKD	0.07578	0.27527	0.21325	2.77232	0.97092
SG20	AIL	0.00534	0.07307	0.05780	0.83995	0.93421
	TBR	0.00423	0.06504	0.05466	0.79410	0.94788
	TOR	0.00432	0.05848	0.04533	0.66193	0.95786
	MTKD	0.00398	0.06305	0.04952	0.71876	0.95102
	ALLMTKD	0.01245	0.11157	0.10012	1.44319	0.84661
	CLMTKD	0.00308	0.05554	0.04265	0.62291	0.96199
SG25	AIL	0.04095	0.20237	0.16139	1.96994	0.93465
	TBR	0.02605	0.16139	0.12953	1.61069	0.95844
	TOR	0.02077	0.14413	0.11386	1.41403	0.96685
	MTKD	0.01336	0.11561	0.08560	1.07310	0.97867
	ALLMTKD	0.10177	0.31900	0.27808	3.44080	0.83762
	CLMTKD	0.01171	0.10819	0.07837	0.98788	0.98132
SG26	AIL	0.02030	0.14246	0.11936	1.47008	0.92845
	TBR	0.01038	0.10186	0.07854	0.97147	0.96342
	TOR	0.01212	0.11010	0.08304	1.02446	0.95727
	MTKD	0.00932	0.09648	0.07726	0.96672	0.96718
	ALLMTKD	0.05261	0.22937	0.19274	2.32723	0.81451
	CLMTKD	0.00766	0.08754	0.06781	0.84433	0.97298

, it can be found that the proposed CLMTKD achieves ideal results on the four datasets. Figure 9 is the radar chart of all algorithms, where the larger the area, the better the performance. It is not difficult to find that our method outperforms the other methods on all indicators.

Further analyzing the experimental results, compared with traditional single-teacher distillation networks (AIL, TBR, TOR), we find that relying solely on a single teacher network to transmit knowledge to students has a single source and limited features that it can cover. In contrast, our method aggregates the advantages of multiple teacher networks through clustering and integrated selection, ensuring performance while maintaining model diversity. This enables the student network to learn more comprehensive and rich knowledge, effectively demonstrating the necessity of introducing multi-teacher networks in knowledge distillation. At the same time, our method dynamically sets weights according to the performance of each teacher network, meaning that more capable teacher networks have more say in the knowledge transfer process, allowing the student network to learn more from the “high-quality information” of its outputs. However, single-teacher networks do not have such a mechanism for dynamically adjusting weights according to performance.

Compared with traditional multi-teacher networks, our method also achieves performance improvements. MTKD uses multiple teacher networks to form a single teacher network through weighted averaging. However, it is difficult to effectively distinguish and utilize the truly valuable parts in multi-teacher networks, and, instead, low-quality or repetitive information may interfere with the learning of the student network. Compared with ALLMTKD, which directly allows all teacher networks to guide students, due to the large number of teacher networks, the guidance weights are overly dispersed, meaning that the influence of each teacher network on the student network is very weak, and it is difficult to effectively transmit their own knowledge to the student network. The student network is likely to be lost in the large amount of weak and messy guiding knowledge, resulting in poor learning effects, effectively demonstrating the importance of directly introducing clustering and integrated selection.

The results of the time and performance comparison of different models are shown in Figure 10. We can see that the training time of the single-teacher distillation method is shorter than that of multi-teacher networks. The main reason is that the multi-teacher distillation process needs to integrate knowledge from multiple teachers. Therefore, it usually requires a longer training time than the single-teacher distillation method. ALLMKD has the longest training time because multiple teacher networks participate in the guidance simultaneously, resulting in chaos in the knowledge transfer process, greatly increasing the training cost, and achieving the worst performance. This also demonstrates that the number of teacher networks in the distillation process is not the more, the better. The training time of the proposed CLMTKD is longer than that of MTKD. The reason is that we introduce a clustering and integration method, which consumes a part of the training time, but we achieve the best distillation results.

5. Conclusions

This paper proposed a multi-teacher distillation method based on clustering integration for dam displacement prediction. We discussed the advantages of multi-teacher networks over single-teacher networks in the knowledge distillation process. On this basis, a clustering selection and integration algorithm for multi-teacher networks was proposed to address issues such as the confusion and inefficiency caused by multi-teacher networks during the distillation process. Knowledge is transferred from the teacher network to the student network by minimizing the loss function. The experiments on all datasets showed that our model could effectively compress the model while improving the accuracy of the student model. Further analysis showed that we could significantly improve students’ performance, and the prediction results were more consistent with the true values. In conclusion, the method we proposed can effectively improve the prediction accuracy, capture the subtle changes in dam deformation in advance, provide strong data support for dam safety management, reduce the risk of safety accidents, and ensure the safety of the lives and property of people downstream as well as the normal operation of water conservancy facilities. At present, this method was only applied to concrete face rockfill dams. In the future, we will consider exploring this framework for use with more general infrastructure such as concrete gravity dams and arch dams.