Integrating InSAR Data and LE-Transformer for Foundation Pit Deformation Prediction

Abstract

The rapid development of urban infrastructure has accelerated the construction of large foundation pit projects, posing challenges for deformation monitoring and safety. This study proposes a novel approach integrating time-series InSAR data with a multivariate LE-Transformer model for deformation prediction. The LE-Transformer model integrates Long Short-Term Memory (LSTM) to capture temporal dependencies, Efficient Additive Attention (EAA) to reduce computational complexity, and Transformer mechanisms to model global data relationships. Deformation monitoring was performed using PS-InSAR and SBAS-InSAR techniques, showing a high correlation coefficient (0.92), confirming the reliability of the data. Gray relational analysis identified key influencing factors, including rainfall, subway construction, residential buildings, soil temperature, and hydrogeology, with rainfall being the most significant (correlation of 0.838). These factors were incorporated into the LE-Transformer model, which outperformed univariate models, achieving a mean absolute percentage error (MAPE) of 2.5%. This approach provides a robust framework for deformation prediction and early warning systems in urban infrastructure projects.

1. Introduction

In the process of urban and rural development, the expansion and modernization of infrastructure have led to a significant increase in the number of large foundation pit projects. The safety of these foundation pit projects plays a pivotal role in maintaining societal stability, as any safety incidents could result in project delays, economic losses, and even severe harm to the surrounding environment and personnel [1]. Traditional monitoring methods for foundation pit projects, such as total stations, inclinometers, and theodolites, face several limitations [2,3]. These limitations include a restricted number of monitoring points, which prevent comprehensive coverage of the entire foundation pit area, and an inability to provide a full and accurate depiction of deformation patterns. Furthermore, these methods are often affected by baseline point shifts during excavation, leading to compromised monitoring accuracy and timeliness [4]. Therefore, there is an urgent need for new monitoring techniques that can enhance monitoring efficiency and overcome the limitations of current technologies. In contrast, InSAR technology provides large-scale, high-precision surface deformation monitoring, making it a promising tool for addressing these challenges. Many researchers have utilized InSAR technology for deformation monitoring [5,6]. For example, Jinhua Zhang et al. employed the PS-InSAR method to acquire ground settlement data along the Shanghai subway lines and validated these results with leveling measurement data. Their findings demonstrated good agreement between the two datasets, highlighting the reliability of InSAR-based monitoring [7].

Surface deformation prediction is critical for determining the development trends and deformation modes of foundation pit projects [8]. Traditional prediction methods include empirical models, semi-empirical models, numerical models, and artificial neural network models [9]. For instance, Juan Du et al. utilized multiple neural network models combined with various influencing factors to predict landslide displacement in the accumulation layers of the Three Gorges Reservoir area. However, these models are only applicable to regions with well-defined deformation mechanisms and are unsuitable for complex study areas [10].

With the development of deep learning methods, models such as LSTM and Transformer have demonstrated outstanding performance in areas like computer vision and time-series forecasting [11,12,13]. For example, Chen Y et al. studied the Hong Kong International Airport and used the LSTM model to predict deformation, comparing the results with InSAR data. While the LSTM model performed well at the point-level scale, it did not account for influencing factors such as meteorological conditions [14]. The Transformer model, however, has proven to be an effective solution to this issue, demonstrating excellent performance in time-series prediction tasks due to its efficient parallel computation capabilities and self-attention mechanism. Mukesh Kumar Bharti et al. focused their research on Transformer models, exploring multivariate time-series prediction and achieving superior results [15]. Jing Wang et al. proposed a deep transformer-based model for InSAR time-series deformation prediction to accurately forecast surface deformation trends around Salt Lake in the permafrost areas of the Qinghai–Tibet Plateau [16]. Therefore, Transformer models can be leveraged for multi-feature pit deformation prediction and deformation mechanism analysis.

Previous research has primarily focused on InSAR time-series deformation prediction using GRU or LSTM models. Although Transformer-based models have demonstrated strong capabilities in InSAR time-series deformation prediction, they still require improvements. A key limitation of the Transformer model is its computational complexity and sensitivity to long-term dependencies, which impacts both the accuracy and efficiency of predictions [17,18]. This issue becomes particularly prominent in the ability to evaluate deformation trends in foundation pit areas, where long-term forecasting accuracy is critical. To address this gap, we propose the LE-Transformer model, which enhances time-series forecasting performance by integrating the strengths of LSTM, Efficient Additive Attention (EAA), and Transformer mechanisms. This model improves computational efficiency by optimizing time and space complexity, refining self-attention computation, and enhancing feature extraction capabilities, resulting in more efficient and accurate long-sequence data predictions. The proposed LE-Transformer model is applied to InSAR time-series deformation prediction for foundation pit deformation in the Nansha District of Guangzhou. The model incorporates environmental factors such as rainfall, subway construction, and hydrogeology to forecast deformation trends, providing a valuable tool for risk assessment and management in foundation pit engineering.

2. Materials and Methods

2.1. Study Area

This study focuses on the 220 kV Port Substation foundation pit project in Nansha District, Guangzhou, located at approximately 113°34′33″E and 22°46′14″N (Figure 1). The foundation pit is part of the region’s power infrastructure, influenced by various geological and environmental factors, including soil composition, groundwater conditions, rainfall, and nearby human activities like subway construction and residential buildings.

The geological conditions consist of a mixture of sand, clay, and peat. Peat and clay layers, with lower shear strength, are particularly prone to deformation when saturated. Sand provides some resistance, but its interaction with clay and peat leads to significant surface settlement, especially during heavy rainfall.

Environmental factors, such as rainfall and groundwater fluctuations, play a major role in soil behavior. Groundwater levels rise during the rainy season, leading to soil saturation, which weakens its strength and accelerates deformation. In the dry season, lower groundwater levels reduce moisture content, increasing soil temperature and causing shrinkage and settlement.

Fluctuating groundwater levels are a key risk factor for foundation pit stability. High groundwater levels weaken the soil, while low levels cause consolidation, but also increase brittleness, leading to cracking and damage. Monitoring these fluctuations is crucial to mitigate their impact.

Human activities, such as subway construction and residential buildings, add extra load to the soil, further contributing to deformation. The interaction of these factors creates a complex environment, requiring careful monitoring and analysis for surface stability management.

2.2. Dataset

The study used C-band Sentinel-1A imagery from the European Space Agency (ESA) as the primary Synthetic Aperture Radar (SAR) data source. A total of 72 SAR images were collected between 20200108 and 20240907, with a time interval of 24 days between each acquisition. The images were captured in IW mode with VV polarization and an incidence angle of 39.73°. PixelSpacingRg was 2.33 m, and PixelSpacingAz was 13.95 m. The study incorporated Precise Orbit Data (POD) to enhance satellite positioning accuracy, ensuring precise alignment of the SAR images with ground coordinates. To mitigate the impact of terrain-induced phase variations, a 30 m resolution Digital Elevation Model (DEM) generated from ALOS PALSAR data was applied. Additionally, the study incorporated tropospheric delay correction products from GACOS.

Beyond the primary SAR data, a range of complementary datasets were integrated to enhance the understanding of the factors influencing foundation pit deformation. These included subway construction data, which provided information on the spatial distribution of subway lines, the locations of subway stations, and other related factors. Residential building data were also incorporated, offering insights into the types, heights, and spatial distribution of buildings surrounding the pit. Rainfall data, obtained from the Nansha District meteorological station, were used to explore the relationship between precipitation and deformation, particularly during the rainy season. Soil temperature data, collected from monitoring stations within and around the foundation pit, were averaged on a monthly basis to assess the effects of temperature variations on soil behavior. Finally, hydrogeological data from local geological surveys provided crucial information on groundwater levels, water content, and other subsurface conditions, all of which were essential for a comprehensive analysis of the deformation process.

2.3. Deformation Monitoring and Model Development

Figure 2 illustrates the comprehensive workflow followed in this study, which consists of three main stages: InSAR data processing, impact factor extraction, and model development and forecasting. Initially, SAR images, DEM data, and POD are collected and processed using SARscape 5.7, with PS-InSAR and SBAS-InSAR techniques to extract deformation features. Then, ArcGIS 10.8 is used for visualization analysis and accuracy verification to ensure reliability. In the second stage, impact factors such as subway construction, housing construction, soil temperature, and rainfall, along with hydrogeological data, are analyzed through gray correlation analysis to evaluate their association with deformation. Finally, the data undergo model development, which includes preprocessing, model architecture design, and hyperparameter tuning, followed by training, inference, and deformation prediction. A subsequent evaluation assesses the model’s accuracy and forecasting capability.

2.3.1. InSAR Technology

(1)

PS-InSAR

The PS-InSAR technology was proposed by Ferretti et al. in 2001 [19]. By analyzing the amplitude and phase information of the images, it identifies permanent scatterers (PS), such as rocks and buildings, with high coherence and strong scattering characteristics in long time-series. The core principle is to establish a differential interferometric phase model for PS points, separating the temporal and spatial characteristics of each phase component, thereby obtaining both linear and nonlinear deformation information for each point. This study adopts an enhanced PS-InSAR technology, which performs adaptive filtering on the interferometric phase to more accurately identify PS and distributed scatterer (DS) pixels. This method effectively extracts surface deformation information, improving the accuracy and coverage of ground deformation monitoring [20,21].

Acquire N + 1 SAR images covering the same region, using one of them as the reference image. After registration, resampling, atmospheric correction, and removal of flat earth and topographic phases, N interferograms are obtained. The interferograms are then subjected to terrain correction using DEM and satellite orbit data. The phase at position x in the i-th interferogram, denoted as ${\varphi }_{x,i}$, can be expressed as follows:

$${\varphi }_{x,i}={\varphi }_{\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{o},x,i}+{\varphi }_{\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{o},x,i}+{\varphi }_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t},x,i}+{\varphi }_{\mathrm{s}\mathrm{l}\mathrm{o}\mathrm{p}\mathrm{e},x,i}+{\varphi }_{\mathrm{a}\mathrm{t}\mathrm{m},x,i}+{\varphi }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},x,i}$$

(1)

where the sum of the three components, ${\varphi }_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t},x,i}$, ${\varphi }_{\mathrm{s}\mathrm{l}\mathrm{o}\mathrm{p}\mathrm{e},x,i}$, and ${\varphi }_{\mathrm{a}\mathrm{t}\mathrm{m},x,i}$, is referred to as the atmospheric phase shift (APS). These components represent the constant phase value, the linear phase shift induced by atmospheric effects, and the nonlinear atmospheric effects, respectively. ${\varphi }_{\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{o},x,i}$ and ${\varphi }_{\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{o},x,i}$ represent the ground deformation phase along the radar line of sight and the terrain phase caused by DEM errors, while ${\varphi }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},x,i}$ represents the noise phase [22,23].

The acquired images are first subjected to differential interferometric processing and registered to the master image from 21 May 2022 (Figure 3). Then, adaptive filtering is applied to identify the DS points from the interferogram. High amplitude pixels are selected as candidate PS points using the amplitude deviation index. Subsequently, the linear deformation rate and elevation errors are inverted based on the linear deformation model. The residual errors are then decomposed to obtain nonlinear deformation information, ultimately providing surface deformation data. The specific process is shown in Figure 4.

(2)

SBAS-InSAR

The SBAS-InSAR technique was proposed by Berardino et al. [24]. Its core idea is to group all SAR images into small baseline sets based on spatial and temporal baselines, ensuring that the baseline distances within each subset are small. Ground surface deformation time-series within each subset are computed using the least squares method, and the singular value decomposition (SVD) method is employed to address possible ill-posed parameter solutions between subsets. By combining multiple interferograms, a linear deformation model is established based on high-coherence points, allowing the inversion of linear deformation rates and elevation errors. Residual errors are then decomposed to obtain nonlinear deformation information. This study adopts an enhanced SBAS-InSAR technique, integrating PS and SBAS methods, and simultaneously processes both PS and DS pixels. This approach enables the measurement of nonlinear displacement trends in both PS and DS pixels, providing more comprehensive surface deformation information [25,26].

Obtain N + 1 SAR images covering the same area, select one image as the reference image, and then perform registration, resampling, and removal of flat earth and topographic phases. Based on reasonable temporal and spatial baseline thresholds, M interferograms are generated, where M interferometric pairs must satisfy the inequality (N + 1)/2 < M < N(N + 1)/2. The phase value of the pixel (x, y) in the i-th (i = 1, …, M) differential interferogram is given by the following:

$${∆\varphi }_x(x,y)=\varphi (t_B,x,y)-\varphi (t_A,x,y)\approx ∆{\varphi }_{\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{o},(x,y),i}+∆{\varphi }_{\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{o},(x,y),i}+∆{\varphi }_{\mathrm{a}\mathrm{t}\mathrm{m},(x,y),i}+∆{\varphi }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},(x,y),i}$$

(2)

where t_A and t_B (t_A > t_B) represent the acquisition times of the SAR images corresponding to the i-th interferogram, $∆{\varphi }_{\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{o},(x,y),i}$ is the deformation phase component corresponding to the displacement in the slant range direction between t_B and t_A, $∆{\varphi }_{\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{o},(x,y),i}$ is the topographic phase error, $∆{\varphi }_{\mathrm{a}\mathrm{t}\mathrm{m},(x,y),i}$ is the atmospheric phase error, and $∆{\varphi }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},(x,y),i}$ is the noise phase error [27,28].

The baseline estimation of the image is first performed, using the image from 21 May 2022, as a master image (Figure 5). It is then registered and resampled with all other images. After applying terrain flattening, interferometry, and phase unwrapping, differential interferograms and unwrapped phase images are obtained. Subsequently, residual phases are removed through orbit refinement and further terrain flattening. A linear deformation model is constructed based on high-coherence points, and the linear deformation rate and elevation errors are inverted. The residual errors are decomposed to extract nonlinear deformation information. Surface deformation time-series data are obtained using the SVD method. The flowchart of this process is shown in Figure 6.

2.3.2. The Gray System Theory

The gray system theory, first proposed by renowned scholar Professor Deng Julong, is a scientific theory that includes various analysis methods. One such method is gray relational analysis, which performs quantitative analysis on the development of dynamic processes. By comparing the geometric relationships of statistical data within the system, the method determines the gray relational degree between a reference sequence and each comparative sequence. Finally, based on the magnitude of the relational degrees, the evaluation indicators are ranked. The specific steps are as follows.

First, assume that {x_t(0)} is the reference sequence (also known as. the parent sequence), {x_t(i)} is the comparison sequence (also known as. the subsequence), (t = 1, 2, …, n, i = 1, 2, …, m), constructs the matrix x.

$$x_i^{\left(0\right)}=\left[\begin{array}{cccc}{x}_1\left(0\right)& x_1\left(1\right)& \cdots & x_1\left(m\right)\\ x_2\left(0\right)& x_2\left(1\right)& \cdots & x_2\left(m\right)\\ ⋮& ⋮& \cdots & ⋮\\ x_n\left(0\right)& x_n\left(1\right)& \cdots & x_n\left(m\right)\end{array}\right]$$

(3)

where i = 0, 1, 2, …, m, are the labels of the reference factor (i = 0) and the m comparison factors; t = 1, 2, …, n, are the sample numbers; and x_t(i) is the dimensionless quantized function value of the ith indicator for the tth sample point.

$${\epsilon }_t\left(i\right)={\displaystyle \frac{△min+\rho △max}{{△}_t\left(i\right)+\rho △max}}$$

(4)

where ${\epsilon }_t\left(i\right)$ is the correlation coefficient between the i-th comparison sequence and the reference sequence at the t-th sample point; and $\rho$ is the discrimination coefficient, with a value in the range of (0, 1), and usually takes the value of 0.1 to 0.5.

$${△}_t\left(i\right)=\mid x_t\left(i\right)-x_t\left(0\right)\mid$$

(5)

$${△}_{max}=\underset{t}{max}\left\{\underset{i}{max}\left|x_t\left(i\right)-x_t\left(0\right)\right|\right\}$$

(6)

$${△}_{min}=\underset{t}{min}\left\{\underset{i}{min}\left|x_t\left(\mathrm{i}\right)-x_t\left(0\right)\right|\right\}$$

(7)

The correlation coefficient ${\epsilon }_t\left(i\right)$ obtained from the above calculation and the correlation between the comparison sequence and the reference sequence is calculated by the following equation:

$$r\left(i\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\epsilon }_t\left(i\right)$$

(8)

where $r\left(i\right)$ is the correlation between the i-th comparison sequence and the reference sequence [29,30].

2.3.3. Surface Deformation Prediction Model

(1)

LSTM

LSTM (Long Short-Term Memory) improves upon traditional RNNs by effectively addressing the issues of gradient explosion and vanishing gradients, making it particularly successful in the field of time-series prediction [31,32]. As shown in Figure 7, LSTM introduces three gating mechanisms—forget gate, input gate, and output gate—along with a memory cell to regulate the flow of information. These mechanisms enable LSTM to selectively retain or discard information, enhancing the model’s ability to process and learn from long-sequence data. This architecture empowers LSTM to capture both short-term and long-term dependencies, making it a robust tool for sequential data modeling [33].

The data are calculated in the forget gate as shown in Equation (9).

$$f_t=\sigma \left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right)$$

(9)

where σ is the sigmoid activation function. h_t−1 and x_t are the hidden state and input vector at the moment t − 1, respectively, and W_f, as well as b_f, represent the weight value and bias value, respectively. When the value of f_t output is close to 0, it means that the previous data are forgotten, while when it is close to 1, it indicates that the previous data are memorized.

The data are calculated in the input gate as shown in Equations (10) and (11).

$$i_t=\sigma \left(W_i\cdot \left[h_{t-1},x_t\right]+b_i\right)$$

(10)

$${\tilde{C}}_t=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(W_C\cdot \left[h_{t-1},x_t\right]+b_c\right)$$

(11)

where σ is the sigmoid activation function. W_i and W_c are the updated weights, b_i is the bias of the input gate, and b_c is the deviation.

After the computation of the forget gate and the input gate is completed and the memory cell is updated, we have the following:

$$C_t=f_t\times C_{t-1}+i_t\times {\tilde{C}}_t$$

(12)

The data are first output at the output gate via the sigmoid layer output O_t, which is used to determine which parts of the cell state values will be output. The cell state is then processed by the tanh function, by which operation of the cell state values are scaled to between −1 and 1. Finally, the result is output by multiplying it with the output of the sigmoid gate. The calculation formula is given below.

$$o_t=\sigma \left(W_o\left[h_{t-1},x_t\right]+b_o\right)$$

(13)

$$h_t=o_t\times \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(C_t\right)$$

(14)

where h_t represents the new output value and W_o and b_o are the deviations of the updated weights and of the output gate, respectively [34,35].

(2)

Transformer

The Transformer is an encoder–decoder architecture, with both the encoder and decoder consisting of multiple stacked Transformer blocks [36,37,38]. At the heart of the Transformer module lies the multi-head attention mechanism, which is composed of multiple independent and parallel attention mechanisms. This design allows the model to focus on information from different subspaces, enabling the attention mechanism to capture the multidimensional features of input data effectively [39,40,41]. Additionally, the Transformer leverages layer normalization (LN) and skip connections to enhance training efficiency and predictive performance. These structural innovations contribute to the model’s robustness and ability to handle complex data dependencies [42,43]. The core computational formula of the Transformer model is as follows:

$$\mathrm{Attention}\left(\mathit{Q},\mathit{K},\mathit{V}\right)=\mathrm{softmax}\left(\mathit{Q}{\mathit{K}}^{\mathit{T}}/\sqrt{{\mathit{d}}_{\mathit{K}}}+\mathit{B}\right)\mathit{V}$$

(15)

$$\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_i=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$$

(16)

$$\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}\left(Q,K,V\right)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_1,\cdots ,\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_h\right)W^O$$

(17)

where Q, K, and V represent query vector, key vector, and value vector, respectively; W^Q, W^K, W^V, and W^O are trainable convolutional kernels; head_i denotes the ith attention head, where i = 1, 2, …, h; h is the number of attention heads; $\sqrt{{\mathit{d}}_{\mathit{K}}}$ denotes the dimensionality of the key vectors; B is the relative positional bias, which serves to enhance the performance of the network model; softmax denotes the activation function; Concat denotes the splicing function; Attention denotes the attention function; and MultiHead denotes the multi-head self-attention function [44,45]. Figure 8 shows the overall structure of the Transformer network model.

(3)

Efficient Additive Attention

Efficient Additive Attention (EAA) is the core technique employed to improve model performance in this study. Its key innovation lies in replacing traditional quadratic matrix multiplication with linear element-level multiplication to reduce computational complexity. In conventional Transformer models, the self-attention operation is computationally expensive due to the quadratic complexity (O(n²)) with respect to the input sequence length. EAA introduces an additive attention mechanism to compute the global context through the interaction of the query and key matrices via linear transformation, thus reducing this high computational cost.

In the EAA mechanism, the input embedding matrix is first transformed into query (Q) and key (K) matrices using separate learnable weight matrices W_q and W_k. The query matrix is then multiplied by a learnable vector W_a to obtain the attention weight of the query, generating a global attention query vector. This attention vector is pooled to produce a single global query vector, which is then used to encode the interaction between the query and key matrices through element-wise multiplication to form a global context. This global context can effectively capture the global information while maintaining linear complexity in relation to the sequence length, making it more efficient than the multi-head self-attention (MHSA) mechanism, which has quadratic complexity.

Specifically (see Figure 9), the input embedding matrix x is transformed into a query matrix Q and a key matrix K via two matrices W_q and W_k, where Q, K ∈ R^n×d and W_q, W_k ∈ R^d×d, n is the labeling length and d is the dimension of the embedding vector. Then, the query matrix Q is multiplied by the learnable parameter vector W_a ∈ R^d to learn the attention weight of the query, generating the global attention query vector α ∈ Rⁿ, computed as follows:

$$Q=x{W}_q, K=x{W}_k$$

(18)

$$\alpha ={\displaystyle \frac{Q\cdot W_a}{\sqrt{d}}}$$

(19)

A pooling operation on the query matrix based on the learned attention weights yields a single global query vector q ∈ R^d, i.e., the following:

$$q={\sum }_{i=1}^n{\alpha }_i \ast Q_i$$

(20)

The interaction between the global query vector q and the key matrix K is encoded through element-wise multiplication, and the resulting context vector is then transformed using a linear transformation to form the global context C.

$$C=K\cdot q$$

(21)

The interaction between the global query vector q ∈ R^d and the key matrix K ∈ R^n×d is encoded using element-by-element multiplication to form the global context R^n×d. This matrix is similar to the attention matrix in MHSA in terms of its ability to capture information from each token and its flexibility to learn the relevance of the input sequence. It is computationally less expensive compared to MHSA and has linear complexity with respect to the marker length [46]. Inspired by the Transformer architecture, a hidden representation of tokens is learned using a linear transformation layer employed for query key interactions. The output of Efficient Additive Attention can be described as follows:

$$\hat{x}=\hat{\mathit{Q}}+T\left(K \ast q\right)$$

(22)

where $\hat{\mathit{Q}}$ denotes the normalized query matrix and T denotes the linear transformation.

The LE-Transformer model, enhanced with the EAA mechanism, predicts foundation pit deformation using time-series InSAR data. By integrating environmental factors such as rainfall, subway construction, and soil temperature, the model effectively captures both local and global contexts. The EAA mechanism enables the model to process large data sequences quickly, making it suitable for real-time foundation pit stability monitoring. By replacing traditional attention mechanisms, the LE-Transformer improves prediction accuracy and reduces computational overhead, providing a robust framework for foundation pit deformation prediction and early warning systems in urban infrastructure projects.

(4)

LE-Transformer Model Architecture

The LE-Transformer network is a deep learning architecture that integrates LSTM, Efficient Additive Attention (EAA), and Transformer modules, designed to process complex sequential data and perform effective feature extraction and prediction. As illustrated in Figure 10, the architecture combines the strengths of its three core components. Below is a detailed description of the network architecture.

(1)

Data preprocessing module: The data preprocessing module normalizes the input data to a range between 0 and 1, improving the training efficiency and accuracy of the model. This operation ensures that the data can be effectively handled in subsequent modules, laying the foundation for accurate feature learning by the model.

(2)

LSTM module: The LSTM module initializes hidden states and cell states before performing forward propagation through LSTM layers. It outputs features that contain rich temporal information about the input sequences, providing a robust representation for the subsequent attention mechanism.

(3)

EAA Module: The EAA module transforms the input embedding matrix into query and key matrices through specific matrix operations, followed by normalization. The query matrix is multiplied by a learnable vector to generate attention weights, producing a global attention query vector. This query vector is pooled into a single global query vector, and element-wise multiplication is applied to encode its interaction with the key matrix to form a global context. EAA replaces traditional matrix multiplications with linear operations, reducing computational complexity. By precisely focusing on key factors in multivariate data, this module provides weighted features to the Transformer module, enabling the model to capture critical information and enhance the accuracy of pit deformation prediction.

(4)

Transformer module: The Transformer module processes the weighted features provided by the EAA module. Using its multi-head self-attention mechanism, it captures global dependencies in the data. The module further employs a feedforward network for nonlinear transformations, while residual connections and layer normalization help mitigate gradient issues during the training of deep networks.

(5)

Output layer: The features processed by the Transformer module are passed to a fully connected layer. This layer adjusts weight based on the training data, mapping the input features to the final predicted values. The output layer generates accurate predictions for pit deformation.

Figure 10. LE-Transformer model architecture.

Figure 10. LE-Transformer model architecture.

To update the parameters of a deep learning network model, the loss function is utilized to evaluate the error between the true values and the predicted values. In this study, which focuses on time-series prediction tasks, the goal is to minimize the error between the true values and the predicted values. Therefore, the mean squared error (MSE) function is chosen as the loss function for model training. The mathematical expression for the MSE loss function is as follows:

$$M_{\mathrm{S}\mathrm{E}}={\displaystyle \frac{1}{r}}\sum _{g=1}^r{d}_g^2$$

(23)

where r is the sample size of the training samples; and d_g denotes the error between the predicted and true values.

(5)

Training and Evaluation

After constructing the LE-Transformer model, it was trained and evaluated to ensure its validity and accuracy. The dataset used in this study consisted of data related to the 220 kV Port Substation foundation pit project in Nansha District, Guangzhou, including InSAR deformation rates and multiple influencing factors. The dataset was divided into training (70%), validation (20%), and test (10%) sets. The data were preprocessed and normalized to 0–1 to improve the training efficiency, and the Adam optimizer was used to minimize the prediction and true value errors. During training, the model learns the prediction based on historical data, calculates the prediction value with forward propagation on the training set, calculates the loss with the loss function, updates the parameters with backward propagation, and calculates the loss and the validation loss on the training and validation sets at the same time to monitor the training effect and the generalization ability. After training, the model is evaluated in the test set with RMSE, MAE, MAPE, R², and other indicators, and the model prediction results are back-normalized and compared with the real values to judge its prediction ability for pit deformation.

3. Results

3.1. InSAR Monitoring Results

The foundation pit project commenced in May 2023 and was completed in December 2023. Using 72 ascending-track Sentinel-1A synthetic aperture radar images, combined with PS-InSAR and SBAS-InSAR techniques, vertical ground deformation monitoring results for the pit site from 2020 to 2024 were obtained. The final deformation results were primarily based on SBAS-InSAR monitoring, which is due to its ability to process multiple small baseline subsets of synthetic aperture radar images. This provides unique advantages in capturing the spatial variation in large-scale ground deformation. Such dense spatial coverage allows for a more comprehensive and detailed presentation of deformation patterns, which is crucial for understanding the complex deformation processes in the foundation pit and its surrounding regions, especially in areas with significant spatial heterogeneity. The ground deformation time-series was analyzed as follows.

Early Stage (January–April 2020): During the preparatory stage, the site was selected for power infrastructure construction, and the region was generally stable. The observed deformation, as shown in the SBAS-InSAR results (Figure 11a), indicated a cumulative settlement ranging from 1 to 4 mm. This minor deformation can be attributed to factors such as soil consolidation, slight groundwater table fluctuations, and initial disturbances from preparatory construction activities, including equipment movements and site clearing. Despite the absence of significant excavation work at this stage, these factors contributed to the observed surface settlement.

Middle Stage (May–December 2023): The soil composition in the pit area consisted of sand, clay, and peat, which are prone to wet collapse and deformation during rainfall. During this period, the total rainfall was 1775 mm, with distinct dry and wet seasons. The rainy season (May–October) accounted for 85% of the annual rainfall. To investigate the impact of rainfall on surface deformation, five subsidence time-series points (P1–P5) were selected within the settlement zones (Figure 11c). The relationship between settlement and monthly rainfall (Figure 11d) shows the following: All five points exhibited nonlinear settlement, which was strongly correlated with rainfall. Deformation fluctuated with rainfall, and higher rainfall led to accelerated deformation. The sudden increase in subsidence during the rainy season was primarily caused by the saturation of the soil, which significantly reduced its strength and load-bearing capacity. In particular, the peat and clay layers, which are highly susceptible to wet collapse, experienced a reduction in shear strength due to the intense rainfall. The excess moisture from the rainfall decreased the friction between soil particles, resulting in accelerated surface deformation. Additionally, soil temperature played a role in influencing the soil’s moisture retention and strength. During the rainy season, higher temperatures, particularly in May and June, increased evaporation rates, affecting the soil’s ability to retain moisture. This caused fluctuating soil moisture levels, further complicating the deformation processes. The combined effects of temperature and moisture fluctuations exacerbated the deformation, especially in the peat and clay layers, which are sensitive to both temperature and moisture changes.

While the effects of rainfall on deformation were noticeable, they were temporary. After the rainfall stopped, surface deformation gradually returned to a normal settlement rate. Notably, points P3 and P5 exhibited the fastest settlement, with maximum settlements of −32.1 mm and −16.6 mm, respectively. This rapid settlement was largely due to the combined effects of nearby subway construction, residential buildings, and the high amount of rainfall. The subway construction added additional load to the soil, compounding the deformation caused by the saturated soil. Residential buildings contributed to local settlement by exerting extra pressure on the soil. Furthermore, rainfall exacerbated the deformation by weakening the soil structure.

In the months of July and August, heavy rainfall did not cause dramatic settlement. This can be explained by the infiltration of rainwater raising the groundwater table, which temporarily reduced the pore water pressure. As a result, the soil became more stabilized, preventing substantial surface settlement in the short term.

Point P4, which exhibited slow settlement under normal conditions, showed accelerated deformation during August–September. This was largely due to the heavy rainfall during this period, which saturated the soil, and its relative distance from the subway lines and high-rise buildings. The absence of nearby structural loading (such as subway tunnels or high buildings) meant that rainfall had a more significant impact on the settlement at this point.

Following the end of the rainy season, reduced rainfall and rising temperatures during November caused higher soil temperatures and surface drought conditions. The increase in soil temperature, combined with the decrease in groundwater levels, led to accelerated evaporation and further reduction in soil moisture content. While the soil’s strength and load-bearing capacity did decline, this did not result in a re-acceleration of surface settlement. Instead, surface deformation stabilized after the rainy season, and the settlement rate gradually returned to normal levels as the soil regained some stability despite the reduced moisture content.

Late Stage (January–September 2024): During the late stage and after the completion of the foundation pit project, the cumulative deformation in the overall region decreased and tended toward stability (Figure 11b). Most of the cumulative settlement in the pit area ranged between −3 mm and 1 mm, indicating that the foundation structure provided support and constraint. Additionally, the cumulative deformation in the surrounding area also decreased and stabilized, possibly due to the foundation structure reducing the influence of external factors on pit deformation.

To evaluate the accuracy of InSAR monitoring results, a correlation analysis was performed between the monitoring outcomes of the PS-InSAR and SBAS-InSAR methods. The results demonstrated a high level of consistency, with a correlation coefficient of 0.92, indicating good monitoring precision. As shown in Figure 12, most data points are concentrated in the 5–15 mm range, with a few scattered between −10 and 5 mm. The differences may arise from variations in the implementation processes of the two methods, such as differences in the construction of connection graphs and the selection of Ground Control Points (GCPs), which lead to variations in deformation rate values. In this study, the SBAS-InSAR pixel values were extracted using highly coherent PS points from PS-InSAR. Since SBAS-InSAR provides a denser distribution of monitoring points, the coherence of PS points may introduce slight biases in the SBAS pixel values. Overall, the mutual validation between PS-InSAR and SBAS-InSAR results supports the reliability of InSAR monitoring to a certain extent. This demonstrates the robustness of InSAR techniques for monitoring surface deformation with high precision.

3.2. Analysis of Influencing Factors

The analysis of ground settlement causes at the 220 kV Port Substation foundation pit in Nansha District, Guangzhou, reveals that the deformation is influenced by both anthropogenic factors (subway construction and residential buildings) and natural factors (rainfall, soil temperature, and hydrogeology). This classification aligns with the findings of scholars such as MA Yuanyuan [47] and LI Hong-xia [48] regarding the causes of ground settlement. To facilitate input into the prediction model, qualitative factors (subway construction and residential buildings) were quantitatively processed. By creating buffer zones of varying widths to cover settlement areas, these qualitative factors were converted into raster data. The processed quantitative data for subway construction and residential buildings are shown in Figure 13.

Settlement points within the pit area and its surroundings were selected, and the corresponding data for subway construction, residential buildings, rainfall, soil temperature, and hydrogeology were extracted to these settlement points using the multi-value extraction-to-point technique. To evaluate whether these five influencing factors are suitable as predictors for urban ground settlement, a gray relational analysis was performed between each factor and the settlement values.

The deformation point sample values of the five main controlling factors—subway construction, residential buildings, rainfall, soil temperature, and hydrogeology—were selected as comparative sequences, with settlement values serving as the reference sequence. The results are shown in

Table 1. Impact factor correlation for foundation pit settlement.

Controlling Factor	Subway Construction	Housing Construction	Rainfall	Soil Temperature	Hydrogeology
Relatedness	0.731	0.725	0.838	0.796	0.824

. It is evident that the selected five influencing factors exhibit a high degree of correlation with settlement values, aligning with the settlement driving factors proposed by scholars. This verifies the rationality and reliability of using these factors as predictors for pit settlement.

3.3. LE-Transformer Prediction Results

The Inverse Distance Weighted (IDW) interpolation method was applied to process pit deformation data to more reasonably estimate the values of unknown points. This method assigns weights based on the distance between the point to be interpolated and known data points, with closer distances receiving higher weights [49]. Using this approach, deformation rate maps from 2020 to 2024 were generated, providing intuitive spatiotemporal variation information for subsequent analysis. Additionally, the settlement time-series of 1556 observation points were selected as training samples.

The grid search method exhaustively evaluates all possible parameter combinations to find the optimal configuration for improving model prediction accuracy [50]. In this study, grid search was applied to optimize the number of network layers (K) and the number of hidden layer nodes (S). Figure 14 presents the prediction heatmap, illustrating the distribution of model prediction errors under different combinations of network layers and hidden layer nodes. The color intensity in the heatmap represents the magnitude of prediction errors. The analysis revealed that the number of layers and hidden nodes were not the only factors influencing prediction accuracy. For a configuration of five network layers and 128 hidden layer nodes, the model achieved a good balance between capturing complex features in the data and avoiding overfitting. Fewer layers might fail to fully capture deep-level information, while excessive layers could lead to overfitting, diminishing the model’s generalization ability. Therefore, the optimal configuration for the model in this study was determined to be five layers with 128 hidden nodes.

To evaluate the model’s prediction performance in various deformation areas, random points under different conditions (including the pit site and surrounding areas) were selected within the study area for analysis. A comparison between the model’s prediction results and the SBAS-InSAR method results (Figure 15) shows that the two exhibit consistent overall trends, although differences were observed in finer details. The model effectively captures the overall deformation trends, while SBAS-InSAR, due to its technical characteristics, is more sensitive to minor deformations, leading to slight deviations. However, the model demonstrates advantages in handling continuous time-series data and integrating the effects of multiple factors, providing a more comprehensive prediction of deformation trends.

This study compares the impact of univariate and multivariate data on deformation prediction results. Univariate data consist solely of InSAR time-series deformation data, while multivariate data incorporate additional factors, such as rainfall. The temporal span of the input deformation monitoring data is from 8 January 2020 to 25 September 2023, with the model’s prediction period extending from 19 October 2023 to 7 September 2024. As shown in Figure 16, the prediction errors between the forecasted and actual cumulative deformation reveal that the multivariate model outperforms the univariate model in regions with significant deformation. It provides a more accurate reflection of deformation trends at key points in major deformation zones and aligns more closely with the SBAS-InSAR results. In contrast, the univariate model, due to the absence of additional influencing factors, often exhibits an “inertia” effect. This results in challenges when adjusting predictions under changing environmental conditions, leading to either overestimation or underestimation of deformation. In areas distant from the pit site, where deformation is minor and the impact of influencing factors is limited, the differences between the two models are negligible. Nevertheless, the univariate model can still partially capture deformation in these regions. However, in areas with complex deformation, the multivariate model demonstrates significant advantages. This analysis underscores the importance of integrating multivariate data to achieve accurate predictions of pit-induced deformation. The selected factors play a critical role in significantly improving the model’s predictive capability in regions experiencing complex deformation.

3.4. Comparison with Other Models

The EAA mechanism offers unique advantages within the LE-Transformer combination model. Unlike traditional quadratic matrix multiplications, EAA uses linear element-wise multiplication, avoiding complex computations. By calculating global context through linear transformations of query and key interactions, it efficiently captures global information. When processing pit deformation data, EAA quickly focuses on critical features and accurately identifies relevant factors in multivariate data, enhancing prediction accuracy in complex deformation regions. Comparative studies were conducted to evaluate its impact on dataset training. As shown in

Table 2. Training effects of different mechanisms.

Self-Attention Type	MSE	MAE	Complexity
EAA	0.085	0.122	O(n × d)
Multi-Head Attention	0.156	0.232	O(n2 × d)
Scaled Dot-Product Attention	0.183	0.283	O(n2 × d)

, EAA’s performance is comparable to Scaled Dot-Product Attention and Transformer’s multi-head self-attention mechanism, with similar MSE and MAE values. However, EAA demonstrates practical improvements in training datasets. Moreover, the computational complexity of EAA, Scaled Dot-Product Attention, and multi-head attention remains efficient, enabling the prediction of extended time-series data under similar hardware conditions.

To assess the superiority of the introduced prediction model, LSTM, Transformer, GRU, and Informer models were trained and evaluated for multivariate sequence deformation prediction. In addition to MSE, other performance metrics such as MAPE, RMSE, and R² were employed to compare model accuracy. The LE-Transformer model demonstrated predictions closest to the SBAS-InSAR results, with an MAPE of only 2.5%, proving its effectiveness. As shown in

Table 3. Effect of different model training.

Models	MSE	MAPE	RMSE	R2
GRU	0.238	0.048	0.412	0.835
LSTM	0.215	0.042	0.384	0.857
Transformer	0.195	0.039	0.369	0.886
LE-Transformer	0.125	0.025	0.292	0.934

, the LE-Transformer model outperformed other models in accuracy on the multivariate dataset. This indicates that the inclusion of the EAA mechanism enhances the model’s capability in handling multivariate sequences, effectively processing long input sequences while achieving high prediction accuracy at the point scale. Overall, the LE-Transformer model exhibited superior performance, making it a reliable tool for predicting surface deformation in foundation pit projects.

4. Discussion

This study successfully applied the LE-Transformer model to predict foundation pit deformation, showing its effectiveness in capturing both temporal and spatial dependencies in time-series data. With an MAPE of 2.5%, the model outperforms traditional methods, including LSTM, GRU, and Transformer, by integrating multiple factors such as rainfall, subway construction, and soil temperature.

The LE-Transformer’s use of EAA improved computational efficiency, enabling the model to process multivariate data with reduced complexity. This approach addresses limitations found in traditional multi-head self-attention mechanisms, which often suffer from quadratic complexity.

However, the model’s generalizability may be affected by data sparsity or extreme environmental conditions. Future studies should focus on enhancing its robustness by incorporating more diverse data sources and improving its performance in less data-rich scenarios. Additionally, the model’s accuracy could be further improved by exploring nonlinear interactions between influencing factors and testing its applicability across varied geological conditions.

Comparison with existing research demonstrates that, unlike traditional empirical and neural network models, the LE-Transformer offers a more comprehensive solution for deformation prediction by integrating InSAR time-series data with environmental factors. This makes it a promising tool for urban infrastructure projects where deformation prediction and early warning systems are crucial.

Future work should focus on improving the model’s integration with real-time monitoring systems and expanding its application to other areas, such as geological disaster monitoring and urban planning. Incorporating factors like vegetation cover and slope orientation could also further enhance its predictive accuracy.

5. Conclusions

This study introduced an innovative approach for monitoring and predicting foundation pit deformation by integrating time-series InSAR data with the LE-Transformer model, a deep learning framework combining LSTM, Efficient Additive Attention (EAA), and Transformer mechanisms. Using the 220 kV Port Substation foundation pit in Guangzhou’s Nansha District as a case study, the key findings and contributions of this research are summarized as follows.

(1)

Deformation data for the 220 kV Port Substation foundation pit in Guangzhou’s Nansha District were obtained using two InSAR techniques, PS-InSAR and SBAS-InSAR. The cumulative deformation ranged from −10 mm to +15 mm, with a high correlation coefficient of 0.92 between the results from both methods. This demonstrates the effectiveness of InSAR-based monitoring for large-scale urban infrastructure projects, offering superior coverage and precision compared to traditional methods.

(2)

Gray relational analysis identified key factors affecting deformation, including rainfall, subway construction, residential buildings, soil temperature, and hydrogeology. Rainfall was the most influential factor, with a correlation of 0.838. These findings underscore the importance of considering both environmental and infrastructural factors in foundation pit settlement, which is crucial for risk management in urban development projects.

(3)

The LE-Transformer model outperformed traditional prediction models, achieving a mean absolute percentage error (MAPE) of 2.5%. By incorporating both spatial and temporal data, the model demonstrated superior capability in predicting complex deformation trends. Its ability to handle multivariate data provides a more comprehensive understanding of the deformation mechanisms in foundation pit projects.

This integrated methodology provides a reliable framework for monitoring foundation pit deformation and can enhance risk prevention and early warning systems. Future work should explore the integration of more factors into the model to further improve prediction accuracy.