STAIT: A Spatio-Temporal Alternating Iterative Transformer for Multi-Temporal Remote Sensing Image Cloud Removal

Highlights

What are the main findings?

• A novel Spatio-Temporal Alternating Iterative Transformer (STAIT) is proposed to explicitly model the dynamic dependencies in multi-temporal remote sensing image cloud removal task.
• An efficient framework combining multi-level feature extraction and a weight-sharing decoder is designed to ensure high-quality, temporally consistent reconstruction.

What are the implication of the main findings?

• The method significantly improves cloud removal accuracy, effectively restoring surface details obscured by thick clouds.
• It provides a robust and efficient solution for generating continuous remote sensing data, enhancing the reliability of Earth observation applications.

Abstract

Multi-temporal remote sensing image cloud removal aims to reconstruct land surface information in regions obscured by clouds and their shadows, thereby mitigating a major constraint on the application of remote sensing imagery. However, existing multi-temporal deep learning methods for cloud removal often fail to model complex spatio-temporal dynamics, leading to suboptimal performance. To address this challenge, we propose a novel framework for multi-temporal cloud removal. In this architecture, the most critical component is the Spatio-Temporal Alternating Iterative Transformer (STAIT), which primarily consists of temporal and spatial attention mechanisms. STAIT is engineered to refine spatio-temporal feature representation by establishing an effective interplay between spatial details and temporal dynamics. Our framework is enhanced by an efficient image token generator with group convolution-based multi-level feature extraction to manage complexity, and a pixel reconstruction decoder with a shared progressive upsampling network to improve reconstruction by learning time-invariant features. Experimental results demonstrate that by explicitly modeling spatio-temporal feature dependencies, our approach achieves superior performance in restoring high-fidelity, cloud-free imagery.

1. Introduction

Optical satellite remote sensing serves as a fundamental technology for observing and comprehending our planet. It enables a wide range of applications, including precision agriculture [1,2], urban dynamic monitoring [3,4], and environmental change tracking [5,6]. However, the pervasive presence of clouds poses severe obstacles to fundamental remote sensing tasks like ground target recognition [7] and image retrieval [8,9]. Furthermore, it plays a crucial role in low-level multi-spectral image reconstruction [10,11,12,13], providing high-quality foundational data for various downstream quantitative analyses. This widespread “cloud interference” phenomenon not only significantly impedes research and decision-making efficiency in critical application domains, but also leads to incomplete datasets and discontinuous time series, thereby compromising the validity of subsequent analyses and potentially leading to flawed interpretations.

The widespread presence of clouds makes simple single-shot data collection ineffective. Therefore, high-fidelity remote sensing usually requires more complex and advanced imaging methods. A prevalent methodology relies on the temporal correlation between multi-temporal images to reconstruct the missing land surface data under thick clouds [14,15,16]. This approach generally relies on multi-temporal observations from a single platform for data acquisition. By inputting a sequence of partially cloudy remote sensing images acquired at short time intervals, it aims to predict cloud-free remote sensing imagery. In practical applications, this methodology requires the accumulation and processing of substantial historical image data to fully exploit temporal variations and consistencies for cloud removal.

Current multi-temporal cloud removal techniques are primarily divided into tensor completion, generative models, and feature-learning frameworks. However, tensor-based optimization often lacks flexibility in handling complex occlusions, while generative approaches (e.g., GANs, DDPMs) face challenges regarding training stability and inference efficiency. Although deep feature-learning methods leveraging CNNs and Transformers have shown promise in capturing contextual dependencies, they share a critical limitation: most approaches simply concatenate images from multiple time nodes as inputs to attention mechanisms. Consequently, they fail to explicitly model the underlying spatio-temporal feature dependencies and the inherent evolutionary relationships across temporal sequences. In other words, while these methods leverage multi-temporal information, they lack a deep, explicit spatio-temporal modeling of the potential dependencies and correlations between cloudy images. Such a simplistic approach of channel-wise concatenation essentially treats temporal snapshots as isolated inputs, neglecting the inherent spatio-temporal continuity and the complex dependencies that connect them. Consequently, the model struggles to capture crucial spatio-temporal dynamics and feature dependencies. Furthermore, the absence of explicit constraints on these dependencies can result in temporally inconsistent predictions, which in turn hampers the overall predictive performance and generalization ability.

To address this deficiency, we propose a novel Transformer module, Spatio-Temporal Alternating Iterative Transformer (STAIT), to explicitly model the spatio-temporal relationships in multi-temporal remote sensing cloud removal. The core of this approach lies in constructing a model framework capable of deeply fitting the complex spatio-temporal dependencies between images from different time nodes. This explicit modeling of spatio-temporal relationships enables our method to more effectively capture the continuity and evolutionary patterns of surface features in the temporal dimension. Consequently, it achieves a more accurate restoration of cloud-free surface information in obscured regions while removing thick clouds, leading to a significant enhancement in the quality of the generated cloud-free imagery.

Our contributions are summarized as follows:

• We propose STAIT to explicitly model the inherent spatio-temporal correlations within multi-temporal data. STAIT innovatively employs an alternating iteration of spatial and temporal attention, enabling precise information aggregation from a global spatio-temporal context. Additionally, our approach offers a more direct and effective pathway to understanding and utilizing the dynamic characteristics of remote sensing imagery.
• To resolve the difficulties in feature tokens extraction and the excessive model complexity arising from high-dimensional multi-temporal inputs, we devise an efficient feature token generator. The generator leverages a group convolution-based multi-level structure. On the one hand, it utilizes group convolutions to reduce channel-wise computational redundancy and effectively control the model’s parameter count. On the other hand, its multi-level design captures feature tokens at various scales, significantly strengthening the model’s representation ability.
• To ensure high temporal consistency in the final cloud-free result, we introduce a novel pixel reconstruction decoder where the parameters of the upsampling module are shared across all timesteps. This constraint is critical as it forces the model to learn a unified generation style, effectively eliminating stylistic and textural inconsistencies.

The rest of this article is structured as follows. Section 2 reviews the related work. Section 3 elaborates on the proposed method. Section 4 reports and analyzes the experimental results, Section 5 discusses the findings and Section 6 provides the conclusion.

2. Related Work

2.1. Multi-Temporal Remote Sensing Image Cloud Removal

Existing approaches for multi-temporal remote sensing image cloud removal can be generally categorized into three groups: tensor completion via optimization algorithms, generative-model-based methods, and feature-learning-based methods. In the first category, the reconstruction of cloudy pixels is formulated as a tensor completion problem [14,17,18,19,20,21,22,23]. Within this framework, time-series images are typically represented as higher-order tensors, and optimization algorithms are utilized to recover cloudy values by exploiting the global low-rank structure inherent in the data. While mathematically rigorous, these methods often struggle with complex cloud patterns where the low-rank assumption may not fully hold. Furthermore, generative models represent another significant stream of research, principally including Generative Adversarial Networks (GANs) [24,25,26] and Denoising Diffusion Probabilistic Models (DDPMs) [27,28,29]. Researchers leverage adversarial training to synthesize cloud-free imagery or employ diffusion processes for gradual denoising, achieving excellent data distribution fitting. However, practical applications are often hindered by the training instability of GANs and the computational inefficiency associated with the iterative sampling of diffusion models.

More recently, feature-learning-based methods have gained prominence by leveraging deep neural networks to learn discriminative features and direct mappings from multi-temporal inputs to cloud-free outputs. This category predominantly employs Convolutional Neural Network (CNN) modules [30,31,32,33] to extract fine-grained local features or Transformer-based modules [34] to capture long-range contextual dependencies. By integrating their complementary strengths, these methods effectively suppress clouds and recover underlying surface information through an end-to-end learning paradigm.

2.2. Transformer in Remote Sensing Image

The evolution of deep learning in remote sensing largely mirrors advancements in general computer vision. Initially, CNNs dominated the field; however, their performance is often bottlenecked by limited receptive fields, which hinders the modeling of long-range dependencies. While architectures such as Feature Pyramid Networks (FPNs) [35], U-Net++ [36] and dense skip connections [37] were developed to fuse multi-scale features and mitigate information loss, they remain inherently limited in modeling long-range dependencies due to the local nature of convolution operations. Consequently, the field has witnessed a shift toward Transformers, which excel at global context modeling. Currently, the most common strategy involves hybrid architectures that combine CNNs for local feature extraction with Transformers for global representation. These hybrid approaches typically employ strategies such as series-parallel splicing [38] or multi-level hybridization [39]. Considering the rapid development of large-scale foundation models, Transformers are poised to become a significant trend in the next generation of RS image interpretation.

3. Methods

3.1. Problem Formulation and Motivation

The task of multi-temporal remote sensing cloud removal aims to reconstruct a cloud-free remote sensing image by leveraging multi-spectral cloudy images acquired from satellites at different time nodes. We define the multi-spectral cloudy images at various time nodes as $\mathbf{X}\in {\mathbb{R}}^{T\times C\times H\times W}$, where T represents the number of time nodes, C is the number of spectral channels, and H and W denote the height and width of the image, respectively. Specifically, $\mathbf{X}(T=i)\in {\mathbb{R}}^{C\times H\times W}$ denotes the multi-spectral cloudy image at a single time node i. Let $\mathbf{Y}\in {\mathbb{R}}^{C\times H\times W}$ and $\widehat{\mathbf{Y}}\in {\mathbb{R}}^{C\times H\times W}$ denote the ground-truth and the model-predicted cloud-free remote sensing images, correspondingly. Assuming that land cover features remain unchanged over a short period, our objective is to restore the underlying cloud-free remote sensing image $\mathbf{Y}$ from $\mathbf{X}$.

For a given region, we classify degraded pixels from the multi-temporal cloudy image sequence into two types, as illustrated in Figure 1. The first type is Temporally Compensated Pixels. These pixels are degraded in some observations, while they are clear (i.e., not degraded) in at least one temporal observation. The second type consists of Blind-Inpainting Pixels. These pixels are consistently degraded across all temporal observations (i.e., they are perpetually obscured by clouds or shadows). The Temporally Compensated Pixels are restored by incorporating prior information gleaned from their corresponding clear observations at different time nodes. In contrast, the Blind-Inpainting Pixels require the model to infer their values by learning the intrinsic prior distributions or structural regularities within remote sensing images.

Our approach strategically integrates both inter-temporal and spatial mechanisms: the former effectively handles Temporally Compensated Pixels, while the latter specifically targets Blind-Inpainting Pixels. Specifically, for Temporally Compensated Pixels, we devise an inter-temporal feature attention mechanism named Cross-Timenode Temporal Attention Module (CT-TAM) to capture the correlations between image features at different time nodes. For Blind-Inpainting Pixels, a spatial feature attention mechanism named Single-Timenode Spatial Attention Module (ST-SAM) is employed to learn the inherent prior feature distributions of remote sensing imagery, such as self-similarity. Finally, to reconstruct these two distinct categories of pixels, we propose a novel STAIT Module, which restores both pixel types in an alternating iterative manner. This iterative design is built upon a synergistic relationship. In each iteration, pixels restored using temporal information provide a reliable spatial context for the cloudy inpainting task. Consequently, the newly inpainted regions create a more complete feature token, which in turn enhances the accuracy of temporal feature alignment in the subsequent iteration. This reciprocal process allows for the progressive refinement of the entire image. Notably, the decision to restore these two pixel types at the feature level is a key advantage of our approach. It allows the model to leverage higher-level semantic information, rather than manipulating raw pixel values directly. This enables the generation of more semantically coherent and visually plausible results, which is particularly crucial for reconstructing the Blind-Inpainting Pixels.

3.2. Overall Framework

As illustrated in Figure 2, the proposed framework in this study is designed to synthesize high-quality cloud-free image from multi-temporal cloudy images (comprising T time nodes). The overall model architecture primarily comprises three core components: Feature Token Generator, STAIT Module, and Pixel Reconstruction Decoder.

Specifically, the processing pipeline unfolds as follows: Firstly, given a set of multi-temporal remote sensing cloudy images tensor $\mathbf{X}$. This tensor is then processed by the Feature Token Generator to produce a sequence of informative feature tokens. Subsequently, the output of the feature extraction module serves as the input to the STAIT module. The STAIT module’s core lies in its unique alternating iterative mechanism, which, through iterative operations between the ST-SAM and CT-TAM, enables the deep excavation and fusion of intricate spatio-temporal patterns across different time nodes. Ultimately, the STAIT module outputs a rich spatio-temporal context-aware feature token representation. Finally, these features processed by the STAIT module are fed into the Pixel Reconstruction Decoder. This decoder is responsible for progressively reconstructing feature tokens into a high-resolution pixel image space, thereby synthesizing the final predicted cloud-free image.

3.3. Feature Token Generator

A significant challenge in applying deep networks to multi-temporal data is the prohibitive computational cost associated with processing long image sequences. To mitigate this limitation, we propose an efficient feature token extraction strategy that employs two key components. Firstly, we use a multi-level feature extraction module to capture and fuse diverse features from shallow, intermediate, and deep layers, ensuring a comprehensive feature representation. Secondly, we utilize group convolution to reduce computational complexity. This dual approach enables both high computational efficiency and superior cloud removal performance. Specifically, for a given cloudy image $\mathbf{X}$, we first employ multi-layer convolutions to extract an initial feature, denoted as ${\mathbf{F}}_{\mathbf{0}}$. Additionally, for input feature $\mathbf{F}$, the grouped convolution operation is denoted as G parallel standard convolutions:

$$\mathrm{GroupConv}\left(\mathbf{F}\right)=\underset{g=1,\dots ,G}{Concat}\left(\mathrm{Conv}({\mathbf{F}}^g,{\mathbf{W}}^g)\right),$$

where ${\mathbf{F}}^g\in {\mathbb{R}}^{T\times (C/G)\times H\times W}$ is the g-th partition of $\mathbf{X}$ along the channel dimension, ${\mathbf{W}}^g$ is the g-th partition of $\mathbf{W}$ (corresponding to the filters for the g-th group), and Concat denotes concatenation along the output channel dimension. Ultimately, our proposed group convolution-based multi-level feature token extraction is represented as follows:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{F}}}_{i+1}& =\mathrm{GroupConv}\left({\mathbf{F}}_i\right),\hfill \\ \hfill {\mathbf{F}}_{i+1}& =\mathrm{Concat}({\widehat{\mathbf{F}}}_{i+1},{\mathbf{F}}_0),\hfill \\ \hfill {\mathbf{F}}_{gen}& ={\mathbf{F}}_N\in {\mathbb{R}}^{T\times C_{gen}\times H_{gen}\times W_{gen}},\hfill \end{array}$$

where $i=0,\dots ,N-1$, $C_{gen},H_{gen}$ and $W_{gen}$ represent the channel, height and width of feature ${\mathbf{F}}_{gen}$. At each level i, the features ${\mathbf{F}}_i$ undergo a group convolution operation to produce ${\widehat{\mathbf{F}}}_{i+1}$, capturing more abstract or discriminative representations. Crucially, these refined features ${\widehat{\mathbf{F}}}_{i+1}$ are then concatenated with the inital feature ${\mathbf{F}}_0$ to form the input for the next level ${\mathbf{F}}_{i+1}$. This repetitive concatenation of deeper, processed features with the foundational input ${\mathbf{F}}_0$ ensures that the network continuously benefits from both deep features semantic information and fine-grained shallow features details.

The cascaded multi-level feature extraction module significantly enhances feature extraction capabilities. By progressively integrating features from different abstraction levels while preserving the original input, it effectively mitigates information loss that may occur in deep networks. This hierarchical and recurrent feedback mechanism allows the module to capture a richer and more comprehensive representation of the input, leading to more robust and discriminative feature tokens.

3.4. Spatio-Temporal Alternating Iterative Transformer

To facilitate the robust reconstruction of occluded pixel regions, our methodology operates at the feature level, explicitly employing attention mechanisms to model inter-feature correlations. This application allows for the granular capture of two critical types of relationships: temporal dependencies among corresponding pixel feature tokens across varying time nodes, and spatial relationships between disparate pixel feature tokens within a given temporal period. By explicitly modeling these distinct contextual tokens, our system can more effectively infer and reconstruct missing or corrupted pixel information, leading to enhanced recovery fidelity.

We propose a novel Spatio-Temporal Alternating Iterative Transformer (STAIT) module designed to efficiently capture complex spatio-temporal dependencies within multi-temporal cloudy images. The core of this module lies in its unique alternating architecture, iteratively composed of a Single-Timenode Spatial Attention Module (ST-SAM) module and a Cross-Timenode Temporal Attention Module (CT-TAM) module. This design enables the simultaneous learning of rich and effective representations from both temporal and spatial dimensions. Specifically, given the input features ${\mathbf{F}}_{gen}$ obtained from a feature extraction layer, we first project them into a higher-dimensional feature space via an embedding layer, yielding ${\mathbf{F}}_{emb}\in {\mathbb{R}}^{T\times c\times d\times d}$. This embedding operation aims to enhance the expressive power of the features and adapt them to the internal dimensionality requirements of the Transformer module. We then transform ${\mathbf{F}}_{emb}$ by unfolding it into the input tokens $\mathbf{M}(T=1), \mathbf{M}(T=2)$ and $\mathbf{M}(T=3)\in {\mathbb{R}}^{d^2\times c}$ for the self-attention mechanism:

$$\begin{array}{cc}\hfill \mathbf{M}(T=i)& =\mathrm{UnFold}[\left({\mathbf{F}}_{emb}(T=i)\right],i=1, 2, 3,\hfill \\ \hfill \mathbf{M}& =\underset{i=1,2,3}{Concat}\left[\mathbf{M}(T=i)\right],\mathbf{M}\in {\mathbb{R}}^{T\times d^2\times c},\hfill \end{array}$$

where each row of ${\mathbf{M}}_i$ corresponds to a token’s representation.

Subsequently, $\mathbf{M}$ is fed into our spatio-temporal alternating iterative process. We initialize ${\mathbf{M}}_{trans}^0=\mathbf{M}$ and for each iteration, with $l=0,1,\dots ,L-1$, the following steps are performed:

• ST-SAM Step: The primary function of this module is to learn correlations among different feature dimensions within a single timestep. By computing and aggregating this spatial contextual information, the module generates the feature representation ${\mathbf{M}}_{trans}^{l+{\displaystyle \frac{1}{2}}}$. This ensures that the network can comprehend and integrate the interactions between various tokens within each specific timestep. The process is described by the following equation:

  $${\mathbf{M}}_{trans}^{l+{\displaystyle \frac{1}{2}}}={\mathrm{ST}\text{-}\mathrm{SAM}}_{\theta }\left({\mathbf{M}}_{trans}^l\right),$$
• CT-TAM Step: Immediately following, ${\mathbf{M}}_{trans}^{l+{\displaystyle \frac{1}{2}}}$ is passed to the Cross-Timestep Temporal Attention Module. This module focuses on capturing correlations between tokens at the same spatial location but across different timesteps. By mixing different time nodes’ tokens, CT-TAM can identify and learn patterns and dependencies of feature evolution over time, thereby yielding ${\mathbf{M}}_{trans}^{l+1}$. This process is also described by the following equation:

  $${\mathbf{M}}_{trans}^{l+1}={\mathrm{CT}\text{-}\mathrm{TAM}}_{\varphi }({\mathbf{M}}_{trans}^{l+{\displaystyle \frac{1}{2}}}),$$

where $\theta$ and $\varphi$ represent the learnable parameters of the ST-SAM and CT-TAM modules, respectively. Notably, the parameters for both modules are shared across all steps of the alternating iteration process.

This spatial-temporal attention alternation constitutes a complete iteration cycle. To thoroughly extract deep spatio-temporal patterns from the data, we repeat the aforementioned process L times. In each iteration, the output from the previous iteration serves as the input for the current one, allowing the network to progressively refine and strengthen its understanding of complex relationships across temporal and spatial dimensions, ultimately producing highly refined feature representations ${\mathbf{M}}_{trans}^L$. The core advantage of this alternating attention mechanism lies in its ability to systematically decouple and integrate spatio-temporal information flows. By processing these two dimensions independently and sequentially, our network mitigates potential dimensional confounding that might arise from traditional monolithic attention mechanisms, thereby enabling more precise and effective learning of both global and local spatio-temporal features. Subsequently, we define the self-attention mechanism as follows:

$$\begin{array}{cc}\hfill & \mathrm{Attention}\left({\mathbf{M}}_i\right)=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d_k}}}\right)\mathbf{V},\hfill \\ \hfill & \mathbf{Q}={\mathbf{M}}_i{\mathbf{W}}_{\mathbf{Q}},\mathbf{K}={\mathbf{M}}_i{\mathbf{W}}_{\mathbf{K}},\mathbf{V}={\mathbf{M}}_i{\mathbf{W}}_{\mathbf{V}},\hfill \end{array}$$

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}\in {\mathbb{R}}^{c\times d^2}$ denote the learnable Query, Key, and Value matrices, respectively, obtained by applying linear transformations ${\mathbf{W}}_{\mathbf{Q}}$, ${\mathbf{W}}_{\mathbf{K}}$, and ${\mathbf{W}}_{\mathbf{V}}\in {\mathbb{R}}^{d\times d}$. The architectures of the ST-SAM and CT-TAM modules, illustrated in Figure 2, are introduced as follows:

• Single-Timenode Spatial Attention Module. The primary function of the ST-SAM module is to enhance feature representations for the inpainting of degraded regions. It operates by exclusively leveraging the spatial dependencies among feature tokens within each individual time step. Given an input feature tensor ${\mathbf{M}}_{trans}^l$ to the SAM, a shared self-attention mechanism is employed to compute the internal feature similarities for each time node. This process enables the module to capture the contextual dependencies among spatial elements within a single time node, thereby optimizing the per-time-nodes feature representation prior to subsequent temporal processing. The process can be formulated as:

  $$\begin{array}{cc}\hfill & {\mathbf{M}}_{trans}^l(T=1),{\mathbf{M}}_{trans}^l(T=2),{\mathbf{M}}_{trans}^l(T=3)=\mathrm{Split}\left[{\mathbf{M}}_{trans}^l\right],\hfill \\ \hfill & {\mathbf{M}}_{trans}^{l+{\displaystyle \frac{1}{2}}}=\underset{i=1,2,3}{Concat}\left[{\mathrm{Attention}}_{\theta }\left({\mathbf{M}}_{trans}^l(T=i)\right)\right].\hfill \end{array}$$
• Cross-Timenode Temporal Attention Module. The CT-TAM module is designed to capture correlations among feature tokens in the same spatial regions across different time nodes. Given an input feature token ${\mathbf{M}}_{trans}^{l+{\displaystyle \frac{1}{2}}}$, we first partition all tokens into P distinct segments for each time node. Subsequently, for each segment index $p\in \{1,2,\dots ,P\}$, we concatenate the p-th segment from all time nodes to form ${\mathbf{m}}_{trans}^{l+{\displaystyle \frac{1}{2}}}(p=i)\in {\mathbb{R}}^{({\displaystyle \frac{d^2}{P\times T}})\times c}$. This step embodies the core mechanism of our approach, which we denote as token mixing. A self-attention mechanism is then applied to these mixed feature tokens ${\mathbf{m}}_{trans}^{l+{\displaystyle \frac{1}{2}}}$, enabling the module to compute the cross-temporal relationships for that specific area. Finally, the inverse of the token mixing operation named token demixing is performed to restore the token representation to the original input state. This process effectively captures the relationships between features in the same spatial regions across varying time nodes. The process can be formulated as:

  $$\begin{array}{cc}\hfill & {\mathbf{m}}_{trans}^{l+{\displaystyle \frac{1}{2}}}(p=1),{\mathbf{m}}_{trans}^{l+{\displaystyle \frac{1}{2}}}(p=2),\cdots ,{\mathbf{m}}_{trans}^{l+{\displaystyle \frac{1}{2}}}(p=P)=\mathrm{Token}\mathrm{Mixing}[{\mathbf{M}}_{trans}^{l+{\displaystyle \frac{1}{2}}}],\hfill \\ \hfill & {\mathbf{M}}_{trans}^{l+1}=\mathrm{Token}\mathrm{Demixing}\{\underset{i=1,2,\dots ,P}{Concat}[{\mathrm{Attention}}_{\varphi }({\mathbf{m}}_{trans}^{l+{\displaystyle \frac{1}{2}}}(p=i)]\}.\hfill \end{array}$$

3.5. Pixel Reconstruction Decoder

To generate the final cloud-free image, we employ a Pixel Reconstruction Decoder to transform the feature tokens from the STAIT module. Initially, these tokens are adapted via a Multi-Layer Perceptron (MLP) and a reshape operation to match the decoder’s input dimensions. The tokens are then processed by a shared upsampling module, which progressively increases their resolution through alternating upsampling and convolutional layers until they reach the target image size. Critically, the parameters of this module are shared across all timesteps, a constraint that forces the model to learn a unified generation style and ensures strong stylistic and textural consistency in the outputs. The final prediction is produced by summing the reconstructed images from each timestep and applying a non-linear activation function to effectively aggregate the multi-temporal information.

3.6. Loss Function

The proposed network is optimized using an ${\ell }_1$ loss function. This function computes the mean absolute error (MAE) between the cloud-free image $\mathbf{Y}\in {\mathbb{R}}^{C\times H\times W}$ and the prediction $\widehat{\mathbf{Y}}\in {\mathbb{R}}^{C\times H\times W}$, as shown in the following formula:

$$\mathcal{L}(\widehat{\mathbf{Y}},\mathbf{Y})={\displaystyle \frac{1}{C\times H\times W}}\sum _{c=1}^C\sum _{i=1}^H\sum _{j=1}^W\left|{\widehat{\mathbf{Y}}}_{c,i,j}-{\mathbf{Y}}_{c,i,j}\right|,$$

(1)

where C, H, and W denote the number of spectral bands, height, and width, respectively. In our implementation, the loss is computed as the mean reduction over all spectral dimensions (bands) and spatial dimensions (pixels). The ${\ell }_1$ loss function is particularly effective due to its robustness to outliers, which prevents large errors from dominating the optimization.

4. Experiments

To validate the effectiveness of our proposed model, we conduct a series of comprehensive experiments in this section. We first compare our method with several state-of-the-art approaches on two benchmark datasets to validate its performance. Furthermore, we conduct extensive ablation studies to analyze the contribution of each key component, thereby justifying the rationale behind our design.

4.1. Datasets and Metrics

To comprehensively evaluate the performance of our proposed method, we conducted experiments on two public multi-temporal remote sensing cloud removal benchmark datasets: STGAN [24] and Sen2_MTC_New [40]. All images in these datasets have a spatial resolution of $256\times 256$ pixels and 3 time nodes, which means $T=3$. The details of these datasets are as follows:

• STGAN is a multispectral dataset derived from the Sentinel-2 satellite. It contains 3130 image sequences from different geographical regions. Each sequence is composed of three cloudy images and a corresponding cloud-free reference image. All images include four spectral bands: Red, Green, Blue (RGB), and Near-Infrared (NIR). In our experiments, we adopted a random splitting strategy, partitioning the dataset into training, validation, and test sets at an 8:1:1 ratio.
• Sen2_MTC_New is structurally similar to STGAN, comprising a total of 3417 multispectral image sequences. Each sequence also consists of three cloudy images, one cloud-free reference image, and four spectral bands. To ensure a fair comparison with previous works, we strictly adhered to the standard partitioning scheme proposed in [40] to construct the training, validation, and test sets.

To quantitatively evaluate the quality of the generated cloud-free images, we adopted three widely used evaluation metrics: Peak Signal-to-Noise Ratio (PSNR), the Structural Similarity Index [41] (SSIM) and Spectral Angle Mapper (SAM).

PSNR is a common metric for measuring the quality of image reconstruction. It assesses fidelity by calculating the pixel-level error between the generated image and the ground-truth reference image. A higher PSNR value indicates less distortion and, therefore, higher quality in the reconstructed image. Its formula is defined as:

$$\mathrm{PSNR}=10·lo{g}_{10}\left({\displaystyle \frac{{\mathrm{MAX}}_I^2}{\mathrm{MSE}}}\right),$$

where ${\mathrm{MAX}}_I^2$ is the maximum possible pixel value of the image, and MSE is the Mean Squared Error between the two images $\widehat{\mathbf{Y}}$ and $\mathbf{Y}$.

SSIM evaluates image quality from the perspective of the human visual system by comparing the similarity of two images in terms of three components: luminance, contrast, and structure. The SSIM value ranges from −1 to 1, where a value closer to 1 signifies greater structural similarity between the two images and a better generation result. It is defined as:

$$\mathrm{SSIM}(\widehat{\mathbf{Y}},\mathbf{Y})={\displaystyle \frac{(2{\mu }_{\widehat{\mathbf{Y}}}{\mu }_{\mathbf{Y}}+c_1)(2{\sigma }_{\widehat{\mathbf{Y}}\mathbf{Y}}+c_2)}{({\mu }_{\widehat{\mathbf{Y}}}^2+{\mu }_{\mathbf{Y}}^2+c_1)({\sigma }_{\widehat{\mathbf{Y}}}^2+{\sigma }_{\mathbf{Y}}^2+c_2)}},$$

where $\widehat{\mathbf{Y}}$ and $\mathbf{Y}$ represent the generated and reference images, respectively; $\mu$ and $\sigma$ are the mean and standard deviation; and $c_1,c_2$ are stabilization constants to avoid division by zero.

The Spectral Angle Mapper (SAM) is a metric used to determine the spectral similarity between two spectra by calculating the angle between them. It treats each spectrum as a vector in an L-dimensional space, where L is the number of spectral bands. This makes SAM insensitive to illumination changes, as it measures the similarity of the spectral shape rather than magnitude. A smaller SAM value indicates a higher similarity. The formula is:

$$\mathrm{SAM}(\widehat{\mathbf{Y}},\mathbf{Y})=arccos\left({\displaystyle \frac{{\widehat{\mathbf{Y}}}^T\mathbf{Y}}{\parallel \widehat{\mathbf{Y}}{\parallel }_2{\parallel \mathbf{Y}\parallel }_2}}\right),$$

where $\mathbf{Y}$ is the ground-truth spectral vector and $\widehat{\mathbf{Y}}$ is the estimated spectral vector. The term ${\parallel ·\parallel }_2$ denotes the ${\ell }_2$-norm.

4.2. Implementation Details

The proposed model was implemented in PyTorch 2.2.1 and trained on an NVIDIA RTX 4090 GPU. Key architectural hyperparameters $K,N,P$ and L were all set to 4. The embedding dimension d is set to 256. For network optimization, the Adam optimizer was employed with a learning rate of $1\times {10}^{-4}$, and coefficients ${\beta }_1$ and ${\beta }_2$ were set to 0.9 and 0.99, respectively. The model was trained for 200,000 iterations with a batch size of 16. For the evaluation, the model checkpoint from the final training epoch was utilized.

4.3. Experiments on STGAN Dataset

To conduct a rigorous evaluation of our proposed method, we adopted the publicly available dataset and the official training, validation, and test splits from STGAN [24]. This standardized protocol ensures both the reproducibility of our experiments and the fairness of the comparison. Our method was comprehensively benchmarked against the traditional temporal Median Filter, which generates the output by taking the median of the multi-temporal inputs, and a suite of state-of-the-art (SOTA) multi-temporal deep learning cloud removal techniques, including DSen2-CR [42], CTGAN [40], STGAN [24], CR-TS-Net [43], PMAA [44] and UnCRtainTS [31]. Regarding the baseline comparisons, we adopt a hybrid strategy to ensure each method achieves its optimal performance. For methods where only training code is provided, we retrain the models from scratch using our dataset partitions. For methods with officially released pre-trained weights that utilize the same dataset partitions as ours, we directly employ the official models to avoid performance degradation caused by sub-optimal retraining. To maintain consistency, all methods are evaluated on the identical testing set using unified metrics. To ensure an equitable comparison, all baseline models were retrained from scratch on the designated training set until convergence, utilizing their official source code and adhering strictly to the hyperparameter configurations recommended in their respective original publications.

The qualitative comparison results are presented in Figure 3, which illustrates the cloud removal performance of different methods across several representative scenes. It is evident that most baseline methods exhibit varying degrees of artifacts and limitations, particularly when processing large areas of thick cloud or the fine details at the edges of thin clouds. For instance, the results generated by compared methods suffer from conspicuous visual artifacts and textural distortions. In stark contrast, our proposed method demonstrates significant superiority in visual quality. Our method effectively preserves the spectral characteristics of the original land cover. The resulting images exhibit natural and coherent colors that are seamlessly integrated with the surrounding context. These advantages underscore our model’s robust capability to effectively capture and leverage multi-temporal information for the reconstruction of high-quality, cloud-free pixels.

As delineated in

Table 1. Quantitative comparison results on STGAN and SEN2_MTC_New datasets. The red and blue markers represent the best and second-best values.

Methods	STGAN Dataset			SEN2_MTC_New Dataset
	PSNR ↑	SSIM ↑	SAM ↓	PSNR ↑	SSIM ↑	SAM ↓
Median Filter	10.495	0.441	7.756	9.485	0.404	11.658
DSen2-CR	25.559	0.788	3.801	17.417	0.576	7.378
CTGAN	25.211	0.776	4.079	18.308	0.609	6.857
STGAN	26.310	0.796	3.376	18.158	0.556	7.018
CR-TS-Net	26.309	0.800	3.346	18.597	0.616	6.892
PMAA	26.930	0.829	3.269	18.369	0.614	7.155
UnCRtainTS	27.244	0.821	3.112	18.770	0.631	6.528
STAIT (Ours)	27.340	0.831	2.949	18.868	0.640	6.392

, our method achieves the highest scores across both key metrics. Specifically, our model obtains a PSNR of 27.340 and an SSIM of 0.831, outperforming all other competing methods by a significant margin. The top-ranking SSIM score corroborates our qualitative findings, confirming that our method not only ensures pixel accuracy but also excels at preserving the intrinsic structure, luminance, and contrast of the original ground features. This aspect is of paramount importance for downstream remote sensing applications, such as change detection and land cover classification, which are highly dependent on the structural integrity of the imagery. To evaluate the stability and reproducibility of our method, we conducted several independent runs with different random seeds. The proposed method demonstrates robust performance with a mean PSNR of 27.321 ± 0.121 dB, SSIM of 0.827 ± 0.014, and SAM of 2.946 ± 0.018. The low standard deviations indicate that our model converges consistently to a high-quality image.

In summary, the convergence of evidence from both qualitative visual comparisons and quantitative metric evaluations consistently demonstrates that our proposed multi-temporal cloud removal method surpasses current mainstream approaches in overall performance. Our model not only generates cloud-free images that are visually more realistic, richer in detail, and more color-consistent, but its superior reconstruction accuracy and structural fidelity are also empirically validated by objective metrics. These comprehensive results strongly affirm the efficacy and advanced nature of our designed model.

4.4. Experiments on SEN2_MTC_New Dataset

We conducted additional experiments on the SEN2_MTC_New dataset. We adhered to the identical experimental protocol established for the STGAN dataset to ensure a consistent and fair comparison. For the baseline methods, it is noteworthy that we directly employed the publicly available, pre-trained models for PMAA and CTGAN. All other competing methods were retrained from scratch on the SEN2_MTC_New training set until convergence.

Figure 4 provides a visual testament to our model’s performance on the more challenging SEN2_MTC_New dataset. The comparative results reveal the limitations of existing methods when faced with diverse and complex cloud scenarios. Many competing models, such as DSen2-CR and STGAN, produce reconstructions that either degrade the visual plausibility with unnatural textures, leading to a “patchwork” effect where the restored area is clearly demarcated. Conversely, our approach yields reconstructions that are markedly more plausible and coherent. Our model excels in rendering fine-scale details with high precision. The generated content maintains spatial coherence and is semantically consistent with the surrounding cloud-free context. A key strength of our method is its ability to achieve seamless tonal and color blending. The restored regions are photorealistically integrated into the image, avoiding the chromatic aberrations that plague other methods. This visual evidence suggests that our model possesses a superior understanding of multi-temporal scene dynamics, enabling it to generate restorations that are not only free of clouds but also visually indistinguishable from genuine observations.

To substantiate these visual observations with empirical data, we performed a quantitative comparison using the PSNR and SSIM metrics. The numerical results, summarized in Table 1, unequivocally demonstrate the quantitative superiority of our proposed method. Our model consistently achieves the highest scores, establishing a clear performance advantage over all evaluated SOTA techniques on the SEN2_MTC_New dataset. The top-tier PSNR score signifies a greater minimization of pixel-level discrepancies between our restored images and the ground truth. This highlights our model’s capacity for precise and accurate pixel synthesis. The leading SSIM score is particularly telling, as it reflects our model’s proficiency in preserving the complex structural information and perceptual quality of the original scene. This is crucial for ensuring that the restored images are not just mathematically close but also visually faithful. The consistent lead in both metrics underscores the robustness and effectiveness of our architectural design, validating its ability to generalize to new and varied data distributions. Similarly, we applied the same rigorous evaluation protocol to the SEN_MTC_New dataset. By performing independent runs with distinct random seeds, our method achieved a mean PSNR of 18.871 ± 0.124 dB, SSIM of 0.642 ± 0.015, and SAM of 6.384 ± 0.182. These consistent results with minimal standard deviations further confirm the stability of our architecture.

To ensure a fair comparison with the state-of-the-art diffusion-based method DiffCR [27], we evaluated it using its officially released pre-trained weights on the standard test set. Since DiffCR is designed to operate on 3-channel RGB data, we correspondingly retrained and tested our proposed STAIT framework under the same 3-channel configuration, which is denoted as STAIT-3. As presented in

Table 2. Comprehensive comparison of model complexity (Efficiency) and restoration quality (Performance) on the SEN2_MTC_New dataset.

Methods	Efficiency Metrics			Performance Metrics
	Params (M)	FLOPs (G)	Latency (ms) ↓	PSNR ↑	SSIM ↑	SAM ↓
DiffCR	22.91	91.72	498.70	19.150	0.671	6.454
STAIT-3 (Ours)	7.68	118.75	6.55	18.880	0.653	6.324

, although DiffCR achieves marginally higher PSNR and SSIM scores, our method outperforms it in terms of SAM, indicating superior spectral fidelity. More importantly, our method demonstrates a significant advantage in computational efficiency. It is worth noting that all efficiency metrics (FLOPs and latency) were evaluated by a fixed input size of $256\times 256$ with 3 time nodes. Compared to DiffCR, STAIT requires significantly fewer parameters and achieves an inference speed that is orders of magnitude faster (6.55 ms vs. 498.70 ms), making it far more practical for real-time applications.

Taken in concert, the qualitative and quantitative results on the SEN2_MTC_New dataset reinforce the conclusions drawn from our initial experiments. Our method sets a new performance benchmark for multi-temporal cloud removal. The generated images are characterized by a remarkable degree of realism, detail preservation, and color accuracy, which is corroborated by their state-of-the-art performance on objective metrics. This sustained success across different datasets highlights the robustness of our approach. It effectively addresses the core challenges of the task, demonstrating its potential as a reliable and highly effective tool for generating analysis-ready, cloud-free satellite imagery for critical downstream applications.

4.5. Efficiency Analysis

In addition to restoration quality, computational efficiency is a critical determinant for the practical deployment of cloud removal algorithms, particularly in real-time remote sensing scenarios. As presented in

Table 3. Params, FLOPs, Inference Time and Memory Comparison. The red and blue markers represent the best and second-best values.

Methods	Params, FLOPs, Latency and Memory Comparison
	Params (M)	FLOPs (G)	Latency (ms)	Memory (MB)
DSen2-CR	18.92	2478.73	44.77	333.39
CTGAN	642.92	1263.40	43.20	3319.50
STGAN	231.93	2186.51	34.25	1353.24
CR-TS-Net	38.39	15082.58	312.30	1575.43
PMAA	3.45	185.09	16.42	370.59
UnCRtainTS	0.56	167.10	38.90	785.32
STAIT (Ours)	7.68	118.81	6.68	574.44

, our proposed STAIT demonstrates a significant advantage in computational costs. Evaluated on 4-channel inputs with a spatial resolution of $256\times 256$ across 3 time nodes, STAIT achieves the lowest FLOPs of 118.81 G and an ultra-low inference latency of 6.68 ms. Regarding memory consumption, our method maintains a competitive footprint of 574.44 MB. While slightly higher than the parameter-sparse DSen2-CR, it remains significantly more memory-efficient than heavy-weight counterparts like CTGAN and CR-TS-Net. More importantly, STAIT represents a 2.5× speedup compared to the second-best method, PMAA, striking a superior balance between memory usage and processing speed. This efficiency is primarily attributed to our specific architectural designs: on the one hand, we utilize grouped convolutions to replace standard dense convolutions in high-dimensional feature extraction stages, which significantly reduces computational redundancy and FLOPs without compromising feature representation; on the other hand, the adoption of a shared-parameter decoder effectively constrains the growth of model parameters and minimizes memory access overhead during inference. Consequently, STAIT achieves an optimal trade-off among performance, latency, and memory costs, rendering it highly suitable for large-scale and real-time on-board processing tasks.

4.6. Analysis of Hyperparameters

In this section, we conduct a series of experiments to determine the optimal settings for our model’s key hyperparameters. We individually assess the impact of: (1) the number of levels N in the Feature Extraction module, and (2) the number of iterations L in the STAIT module to find a balance between performance and computational cost.

Firstly, we investigate the importance of the hierarchical feature extraction mechanism by varying the number of levels N, within the Feature Extraction module, setting N to 2, 4, 6, and 8. As presented in

Table 4. Ablation analysis of hyperparameters N and L on the SEN2_MTC_New dataset. The red markers represent the best values.

Analysis of Hyperparameter N				Analysis of Hyperparameter L
N	PSNR ↑	SSIM ↑	SAM ↓	L	PSNR ↑	SSIM ↑	SAM ↓
2	17.842	0.588	7.157	2	17.754	0.583	7.352
4	18.868	0.640	6.392	4	18.868	0.640	6.392
6	18.859	0.637	6.458	6	18.851	0.639	6.405
8	18.731	0.638	6.437	8	18.865	0.635	6.428

, the results exhibit a clear and positive correlation between the number of levels N and the model’s performance when N = 2, 4. The steady improvement in both PSNR SSIM and SAM scores with increasing N demonstrates that a deeper feature hierarchy enables the model to capture a richer and more diverse set of spatial features, which is crucial for high-fidelity image reconstruction. It is observed that the performance peaks at N = 4. Secondly, we aim to quantify the effect of the iterative refinement process within our STAIT module. We configured the model with a varying number of alternating iterations L, specifically testing L = 2, 4, 6, and 8. The quantitative results detailed in Table 4 reveal that performance initially improves as L increases from 2 to 4, but subsequently degrades when L exceeds 4. This indicates that L = 4 strikes an optimal balance. Therefore, considering the trade-off between performance and computational complexity, we select N = 4 and L = 4 as the optimal setting for our model.

4.7. Ablation Study on STAIT Module

Having determined the optimal hyperparameters, we conducted a comprehensive ablation study to validate the architectural effectiveness of STAIT, ranging from the fundamental backbone choice to specific module designs. First and foremost, to substantiate the advantage of our iterative Transformer architecture over traditional convolutional methods, we established a baseline named “w/o STAIT,” where the entire STAIT module (including ST-SAM and CT-TAM) was replaced by standard residual convolutional blocks of equivalent depth. As detailed in

Table 5. Ablation Experiments on SEN2_MTC_New Dataset. The red markers represent the best values.

Ablation Experiment
Exp	PSNR ↑	SSIM ↑	SAM ↓
w/o STAIT	17.535	0.541	7.662
CT-TAM only	18.112	0.577	7.395
ST-SAM only	17.631	0.538	7.928
w/o Feature Token Generator	18.331	0.614	6.883
w/o Shared Decoder	18.742	0.632	6.487
Ours	18.868	0.640	6.392

, this configuration suffered the most severe performance degradation, with a PSNR drop of 1.33 dB compared to the full model. This sharp decline underscores the limitation of standard CNNs in capturing the complex, long-range spatio-temporal dependencies required for cloud removal, thereby validating the necessity of the proposed attention-based architecture.

Subsequently, we dissected the internal structure of the STAIT module to demonstrate the synergistic contributions of its two core sub-modules: the ST-SAM and the CT-TAM. We constructed two homogeneous variants: “ST-SAM only,” where the CT-TAM was replaced by a second ST-SAM to force exclusive reliance on spatial processing, and “CT-TAM only,” where the ST-SAM was replaced by an additional CT-TAM to rely solely on temporal aggregation. As evidenced by the results, both configurations experienced significant performance drops compared to the heterogeneous full model. This outcome decisively demonstrates that ST-SAM and CT-TAM fulfill complementary and non-interchangeable roles. Neither module, even when duplicated, can effectively substitute for the other, confirming that the dynamic interplay between spatial texture refinement and temporal consistency aggregation is fundamental to STAIT.

Finally, we investigated the impact of specific component designs regarding feature embedding and decoding strategies. The experiment “w/o Feature Token Generator,” which replaced our tokenization module with a standard dense CNN for feature extraction, resulted in a clear performance decline. This indicates that our token generator effectively encapsulates high-level semantic information into tokens, which are more suitable for subsequent Transformer processing than simple convolutional features. Furthermore, the “w/o Shared Decoder” variant, which utilized independent decoders for each time point instead of a shared-parameter strategy, also failed to match the optimal performance. This suggests that the shared-parameter design not only reduces model complexity but also acts as a regularization mechanism, implicitly enforcing consistency across temporal outputs. In summary, the superior performance of the full STAIT model confirms that each component is indispensable for optimal restoration results.

5. Discussion

In this study, our primary objective was to overcome the limitations of existing deep learning paradigms that fail to capture complex spatio-temporal dynamics in multi-temporal cloud removal. The experimental results demonstrate that the proposed Spatio-Temporal Alternating Iterative Transformer (STAIT) significantly outperforms traditional concatenation-based approaches. Unlike previous methods that often treat multi-temporal images as isolated inputs or simple channel-wise stacks, STAIT explicitly models the evolutionary relationships across temporal sequences. By employing an alternating mechanism between spatial and temporal attention, our model effectively disentangles latent true surface information from cloud interference. This explicit modeling ensures that the reconstructed regions are not only visually plausible but also maintain strong temporal consistency with the surrounding environment, validating our hypothesis that capturing dynamic dependencies is crucial for high-fidelity restoration.

Beyond reconstruction accuracy, the proposed framework successfully addresses the practical challenges of model complexity and output coherence. The integration of the group convolution-based image token generator plays a vital role in mitigating the computational redundancy typically associated with high-dimensional multi-temporal inputs. This design allows for efficient multi-level feature extraction without compromising the model’s representational capacity. Furthermore, the introduction of a shared progressive upsampling network in the decoder significantly enhances temporal coherence. By enforcing shared parameters across different timestamps, the model is constrained to learn a unified generation style. This effectively eliminates the textural and stylistic inconsistencies that frequently degrade the quality of results in prior feature-learning frameworks, ensuring that the cloud-free output remains natural and artifact-free.

While the proposed framework successfully optimizes computational efficiency by significantly reducing parameters and FLOPs, its performance remains constrained by the intrinsic information availability within the input sequence. The restoration of high-fidelity imagery relies heavily on extracting complementary cues from the temporal dimension. In scenarios characterized by extreme temporal intervals or persistent, extensive cloud cover across all available time points, the retrieval of fine-grained surface details becomes an ill-posed problem. Under such conditions, the lack of visible reference pixels and weak temporal correlations can impede the reconstruction of high-frequency textures. Consequently, future work will focus on addressing these data-dependent challenges, potentially by incorporating stronger external priors or generative constraints to enhance feature recovery in highly occluded or temporally sparse sequences.

6. Conclusions

To address the challenge of thick cloud removal in multi-temporal imagery, we propose a novel framework that effectively addresses the absence of explicit spatio-temporal dependency modeling, a critical limitation in existing methods. At the core of our proposed framework is the Spatio-Temporal Alternating Iterative Transformer (STAIT), a novel module that captures the dynamic evolution of surface features through an alternating application of spatial and temporal attention. Complemented by an efficient multi-level feature extractor and a weight-sharing reconstruction decoder, our framework effectively manages model complexity while ensuring high-fidelity and consistent image synthesis. Experimental results demonstrate that our method significantly improves the accuracy and visual quality of cloud-free images, offering a robust, parameter-efficient solution that marks a significant step forward in achieving continuous and reliable Earth observation.