Deep Learning for Regular Raster Spatio-Temporal Prediction: An Overview

Abstract

The raster is the most common type of spatio-temporal data, and it can be either regularly or irregularly spaced. Spatio-temporal prediction on regular raster data is crucial for modelling and understanding dynamics in disparate realms, such as environment, traffic, astronomy, remote sensing, gaming and video processing, to name a few. Historically, statistical and classical machine learning methods have been used to model spatio-temporal data, and, in recent years, deep learning has shown outstanding results in regular raster spatio-temporal prediction. This work provides a self-contained review about effective deep learning methods for the prediction of regular raster spatio-temporal data. Each deep learning technique is described in detail, underlining its advantages and drawbacks. Finally, a discussion of relevant aspects and further developments in deep learning for regular raster spatio-temporal prediction is presented.

1. Introduction

In several scientific areas [1,2,3,4,5], data measurements are generally accompanied by additional metadata denoting the location and the time in which data were recorded. Recently, in statistics [6,7] and machine learning [8,9], particular attention has been paid on how to effectively model spatio-temporal data while fully leveraging the additional knowledge on acquisition location and time.

There are different types of spatio-temporal data, and each can be properly analysed by specific methodologies. The most common spatio-temporal data type is the raster spatio-temporal data [8], characterised by a set of spatio-temporal observations acquired at fixed locations and timestamps. Both locations and timestamps can be regularly or irregularly sampled, meaning that the distance between neighbouring locations, and timestamps, can be either constant (regular case) or not constant (irregular case) over the whole spatio-temporal domain. Spatial sampling is crucial, since, depending on how raster spatio-temporal data are spaced, some deep learning models may be fitter than others for their processing. As shown in Figure 1, regularly spaced spatio-temporal data can be organised on a regular three-dimensional lattice, with one dimension being time and the remaining dimensions being space. Observations are localized within cells of a grid (e.g., pixels of a multichannel image), defining the spatio-temporal neighbourhood of each observation. On the contrary, irregularly spaced data are better represented as a time series of spatial graphs, whose edges encode space–time neighbouring relations among observations.

Some deep learning architectures, such as Convolutional Neural Networks [10] and transformers [11], can leverage spatial regularity and are, therefore, better suited to process regularly spaced spatio-temporal data.

Regular raster spatio-temporal data are widely encountered in several applicative fields, such as environmental monitoring [12,13,14,15], remote sensing [16,17,18,19], traffic modelling [20,21,22,23], energy management [24,25] and video processing [26,27,28,29], among others. In recent years, deep learning has been widely applied to tasks from the aforementioned fields. To address properties and challenges of each specific task, several deep learning architectures have been employed. For example, Convolutional Neural Networks, transformers and Diffusion models have been widely applied to video processing, while Recurrent Neural Networks and hybrid convolutional–recurrent models have found great application in the environmental field.

Although there are a few surveys addressing the use of deep learning for spatio-temporal data in general, a dedicated survey focusing specifically on regular raster data is absent in the literature. Such a survey, therefore, presents a remarkable degree of novelty and can be potentially relevant for scholars and researchers working in the aforementioned fields, where grid-based spatio-temporal data are most commonly encountered. The aim of this work is to provide a review of the deep learning methodologies commonly applied to regularly spaced spatio-temporal raster data. Instead of focusing on single specific methodologies, this work provides a detailed description of the key concepts and core techniques behind deep-learning-based approaches for spatio-temporal prediction. Moreover, the work discusses specific benefits and drawbacks of the reviewed deep learning methodologies, as well as the applicative domains in which they are more effective. Finally, tables summarising related papers, along with their applicative domains and used datasets, are reported.

The manuscript is organised in the following structure. Section 2 formalises the foundations for spatio-temporal prediction; Section 3 gives an overview of Convolutional Neural Networks for regular raster spatio-temporal prediction; Section 4 describes Recurrent Neural Networks and their application to spatio-temporal prediction; Section 5 presents hybrid models combining both recurrent and Convolutional Neural Networks; Section 6 reviews transformers and their application to spatio-temporal prediction and video processing; Section 7 describes diffusion models for the same task; Section 8 provides an application-oriented discussion of the presented methodologies and outlines future research directions; finally, Section 9 draws conclusions and suggests some directions for future investigations.

2. Preliminaries

A spatio-temporal process describes a phenomenon evolving through space and time and it can be generally formalised as

$$z(\overrightarrow{s},t)\forall \overrightarrow{s}\in S,t\in T,$$

(1)

where locations $S=\{{\overrightarrow{s}}_1,{\overrightarrow{s}}_2,{\overrightarrow{s}}_3,\dots ,{\overrightarrow{s}}_N\}$ and timestamps $T=\{t_1,t_2,t_3,\dots ,t_M\}$ are the spatial and temporal domains of the spatio-temporal process being observed. Spatio-temporal processes have been widely studied in statistics, and they have been modelled following the so-called conditional (or dynamic) approaches [30]. These approaches model the dynamics of the spatio-temporal process by grasping how spatio-temporal variables influence each other at different locations $\overrightarrow{s}$ and times t. Statistical spatio-temporal autoregressive models [7], such as STAR, STARMA, STARIMA and STARMAX [6,31,32,33], follow this modelling approach by trying to capture spatio-temporal dynamics in a data-driven way.

In an analogy with statistical approaches, a spatio-temporal series can be organised as a temporal collection of vectors ${\left\{{\overrightarrow{z}}_t\right\}}_{t=1}^T,{\overrightarrow{z}}_t\in {\mathbb{R}}^N$, where each dimension corresponds to a location, and N is the number of locations being observed. The behaviour of spatio-temporal series, generated by the underlying nonlinear dynamical system [7], can be modelled by a nonlinear spatio-temporal autoregressive (NSTAR) model as follows:

$${\overrightarrow{z}}_t=\mathcal{F}({\overrightarrow{z}}_{t-1},{\overrightarrow{z}}_{t-2},\dots ,{\overrightarrow{z}}_{t-p})+{\overrightarrow{ϵ}}_t,$$

(2)

where p is the model order (i.e., how many past samples are required to model the spatio-temporal series adequately), and $\mathcal{F}(·)$ is the skeleton, a generic nonlinear function modelling the spatio-temporal relationships. Following such a data-driven paradigm, machine learning techniques, such as Support Vector Machines [34] and Random Forests [35], and more recently deep learning methodologies [36], have been employed to approximate $\mathcal{F}(·)$ by effectively catching complex dependencies in spatio-temporal data.

The remaining sections of this work are devoted to describe the various deep learning methods applied to spatio-temporal prediction.

3. Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are deep learning models that rely on convolutions to extract spatio-temporal relationships from spatial time series. Thanks to the use of convolutions, CNNs have three peculiar characteristics: local connectivity, weight sharing and translation equivariance [10]. Furthermore, it is common in CNNs to employ not only convolutions but also other kinds of layers, such as pooling [10], fully connected layers and residual connections [37]. In CNNs, different types of convolutions can be adopted to model spatio-temporal data, according to the input structure, namely, mono-dimensional, bi-dimensional and three-dimensional convolutions.

CNNs, based on 1D convolutions, can process data evolving along a single dimension, e.g., usual time series. Given an input sequence $Z={\left\{{\overrightarrow{z}}_t\right\}}_{t=1}^T$, with ${\overrightarrow{z}}_t\in {\mathbb{R}}^D$, and a kernel $W\in {\mathbb{R}}^Q\otimes {\mathbb{R}}^D$, where Q is the kernel size, 1D convolution between W and Z is

$$y_t={(W\ast Z)}_t=\sum _{q=0}^{Q-1}\langle {\overrightarrow{w}}_q,{\overrightarrow{z}}_{t-q+\lfloor Q/2\rfloor }\rangle \forall t.$$

(3)

$(W\ast Z)$ denotes the convolution operator, ${\overrightarrow{w}}_q\in {\mathbb{R}}^D$ is an element of the kernel and $\langle ·,·\rangle$ is the dot product. Standard 1D convolution has a limited receptive field, i.e., while processing the t-th time step, 1D convolution only considers ${\overrightarrow{z}}_t$ and a few, contiguous elements in the close past and future. Therefore, it might not be a good choice when modelling temporal dependencies that can span many non-contiguous time steps. The receptive field of a CNN can be enlarged by increasing kernel sizes and stacking multiple convolutional kernels. However, the increase in the receptive field size implies a larger computational cost. In these cases, causal dilated convolutions can be a better choice to model temporal data. Unlike standard 1D convolutions, applying the kernel over both past and future elements, causal dilated convolutions only consider elements from the past, motivating the term causal. Moreover, while standard 1D convolutions apply the kernel to contiguous elements, in causal dilated convolutions, a fixed step between element pairs is considered by introducing a positive dilation factor $d\in \mathbb{N}$, thus dilated. A 1D causal dilated convolution is defined as

$$y_t={(W\ast Z)}_t=\sum _{q=0}^{Q-1}\langle {\overrightarrow{w}}_q,{\overrightarrow{z}}_{t-dq}\rangle \forall t\in [1,T].$$

(4)

Causal dilated convolutions are the core of Temporal Convolutional Networks (TCNs) [38], CNNs specifically designed to process temporal data using convolutions. TCNs are made of multiple stacked dilated convolution layers with different dilation factors at each layer. Typically, the i-th layer uses a dilation factor $d_i=2^i,i\ge 0$. Causal dilated convolutions provide TCNs two advantages. The former enables the modelling of temporal dependencies across multiple time scales by applying different dilation factors at each layer. The latter allows grasping long-range dependencies using fewer stacked layers compared with standard convolutions. Due to its inherent limitations, 1D convolution cannot cope with spatial time series. Nevertheless, they can be useful building blocks in other deep learning architectures for spatio-temporal prediction since they can model temporal information effectively.

Two-dimensional convolutions are used with spatially evolving data, e.g., still images. Given a tensor $Z\in {\mathbb{R}}^U\otimes {\mathbb{R}}^V\otimes {\mathbb{R}}^D$, where ${\overrightarrow{z}}_{u,v}\in {\mathbb{R}}^D$ is the observation at location $u,v$, and a kernel $W\in {\mathbb{R}}^Q\otimes {\mathbb{R}}^K\otimes {\mathbb{R}}^D$, 2D convolution between Z and W is defined as follows:

$$\begin{array}{cc}\hfill y_{u,v}& ={(W\ast Z)}_{u,v}\hfill \\ \hfill & =\sum _{q=0}^{Q-1}\sum _{k=0}^{K-1}\langle {\overrightarrow{w}}_{q,k},{\overrightarrow{z}}_{u-q+\lfloor Q/2\rfloor ,v-k+\lfloor K/2\rfloor }\rangle \forall u,v.\hfill \end{array}$$

(5)

$(W\ast X)$ is the 2D convolution operator, and ${\overrightarrow{w}}_{q,k}\in {\mathbb{R}}^D$ is an element of the kernel. Q and K are the spatial dimensions of the kernel, with typically $K=Q$. In regular raster spatio-temporal modelling, 2D convolutions can be used to jointly process the spatial and temporal domains by considering the first two dimensions of the input tensor X as spatial dimensions and the third one as a temporal dimension [39,40,41,42,43]. Given a sequence of spatial data, e.g., remote sensing images, evolving over time ${\left\{I_t\right\}}_{t=1}^T,I_t\in {\mathbb{R}}^U\otimes {\mathbb{R}}^V$, spatio-temporal tensors $Z_t$ are built by concatenating, along the third dimension, the spatial data at time t, $I_t$ with the previous $\mathcal{T}-1$ ones:

$$Z_t=concatenate(I_t,I_{t-1},\dots ,I_{t-\mathcal{T}+1})\in {\mathbb{R}}^U\otimes {\mathbb{R}}^V\otimes {\mathbb{R}}^{\mathcal{T}},$$

where $\mathcal{T}$ is a fixed time lag. Spatio-temporal features can be extracted by 2D convolutions between $Z_t$ and a kernel $W\in {\mathbb{R}}^Q\otimes {\mathbb{R}}^K\otimes {\mathbb{R}}^{\mathcal{T}}$, according to Equation (5). With this approach, 2D convolutions can be used to model spatio-temporal dynamics within a limited time window of length $\mathcal{T}$, which might not be optimal. Furthermore, this approach requires a kernel whose depth is exactly equal to $\mathcal{T}$, making the resulting 2D convolution operation computationally burdensome when $\mathcal{T}$ is large. This limits the size of the temporal window that can be processed by 2D convolutions. Furthermore, 2D convolution transforms the 3D spatio-temporal input tensor into a 2D feature map, flattening the temporal dimension and thus losing temporal localisation of CNN-extracted features. Figure 2 shows an example application of 2D convolution to a spatio-temporal input tensor. Notice how the kernel has the same size of the input tensor along the temporal dimension and how this dimensions is lost in the convolution output.

Three-dimensional convolutions overcome the aforementioned limitations since the kernel does not need to have a depth equal to the window length $\mathcal{T}$ and can be much larger. Furthermore, the kernel is allowed to move not only along the spatial dimensions but along the temporal one, too. Three-dimensional convolutions [20,44,45,46] can be applied to spatio-temporal tensors $Z\in {\mathbb{R}}^U\otimes {\mathbb{R}}^V\otimes {\mathbb{R}}^T\otimes {\mathbb{R}}^D$, where each space–time observation ${\overrightarrow{z}}_{u,v,t}\in {\mathbb{R}}^D$ is a multidimensional vector. Three-dimensional convolutions can be defined as

$$\begin{array}{cc}\hfill y_{u,v,t}& ={(W\ast Z)}_{u,v,t}\hfill \\ \hfill & =\sum _{q=0}^{Q-1}\sum _{k=0}^{K-1}\sum _{r=0}^{R-1}\langle {\overrightarrow{w}}_{q,k,r},{\overrightarrow{z}}_{u-q+\lfloor Q/2\rfloor ,v-k+\lfloor K/2\rfloor ,t-r+\lfloor R/2\rfloor }\rangle \forall u,\forall v,\forall t,\hfill \end{array}$$

(6)

where $W\in {\mathbb{R}}^Q\otimes {\mathbb{R}}^K\otimes {\mathbb{R}}^R\otimes {\mathbb{R}}^D$ is the kernel; Q and K are the spatial dimensions of the kernel; and $R\ll T$ is the temporal dimension. Unlike 2D convolution, which outputs a 2D feature map, 3D convolution results in a 3D feature map, preserving the temporal dimension. In 3D convolutions, the kernel also moves along the temporal dimension, and thus the 2D convolution properties in the space domain (e.g., weight sharing and translation equivariance) are inherited by 3D convolution in time, too. Figure 3 shows an example application of 3D convolution to a spatio-temporal input tensor. Notice how, unlike 2D convolution, the kernel does not need to have the same size of the input tensor along the temporal dimension. Moreover, since it moves in time as well, the temporal dimension is preserved in the output tensor, although shrunk.

Another type of convolution relevant for spatio-temporal prediction is the transposed convolution [47]. Transposed convolutions are used to upsample input images through learnable weights. They work the opposite to convolutions: while convolution with a kernel of size $Q\times Q$ maps an equally sized input patch onto a single value of the output feature map, transposed convolutions map a single input value onto a patch of size $Q\times Q$ on the output feature map through an equally sized kernel.

The approaches analysed herein can be categorised into different taxonomies, according to the type of used convolution, i.e., 2D or 3D, and how convolution impacts on spatial time-series modelling. More precisely, in some cases, convolutions in CNNs are used as spatio-temporal feature extraction mechanisms. Then, extracted features are used for non spatio-temporal tasks, e.g., human action recognition, video classification or index prediction. On the other hand, convolutions are used for end-to-end spatio-temporal tasks, i.e., to map spatio-temporal inputs onto spatial or spatio-temporal outputs, such as in spatio-temporal forecasting tasks. In this context, encoder–decoder architectures are also used, which first spatially downsample the input images, e.g., using valid convolutions and pooling operations, and then produce pixel-wise predictions over all the input spatial cells through a series of upsampling layers. For instance, a CNN based on U-Net [48] performs the spatio-temporal forecasting of arctic sea ice [43], whereas an encoder–decoder CNN based on transposed convolutions through its upsampling module carries out missing data imputation on remote sensing imagery [49].

Table 1. List of papers where CNNs are used, underlining the type of convolution employed.

Ref.	Year	Type	Applications	Datasets
[50]	2010	2D	Video Processing; Action Recognition	NORB Synthetic; KTH Action; Hollywood2
[44]	2013	3D	Action Recognition	Trecvid
[45]	2014	3D	Video Classification; Action Recognition	Sports-1M; UCF-101
[46]	2015	3D	Video Processing; Action Recognition; Scene Recognition; Object Recognition	Sports-1M; UCF-101; YUPENN; Maryland; Egocentric
[39]	2018	2D	Crowd Flow Forecasting	TaxiBJ; BikeNYC
[40]	2019	2D	El Niño Index Prediction	CMIP5; SST Reanalysis Data
[49]	2019	2D	Satellite Image Missing Data Reconstruction	FY-2G LST; MSG-SEVIRI LST
[20]	2019	3D	Traffic Forecasting	TaxiBJ; BikeNYC
[41]	2019	2D	Precipitation Forecasting	DWD RY Product
[51]	2019	2D	Action Recognition	Kinetics; AVA
[42]	2020	2D	Sea Surface Temperature Modelling	CMEMS SST
[43]	2021	2D	Sea Ice Forecasting	EUMETSAT OSI-SAF SIC
[52]	2022	2D	Video Prediction	MovingMNIST; TaxiBJ; Human3.6M; KTH
[18]	2023	3D	Cloud Removal	SEN12MSCR; SEN12MS-CR-TS
[25]	2024	2D	Solar Potential Forecasting	Custom Datasets

summarises all those articles, analysed in this work, that rely exclusively on convolutions to model spatio-temporal dynamics.

Benefits and Drawbacks of Convolutional Neural Networks

Two-dimensional and three-dimensional convolutions can be effectively used to extract spatio-temporal dependencies among data evolving through space and time. The convolution operation has some properties that can benefit many applications, such as translation equivariance, allowing CNNs to detect features regardless of their location in space and time, and weight sharing, i.e., reusing the same kernel weights across spatio-temporal locations, thus reducing the overall parameter number.

Two-dimensional convolutions are naturally suited to extracting spatial dependencies, and they can easily be extended to catch temporal dependencies as well. This extension involves gathering multiple temporal observations into a single tensor and making the convolution kernel as deep as the number of stacked temporal steps. This last aspect limits the number of temporal steps that can be gathered since aggregating too many steps would make the application of 2D convolution computationally unfeasible. Furthermore, the application of 2D convolution to a spatio-temporal tensor flattens the temporal dimension, thus losing temporal localisation in the convolution output while hindering model interpretability too. A possible solution could process small batches of temporal observations and then stack along the temporal dimensions the spatio-temporal features extracted by 2D convolutions for different temporal batches. However, this mechanism risks losing temporal coherence; thus, it may make the model unable to grasp spatio-temporal dynamics correctly.

Three-dimensional convolutions overcome these issues by allowing the kernel to move along both space and time, thus capturing spatio-temporal dependencies jointly and providing richer feature representations. This, however, implies a higher computational cost than 2D convolution for an equally sized kernel.

Two-dimensional and three-dimensional convolutions share some limitations. Both rely on Euclidean kernels [53] with a rigid structure, where only a continuous and commonly small local spatio-temporal neighbourhood is considered at each kernel application. While in some cases this can be a strong inductive bias, in some others, it may not be. Although local, spatio-temporal dependencies can be non-Euclidean and follow irregular spatio-temporal patterns. In such cases, some techniques, e.g., deformable convolutions [54], can help. Furthermore, in many spatio-temporal scenarios, especially Earth and climate domains, there are the so-called teleconnections [55], that is, a strong dependency can exist between events occurring very far in space and time. Two-dimensional and three-dimensional convolutions can hardly cope with teleconnections due to their Euclidean local neighbourhoods. Although, these phenomena could be modelled by stacking multiple convolutional layers and using larger kernel sizes, a better choice could be using dilated convolutions (see Section 3) or attention mechanisms (see Section 6).

4. Recurrent Neural Networks

Recurrent Neural Networks (RNNs) [10,36] are deep learning architectures designed to model temporal relationships among elements in a sequence. One of the earliest RNN formulations is commonly referred to as the vanilla RNN [56]. A vanilla RNN layer with K units sequentially processes an input sequence $Z=\{{\overrightarrow{z}}_1,{\overrightarrow{z}}_2,\dots ,{\overrightarrow{z}}_T\}$, handling one element ${\overrightarrow{z}}_t\in {\mathbb{R}}^D$ at a time, and updates a hidden representation ${\overrightarrow{h}}_t\in {\mathbb{R}}^K$ at each time step according to

$${\overrightarrow{h}}_t=g\left(W{\overrightarrow{h}}_{t-1}+U{\overrightarrow{z}}_t+\overrightarrow{b}\right)\forall t\in [1,T],$$

(7)

where $W\in {\mathbb{R}}^K\otimes {\mathbb{R}}^K$, $U\in {\mathbb{R}}^K\otimes {\mathbb{R}}^D$ and $\overrightarrow{b}\in {\mathbb{R}}^K$ are a hidden–hidden weights matrix, input–hidden weights matrix and bias, respectively; $g(·)$ can be any generic nonlinear function. Although the vanilla RNN has been widely adopted in time-series modelling, two RNN variants are preferred: Long Short-Term Memory (LSTM) networks [57] and Gated Recurrent Units (GRUs) [58]. The main advantage of LSTM networks and GRUs over the vanilla RNN consists of their increased stability during training and, therefore, of their better modelling temporal dependencies. Vanilla RNNs, trained with backpropagation, can suffer of vanishing or exploding gradients [10], namely, as the error gradient is back-propagated through a deep computational graph, some of its components might either get very small, giving an equally small contribution to the corresponding weight update, or very large, making training unstable. Both LSTM networks and GRUs adopt gating mechanisms to control the information flow in order to prevent derivatives from getting too small or too large.

Information flow within a LSTM unit is controlled through three gates, namely, input, forget and output gates. Given the previous LSTM output ${\overrightarrow{h}}_{t-1}$, an encoded representation ${\overrightarrow{i}}_t$ of the current sequence element ${\overrightarrow{z}}_t$ is first computed as

$${\overrightarrow{i}}_t=\beta (W_i{\overrightarrow{z}}_t+U_i{\overrightarrow{h}}_{t-1}+{\overrightarrow{b}}_i).$$

(8)

where $\beta (·)$ can be any sigmoidal function, and $\{W_i,U_i,{\overrightarrow{b}}_i\}$ are learnable parameters. Then, activations of the three gates are computed according to the following equations:

$$\begin{array}{cc}\hfill {\overrightarrow{g}}_t& =\sigma (W_g{\overrightarrow{z}}_t+U_g{\overrightarrow{h}}_{t-1}+{\overrightarrow{b}}_g),\hfill \\ \hfill \overrightarrow{f_t}& =\sigma (W_f{\overrightarrow{z}}_t+U_f{\overrightarrow{h}}_{t-1}+{\overrightarrow{b}}_f),\hfill \\ \hfill {\overrightarrow{o}}_t& =\sigma (W_o{\overrightarrow{z}}_t+U_o{\overrightarrow{h}}_{t-1}+{\overrightarrow{b}}_o),\hfill \end{array}$$

(9)

where $\sigma (·)$ is the logistic function, ${\overrightarrow{g}}_t$, ${\overrightarrow{f}}_t$, ${\overrightarrow{o}}_t$ are the input, forget and output gates, respectively, and $\{W_g,U_g,{\overrightarrow{b}}_g\}$, $\{W_f,U_f,{\overrightarrow{b}}_f\}$ and $\{W_o,U_o,{\overrightarrow{b}}_o\}$ are the corresponding sets of parameters. An inner state ${\overrightarrow{c}}_t$ is then updated by a linear combination of the encoded input ${\overrightarrow{i}}_t$ and ${\overrightarrow{c}}_{t-1}$:

$${\overrightarrow{c}}_t={\overrightarrow{g}}_t\odot {\overrightarrow{i}}_t+{\overrightarrow{f}}_t\odot {\overrightarrow{c}}_{t-1},$$

(10)

where ⊙ is the element-wise product. Finally, the LSTM output ${\overrightarrow{h}}_t$ is given by the current inner state, weighted by the output gate:

$${\overrightarrow{h}}_t={\overrightarrow{o}}_t\odot tanh\left({\overrightarrow{c}}_t\right).$$

(11)

A GRU is simpler than a LSTM unit as it is composed of just two gates: a reset and an update gate, respectively, weighting the influence of the previous GRU output ${\overrightarrow{h}}_{t-1}$ and the current input ${\overrightarrow{z}}_t$ on the current GRU output ${\overrightarrow{h}}_t$. In a GRU layer, the two gates are computed first, according to the following equations:

$$\begin{array}{cc}\hfill {\overrightarrow{r}}_t& =\sigma \left(W_r{\overrightarrow{h}}_{t-1}+U_r{\overrightarrow{x}}_t+{\overrightarrow{b}}_r\right),\hfill \\ \hfill {\overrightarrow{u}}_t& =\sigma \left(W_u{\overrightarrow{h}}_{t-1}+U_u{\overrightarrow{x}}_t+{\overrightarrow{b}}_u\right).\hfill \end{array}$$

(12)

${\overrightarrow{r}}_t$ and ${\overrightarrow{u}}_t$ are the reset and update gates, respectively; $\{W_r,U_r,{\overrightarrow{b}}_r\}$ and $\{W_u,U_u,{\overrightarrow{b}}_u\}$ are dedicated sets of parameters; $\sigma (·)$ is a sigmoid activation function. An intermediate output ${\overrightarrow{h}}_t^{\prime }$ is then computed using the reset gate:

$${\overrightarrow{h}}_t^{\prime }=tanh\left(W({\overrightarrow{r}}_t\odot {\overrightarrow{h}}_{t-1})+U{\overrightarrow{z}}_t++\overrightarrow{b}\right),$$

(13)

The final output ${\overrightarrow{h}}_t$ of the GRU is then computed using the update gate as follows:

$${\overrightarrow{h}}_t={\overrightarrow{u}}_t\odot {\overrightarrow{h}}_{t-1}+(\overrightarrow{e}-{\overrightarrow{u}}_t)\odot {\overrightarrow{h}}_t^{\prime },$$

(14)

where $\overrightarrow{e}$ is a column vector of ones.

Another relevant variant of the vanilla RNN is the Echo State Network (ESN) [59], belonging to the reservoir computing paradigm. In this framework, the latent input representation is generated by a fixed reservoir, which, in the case of ESNs, corresponds to the recurrent layer. ESNs employ the same update equations as a vanilla RNN layer (see Equation (7)). However, the matrices W and U are non-trainable and randomly fixed parameters. Additionally, a predefined sparsity level is enforced on these matrices by setting a subset of their elements to zero. The adoption of a fixed recurrent reservoir confers two primary advantages: ESNs are largely unaffected by vanishing or exploding gradients since no backpropagation is performed through the recurrent layer, and they can be trained significantly faster than vanilla RNNs, LSTMs or GRUs.

Despite being naturally suited to process temporal data, the RNN and its variants have also been adopted to process spatio-temporal data. For instance, a Spatio-Temporal RNN (STRNN) [60] can model spatio-temporal dependencies by cascading two vanilla RNNs: the former processes spatial dependencies and the latter the temporal ones. The spatial RNN, firstly applied, is a quad-directional RNN processing spatial data at each time step through recurrent formulations applied along four spatial directions. The temporal RNN is a bidirectional RNN processing the output of the spatial RNN in the time domain. Another approach is implemented by a Structural RNN (S-RNN) [61]. S-RNN was developed to tackle spatio-temporal tasks that can be modelled as generic spatio-temporal graphs. Hence, S-RNN can be seen as an alternative to spatio-temporal Graph Neural Networks [62]. S-RNN models relationships among graph nodes by assigning a dedicated RNN, in particular, a LSTM network, to each node and edge of the graph. In order to reduce the computational cost of S-RNN, RNNs are shared among nodes and edges with the same semantic meaning according to the task at hand. RNNs are used for the temporal processing of node and edge properties and are distinguished into nodeRNNs and edgeRNNs. When processing a given node v of the graph, edgeRNNs, assigned to all edges that are incident on v, are used to process the temporal sequence of the corresponding edge features first. Then, the nodeRNN assigned to v is used to process the temporal sequence of features of v, along with the output previously computed by edgeRNNs.

Echo State Networks have also been used for spatio-temporal forecasting tasks. For example, a quadratic ESN [63] has been employed to forecast sea surface temperature. Additionally, uncertainty quantification has been achieved through a bootstrap ensemble of ESNs built by sampling multiple random reservoirs and by averaging their prediction. The work has been extended [64] by considering deep ESNs, along with a Bayesian approach to uncertainty quantification, applied to the problem of soil moisture forecasting.

Table 2. List of publications based on RNNs along with the type of recurrent unit they use.

Ref.	Year	Type	Applications	Datasets
[65]	2015	LSTM	Human Motion Prediction	Human3.6M
[66]	2015	LSTM	Video Representation Learning; Video Action Recognition	Sports-1M; UCF101; HMDB51
[63]	2017	ESN	Sea Surface Temperature Modelling	NOAA ERSST
[67]	2017	RNN; LSTM	Land Cover Prediction	MODIS; RSPO; Tree Plantation
[68]	2017	LSTM	Land Cover Prediction	MODIS; RSPO; Tree Plantation
[69]	2018	LSTM	Vegetation Dynamics Forecasting	MODIS NDVI
[70]	2018	LSTM; GRU	Land Cover Classification	Sentinel-1A/1B SAR
[64]	2019	ESN	Soil Moisture Forecasting	NOAA CPC GMSM
[71]	2019	Bayesian RNN	Sea Surface Temperature Modelling	Lorenz-96; NOAA ERSST
[72]	2020	LSTM; GRU; ESN	Spatio-temporal Forecasting	Lorenz-96; Kuramoto–Sivashinsky
[21]	2020	LSTM	Traffic Forecasting; Missing Data Imputation	Loop-Sea; PeMS
[73]	2022	LSTM	Vegetation Health Forecasting	MODIS NDVI
[74]	2022	LSTM	Crop Yield Prediction	USDA NASS
[24]	2022	ESN	Wind Power Forecasting	WRF Simulation Data

summarises the works analysed herein that rely exclusively on RNNs and their variants to model spatio-temporal dependencies.

Benefits and Drawbacks of Recurrent Neural Networks

RNNs excel at processing sequences. While they can naturally model temporal dependencies, adaptations are required to properly capture spatial dependencies too.

In the literature, LSTMs, GRUs and ESNs are the most widely used RNN variants since Vanilla RNNs are characterised by unstable training, impeding neural networks to capture long-term temporal dependencies (see Section 4). Such drawback is mostly overcome by LSTM networks and GRUs, using gating mechanisms to control the information flow within recurrent cells. Nevertheless, it must be remarked that these models still suffer from short-term memory, i.e., the inability in grasping long-term dependencies. Consider the RNN output $y_t$ at time t. It is predicted starting from the current input ${\overrightarrow{z}}_t$ and the hidden state ${\overrightarrow{h}}_t$, which is a rough condensation of the past inputs. ${\overrightarrow{h}}_t$ is the only way for the output layer to have knowledge about past inputs in the sequence. Unfortunately, there is no certainty that, after several recurrent steps ${\overrightarrow{h}}_1,{\overrightarrow{h}}_2,\dots ,{\overrightarrow{h}}_{t-1}$, adequate information about temporally distant inputs survives into ${\overrightarrow{h}}_t$. This effect gives the RNN and its most common variants a short-term memory, making them unable to preserve information from the distant past. A solution to this drawback is to use methods that can efficiently consider the whole temporal sequence (or most of it, at least) as one. Attention mechanisms are an example of such a method (see Section 6).

The short-term memory of RNNs is partially due to their sequential nature, which represents yet another limitation. Sequential processing in RNNs, in fact, limits parallelization of such models, making them hard to scale to longer sequences or more data.

5. Hybrid Convolutional–Recurrent Networks

A major research direction in spatio-temporal modelling has been the integration of convolutions with RNNs. On one hand, convolutions are naturally suited to model spatial data (e.g., images); on the other, RNNs give their best when modelling temporal sequences. The idea is to join both approaches into a single, hybrid convolutional–RNN. In this domain, there are some works that apply convolutions and RNNs as two separate modules [16,75,76,77], either convolution-first or RNN-first. A lot of attention, however, has been focused into directly integrating convolutions within recurrent units. One of the earliest works in this direction is the Convolutional LSTM (ConvLSTM) [12]. ConvLSTM was proposed to tackle precipitation nowcasting tasks, but it is applicable to general spatio-temporal series modelling tasks. The core idea is to nest convolutions within recurrent LSTM equations, i.e., Equations (8) and (9), by substituting matrix products with convolution operators. ConvLSTM is a modification of an alternative LSTM formulation [78], slightly different from the one described in Section 4. A ConvLSTM layer is controlled by the following equations:

$$\begin{array}{cc}\hfill i_t& =tanh(W_i\ast z_t+U_i\ast h_{t-1}+b_i),\hfill \\ \hfill g_t& =\sigma \left(W_g\ast z_t+U_g\ast h_{t-1}+V_g\odot c_{t-1}+b_g\right),\hfill \\ \hfill f_t& =\sigma \left(W_f\ast z_t+U_f\ast h_{t-1}+V_f\odot c_{t-1}+b_f\right),\hfill \\ \hfill c_t& =g_t\odot i_t+f_t\odot c_{t-1},\hfill \\ \hfill o_t& =\sigma \left(W_o\ast z_t+U_o\ast h_{t-1}+V_o\odot c_t+b\right),\hfill \\ \hfill h_t& =o_t\odot tanh\left(c_t\right).\hfill \end{array}$$

(15)

ConvLSTM processes a series of spatial data, such as multichannel images, thus $z_t\in {\mathbb{R}}^C\otimes {\mathbb{R}}^H\otimes {\mathbb{R}}^W$, where C is the number of channels (e.g., RGB), and H and W are the spatial dimensions. The variables $c_t$ and $h_t$ denote, respectively, the inner state and output of ConvLSTM and have the same spatial dimensions of the input $z_t$ ($i_t$, $g_t$, $f_t$ and $o_t$ also share the same dimensionality). Their number of channels depends on the number of filters used in the convolution (i.e., ∗). The triplets $(W_i,U_i,b_i)$, $(W_g,U_g,b_g)$, $(W_f,U_f,b_f)$ and $(W_o,U_o,b_o)$ are learnable parameters; the first two elements of each triplet are convolutional kernels, while the last one of each triplet is a bias tensor. Figure 4 shows a simplified example of ConvLSTM’s inner mechanisms. Each cell in the hidden state $h_t$ depends on a small local neighbourhood from both the hidden state at the previous time step $h_{t-1}$ and the input at the current time step $z_t$. Such dependency is modelled through convolutions, which preserves spatial structure.

An extension of ConvLSTM is Trajectory GRU (TrajGRU) [79]. TrajGRU is based on ConvGRU, a GRU where all matrix products in its reset and update gates have been replaced with convolutions similar to ConvLSTM. A key innovation of TrajGRU resides in using location-variant convolutions, where the neighbourhood structure at a given position is not fixed by the kernel size and shape, but it is learned during training. This is similar to the concept of deformable convolutions, and it is implemented through a trainable structure generating network, that is, a small CNN that receives the current input $z_t$ and the previous GRU output $h_{t-1}$ and produces a flow field indicating the structure of the neighbourhood at any given location.

Another extension of ConvLSTM is the Predictive RNN (PredRNN) [14,80]. The key innovation of PredRNN resides in how the information flows across time and between multiple ConvLSTM layers. In a multilayer scenario, the l-th ConvLSTM layer, at the t-th time step, receives two inputs: its own output from the previous time step, $h_{t-1}^l$, and the output of its preceding ConvLSTM layer, $h_t^{l-1}$, which is the current input $z_t$ if $l=0$ (i.e., the first layer). In this way, layers are loosely interconnected and almost mutually independent. In fact, each layer tracks its own temporal evolution but has almost no clue on the features extracted so far by the whole network. The main finding of PredRNN is that a unified memory pool, tracking the evolution of information in time and across all layers, is useful for spatio-temporal predictive learning. It proposes to implement such mechanism by adding an additional path along which information can flow: at each time step, the first ConvLSTM layer also receives the output of the last layer of the network from the previous time step. This new path is integrated in the canonical ConvLSTM layer through dedicated equations that can be found in the original papers [14,80].

The Memory in Memory (MiM) network [81] further extends PredRNN in order to better model non-stationary dynamics in spatio-temporal processes. MiM authors noted that the forget gate of ConvLSTM (and other LSTM-like units) often gets saturated, indicating that they cannot adapt to non-stationary dynamics. They propose to overcome this limitation by replacing the forget gate with two cascaded modules, dedicated, respectively, to capture non-stationary and stationary behaviours.

Finally, the aforementioned works are summarised in

Table 3. Articles proposing hybrid convolutional–recurrent models.

Ref.	Year	Type	Applications	Datasets
[12]	2015	LSTM	Precipitation Nowcasting	MovingMNIST; Radar Echo
[79]	2017	GRU	Precipitation Nowcasting	MovingMNIST++; HKO-7
[80]	2017	LSTM	Video Prediction	MovingMNIST; KTH; Radar Echo
[82]	2018	LSTM	Sea Surface Temperature Modelling	NOAA OISST; NASA AVHRR
[81]	2019	LSTM	Spatio-temporal Forecasting	MovingMNIST; TaxiBJ; Human3.6M; Radar Echo
[75]	2019	GRU	Land Cover Classification	Sentinel-2 Data
[76]	2019	LSTM	Urban Land Cover Classification	Sentinel-2 Data
[83]	2019	GRU	Land Cover Classification	Sentinel-1 Data; Sentinel-2 Data
[1]	2019	LSTM	Sea Surface Temperature Modelling	NOAA OISST
[16]	2019	RNN	Remote Sensing Change Detection	Landsat ETM+
[84]	2021	LSTM	Urban Expansion Prediction	SPOT Satellite Data
[85]	2021	Various	Video Prediction; Precipitation Nowcasting	Human3.6M; Shangai Precipitation Data; MovingMNIST
[77]	2022	LSTM	Crop Yield Prediction	MODIS; Landsat8; Sentinel-2
[86]	2022	LSTM	NDVI Forecasting	Sentinel-2; ERA5; SMAP; SRTM
[14]	2023	LSTM	Spatio-temporal Forecasting	MovingMNIST; KTH; Radar Echo; Traffic4Cast; BAIR

.

Benefits and Drawbacks of Hybrid Convolutional–Recurrent Networks

Convolutional and recurrent neural networks are suited for the processing of spatial and temporal data, respectively. Hybrid CNN-RNN models combine the best of both worlds, becoming a reasonable choice for the modelling of spatio-temporal data. Unlike fully connected RNNs, such hybrid models can process images as tensors directly, without the need of linearizing them into vectors and, therefore, preserving the input spatial structure. Furthermore, the use of convolutional kernels gives these hybrid models the additional weight sharing property, typical of CNNs. Therefore, hybrid models can generally have a smaller number of parameters than fully connected RNNs. However, since they are often based on Euclidean kernels, they may not capture long range or irregular spatial dependencies (see Section 3).

Unlike convolutional models, thanks to the use of recurrent units, CNN-RNN hybrid models can capture temporal dependencies more precisely and at a finer level. However, similar to RNNs, these hybrid models can still suffer from vanishing/exploding gradients and from short-term memory problems when modelling very long sequences (see Section 4).

6. Transformers

Transformers [11] were originally designed for sequence-to-sequence transduction tasks in natural language processing, and they have soon been adapted for different tasks in the spatio-temporal field as well, such as spatio-temporal forecasting and video processing. Videos are actual examples of regular raster spatio-temporal data since any temporal sequence of spatial maps (i.e., images) can be seen as a video. Like RNNs, transformers can process sequences. Transformers, however, compare favourably against RNNs in sequence modelling tasks as they overcome two main RNN drawbacks: the difficulty in parallelising computations due to RNNs’ inherent sequential nature and short-term memory [11] (see Section 4). Transformers address these limitations through the use of attention mechanisms, which enable models to selectively focus on specific parts of the input depending on the information being processed. An attention mechanism operates on a query vector $\overrightarrow{q}$ and a set of N value vectors ${\overrightarrow{v}}_i$, each associated with a corresponding key vector ${\overrightarrow{k}}_i$. The query encodes the information currently being processed, while the keys and values represent the input elements over which attention is applied. Typically, keys and values are the same vectors. Among several definitions of attention, transformers adopt the scaled dot-product attention [11], which implements a dot product between query and key vectors. A transformer operates on sequences of queries, keys and values, which are generally referred to as tokens, a term borrowed from natural language processing, indicating an element of a sequence as processed by a transformer. Typically, there are multiple queries, and an attention mechanism is applied to each one of them, yielding multiple output vectors as follows:

$$Y=\mathrm{Attention}(K,V,Q)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{D_k}}}\right)V.$$

(16)

Here, $Q\in {\mathbb{R}}^M\otimes {\mathbb{R}}^D$ denotes the matrix whose rows correspond to the query vectors. Analogously, the keys and values are arranged as rows of the matrices $K\in {\mathbb{R}}^N\otimes {\mathbb{R}}^D$ and $V\in {\mathbb{R}}^N\otimes {\mathbb{R}}^D$, respectively. The attention output $Y\in {\mathbb{R}}^M\otimes {\mathbb{R}}^D$ is a matrix whose i-th row represents the output vector associated with the i-th query. Within transformers, attention mechanisms are typically distinguished into self-attention and cross-attention. In self-attention, queries, keys and values originate from the same sequence, enabling attention among its elements, whereas in cross-attention, queries and keys/values come from different sequences.

In practice, transformers employ Multi-Head Attention (MHA) in which queries, keys and values are linearly projected H times into subspaces of dimensions $D_k$, $D_v$ and $D_k$, respectively, using learned projection matrices. Scaled dot-product attention is then applied independently to each set of projections, after which the resulting outputs are concatenated and linearly mapped back to the original space. Each projection-attention couple defines an attention head $h_i$ with three learned matrices $W_i^K\in {\mathbb{R}}^D\otimes {\mathbb{R}}^{D_k}$, $W_i^V\in {\mathbb{R}}^D\otimes {\mathbb{R}}^{D_v}$ and $W_i^Q\in {\mathbb{R}}^D\otimes {\mathbb{R}}^{D_k}$, used to project keys, values and queries, respectively. Each attention head computes scaled dot-product attention, as defined in Equation (16), over its projected inputs:

$$h_i=\mathrm{Attention}(K{W}_i^K,V{W}_i^V,Q{W}_i^Q)\forall i\in [1,H]$$

(17)

The outputs $h_i$ from all attention heads are concatenated into a single matrix and linearly re-projected to the original D-dimensional space via an additional projection matrix $W^o\in {\mathbb{R}}^{H{D}_v}\otimes {\mathbb{R}}^D$.

$$\mathrm{MHA}(K,V,Q)=\mathrm{Concatenate}(h_1,h_2,\dots ,h_H)W^o$$

(18)

Standard transformers adopt an encoder–decoder architecture [11]. The encoder processes the input sequence and, through self-attention, transforms it into a hidden representation. The decoder generates the output sequence by combining cross-attention over the encoder’s hidden representation with self-attention over its previous outputs. Variants of the architecture often rely exclusively on either the encoder [87] or the decoder [88], depending on the specific task and the modelling paradigm. It is worthwhile to remark that transformers assume no intrinsic ordering of inputs, that is, input is considered as a collection of samples rather than an ordered stream of samples. If a specific input ordering must be considered for the task at hand, it must be explicitly encoded in the input embedding. A common approach to encode sample position within a sequence consists in summing a positional embedding $E_{pos}$ to the main sample embedding $\mathcal{X}\in {\mathbb{R}}^L\otimes {\mathbb{R}}^D$ [11]:

$$input=\mathcal{X}+E_{pos}.$$

(19)

Traditional transformers are naturally suited to process sequences. Efforts have been spent to adapt the transformer architecture for the processing of spatio-temporal data, mostly in the video processing domain. The extension has been gradual: first, transformers have been adapted to process images (i.e., regularly sampled spatial data), with models such as the vision transformer [89] and the Swin transformer [90]. Then, these models have been extended for the processing of videos [27,28], which are a further example of spatio-temporal data. In this section, two families of spatio-temporal transformers are discussed: spatio-temporal transformers developed for video processing tasks, primarily video classification tasks (Section 6.1), and spatio-temporal transformers built for specific, spatio-temporal tasks, such as Earth systems modelling or traffic flow prediction (Section 6.2).

6.1. Vision Transformers and Transformers for Video Processing

Vision transformers [89] (ViT) were introduced to tackle the problem of image classification without relying on convolution-based models. ViTs are based exclusively on the transformer architecture [11], and their main innovation concerns how their input is tokenized. Unlike the Natural Language Processing domain, in a vision transformer, representing an image as a set of tokens is not straightforward. ViT tokenizes an image by dividing it into small, non-overlapping 2D patches of fixed size. Each patch matrix is flattened and encoded through a linear embedding. A position embedding is added to encode the patch position within the original image. The sequence of patches is then processed by self-attention through a standard transformer encoder [11].

A video-transformer-based model, inspired by ViT, is the time–space transformer (TimeSformer) [91]. TimeSformer tokenizes videos by sampling N frames at random and encoding each frame as a sequence of small patches, as in ViT. Patch sequences from all frames are concatenated in a single sequence, representing the whole video, which is processed by a standard transformer encoder, as in ViT. The computational efficiency of TimeSformer is improved by a divided space–time attention mechanism, which applies attention separately in time and space. Firstly, TimeSformer applies time attention among all tokens from the same temporal index; then, it applies space attention among all tokens with the same spatial index. A similar approach is followed by the video vision transformer (ViViT) [27]. It tokenizes the video in one of two ways: uniform frame sampling, i.e., the same tokenization procedure adopted in TimeSformer, and tubelet embedding, which extracts tokens from small non-overlapping spatio-temporal boxes that cover the entire input video. Each spatio-temporal box is flattened, encoded through a linear embedding and concatenated into one long token sequence representing the whole video. The resulting sequence is finally processed by a standard transformer encoder, as in ViT. However, ViViT offers some less computationally demanding variants, which factorise the processing of the spatial and temporal dimensions of the video at either architecture level or self-attention level. Figure 5 shows an example pipeline of a generic video vision transformer inspired by ViViT. First, patches are extracted from video frames according to the patching mechanism adopted by the specific model. A D-dimensional embedding is then computed for each patch, and positional embedding is added to encode patch position within the video. The sequence of patches is then input to a stack composed of L transformer encoder layers. Finally, a prediction head produces the output according to the task being tackled.

The multiview transformer (MTV) [92] is another spatio-temporal transformer model for video processing. MTV, based on ViViT, introduces a multiview architecture to process videos by considering multiple temporal scales. MTV tokenizes videos using the aforementioned tubelet embedding. Multi-scale views of the input video are constructed by tokenizing the video multiple times, and tubelets with different temporal sizes are used at each time. Views with smaller tubelets capture fine-grained video details, while larger tubelets capture coarser aspects of the video. Thus, MTV builds a sequence of multiscale views ordered from the coarsest to the finest scale. Each view is processed with a proper transformer encoder, just as ViT and ViViT. Furthermore, in MTV, each encoder has a cross-view fusion module that aggregates information sequentially between encoders of two adjacent views. Since MTV was proposed for video classification, the encoder of each view maintains a classification token that aggregates from all other tokens in the view the information required for the classification. Finally, classification tokens from each view are concatenated and processed by a further transformer encoder to yield the final classification.

Unlike the aforementioned ViT-based models, the unified transformer (UniFormer) [93] unifies under one transformer architecture the spatio-temporal locality awareness (typical of 3D convolutions) and the skill of capturing long-term dependencies, typical of transformers. UniFormer introduces a specific relation aggregator block, with two variants: a local variant, capturing local spatio-temporal dependencies, and a global one, grasping global spatio-temporal dependencies across the whole video. The former block type is based on 3D convolutions, which, in a less computationally demanding way, model dependencies efficiently in small spatio-temporal neighbourhoods, while the latter is based on self-attention. Both blocks work with multiple heads, each, with its own parameters, computing an output according to the block type. Outputs of all heads are concatenated and linearly projected back to the input space. These relation aggregator blocks are organised in such a way that local dependencies are grasped in shallow layers of the network, while global ones are modelled by deeper layers. In fact, UniFormer is composed of four relation aggregator blocks: the first two are local while the remaining two, the deepest in the network, are global. Three-dimensional convolution layers are employed between each block pair in order to downsample both spatially and temporally the original input video.

The Swin transformer [90] is an alternative to ViT in the image processing domain. Its main novelty is the shifted window attention. As in ViT, Swin tokenizes images from small non-overlapping patches, which are concatenated along the depth and encoded through a linear embedding. Tokens are then divided into small non-overlapping, fixed-size squares (windows), and multi-head attention is applied just among tokens of the same window. To connect tokens of different windows, attention is applied in two steps. First, attention is applied as just described; then, each window is shifted by half its size towards the bottom-right corner of the image, and attention is reapplied on these shifted windows. In this way, tokens take into account the content of adjacent windows thanks to the application of the just-described procedure. The application of windowed attention is, therefore, always followed by the application of a shifted window attention. Furthermore, patch merging mechanisms downsample the input image in deeper layers. The video Swin transformer [28] is a straightforward extension of the Swin transformer to the video processing domain. First, video Swin tokenizes videos by extracting small, non-overlapping spatio-temporal boxes. Just like Swin, these boxes are concatenated along the depth and projected linearly. Video Swin also adopts the shifted window attention, but, in this case, 3D windows of a fixed size are considered. Attention is again applied in two phases: first, using a spatio-temporal windowing of all tokens and then considering a shifted window configuration in order to connect the neighbouring windows. Patch merging is employed before each attention block, and it works exactly as in Swin transformers, namely, spatial dimensions are each downsampled, while the temporal dimension is left untouched.

There are several other relevant transformer-based models for video processing. For instance, there is the Video Transformer Network (VTN) [94], for video classification, that is based on the well-known Longformer model [95] to process longer video clips; the Video Action Transformer Network [26] for action identification in videos; Trackformer [96] for multiple object tracking in videos; the Spatio–temporal Masked Autoencoder [97], which explores unsupervised pre-training and transfer learning on videos.

Table 4. Transformer-related publications for video processing along with their publication year, their applications and the used datasets.

Ref.	Year	Applications	Datasets
[98]	2017	Video Super-Resolution	CDVL
[26]	2019	Video Action Recognition	AVA
[99]	2020	Video Inpainting	YouTube-VOS; DAVIS
[94]	2021	Video Action Recognition	Kinetics; MiT
[91]	2021	Video Action Recognition	Kinetics; Something-Something; Diving-48
[27]	2021	Video Action Recognition; Video Classification	Kinetics; Something-Something; Epic-Kitchens; MiT
[100]	2021	Object Tracking	LaSOT; GOT-10K; COCO2017; TrackingNet
[96]	2022	Object Tracking	MOT17; MOTS20
[92]	2022	Video Classification	Kinetics; Something-Something; Epic-Kitchens; MiT
[93]	2022	Video Classification	Kinetics; Something-Something
[101]	2022	Video Action Recognition	Kinetics; Something-Something
[28]	2022	Video Classification	Kinetics; Something-Something
[102]	2022	Video Captioning	MSVD; YouCookII; MSRVTT; TVC; Vatex
[97]	2022	Video Representation Learning; Video Classification	Instagram Video Data; Kinetics
[103]	2023	Deep Fake Video Detection	FaceForensics++; DeepFakeDetection; Celeb-DF-v2; DeeperForensics-1.0; WildDeepfake
[104]	2023	Video Depth Estimation	dVPN; SCARED
[105]	2023	Video Object Segmentation	Youtube-VOS; DAVIS
[106]	2023	Video Summarization	SumME; TVSum
[107]	2023	Video Prediction	BAIR; KITTI; RoboNet
[108]	2024	Video Super-Resolution; Video Deblurring; Video Denoising	Multiple datasets
[109]	2024	Video Action Recognition	AVA; UCF101; Epic-Kitchens
[110]	2024	Object Tracking	LaSOT; GOT-10K; UAV123; NfS; OTB2015; VOT2018; TempleColor128
[111]	2024	Video Action Recognition	Kinetics; Something-Something

summarises all transformer-based models for video processing that have been analysed herein.

6.2. Transformers for Spatio-Temporal Processing in Heterogeneous Domains

In addition to video processing, the transformer architecture has also been adapted for spatio-temporal modelling in other applicative domains, such as Earth systems forecasting and traffic flow prediction. A relevant work in this area is Earthformer [13], a transformer-based model for spatio-temporal Earth systems forecasting. Earthformer proposes a generic space–time attention, the cuboid attention, which can capture long-term dependencies between input values while maintaining a relatively low computational complexity. Another relevant work using transformers to tackle Earth-related tasks is Contextformer [15], specifically proposed to forecast vegetation health from satellite imagery by leveraging the GreenEarthNet dataset [15], proposed in the same article. Contextformer relies solely on transformers to model both space and time. It follows a similar methodology to MMST-ViT [112], and it is based on pyramid vision transformers [113,114] and Presto [115], a pre-trained transformer for satellite time series, to encode visual inputs and to model temporal relationships, respectively. The SwinLSTM Network [116] is yet another architecture that tackles heterogeneous spatio-temporal prediction tasks, such as traffic flow prediction, by integrating the Swin tranformer attention block (see Section 6.1) in the LSTM unit (see Section 4). Papers using transformer-based deep learning models to tackle spatio-temporal tasks in heterogeneous domains are summarised in

Table 5. Transformer-related publications for spatio-temporal modelling in heterogeneous domains.

Ref.	Year	Applications	Datasets
[117]	2020	Crop Classification	Sentinel-2; Landsat-8
[118]	2021	3D Human Motion Prediction	Human3.6M; AMASS
[119]	2022	Remote Sensing Change Detection	LEVIR; WHU-CD
[17]	2022	Remote Sensing Super Resolution	Jilin-1 Custom
[13]	2022	Earth Systems Forecasting	MovingMNIST; SEVIR; ICAR-ENSO
[120]	2022	Multivariate Spatial Time Series Forecasting	PeMS; Electricity; Traffic
[22]	2022	Traffic Forecasting	METR-LA; Urban-BJ; Ring-BJ
[121]	2022	Remote Sensing Change Detection	CDD; WHU-CD; OSCD; HRSCD
[122]	2022	Remote Sensing Change Detection	Farmland-CD; Barbara-CD; BayArea-CD
[123]	2023	Spatio-temporal PM2.5 Forecasting	Custom EPA AQS
[116]	2023	3D Human Motion Prediction; Traffic Forecasting; Action Recognition	MovingMNIST; Human3.6M; TaxiBJ; KTH
[112]	2023	Crop Yield Prediction	USDA Crop Data; HRRR Data; Sentinel-2
[23]	2024	Traffic Forecasting	PeMS; Zhengzhou
[15]	2024	Vegetation Forecasting	GreenEarthNet

.

6.3. Benefits and Drawbacks of Transformers

Transformers offer several advantages for regular raster spatio-temporal prediction tasks. First, the use of attention mechanisms allows transformers to efficiently catch global dependencies, which is fundamental in many spatio-temporal prediction tasks, where long-range dependencies exist among input data. Moreover, attention mechanisms can be easily parallelized, making transformers generally more scalable than recurrent models. This property is particularly useful in the regular raster spatio-temporal context, where dependencies need to be captured simultaneously across space and time. However, the computational cost of self-attention scales quadratically with the number of queries involved, making it crucial to limit sequence length or adopt strategies that reduce the effective number of queries. Over the last years, several works have focused on dealing with longer sequences while keeping a relatively low computational cost, e.g., Longformer [95], to quote one.

Transformers easily support the integration of data from multiple sources and modalities through attention mechanisms, particularly cross-attention. This capability is very useful in spatio-temporal prediction tasks, such as those in the environmental domain, where data are often heterogeneous and multimodal.

Transformers are also well known for being massive data hungry, and their performance typically depends on the availability of large datasets. Fortunately, nowadays, there are many pre-trained transformers, especially in the vision domain, which have been trained on extensive datasets and can be fine-tuned to regular raster spatio-temporal prediction tasks, e.g., video analysis or environmental forecasting.

7. Diffusion Models for Spatio-Temporal Modelling

Generative models have been a breakthrough in machine learning due to their large applicability and their high-performance in various applicative domains, such as computer vision [124,125,126], natural language processing [87,88] and time-series prediction [127,128,129]. Generative Adversarial Networks (GANs) [130] were, historically, at the forefront of generative modelling as they showed a superior performance compared to other deep learning methods in data generation tasks. However, In recent years, diffusion models (DMs) [131,132] have noticeably outperformed GANs in generative modelling tasks, gaining significant attention not only for their performance but also for several key advantages. First, they are relatively simple: the neural networks employed in DMs often rely on simple architectures, e.g., autoencoders, and require comparatively few parameters given the complexity of the problems that they address. In addition, they are firmly grounded in statistical physics, dynamics and Bayesian theory, which provides them strong theoretical foundations. Finally, their versatility makes them highly applicable to many relevant real-world tasks, often requiring simple and limited modifications to their original architectures. Such large applicability of DMs has also manifested in a rising interest in spatio-temporal problems, such as generation [133], forecasting [134], classification [135] and anomaly detection [136].

The goal of generative models is to learn the underlying probability distribution $p\left(x\right)$, generating the observed data x. If this probability distribution is known, generating new samples $x^{\prime }$ simply consists of sampling from $p\left(x\right)$. Diffusion models can be considered as an extension of Variational Auto Encoders (VAEs) [137] with three additional characteristics. First, a diffusion model is a Markovian cascade of VAEs, yielding a sequence of latent variables $x_t\forall t\in [1,T]$ whose dimensionality is the same of the one of the input space. Second, the structure of the encoder is fixed as a linear Gaussian model that iteratively uses as the mean of the Gaussian distribution at the next iteration the output sampled from the Gaussian distribution at the previous iteration. Finally, the encoder process should guarantee that the latent distribution of the final iteration T is a standard Gaussian distribution $\mathcal{N}(\overrightarrow{0},\mathbb{I})$. According to the first characteristic, the encoder distribution of a DM can be written as

$$q\left(x_{1:T}\right|x_0)=\prod _{t=1}^{T}q\left(x_t\right|x_{t-1}),$$

(20)

where $x_0$ is the input sample, and T is the number of diffusion iterations. Based on the second point, the distribution $q\left(x_t\right|x_{t-1})$ is a Gaussian distribution whose mean depends on $x_{t-1}$ due to the Markovian assumption. Specifically, the mean ${\mu }_t\left(x_t\right)=\sqrt{{\alpha }_t}{x}_{t-1}$ and the variance ${\Sigma }_t\left(x_t\right)=(1-{\alpha }_t)\mathbb{I}$ of the t-th distribution incorporate the coefficient ${\alpha }_t$ that preserves the variance among all the latent variables. In short, the second assumption can be mathematically defined as

$$q\left(x_t\right|x_{t-1})=\mathcal{N}(x_t;\sqrt{{\alpha }_t}{x}_{t-1},(1-{\alpha }_t)\mathbb{I})$$

(21)

The final characteristic is guaranteed by letting ${\alpha }_t$ evolve gradually and sufficiently over time in a way such that $x_T\sim \mathcal{N}(0,\mathbb{I})$. The application of such an iterative Markovian process on the input sample $x_0$ defines the so-called forward process, which produces, after T iterations, the final latent sample $x_T$. The goal is now to define the reverse process, whose objective is to generate a sample $x_0^{\prime }$, starting from $x_T$, which is as similar as possible to the original one. Mathematically, this process is defined by the joint distribution:

$$p\left(x_{0:T}\right)=p\left(x_T\right)\prod _{t=1}^T{p}_{\theta }\left(x_{t-1}\right|x_t),$$

(22)

where $p\left(x_T\right)$ is a standard Gaussian, as stated before, and $p_{\theta }\left(x_{t-1}\right|x_t)$ is a learned conditional distribution that effectively models the unknown transition distribution of the reverse process, $q\left(x_{t-1}\right|x_t)$. The distribution $p_{\theta }$ is generally implemented as a neural network such that

$$p_{\theta }\left(x_{t-1}\right|x_t)=\mathcal{N}(x_{t-1};{\mu }_t^{\theta }\left(x_t\right),{\sigma }_t^2\mathbb{I}),$$

(23)

where ${\mu }_t^{\theta }\left(x_t\right)$ is the mean predicted by a $\theta$-parameterised neural network, and ${\sigma }_t^2\mathbb{I}$ defines a fixed noise scale according to the schedule ${\sigma }_t=1-{\alpha }_t$.

In principle, an Evidence Lower Bound (ELBO) objective function could be used to train DMs. Unfortunately, it can be proven that such an objective has a high variance, causing instabilities during training [131]. To avoid such drawback, Denoising Diffusion Probabilistic Models (DDPMs) [132] are proposed, recasting DMs’ training as a denoising task according to the following objective:

$$\underset{\theta }{argmin}\gamma \left({\alpha }_t\right)\left[\left|\right|ϵ-{ϵ}_t^{\theta }\left(x_t\right){\left|\right|}_2^2\right],$$

(24)

where $ϵ\sim \mathcal{N}(0,\mathbb{I})$ is the noise amount used to corrupt the original sample $x_0$, ${ϵ}_t^{\theta }\left(x_t\right)$ is the amount of noise to subtract predicted by a $\theta$-parameterised neural network and $\gamma \left({\alpha }_t\right)$ indicates a positive coefficient that depends on the noise schedule parameter ${\alpha }_t$.

A crucial limitation of DDPMs is their slow sampling speed. To generate a single sample $x_0^{\prime }$, the entire sequence of reverse diffusion steps must be performed as the forward process is assumed to be Markovian. If the number T of diffusion iterations is large, the inference time of DMs can become unfeasible. Denoising Diffusion Implicit Models (DDIMs) [138] overcome this limitation, reformulating the forward transition $q\left(x_t\right|x_{t-1})$ by conditioning it on the original sample $x_0$ via Bayes’ rule:

$$q\left(x_t\right|x_{t-1},x_0)={\displaystyle \frac{q\left(x_{t-1}\right|x_t,x_0)q\left(x_t\right|x_0)}{q\left(x_{t-1}\right|x_0)}}.$$

(25)

The resulting forward process is, therefore, non-Markovian; however, it must be noted that DDIM’s forward process retains the same marginal distributions $q\left(x_t\right|x_{t-1})$ as DDPMs. DDIM’s sampling procedure is derived by conditioning the learned reverse distribution on the original sample $x_0$:

$$p_{\sigma ,\theta }\left(x_{t-1}\right|x_t,x_0)=\mathcal{N}\left(\sqrt{{\alpha }_{t-1}}{x}_0+\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}{\displaystyle \frac{x_t-\sqrt{{\alpha }_t}{x}_0}{\sqrt{1-{\alpha }_t}}},{\sigma }_t^2\mathbb{I}\right),$$

(26)

where ${\sigma }_t\in {\mathbb{R}}^{+}$ introduces a controllable level of stochasticity in the reverse process. Notably, assuming the denoising setting, the term $x_t$ is only used to estimate the source noise $ϵ$, expressed, in Equation (26), as follows:

$$ϵ={\displaystyle \frac{x_t-\sqrt{{\alpha }_t}{x}_0}{\sqrt{1-{\alpha }_t}}}.$$

(27)

Due to the unavailability of the original sample $x_0$ in the reverse process, it has to be estimated. This can be achieved by inverting Equation (27) and by introducing a neural network ${ϵ}_{\theta }\left(x_t\right)$ that predicts the noise $ϵ$:

$$x_0\simeq f_{\theta }\left(x_t\right)={\displaystyle \frac{(x_t-\sqrt{1-{\alpha }_t}·{ϵ}_{\theta }\left(x_t\right))}{\sqrt{{\alpha }_t}}}.$$

(28)

Equations (27) and (28) provide the first interpretation of DMs, where the training goal is to learn the source signal noise. However, other equivalent DM interpretations can be found in the literature [131]. In

Table 6. Publications for spatio-temporal prediction using diffusion models.

Ref.	Year	Applications	Datasets
[134]	2023	Forecasting	SST; Navier–Stokes; Spring-Mesh
[136]	2024	Anomaly Detection	GOES-16; GOES-17
[139]	2024	Video Prediction	KITTI; Cityscapes; KTH; BAIR; MovingMNIST
[140]	2024	Precipitation Nowcasting	SEVIR
[141]	2024	Cloud Removal	Sen2MTCNew; WHUS2-CRv; SEN12MS-CR
[142]	2025	Wind Speed Prediction	Wind Toolkit Data V2
[19]	2025	Satellite Image Inpainting	Massachussets Roads; DeepGlobe 2018
[29]	2025	Video Generation	RealEstate; ACID; DL3DV
[143]	2025	Video Generation	Sky Time-Lapse; UUCF101; MHAD; WebVid

, some applications of DMs for spatio-temporal prediction tasks are reported.

Benefits and Drawbacks of Diffusion Models

DMs are powerful probabilistic generative models that have attracted significant attention for their dramatic outperformance in generative modelling and beyond. In addition to their strong performance, their probabilistic nature enables them to estimate prediction uncertainty, which can greatly enhance the value of the model predictions. Furthermore, the DM framework is highly flexible as models trained only for data generation task can be easily readapted to different tasks, such as inpainting or missing data imputation.

DMs are not free of drawbacks. Because of their nature, they are bounded to have a Gaussian distribution as a starting distribution [144]. This limits the solution space for the Markovian process that DMs have to approximate since it is constrained to always start from a normal distribution. This limitation is overcome by Flow Models that can start from any distribution [145,146].

The generation process of DMs is stochastic as randomness is inherent in their generative process. Unlike other models, such as Flow Models, this property causes that the reverse process in DMs converges more slowly to the target, often requiring several reverse diffusion steps. To address that, alternative sampling methods have been developed to reduce the number of steps during the reverse process and thereby accelerating sampling while still preserving the marginal distributions [138,147].

DMs, and Flow Models alike, enforce a very specific instantaneous transformation of the sample during each time step in the generation process. This aspect, which may limit model expressiveness, is overcome by the latest frontier of generative modelling, Generator Matching [148], which encompasses DMs, Flow Models and jump processes [146], all in one framework.

8. Discussions and Future Research Directions

Previous sections presented the most commonly adopted deep learning architectures for regular raster spatio-temporal prediction. As shown in Figure 6, there is a fairly even distribution of articles with respect to the reviewed architectures, with a slightly higher concentration of works on transformers for video processing.

Literature analysis highlights five applicative domains in which regular raster spatio-temporal data are most popular, namely, video processing [26,27,28,29], environmental monitoring [12,13,14,15], remote sensing [16,17,18,19], traffic modelling [20,21,22,23] and energy production prediction [24,25]. As displayed in Figure 7, video processing is the realm where spatio-temporal prediction on regular raster data is most often encountered, followed by environmental monitoring, remote sensing, traffic modelling and energy prediction. It should be noted that environmental monitoring and remote sensing fields may sometimes overlap as environmental applications often handle remote sensing data. In this setting, all the works categorised as remote sensing are those applying their methodology on remote sensing data without necessarily performing environmental monitoring (e.g., super resolution of satellite images).

Each architecture relies on specific functional mechanisms and principles, making them more or less suitable to be applied to a specific task. Figure 8 shows, for each reviewed architecture, the percentage of articles related to particular domains.

Since the introduction of ViTs, the transformer architecture has become state of the art in the video processing field. The reason of their success is twofold: the former resides in the flexibility of attention mechanisms, making transformers a solid foundation for building customised and more complex models for video processing. The latter reposes in the attention mechanisms’ capability to adaptively model long range spatio-temporal dependencies. This property makes transformers ideal candidates for environmental modelling tasks, where teleconnections and long-term dependencies generally exist within spatio-temporal data. Diffusion models have also achieved remarkable performance on video processing tasks, especially in the field of video generation. It is important to note, however, that diffusion models denote a deep learning framework rather than specific architectures. Early diffusion models were commonly built around CNNs based on U-Net [48] and, more recently, on diffusion transformers [149], a transformer architecture specifically designed for generative modelling and the integration of data from multiple modalities. Despite their effectiveness, transformer models also have some major drawbacks (see Section 6.3), making more traditional models, such as CNNs, RNNs and CNN-RNN hybrids, still relevant nowadays, especially in environmental sciences, as can be seen from Figure 8. Despite being limited in capturing global dependencies or long-range dependencies, the inductive biases of these models (e.g., the locality of CNNs) can make them competitive against more advanced architectures under different scenarios, e.g., when data availability is scarce and when having an estimate of prediction reliability is not crucial.

For the sake of completeness, it must be remarked that, although all the aforementioned architectures are presented for the processing of regularly spaced raster spatio-temporal data, they can also be adapted to process irregularly spaced data. However, this type of data is more naturally modelled and handled using spatio-temporal extensions of Graph Neural Networks [150,151], which are out of scope for this review. For an in-depth discussion on Graph Neural Networks for spatio-temporal prediction, the reader is referred to [62]. The topic of regular raster spatio-temporal prediction is wide, disparate and complex. Hence, there are some aspects, not analysed herein, that deserve much attention. In the following, major future lines of research are reported.

• Uncertainty estimation for spatio-temporal predictions. With the only exception of diffusion models, other reviewed deep learning approaches do not yield uncertainty estimates on their spatio-temporal predictions. This aspect is crucial in many spatio-temporal prediction tasks concerning environmental safety [152,153] or extreme events forecasting [154]. To introduce prediction uncertainty quantification, the Bayesian learning framework [155] can be used, or it can be achieved using generative AI models, such as Diffusion or Flow Models (see Section 7).
• Physics constraint in spatio-temporal prediction. The spatio-temporal dynamics of many real-world phenomena behave according to some physical laws and properties. Deep learning is of great utility to infer such spatio-temporal dynamics, but, in many cases, its data-driven nature can be empowered by embedding physical constraints, governing equations and first principles directly into the training process [156]. In this way, predictions can be not only statistically accurate but also coherent with known physical laws. This property is particularly important in many regular raster spatio-temporal prediction tasks, such as climate modelling [157,158] and fluid dynamics, where respecting physical laws is critical.
• Model explainability. While deep learning models can achieve strong predictive performance, they act as black boxes, making it difficult, especially for users that are not familiar with deep learning, to understand why certain predictions are obtained and thus limiting trust and applicability. Therefore, it is important to have strategies and methods that allow explainability of deep learning models in spatio-temporal prediction tasks [159]. A relevant direction towards model explainability is causality [160], namely, discovering cause–effect relationships within spatio-temporal input data. Integrating causality into deep learning models can provide not only better interpretable insights into the underlying spatio-temporal dynamics, but it can also yield more robust models with respect to generalisation.
• Foundation models. Foundation models [161] are a relevant recent topic in deep learning. These, representing the next generation of pre-trained models, are trained on massive and heterogeneous datasets, providing general-purpose backbones that can be fine-tuned or extended to diverse tasks. Recently, pre-trained foundation models have also been proposed for spatio-temporal data, especially for Earth observation. Some of the most relevant models in this field are Aurora [162], Prithvi [163], Clay [164], Presto [115] and DOFA [165], which leverage the concept of neural plasticity to adaptively integrate multiple data modalities according to the task being tackled. In fact, multimodal learning is another relevant topic in regular raster spatio-temporal prediction. Multimodal learning refers to the ability of integrating and using multiple data sources to address a given task, leveraging the diverse complementary information provided by heterogeneous data modalities. This capability plays a crucial role in many Earth observation tasks, such as in cloud removal, where the combined use of optical and radar data has proven particularly advantageous [166]. Furthermore, it is worthwhile to remark that a few foundation models based on Large Language Models (LLMs) have been applied to spatio-temporal prediction [167]. Finally, in the field of pre-trained models, the Mixture-of-Experts [168] is a further relevant topic. The mixture-of-experts, generally empowered by transformers [169], can be used to build large pre-trained models while maintaining a moderate computational complexity. Such models are deep learning architectures where different experts, which are still neural networks, process different parts of the input. A routing network decides which experts should process a given input, activating only a few at a time. Such a sparse computation guarantees mixture-of-experts models to have similar performance to traditional models while also being less computationally demanding and scalable.

9. Conclusions

Herein, a survey of the most relevant deep learning architectures for regular raster spatio-temporal prediction is provided. For each reviewed deep learning architecture, the following aspects have been analysed and discussed:

• Methodology and techniques underlying reviewed models;
• Benefits and drawbacks, allowing the identification of domains where some models can be more effective than others;
• Diverse applicative realms, with their respective datasets, where regular raster spatio-temporal prediction is relevant.

Specifically, literature analysis highlighted that the most commonly used deep learning methodologies for processing regular raster spatio-temporal data are CNNs, RNNs, Hybrid CNN-RNN models, transformers and diffusion models. Moreover, the fields in which regular raster spatio-temporal prediction is most often encountered are environmental monitoring and video processing, even if it is still popular in remote sensing, traffic modelling and energy production prediction. Finally, a brief discussion on other very recent and future research directions concerning regular raster spatio-temporal prediction is given, paying particular attention to prediction uncertainty estimation, physics-constrained models, explainability, pre-trained foundation models and multimodal learning.