Robust Recursive Fusion of Multi-Resolution Multispectral Images with Location-Aware Neural Network

Highlights

What are the main findings?

• The development of a robust Kalman filter (KF)-based image fusion method using location-aware neural networks (NNs).
• A scalable recursive operation framework that can process a stream of incoming images on the fly.

What are the implications of the main findings?

• The robust KF-based image fusion method can improve robustness to measurement outliers under minimal assumption on their statistical distribution, and the proposed location-aware NNs improve estimation accuracy while reducing efforts of collecting large datasets for NN training.
• The scalable recursive operation can process images as they are observed, with a complexity that is linear in the length of the image sequence.

Abstract

Multi-resolution image fusion has been studied for years to solve the trade-off between temporal and spatial resolution in remote sensing instruments and has been widely applied to detect and monitor natural phenomena like floods. Despite the considerable research on this topic, the mitigation of the influence of outliers, such as cloud and shadow miscorrections, on satellite image fusion has not been fully developed. Moreover, strategies that integrate robustness, recursive operation and learned models are missing. In this paper, we design a robust recursive image fusion framework leveraging a location-aware neural network (NN) to model image dynamics. Outliers are modeled by representing the probability of contamination of a given pixel and band. An NN model trained on a small dataset provides accurate predictions of stochastic image time evolution, which improves both the accuracy and robustness of the method. A recursive solution is proposed to estimate high-resolution images by using a Bayesian variational inference framework. Experiments fusing images from the Landsat 8 and MODIS instruments show that the proposed method generally reduces the root mean square error (RMSE) and misclassification percentage of the estimated image by over $10\%$ without cloud cover and over $20\%$ with cloud cover compared with the benchmark KF algorithm. This indicates that the proposed approach is significantly more robust against cloud cover, without losing performance when no clouds are present.

1. Introduction

Satellite-based remote sensing is an essential tool for many applications, such as the monitoring of land-cover use [1], deforestation [2] or water quality [3]. A major concern when leveraging satellite-based imaging regards the trade-off among temporal, spatial, and spectral resolution. Such trade-off is due to the large distances between space-borne instruments and target scenes and limitations of multiband imaging systems. These limitations imply that higher spatial resolution leads to longer revisit times. For instance, the Moderate-Resolution Imaging Spectroradiometer (MODIS) has pixel sizes of 250/500/1000 m with a daily revisit period, while Landsat 8 captures images with pixel sizes of 30/100 m once every 16 days [4]. This is a major issue when monitoring events that change rapidly and at the same time require high spatial resolution to be properly characterized.

To cope with these limitations, image fusion approaches have been proposed to generate high-spatial–temporal resolution images by fusing image data from multiple space-borne instruments [5] to generate daily high-resolution (e.g., 30 m) images [6]. Spatial–temporal image fusion approaches can be divided into four categories. Unmixing-based algorithms decompose the images into the spectral signatures of the constituent materials and their pixelwise abundances, which reduces the dimensionality of the problem [7,8]. Weighted fusion predicts temporal changes in high-spatial-resolution images by using a weighted linear combination of the temporal changes in low-spatial-resolution pixels in the observed area [9]. Bayesian approaches can model the uncertainty of satellite images and further estimate high-resolution images [10,11]. Furthermore, learning-based approaches using neural networks (NNs) leverage some kind of training data to learn a mapping from low-resolution to high-resolution images [12,13]. Finally, [14] proposed SkyEyeGPT, which formulates remote sensing image understanding within a unified vision–language framework and supports diverse tasks, such as image captioning, visual question answering, visual grounding, and scene classification. These developments reflect a broader trend in remote sensing and GeoAI toward more general, language-guided, and multimodal Earth observation systems. Additionally, NNs have also been applied to other areas of image fusion; for example, [15] proposed an NN for infrared and visible image fusion based on salient target detection.

Despite the efficiency of these methods, outliers caused by effects such as cloud contamination can severely impact their image fusion performance. Thus, the detection and removal of pixels contaminated by clouds is an essential step of image fusion pipelines. Different algorithms for cloud (and cloud shadow) detection and removal have been proposed and can be divided into three categories [16]. The first kind restores cloudy/shadow-contaminated pixels by assuming that they share the same statistical distribution or geometric structures as the surrounding cloudless ones [17,18]. The second kind uses auxiliary information from different sensors, such as synthetic aperture radar [19] or MODIS images [20]. The third kind uses cloudless images from the same sensor on other dates as reference images [21].

However, the quality of cloud removal is limited, and the images used in the fusion process might still contain outliers. Thus, the development of robust image fusion methods is paramount. Moreover, although noise-robust image fusion methods have been proposed [22], there is a lack of robust online image fusion approaches.

Recursive image fusion methods process low-resolution image sequences on the fly to generate high-resolution images, benefiting from the information contained in long sequences while retaining fixed computational complexity. The Kalman filter framework [23] has been adopted in several works, as it provides a statistically principled solution to this problem. Methods based on such framework were developed to estimate time series of vegetation indices [24] and land surface temperature [10] and also to fuse Landsat and MODIS images in an offline manner [25]. A key element of this framework is a model for the time evolution of high-resolution images. This is related to a multispectral video prediction problem. Recently, a Bayesian filtering approach for image fusion was proposed in [11] using a linear and Gaussian model for image evolution, whose covariance was estimated from a small number of historical data. The flexibility of this model, however, is very limited. For computer vision, several works have proposed black-box NN-based approaches for video prediction [26,27]. However, such approaches require a large number of training data.

In this paper we propose a robust recursive image fusion framework using location-aware NNs to tackle the aforementioned limitations. The recursion is performed in the image space: at each time step k we treat the posterior distribution $p\left({\mathit{s}}_k\right|{\mathit{y}}_{1:k})$ (represented by the fused image mean and its posterior covariance from our variational update) as the prior for the next step $p\left({\mathit{s}}_{k+1}\right|{\mathit{y}}_{1:k})$. In other words, we propagate and update the fused image and its uncertainty in the image space rather than in a learned intermediate feature space. The proposed recursive Bayesian formulation improves robustness and estimation quality since it can leverage temporal information and historical data, which can help to mitigate the influence of outliers (e.g., clouds) and improve robustness. By recursively updating the estimated high-resolution image, the impact of outliers (e.g., residual clouds and noise) can be reduced and estimation quality improved. In particular, the recursive process yields a statistically optimal fusion of all available information up to time k, effectively integrating information from multiple observations. Moreover, the explicit beta–Bernoulli outlier modeling yields indicator functions that down-weight or mask contaminated pixels in the likelihood, reducing their influence in the fusion process and preventing outliers from corrupting the estimated state. This yields more accurate estimates that can be updated in real time.

An illustration of the proposed method is shown in Figure 1. First, a probabilistic image acquisition model is presented in Section 2, where the measurement outliers such as clouds are represented as a statistical hypothesis of a large-magnitude outlier affecting a pixel, not requiring a rigid statistical model. Then, in Section 3 a location-aware NN-based stochastic model is proposed to represent the dynamical evolution of the high-resolution images. The NNs, which represent the mean and variance of the image evolution, are based on interpretable structures that use auxiliary variables to aid in the predictions. The NNs are trained on a small dataset of historical images of a given scene within a maximum-likelihood framework. Images from different modalities are then recursively fused in a Bayesian paradigm leveraging a variational inference framework previously developed in [28], as discussed in Section 4. To handle the high computational complexity of the Bayesian solution, a new computationally efficient distributed algorithm is developed in Section 5 by using different statistical approximations. This makes it possible to employ this solution over large datasets and geographical areas. Section 6 illustrates the experimental setup. The experimental results with real Landsat-8 and MODIS data shown in Section 7 illustrate the superior performance of the proposed approach. Section 8 discusses the proposed algorithm compared with the comparison methods, which is further concluded in Section 9.

Figure 1. Overview of the proposed method. Time series of multimodal (e.g., Landsat and MODIS) images are recursively fused by the proposed robust Variational Distributed Multimodal Bayesian Fusion algorithm. The fused results are high-spatial–temporal-resolution estimated images, while the contamination caused by cloud/shadow influence (marked in the yellow box) is mitigated. A location-aware NN-based dynamical evolution model is estimated through a supervised strategy based on a small number of high-resolution historical data.

The proposed framework addresses key limitations of state-of-the-art algorithms, as explained in the following. For instance, ref. [22] proposes an NN-based satellite image fusion method robust against cloud influence. Despite the efficiency of this algorithm, the NN structure is complex, with about 86 convolutional layers, and the algorithm needs reference images at the preceding and succeeding times of the predicted image, meaning that it cannot be applied as an online algorithm. On the other hand, the proposed algorithm can be applied online, using only current and past observations, with NNs built using six layers in total. Similarly, refs. [29,30] propose offline NN-based satellite image fusion algorithms, but they cannot detect or mitigate cloud and shadow influence efficiently, while our proposed method can detect cloud and shadow influence directly given that it includes a robust Kalman filter. Furthermore, ref. [31] addresses clouds/outliers but does not accommodate learning/NN models, and it needs images dated a year after the processed image for better performance, meaning that it is an offline method instead of an efficient online method such as ours. Also, ref. [32] applies an online satellite image fusion method under a robust Kalman filter framework for cloud/outlier mitigation, but the dynamical model is simple and does not use all the features from the observed images efficiently. More details will be introduced for this specific comparison later in Section 2.3 as the motivation of the deep learning part of this algorithm.

In summary, the contribution of the present study is the design of a convolutional neural network (CNN)-based Bayesian variational inference framework to mitigate cloud cover influence for more accurate multi-resolution and multispectral image fusion. Furthermore, the novelty of the proposed method lies in a principled estimation framework which combines: (1) robustness to measurement outliers under minimal assumption on their statistical distribution, (2) scalable recursive operation, (3) approximated distributed implementation to reduce computational cost, and (4) integration of location-aware NNs to model image dynamics which require only a small number of training data.

2. Dynamical Imaging Model

2.1. Definitions and Notation

In this paper, we define ${\mathit{y}}_{k,\mathsf{\ell }}^m\in {ℝ}^{N_{m,\mathsf{\ell }}}$ as the k-th acquired image reflectances at the ℓ-th band with modality under $m\in \Omega$, $N_{m,\mathsf{\ell }}$ pixels for each of the bands $\mathsf{\ell }=1,\dots ,L_m$, and $\Omega$ as the set of image modalities. To be specific, we consider Landsat-8 and MODIS images in this paper, which are represented by $\Omega =\{\mathsf{L},\mathsf{M}\}$. We denote the high-resolution latent reflectances by ${\mathit{S}}_k\in {ℝ}^{N_H\times L_H}$, with $N_H$ pixels and $L_H$ bands, with $L_H\ge L_m$ and $N_H\ge N_{m,\mathsf{\ell }}$, $\forall \mathsf{\ell },m$. The subindex $k\in {\mathbb{N}}_{\ast }$ indicates the acquisition date. Furthermore, vectorization, vector stacking, diagonal and block diagonal matrix operators are denoted by $vec(·)$, $col\{·\}$, $diag\{·\}$ and $blkdiag\{·\}$, respectively. The notation ${\mathit{x}}_{a:b}$ for $a,b\in {\mathbb{N}}_{\ast }$ represents the set $\{{\mathit{x}}_a,{\mathit{x}}_{a+1},\dots ,{\mathit{x}}_b\}$. $\mathcal{N}(\mathit{\mu },\mathbf{\Sigma })$ is the Gaussian distribution, with $\mathit{\mu }$ the mean and $\mathbf{\Sigma }$ the covariance matrix.

2.2. Measurement Model

The images acquired at each time k consist of spatially degraded, noisy versions of a high-resolution image ${\mathit{S}}_k$. Following this assumption, traditional methods consider a measurement model that can be expressed according to

$$\begin{array}{c}\hfill {\mathit{y}}_{k,\mathsf{\ell }}^m={\mathcal{H}}_{\mathsf{\ell }}^m\left({\mathit{S}}_k\right)+{\mathit{r}}_{k,\mathsf{\ell }}^m,\end{array}$$

(1)

for each band $\mathsf{\ell }=1,\dots ,L_m$ and for each modality $m\in \Omega$. The function ${\mathcal{H}}_{\mathsf{\ell }}^m:{ℝ}^{N_H\times L_H}\to {ℝ}^{N_{m,\mathsf{\ell }}}$ is a linear operator representing the spectral and spatial degradation occurring at the ℓ-th band of the m-th imaging modality; it represents the effects of blurring and downsampling of the high-resolution image bands, as well as the spectral response function of the ℓ-th sensor band. ${\mathit{r}}_{k,\mathsf{\ell }}^m$ is the measurement noise. Note that we assume in this paper that different bands in ${\mathit{S}}_k$ hold the same spatial resolution, but the resolution can be different over bands. Normally, the measurement noise is assumed as uncorrelated Gaussian noise among bands, that is, ${\mathit{r}}_{k,\mathsf{\ell }}^m\sim \mathcal{N}(\mathbf{0},{\mathit{R}}_{\mathsf{\ell }}^m)$, where ${\mathit{R}}_{\mathsf{\ell }}^m\in {ℝ}^{N_{m,\mathsf{\ell }}\times N_{m,\mathsf{\ell }}}$ is the time-invariant covariance matrix for this Gaussian distribution. At each time index k, the scene is measured through one of the imaging modalities $m\in \Omega$.

Using (1) and properties of the vectorization of matrix products, we can stack all bands of the m-th modality in the vector ${\mathit{y}}_k^m\in {ℝ}^{n_y^m}$, with $n_y^m={\sum }_{\mathsf{\ell }}{N}_{m,\mathsf{\ell }}$ being the total number of pixels observed in the m-th image modality, leading to the equivalent reformulation of Model (1) as

$$\begin{array}{c}\hfill {\mathit{y}}_k^m={\mathit{H}}^m{\mathit{s}}_k+{\mathit{r}}_k^m,\end{array}$$

(2)

where ${\mathit{y}}_k^m=col\left\{{\mathit{y}}_{k,1}^m,\dots ,{\mathit{y}}_{k,L_m}^m\right\}$, ${\mathit{r}}_k^m=col\left\{{\mathit{r}}_{k,1}^m,\dots ,{\mathit{r}}_{k,L_m}^m\right\}$, ${\mathit{R}}^m=blkdiag\left\{{\mathit{R}}_1^m,\dots ,{\mathit{R}}_{L_m}^m\right\}$, ${\mathit{H}}^m={\left[{\left({\mathit{H}}_1^m\right)}^{\top },\dots ,{\left({\mathit{H}}_{L_m}^m\right)}^{\top }\right]}^{\top }$, and ${\mathit{H}}_{\mathsf{\ell }}^m$ is a matrix form representation of the operator ${\mathcal{H}}_{\mathsf{\ell }}^m$, such that $vec\left({\mathcal{H}}_{\mathsf{\ell }}^m\left({\mathit{S}}_k\right)\right)={\mathit{H}}_{\mathsf{\ell }}^m{\mathit{s}}_k$. ${\mathit{s}}_k\in {ℝ}^{L_H{N}_H}$ denotes a vector ordering of high-resolution image ${\mathit{S}}_k$, which is obtained by reorganizing all pixels to have the bands of a single high-resolution pixel, and positions corresponding to nearby pixels are adjacent to each other (see [11] for details about pixel ordering).

In satellite imaging, the pixel values can be degraded by several potential influences in the image fusion process. For example, they can be affected by the dead pixels in the sensor, atmospheric compensation error and the presence of clouds and shadow. Most existing algorithms ignore the existence of such outlier pixels, which can lead to a considerable loss of performance when they are applied in real settings. Thus, we address this issue by considering two hypotheses for the measurements. Under the first hypothesis, denoted by ${\mathcal{C}}_0$, the pixels are only affected by Gaussian noise ${\mathit{r}}_k^m$, whereas under the second hypothesis, denoted by ${\mathcal{C}}_1$, the pixels are corrupted, being affected by a vector of outliers ${\mathit{o}}_k^m\in {ℝ}^{n_y^m}$. This leads to the following measurement model:

$$\begin{array}{cc}\hfill y_k^{m,\left(i\right)}& =\left\{\begin{array}{cc}{\mathit{h}}^{m,\left(i\right)}{\mathit{s}}_k+r_k^{m,\left(i\right)},\hfill & \mathrm{under}{\mathcal{C}}_0\hfill \\ {\mathit{h}}^{m,\left(i\right)}{\mathit{s}}_k+r_k^{m,\left(i\right)}+o_k^{m,\left(i\right)},\hfill & \mathrm{under}{\mathcal{C}}_1\hfill \end{array}\right.\hfill \end{array}$$

(3)

for $i=1,\dots ,n_y^m$, where $y_k^{m,\left(i\right)}$, $r_k^{m,\left(i\right)}$ and $o_k^{m,\left(i\right)}$ denote the i-th element of ${\mathit{y}}_k^m$, ${\mathit{r}}_k^m$ and ${\mathit{o}}_k^m$ respectively, and ${\mathit{h}}^{m,\left(i\right)}$ denotes the i-th row of ${\mathit{H}}^m$. Note that we have one hypothesis for each band and pixel in the measurement of modality m, which might be affected by an outlier. Moreover, the approach we will consider will not need a rigid statistical model for ${\mathit{o}}_k^m$, as will be shown in the following section.

2.3. The Motivation for Using Learning-Based Dynamical Image Models

A precise model description is necessary to keep the Bayesian filtering estimation accurate. However, considering the fact that the time periods between observed images are not fixed, and the fact that the variation between pixels would change in different locations and dates, the design of a dynamical model can be a challenging task. Under the assumption that current variations in the same location can be similar to the past years, ref. [32] built a simple dynamic noise covariance estimation strategy, which is to search for the most similar images in the past and use the corresponding variation as the estimated dynamic noise covariance. There are two main limitations for this strategy: (1) the current observed image cannot be exactly the same as the historical images, and the same applies to the corresponding variation; (2) only the dynamic noise variance is estimated, while the corresponding mean value still depends fully on the measurement update phase.

To address these two limitations, we propose to use an NN to estimate both mean and variance in the dynamical model. By using the NN to extract potential features from the historical dataset instead of directly using a past variation, the dynamical model can be learned more precisely and thus leads to higher accuracy in Bayesian filtering estimation. We describe the proposed model in more detail in the following section.

3. Learning a Scene-Adapted Dynamical Model

To properly exploit the temporal information in the image sequence, an adequate dynamical model is necessary to describe the dynamics of HR images. Model-based frameworks aim to construct priors for dynamical image evolution solely based on prior knowledge, such as assuming that changes in video sequences are of low magnitude, have zero mean and are spatially smooth [33]. However, the reconstruction quality achieved using such models is limited.

Recently, the use of data-driven models has become the predominant approach to performing video prediction [34]. Such approaches are typically based on end-to-end learnable neural network architectures such as CNNs [26], recurrent neural networks (RNNs) [35] and transformers [36]. The flexibility of such models allows for accounting for long-term temporal dependencies. Stochastic models for video prediction have also been proposed to account for the uncertainty in the predictions [37]. These models are trained in a variational inference framework [38]. More recently, diffusion models have been applied to stochastic video prediction [27], owing to their great success in video generation tasks [39]. Other learning approaches leveraging physical knowledge through partial differential equations have also been recently explored [40].

For remote sensing applications, neural network models based on the transformer architecture have become well-established for forecasting tasks [41]. However, despite their flexibility, such black-box models require a large number of training data and are not easily interpretable. In [11], a simple data-driven model was proposed based on a linear and Gaussian model with a data-driven selection of an innovation covariance matrix, which needs only a small number of historical images.

Existing prediction models for videos are either black-box models which use a large number of data for training [35] or rely on analytical models [11] which are interpretable but too simplistic to properly describe the changes seen in real multispectral image sequences. To overcome this limitation, we propose a prediction model for high-resolution images based on location-aware neural networks which incorporates an interpretable architecture design. Moreover, the data-driven parts of the model, consisting of CNNs, are trained using only a small number of data consisting of a historical dataset $\mathcal{D}$ of high-resolution images of the scene to be processed.

3.1. Supervised Dynamical Evolution Model Learning

In this section we propose to learn a scene-adapted dynamical model $p_{\varphi }\left({\mathit{s}}_k\right|{\mathit{s}}_{k-1};{\mathit{u}}_k)$, which is assumed to be a conditional Gaussian distribution represented as

$$\begin{array}{cc}\hfill & p\left({\mathit{s}}_k\right|{\mathit{s}}_{k-1};{\mathit{u}}_k)=\mathcal{N}({\mathit{\mu }}_{\varphi }\left({\mathit{s}}_{k-1},{\mathit{u}}_k),diag\left({\mathit{\sigma }}_{\varphi }^2({\mathit{s}}_{k-1},{\mathit{u}}_k)\right)\right),\hfill \end{array}$$

(4)

where vector ${\mathit{u}}_k$ contains additional variables used in the prediction and functions ${\mathit{\mu }}_{\varphi }$ and ${\mathit{\sigma }}_{\varphi }^2$ compute the mean and the diagonal of the covariance matrix of the distribution. This corresponds to the following equivalent dynamical model:

$${\mathit{s}}_k={\mu }_{\varphi }({\mathit{s}}_{k-1},{\mathit{u}}_k)+{\mathit{q}}_{k-1},$$

(5)

with ${\mathit{q}}_{k-1}\sim \mathcal{N}(\mathbf{0},diag\left({\mathit{\sigma }}_{\varphi }^2({\mathit{s}}_{k-1},{\mathit{u}}_k)\right))$.

3.2. Additional Input Variables ${\mathit{u}}_k$

Additional variables provide useful information for video prediction in several tasks, such as Atari games [42] or robot agents motion [43]. In remote sensing, environmental variables have been used to improve the estimation of vegetation indices from satellite data [44]. In this work, to help model the image dynamics, we consider a vector of auxiliary deterministic variables, denoted by ${\mathit{u}}_k$ and defined as

$$\begin{array}{c}\hfill {\mathit{u}}_k={\left[{\mathit{pos}}^{\top },{\mathit{q}}_0^{\top },dat{e}_k,{\Delta }_k\right]}^{\top },\end{array}$$

(6)

where $dat{e}_k$ denotes the date of the k-th acquisition (number of days since the first day of the corresponding year for the earliest acquisition date), ${\Delta }_k$ is the number of days between the $k-1$-th and the k-th acquisitions, $\mathit{pos}$ is a vector containing the spatial position of each pixel, and ${\mathit{q}}_0$ is the variance of the image difference computed from historical data. It is inspired by the weakly supervised covariance estimation method in [11], aiming to provide a rough estimation for the average transition rate. To compute it, let us define a dataset of historical images of the scene under consideration by $\mathcal{D}=\{{\mathit{y}}_i^{L,h}:i=1,\dots ,n_h\}$, with $n_h$ Landsat images at previous time instants. Based on $\mathcal{D}$, ${\mathit{q}}_0$ is computed as

$${\mathit{q}}_0={\displaystyle \frac{1}{n_{\mathcal{D}}}}{\displaystyle \frac{1}{{\Delta }_{\mathrm{median}}}}\sum _{\mathit{s}\in \mathcal{D}}{\left(\mathit{s}-{\displaystyle \frac{1}{n_h}}\sum _{{\mathit{s}}^{\prime }\in \mathcal{D}}{\mathit{s}}^{\prime }\right)}^2,$$

(7)

where ${\Delta }_{\mathrm{median}}$ denotes the median time between acquisition dates of consecutive images in $\mathcal{D}$, which is used to normalize the result to a per-day variance estimate, and $n_{\mathcal{D}}$ is the total number of historical images in historical dataset. The squaring in (7) is computed elementwise.

The input structure is inspired by the simple dynamic noise covariance estimation strategy in [32], where the basic idea is to find the most similar historical image and to calculate the corresponding variation among the chosen image. For this reason, $dat{e}_k$ and ${\Delta }_k$ are introduced to let the NN learn the temporal features more easily. Furthermore, given that the idea is based on the assumption that variations in a certain place follow a pattern, location features are also necessary, leading to $pos$. $q_0$ is one of the inputs to the strategy in [32] proved to contribute to covariance estimation; therefore we also include it into the NN to provide conventional solutions as additional features.

3.3. NN Model Structure

We propose to parameterize the functions ${\mathit{\mu }}_{\varphi }$ and ${\mathit{\sigma }}_{\varphi }^2$ by combining an interpretable model with CNNs learned based on the historical dataset. ${\mathit{\mu }}_{\varphi }$ is given by

$$\begin{array}{c}\hfill {\mathit{\mu }}_{\varphi }({\mathit{s}}_{k-1},{\mathit{u}}_k)=\mathrm{ReLU}\left({\mathit{s}}_{k-1}+{\mathrm{NN}}_{\varphi }^s({\mathit{s}}_{k-1},{\mathit{u}}_k)\right),\end{array}$$

(8)

which is composed of the sum of the image estimate at the previous time instant ${\mathit{s}}_{k-1}$ and a residual computed by the CNN estimate ${\mathrm{NN}}_{\varphi }^s({\mathit{s}}_{k-1},{\mathit{u}}_k)$, where $\varphi$ represents the parameters of the CNN. $\mathrm{ReLU}(·)$ denotes the rectified linear unit function, which is used to guarantee that the predicted mean is nonnegative. Note that (8) is a general expression for estimating ${\mathit{\mu }}_{\varphi }$. The diagonal of the covariance of the predictive distribution ${\mathit{\sigma }}_{\varphi }^2$ is given by

$$\begin{array}{cc}\hfill {\mathit{\sigma }}_{\varphi }^2({\mathit{s}}_{k-1},{\mathit{u}}_k)=& {\Delta }_k\times \mathrm{ReLU}\left(W_1\times {\mathit{q}}_0+W_2\times {\mathrm{NN}}_{\varphi }^Q({\mathit{s}}_{k-1},{\mathit{u}}_k)\right),\hfill \end{array}$$

(9)

where $W_1,W_2\in ℝ$ are weighting parameters. This is a weighted combination between the variances ${\mathit{q}}_0$ computed a priori based on the historical dataset and a residual term ${\mathrm{NN}}_{\varphi }^Q({\mathit{s}}_{k-1},{\mathit{u}}_k)$ computed by a CNN. The $\mathrm{ReLU}(·)$ function ensures that the computed variances are nonnegative, and the result is scaled by ${\Delta }_k$, so that the variance becomes larger (resp., smaller) the longer (resp., shorter) the time interval between the acquisitions $k-1$ and k. The structure of the NNs is shown in Figure 2, where we use ${\mathrm{NN}}_{\varphi }^s$ and ${\mathrm{NN}}_{\varphi }^Q$ to represent the neural networks generally. For details about the architecture of the NNs (${\mathrm{NN}}_{\varphi }^s$ and ${\mathrm{NN}}_{\varphi }^Q$) and the corresponding input, please refer to Appendix A.2.

Figure 2. Structure of the NN model used to compute the pixelwise mean and variance of the distribution of time evolution of the high-resolution images.

3.4. Cost Function

To learn the parameters of the NN-based models in (8) and (9), we consider two objectives. The main objective is to maximize the expected log-likelihood of the corresponding transition PDF $p\left({\mathit{s}}_k\right|{\mathit{s}}_{k-1};{\mathit{u}}_k)$, given by ${\mathsf{𝔼}}_{p({\mathit{s}}_{k-1},{\mathit{s}}_k)}\left\{log\left(p\left({\mathit{s}}_k\right|{\mathit{s}}_{k-1};{\mathit{u}}_k)\right)\right\}$. When the expectation is approximated using the samples contained in the historical dataset $\mathcal{D}$, this leads to the following objective:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{LL}}\left(\varphi \right)& ={\displaystyle \frac{1}{n_{\mathit{s}}}}\sum _{({\mathit{s}}_{k-1},{\mathit{s}}_k)\in \mathcal{D}}logp({\mathit{s}}_k|{\mathit{s}}_{k-1};{\mathit{u}}_k)\hfill \\ \hfill & ={\displaystyle \frac{1}{2n_{\mathit{s}}}}\sum _{({\mathit{s}}_{k-1},{\mathit{s}}_k)\in \mathcal{D}}[\parallel {\mathit{s}}_k-{\mathit{\mu }}_{\varphi }({\mathit{s}}_{k-1},{\mathit{u}}_k){\parallel }_{{[diag\left({\mathit{\sigma }}_{\varphi }^2({\mathit{s}}_{k-1},{\mathit{u}}_k)\right)]}^{-1}}^2\hfill \\ \hfill & -logdet\left(diag\left({\mathit{\sigma }}_{\varphi }^2({\mathit{s}}_{k-1},{\mathit{u}}_k)\right)\right)+\kappa ],\hfill \end{array}$$

(10)

where $\kappa$ is a constant. Note that, with a slight abuse of notation, we use $\{({\mathit{s}}_{k-1},{\mathit{s}}_k)\in \mathcal{D}\}$ to represent the set of image pairs observed at consecutive time instants in the historical dataset, with $n_{\mathit{s}}$ being the number thereof.

Based on ${\mathcal{L}}_{\mathrm{LL}}\left(\varphi \right)$, we consider the following objective function to train the neural network model:

$$\mathcal{G}(\varphi ,W_1,W_2)={\mathcal{L}}_{\mathrm{LL}}\left(\varphi \right)+{\lambda }_1\left[{(W_1-1)}^2+W_2^2\right]+{\lambda }_2\mathcal{R}\left(\varphi \right),$$

(11)

where ${\lambda }_1,{\lambda }_2$ are regularization parameters. The first regularization term in (11), $\left[{(W_1-1)}^2+W_2^2\right]$, penalizes the magnitude of weight $W_2$ and forces $W_1$ to be close to one; thus, the larger its contribution to the loss term is, the more the covariance model in (9) will approximate the a priori estimate given by ${\Delta }_k\times {\mathit{q}}_0$, reducing the contribution of the NN. The last term, $\mathcal{R}\left(\varphi \right)$, is a weight decay which penalizes the ${\mathrm{L}}_1$ norm of the parameters of neural networks ${\mathrm{NN}}_{\varphi }^Q$ and ${\mathrm{NN}}_{\varphi }^s$.

4. Robust Image Fusion Through Variational Kalman Filter

Given the probabilistic model for the image generation and its temporal dynamics, the online image fusion consists of computing the posterior distribution of the high-resolution image conditioned on all past measurements, $p\left({\mathit{s}}_k|{\left\{{\mathit{y}}_{1:k}^m\right\}}_{m\in \Omega }\right)$. To simplify the notation, we will suppress the dependence of some distributions on the input variables ${\mathit{u}}_k$ whenever this does not impair clarity. When the model is linear and Gaussian, this PDF can be computed efficiently using the Kalman filter [11,23]. However, the proposed model tackles the presence of outliers and uses a learned temporal evolution model, which such techniques cannot address.

To address this issue, we consider an approach based on the General Variational Bayesian Kalman Filter (GVBKF) proposed in [28]. This section simply provides an overview of the proposed GVBKF presented in [28]. First, let us introduce the outlier indicator vector ${\mathit{z}}_k^m={\left(z_k^{m,\left(1\right)},\dots ,z_k^{m,\left(n_y^m\right)}\right)}^{\top }\in \mathcal{Z}={\{0,1\}}^{n_y^m}$, such that $z_k^{m,\left(i\right)}=0$ if there is an outlier on the i-th (corrupted) element of ${\mathit{y}}_k^m$, i.e., $y_k^{m,\left(i\right)}$, and $z_k^{m,\left(i\right)}=1$ if the i-th element is otherwise clean (not corrupted). The clean elements of ${\mathit{y}}_k^m$ can be used nominally in the image fusion process, whereas the contribution of the corrupted ones should be down-weighted. This is performed by modifying the observation model such that the i-th position of the indicator vector, $z_k^{m,\left(i\right)}$, adjusts the variance of a modified (referred to as improper) Gaussian noise distribution, leading to

$$\begin{array}{cc}\hfill p\left({\mathit{y}}_k^m|{\mathit{s}}_k,{\mathit{z}}_k^m\right)& \propto exp\left(-{\displaystyle \frac{1}{2}}\parallel {\mathit{y}}_k^m-{\mathit{H}}^m{\mathit{s}}_k{\parallel }_{{\mathbf{\Sigma }}_k^{-1}\left({\mathit{z}}_k^m\right)}^2\right),\hfill \end{array}$$

(12)

where each element of the covariance matrix in the i-th row and j-th column is defined as ${\left[{\mathbf{\Sigma }}_k\left({\mathit{z}}_k^m\right)\right]}_{ij}={\left[{\mathit{R}}^m\right]}_{ij}$ for $i\ne j$ and ${\left[{\mathbf{\Sigma }}_k\left({\mathit{z}}_k^m\right)\right]}_{ij}={\left[{\mathit{R}}^m\right]}_{ij}/z_k^{m,\left(i\right)}$ for $i=j$, where ${\left[{\mathit{R}}^m\right]}_{ij}$ is the $ij$-th element of ${\mathit{R}}^m$.

To solve the image fusion problem under the new model, we need to approximate the posterior distribution $p\left({\mathit{s}}_k,{\mathit{z}}_k^m|{\mathit{y}}_k^m\right)$. Under the assumption of a Bayesian framework, we impose a beta–Bernoulli hierarchical prior to each indicator in ${\mathit{z}}_k^m$, to estimate the posterior of ${\mathit{z}}_k^m$. Specifically, we assume that $p\left(z_k^{m,\left(i\right)}|{\pi }_k^{m,\left(i\right)}\right)=B(z_k^{m,\left(i\right)},{\pi }_k^{m,\left(i\right)})$ is a Bernoulli distribution and ${\pi }_k^{m,\left(i\right)}$ follows a beta distribution, defined by (unknown) shape hyperparameters $e_0^{\left(i\right)}$ and $f_0^{\left(i\right)}$, i.e., $p\left({\pi }_k^{m,\left(i\right)}\right)=\mathrm{Beta}(e_0^{\left(i\right)},f_0^{\left(i\right)})$, for $i=1,\dots ,n_y^m$.

Note that we assume that the indicators are independent to each other in this paper. To estimate the posterior distribution of the latent variables $\mathit{\theta }=\{{\mathit{s}}_k,{\mathit{\pi }}_k^m,{\mathit{z}}_k^m\}$, that is $p\left(\mathit{\theta }\right|{\mathit{y}}_{1:k}^m)$, we apply the Variational Inference (VI) principle, and resort to an auxiliary distribution $q\left(\mathit{\theta }\right)$ using independence assumptions such that:

$$q\left(\mathit{\theta }\right)=q\left({\mathit{s}}_k\right)q\left({\mathit{\pi }}_k^m\right)q\left({\mathit{z}}_k^m\right)=q\left({\mathit{s}}_k\right){\displaystyle \prod _{i=1}^{n_y^m}}q\left({\pi }_k^{m,\left(i\right)}\right)q\left(z_k^{m,\left(i\right)}\right).$$

(13)

Then we apply the mean-field VI method to the joint distribution and acquire the various marginal distributions, $q(·)$,

$$\begin{array}{cc}\hfill p({\mathit{s}}_k,{\mathit{\pi }}_k^m,& {\mathit{z}}_k^m,{\mathit{y}}_{1:k}^m)\propto p\left({\mathit{s}}_k|{\mathit{y}}_{1:k-1}^m\right)p\left({\mathit{y}}_k^m|{\mathit{s}}_k,{\mathit{z}}_k^m\right)p({\mathit{z}}_k^m,{\mathit{\pi }}_k^m),\hfill \end{array}$$

(14)

such that $ln\left[q\left({\mathit{s}}_k\right)\right]$, $ln\left[q\left({\mathit{\pi }}_k^m\right)\right]$ and $ln\left[q\left({\mathit{z}}_k^m\right)\right]$ can be obtained by marginalizing the logarithm of the joint distribution in (14). Within the Gaussian filtering framework, the first term $p\left({\mathit{s}}_k|{\mathit{y}}_{1:k-1}^m\right)$ on the right-hand side of (14) is a predictive density, which can be approximated as $p\left({\mathit{s}}_k|{\mathit{y}}_{1:k-1}^m\right)\approx \mathcal{N}\left({\widehat{\mathit{s}}}_{k|k-1},{\widehat{\mathit{P}}}_{k|k-1}\right)$, where ${\widehat{\mathit{s}}}_{k|k-1}$ and ${\widehat{\mathit{P}}}_{k|k-1}$ are the mean and covariance matrix of $p\left({\mathit{s}}_k|{\mathit{y}}_{1:k-1}^m\right)$, which can be computed using cubature integration rules [45].

The second and third terms $p\left({\mathit{y}}_k^m|{\mathit{s}}_k,{\mathit{z}}_k^m\right)p({\mathit{z}}_k^m,{\mathit{\pi }}_k^m)$ on the right-hand side of (14) correspond to the update phase in a Kalman filter, which can be approximated using (13), with $q\left({\mathit{s}}_k\right)\approx \mathcal{N}({\widehat{\mathit{s}}}_{k|k},{\widehat{\mathit{P}}}_{k|k})$, where ${\widehat{\mathit{s}}}_{k|k}$ and ${\widehat{\mathit{P}}}_{k|k}$ are the mean and covariance of the filtering posterior at time k, that is $p\left({\mathit{s}}_k|{\mathit{y}}_{1:k}^m\right)\approx \mathcal{N}\left({\widehat{\mathit{s}}}_{k|k},{\widehat{\mathit{P}}}_{k|k}\right)$, while $q\left({\mathit{z}}_k^m\right)$ and $q\left({\mathit{\pi }}_k^m\right)$ are approximated as Bernoulli and beta distribution, respectively. To be precise, we specify term $p({\mathit{z}}_k^m,{\mathit{\pi }}_k^m)$ as

$$p\left(z_k^{m,\left(i\right)}|{\pi }_k^{m,\left(i\right)}\right)={\left({\pi }_k^{m,\left(i\right)}\right)}^{z_k^{m,\left(i\right)}}{\left(1-{\pi }_k^{m,\left(i\right)}\right)}^{1-z_k^{m,\left(i\right)}},$$

(15)

where ${\pi }_k^{m,\left(i\right)}$ follows beta distribution, defined by (unknown shape hyperparameters (notice that a beta distribution is usually defined as $p(x;\alpha ,\gamma )\propto x^{\alpha -1}{(1-x)}^{\gamma -1}$, where $\alpha$ and $\gamma$ are two shape parameters; for simplicity we omit the dependence on $\alpha$ and $\gamma$ and keep the form as $p\left(x\right)$)) $e_0^{\left(i\right)}$ and $f_0^{\left(i\right)}$ as

$$p\left({\pi }_k^{m,\left(i\right)}\right)={\displaystyle \frac{{\left({\pi }_k^{m,\left(i\right)}\right)}^{e_0^{\left(i\right)}-1}{\left(1-{\pi }_k^{m,\left(i\right)}\right)}^{f_0^{\left(i\right)}-1}}{\beta \left(e_0^{\left(i\right)},f_0^{\left(i\right)}\right)}},$$

(16)

and $\beta (·,·)$ is the beta function. Note that in this paper we assume that all the indicators are independent of each other:

$$\begin{array}{cc}\hfill p({\mathit{z}}_k^m,{\mathit{\pi }}_k^m)& ={\displaystyle \prod _{i=1}^{n_y^m}}p\left(z_k^{m,\left(i\right)}|{\pi }_k^{m,\left(i\right)}\right)p\left({\pi }_k^{m,\left(i\right)}\right).\hfill \end{array}$$

(17)

The auxiliary distributions in (13), $q\left({\mathit{s}}_k\right)$, $q\left({\mathit{\pi }}_k^m\right)$ and $q\left({\mathit{z}}_k^m\right)$, are computed by updating them sequentially and iteratively under the Bayesian filtering scheme, defined as the GVBKF in [28]. For details of the algorithm, please refer to [28] for its full derivation and Appendix A for detailed practical setup.

5. An Efficient Distributed Algorithm

A significant limitation of the variational Kalman filter presented in the previous section is that the computation and storage requirements increase quickly with the image size, as can be attested by considering that the dimension of ${\mathit{P}}_{k|k}$ is quadratic in the image size. To solve this problem, we split the state ${\mathit{s}}_k$ into multiple groups [11]. The split groups are assumed to be statistically independent, following [46,47,48].

Note that in this section, we introduce an approximation to the distributed implementation, which is part of the novelties of this paper. It is illustrated with more details starting from Equation (23). To this end, we divide ${\mathit{s}}_k$ into G groups:

$$\begin{array}{c}\hfill {\mathit{s}}_k=vec\left([{\mathit{s}}_k^{\left(1\right)},\dots ,{\mathit{s}}_k^{\left(G\right)}]\right),\end{array}$$

(18)

where the elements in each block ${\mathit{s}}_k^{\left(g\right)}$ can be regarded as dependent, but different blocks ${\mathit{s}}_k^{\left(g_1\right)}$ and ${\mathit{s}}_k^{\left(g_2\right)}$ are assumed to be independent for $g_1\ne g_2$. Based on this assumption, the covariance matrices ${\mathit{P}}_{k|k-1}$ and ${\mathit{P}}_{k|k}$ can be defined as block diagonal matrices:

$$\begin{array}{cc}\hfill {\mathit{P}}_{k|k-1}& =blkdiag\left\{{\mathit{P}}_{k|k-1}^{\left(1\right)},\dots ,{\mathit{P}}_{k|k-1}^{\left(G\right)}\right\},\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\mathit{P}}_{k|k}& =blkdiag\left\{{\mathit{P}}_{k|k}^{\left(1\right)},\dots ,{\mathit{P}}_{k|k}^{\left(G\right)}\right\}.\hfill \end{array}$$

(20)

A natural question is which group structure should be selected. We assume a model where each block consists of all the bands of one single high-resolution pixel (resulting in $G=N_H$ blocks). Following the procedures in [46,47,48], the distributed implementation of the General VBKF under the specific independence assumption can then be derived as described in the following.

In the prediction phase of the filter, which amounts to computing the mean and covariance of $p\left({\mathit{s}}_k|{\mathit{y}}_{1:k-1}^m\right)$, according to [47], the conditional independence assumption between groups allows us to write them as

$$\begin{array}{cc}\hfill {\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}& =\int \int {\mathit{\mu }}_{\varphi }^{\left(g\right)}\left({\mathit{s}}_{k-1}^{\left(g\right)},{\mathit{s}}_{k-1}^{(-g)},{\mathit{u}}_k\right)p\left({\mathit{s}}_{k-1}^{\left(g\right)}|{\mathit{y}}_{1:k-1}^m\right)\times p\left({\mathit{s}}_{k-1}^{(-g)}|{\mathit{y}}_{1:k-1}^m\right)d{\mathit{s}}_{k-1}^{\left(g\right)}d{\mathit{s}}_{k-1}^{(-g)},\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill {\widehat{\mathit{P}}}_{k|k-1}^{\left(g\right)}& =\int \int [\left({\mathit{\mu }}_{\varphi }^{\left(g\right)}\left({\mathit{s}}_{k-1}^{\left(g\right)},{\mathit{s}}_{k-1}^{(-g)},{\mathit{u}}_k\right)-{\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}\right){\left({\mathit{\mu }}_{\varphi }^{\left(g\right)}\left({\mathit{s}}_{k-1}^{\left(g\right)},{\mathit{s}}_{k-1}^{(-g)},{\mathit{u}}_k\right)-{\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}\right)}^{\top }\hfill \\ \hfill & +diag\left({\left({\mathit{\sigma }}_{\varphi }^2\right)}^{\left(g\right)}\left({\mathit{s}}_{k-1}^{\left(g\right)},{\mathit{s}}_{k-1}^{(-g)},{\mathit{u}}_k\right)\right)]p\left({\mathit{s}}_{k-1}^{\left(g\right)}|{\mathit{y}}_{1:k-1}^m\right)p\left({\mathit{s}}_{k-1}^{(-g)}|{\mathit{y}}_{1:k-1}^m\right)d{\mathit{s}}_{k-1}^{\left(g\right)}d{\mathit{s}}_{k-1}^{(-g)},\hfill \end{array}$$

(22)

where ${\mathit{\mu }}_{\varphi }^{\left(g\right)}$ and ${\left({\mathit{\sigma }}_{\varphi }^2\right)}^{\left(g\right)}$ denote the subset of outputs of functions ${\mathit{\mu }}_{\varphi }$ and ${\mathit{\sigma }}_{\varphi }^2$ corresponding to the elements of the state vector in group g, while the notation ${\mathit{s}}_{k-1}^{(-g)}$ indicates the subset of elements of vector ${\mathit{s}}_{k-1}$ except for those belonging to group g.

These equations involve two integrals, one with respect to the variables inside group g, which are expected to be important in the computation of the moments of group g, and another which involves the variables outside the group, which are expected to be less informative. Thus, we use two different integration schemes. In particular, the integration over $p\left({\mathit{s}}_{k-1}^{\left(g\right)}|{\mathit{y}}_{1:k-1}^m\right)$ is based on a cubature rule, while the integration over $p\left({\mathit{s}}_{k-1}^{(-g)}|{\mathit{y}}_{1:k-1}^m\right)$ is based on random sampling. To generate cubature samples, we have:

$$\begin{array}{cc}\hfill {\mathit{X}}_{i,j,k-1|k-1}^{\left(g\right)}=& {\mathit{L}}_{k-1|k-1}^{\left(g\right)}{\mathit{\xi }}_i+{\widehat{\mathit{s}}}_{k-1|k-1}^{\left(g\right)},\hfill \end{array}$$

(23)

where ${\mathit{\xi }}_i$ is the cubature point, for $i=1,\dots ,2n_s$, and ${\mathit{L}}_{k-1|k-1}^{\left(g\right)}$ is the Cholesky decomposition of matrix ${\widehat{\mathit{P}}}_{k-1|k-1}^{\left(g\right)}$. The random samples for variable ${\mathit{s}}_{k-1}^{(-g)}$ are generated following

$${\tilde{\mathit{X}}}_{i,j,k-1|k-1}^{(-g)}\sim \mathcal{N}\left({\widehat{\mathit{s}}}_{k-1|k-1}^{(-g)},{\widehat{\mathit{P}}}_{k-1|k-1}^{(-g)}\right),$$

(24)

where $j=1,\dots ,{\tilde{n}}_s$, with ${\tilde{n}}_s$ representing the number of random samples drawn from $\mathcal{N}\left({\widehat{\mathit{s}}}_{k-1|k-1}^{(-g)},{\widehat{\mathit{P}}}_{k-1|k-1}^{(-g)}\right)$. Following (21) and (22), the state and covariance in the distributed form are estimated as

$$\begin{array}{cc}\hfill {\mathit{X}}_{i,j,k|k-1}^{\ast }=& {\mathit{\mu }}_{\varphi }\left({\tilde{\mathit{X}}}_{i,j,k-1|k-1},{\mathit{u}}_k\right)\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}=& {\displaystyle \frac{1}{2n_s{\tilde{n}}_s}}{\displaystyle \sum _{i=1}^{2n_s}}{\displaystyle \sum _{j=1}^{{\tilde{n}}_s}}{\mathit{X}}_{i,j,k|k-1}^{\ast \left(g\right)}\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill {\widehat{\mathit{P}}}_{k|k-1}^{\left(g\right)}=& {\displaystyle \frac{1}{2n_s{\tilde{n}}_s}}{\displaystyle \sum _{i=1}^{2n_s}}{\displaystyle \sum _{j=1}^{{\tilde{n}}_s}}\left[{\mathit{X}}_{i,j,k|k-1}^{\ast \left(g\right)}-{\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}\right]\times {\left[{\mathit{X}}_{i,j,k|k-1}^{\ast \left(g\right)}-{\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}\right]}^{\top }\hfill \\ \hfill & +{\left({\mathit{\sigma }}_{\varphi }^2\right)}^{\left(g\right)}\left({\tilde{\mathit{X}}}_{i,j,k-1|k-1},{\mathit{u}}_k\right),\hfill \end{array}$$

(27)

where ${\tilde{\mathit{X}}}_{i,j,k-1|k-1}$ is the vector whose elements inside and outside group g are equal to those of ${\mathit{X}}_{i,j,k-1|k-1}^{\left(g\right)}$ and ${\mathit{X}}_{i,j,k-1|k-1}^{(-g)}$, respectively.

Since the proposed distributed General VBKF is based on the Kalman filter scheme, the distributed implementation in terms of measurement update phase is quite similar to the one in [11], while the difference appears in the Kalman filter gain and a posterior covariance, caused by the robust definition of observation covariance. The update equations are given by

$${\widehat{\mathit{s}}}_{k|k}^{\left(g\right)}={\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}+{\mathit{K}}_k^{\left(g\right)}\left({\mathit{y}}_k^m-{\mathit{H}}^m{\widehat{\mathit{s}}}_{k|k-1}\right),$$

(28)

$${\widehat{\mathit{P}}}_{k|k}^{\left(g\right)}={\widehat{\mathit{P}}}_{k|k-1}^{\left(g\right)}-{\mathit{K}}_k^{\left(g\right)}\left({\mathit{S}}_k+{⟨{\mathbf{\Sigma }}_k^{-1}\left({\mathit{z}}_k^m\right)⟩}^{-1}\right){\left({\mathit{K}}_k^{\left(g\right)}\right)}^{\top },$$

(29)

$${\mathit{K}}_k^{\left(g\right)}={\mathit{C}}_k^{\left(g\right)}{\left({\mathit{S}}_k+{⟨{\mathbf{\Sigma }}_k^{-1}\left({\mathit{z}}_k^m\right)⟩}^{-1}\right)}^{-1},$$

(30)

where

$$\begin{array}{cc}\hfill {\mathit{C}}_k^{\left(g\right)}& =\int \left({\mathit{s}}_k^{\left(g\right)}-{\widehat{\mathit{s}}}_{k|k-1}^{\left(g\right)}\right){\left({\mathit{H}}^m{\mathit{s}}_k-{\mathit{H}}^m{\widehat{\mathit{s}}}_{k|k-1}\right)}^{\top }p\left({\mathit{s}}_k|{\mathit{y}}_{1:k-1}^m\right)d{\mathit{s}}_k,\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\mathit{S}}_k& =\int \left({\mathit{H}}^m{\mathit{s}}_k-{\mathit{H}}^m{\widehat{\mathit{s}}}_{k|k-1}\right){\left({\mathit{H}}^m{\mathit{s}}_k-{\mathit{H}}^m{\widehat{\mathit{s}}}_{k|k-1}\right)}^{\top }p\left({\mathit{s}}_k|{\mathit{y}}_{1:k-1}^m\right)d{\mathit{s}}_k,\hfill \end{array}$$

(32)

and ${\mathit{z}}_k^m$ represents one of the $2^{n_y^m}$ possible combinations of ${\left\{z_k^{m,\left(i\right)}\right\}}_{i=1}^{n_y^m}$ binary values; the set of all possible combinations is given by $\mathcal{Z}={\{0,1\}}^{n_y^m}$ such that $\left|\mathcal{Z}\right|=2^{n_y}$. The expectation of ${\mathbf{\Sigma }}_k^{-1}\left({\mathit{z}}_k^m\right)$ with respect to $q\left({\mathit{z}}_k^m\right)$ is defined as

$$⟨{\mathbf{\Sigma }}_k^{-1}\left({\mathit{z}}_k^m\right)⟩=\sum _{\mathit{z}\in \mathcal{Z}}{\mathbf{\Sigma }}_k^{-1}\left(\mathit{z}\right)q({\mathit{z}}_k^m=\mathit{z}).$$

(33)

6. Experimental Setup

In this section, we use the proposed algorithm to fuse Landsat-8 and MODIS image sequences to generate images with high spatial and temporal resolution. We compare the proposed approach, called GVBKF-NN, to three competing recursive algorithms, illustrating the importance of both the proposed outlier and temporal evolution models. The first is a linear Kalman filter with a simple data-driven model for the dynamical image evolution proposed in [11], which we refer to as KF. The second is an extension of a Kalman filter with the learned temporal evolution model proposed in Section 3, which we refer to as KF-NN. The third is the distributed variational Bayesian fusion framework proposed in Section 4 but using the simple dynamical evolution model of [11] instead of the proposed learned dynamical model, which we refer to as GVBKF. In the following, we describe the data and experimental setup and analyze the experimental results.

6.1. Study Region

We consider two sites and two settings (cloudless and cloudy cases). The first is the Oroville Dam. It is located on the Feather River, in the Sierra Nevada Foothills (38°35.3′N and 122°27.8′W), as the tallest dam in the USA. It acts as a major water storage facility in the California State Water Project, with a maximum storage capacity of $1.54\times {10}^{11}$ ${\mathrm{ft}}^3$.

We focus on two particular areas of the Oroville Dam delimited by the blue and orange boxes in the left panel of Figure 3, for cloudy and cloudless cases, respectively.

Figure 3. The left panel shows the Oroville Dam site. The blue and orange boxes delimit the study area used in the cloudy and cloudless datasets, respectively. The right panel shows the Elephant Butte site. The blue box delimits the specific study area used in the cloudy and cloudless datasets.

The other site is the Elephant Butte reservoir (Figure 3, right panel). Located in the southern part of the Rio Grande river, in NM, USA (33°19.4′N and 107°26.2′W), it is the largest reservoir in New Mexico, providing power and irrigation to southern New Mexico and Texas. It is at an elevation of 4414 ft and has a surface area of 36,500 acres. We focus on a particular area of the Elephant Butte reservoir delimited by the blue box in the lower panel of Figure 3, for both cloudy and cloudless cases.

6.2. Remotely Sensed Data

For the Elephant Butte site, we evaluated the performance of the algorithms when processing a larger geographical area of approximately 9 km × 9 km, corresponding to Landsat and MODIS images with $324\times 324$ and $36\times 36$ pixels, respectively. We focused on the red and near-infrared (NIR) bands of the Landsat and MODIS instruments since they are often used to distinguish water from other land-cover elements in the image [49]. To train and test the NNs and set up the algorithm, we collected 47 Landsat images as the historical dataset $\mathcal{D}$, with observed dates varying from 16 January 2014 to 24 November 2017.

In the cloudless case, after removing images with significant cloud cover between 19 March 2019 and 9 July 2019, we obtained a set of five Landsat and six MODIS images to process. The first Landsat image was used to initialize the algorithms, while two others acquired on 7 June and 23 June were set aside as ground truth to measure the performance of different algorithms. Therefore, the remaining six MODIS and two Landsat images were processed by the algorithms, and we evaluated their performance through the fused image results for 14 June and 27 June (where the MODIS observations were available), since the MODIS images from 7 June and 23 June (matching the ground truth) had to be discarded due to significant cloud cover. As for the cloudy case, the Landsat data were the same as for the cloudless case, while for the MODIS data, we substituted a clean image from 27 June with another one acquired on 19 June, which is contaminated by a cloud.

For the experiments with the Oroville Dam site, we focused on a smaller area, exploring the performance of proposed algorithms in a more extreme case, with clouds covering the whole observed area. We also evaluated the methods on the cloudless dataset of the Oroville Dam site for comparison. The study region, shown in the left panel of Figure 3, corresponded to Landsat and MODIS images with $81\times 81$ and $9\times 9$ pixels for the cloudy case (marked by an orange square) and $162\times 162$ and $18\times 18$ pixels for the cloudless case (marked by a blue square), respectively. We collected 50 Landsat data from 9 April 2013 to 7 December 2017 to serve as the historical dataset $\mathcal{D}$, used for training the NNs and setting up the algorithms. In this experiment we also focused on the red and NIR bands.

For the cloudless case, we collected MODIS and Landsat data acquired on the interval ranging from 3 July 2018 to 21 September 2018. After filtering for heavy cloud cover, a set of 6 Landsat and 15 MODIS images were obtained. We used the first Landsat image to initialize the algorithms. Of the remaining five Landsat images, three of them, acquired on 19 July, 20 August and 5 September, were not processed by the algorithm but only used as ground truth, while the corresponding MODIS images were used to estimate the fused image for comparison.

For the cloudy case, we collected MODIS and Landsat data acquired from 29 March 2018 to 19 July 2018. We filtered the images in this interval for heavy cloud cover, with the exception of the date 16 May 2018, for which the corresponding Landsat image is contaminated by heavy cloud cover. This led to 6 Landsat and 10 MODIS images. The first Landsat image was used as initialization for the algorithms, and two out of the remaining five Landsat images (acquired on 1 June and 3 July) were set aside for evaluation and not processed by the algorithms. They were used to evaluate the images reconstructed by the algorithms from the MODIS observations from the same dates.

6.3. Algorithm Setup

We initialized all algorithms using a high-resolution Landsat observation, i.e., ${\mathit{s}}_{0|0}={\mathit{y}}_0^{\mathsf{L}}$, and set all the hyperparameters according to experiments performed over the historical dataset $\mathcal{D}$ and statistics computed from it. From several tests we set the parameters of the algorithm as shown in the following.

We set ${\mathit{P}}_{0|0}={10}^{-10}{\mathit{P}}_0$, where ${\mathit{P}}_0={\displaystyle \frac{1}{10}}\mathbf{1}+{\displaystyle \frac{9}{10}}\mathit{I}$ for the Oroville Dam and ${\mathit{P}}_0={\displaystyle \frac{1}{2}}\mathbf{1}+{\displaystyle \frac{1}{2}}\mathit{I}$ for Elephant Butte, with $\mathbf{1}$ being an all-ones matrix. For the Oroville Dam, the noise covariance matrices were set as ${\mathit{R}}^{\mathsf{L}}=3\times {10}^{-2}{\mathit{P}}_1^{\mathsf{L}}$ and ${\mathit{R}}^{\mathsf{M}}={10}^{-4}{\mathit{P}}_1^{\mathsf{M}}$, where ${\mathit{P}}_1^m={\mathit{I}}_m\otimes \left[\begin{array}{cc}1& 0.1\\ 0.1& 2\end{array}\right]$, with ⊗ denoting the Kronecker product, ${\mathit{I}}_m$ representing the identity matrix, and the superscript $m\in \{\mathsf{M},\mathsf{L}\}$ representing the adjustment of the corresponding covariance size for MODIS and Landsat images. In the Elephant Butte dataset, the noise covariance matrices were set as ${\mathit{R}}^{\mathsf{L}}=7.5\times {10}^{-3}{\mathit{P}}_0^{\mathsf{L}}$ and ${\mathit{R}}^{\mathsf{M}}=2.5\times {10}^{-5}{\mathit{P}}_0^{\mathsf{M}}$, where the superscript $m\in \{\mathsf{M},\mathsf{L}\}$ of ${\mathit{P}}_0^m$ represents the adjustment of the corresponding covariance size for MODIS and Landsat images. The blurring and downsampling matrices were set as ${\mathit{H}}^{\mathsf{L}}=\mathit{I}$ for Landsat, while for MODIS, ${\mathit{H}}^{\mathsf{M}}$ is defined as a spatial convolution based on a uniform $9\times 9$ filter, defined by (where matrix of ones), followed by decimation by a factor of 9 and a bandwise gain factor to compensate for scaling differences between Landsat and MODIS sensors. This represents the degradation occurring at the sensor (see, e.g., [50]).

The hyperparameter of the GVBKF represents the prior knowledge of the outlier information, which is different between the two datasets. Histograms are shown in Appendix B to illustrate the difference in the outlier magnitudes between the two datasets. In this paper, $e_0=0.98$ and $f_0=0.02$ for Elephant Butte, while $e_0=0.5$ and $f_0=0.5$ for Oroville Dam. To limit the computational cost, the total number of random samples was set as ${\tilde{n}}_s=8$ for each ${\mathit{s}}_k^{(-g)}$ in the distributed implementation. Further details on the setup of the proposed algorithm can be found in Appendix A.

To measure the performance of the methods, two metrics are introduced in this paper, the root mean square error (RMSE) and misclassification percentage (MP), defined as ${\mathrm{RMSE}}_k=\sqrt{{\displaystyle \frac{1}{L_H{N}_H}}\parallel {\mathit{s}}_k-{\widehat{\mathit{s}}}_{k|k}{\parallel }_2^2}$ and ${\mathrm{MP}}_k={\displaystyle \frac{100\%}{N_H}}{\parallel {\mathit{e}}_k-{\widehat{\mathit{e}}}_{k|k}\parallel }_1$, where ${\mathit{s}}_k$ represents the reference (ground truth) image with $L_H$ bands and $N_H$ pixels, ${\widehat{\mathit{s}}}_{k|k}$ represents the images estimated by the various methods, and ${\mathit{e}}_k$ and ${\widehat{\mathit{e}}}_{k|k}$ represent binary classification results generated by the K-means algorithm applied to ${\mathit{s}}_k$ and ${\widehat{\mathit{s}}}_{k|k}$, respectively.

7. Experimental Results

7.1. Results and Discussion for the Elephant Butte Site

We fused the red and NIR bands of MODIS and Landsat for the Elephant Butte site in both cloudless and cloudy scenarios. Note that we only show the NIR reflectance band for the reconstruction result for succinctness. The results for the red band are included in Appendix B. In the cloudless scenario, the fusion results for the NIR band and all algorithms are shown in Figure 4 along with the acquired MODIS and Landsat images. In the top labels, acquisition dates represent the acquired date. If the images are processed by the fusion algorithm, we have M for MODIS and L for Landsat, following the date in the top label. As shown in the figure, only the first and last Landsat images were used in the fusion process, while the remaining two images from 7 June and 23 June act as ground truth. They are used to evaluate the accuracy of fused images from 14 June and 27 June to compare the performance of different methods (note that the MODIS images from 7 June and 23 June were not available due to cloud cover). Figure 5 shows the misclassification maps (i.e., the absolute error between the water maps by different algorithms and the ground truth). It can be seen that all four methods produce visually similar image reconstructions.

Figure 4. Fusion results for the NIR band from MODIS (band 2) and Landsat (band 5) in the Elephant Butte cloudless case. The first two rows show the observed images at MODIS and Landsat bands. At each time index, the fused images by GVBKF-NN, GVBKF, KF-NN, KF, ESTARFM, PSRFM and EDCSTFN are presented at the following rows. Note that some Landsat images were used solely as ground truth, with only the acquisition date as the top label, while images processed by the algorithms are indicated by the top labels, where “M” stands for MODIS and “L” for Landsat.

Figure 5. Absolute error of water map of images for the Elephant Butte example in the cloudless case based on the K-means clustering strategy. In this plot, a 0 pixel value indicates correct classification and a 1 pixel value indicates misclassification. For comparison, the acquired Landsat images are shown in the first column as the ground truth.

For comparison purposes, we include the KF, KF-NN, GVBKF, ESTARFM [9], PSRFM [51], and EDCSTFN [52] methods to estimate the fused images and compare them with the ones estimated by the proposed algorithm, GVBKF-NN. Note that all the methods except PSRFM methods, including the proposed one, are online methods, which only estimate the fused images based on the current and past observations. However, PSRFM is an offline algorithm which not only depends on the current and past observations but also on future observations; thus, it is included only as a reference which illustrates what performance can be achieved if more measurement information is available. The acquisition of more information makes offline methods hold the best performance in theory in the normal case [23], but not under outlier influence, and it will be shown in the following experimental results.

To evaluate the performance of different methods more clearly, we introduce RMSE and misclassification percentage as our metrics. RMSE is a widely used metric for evaluating image fusion performance, and it is listed in references such as [53,54]. The misclassification performance metric is also a widely used performance metric for image fusion, as it is less sensitive to variability effects such as illumination differences which can affect the RMSE; it has been considered in impactful works such as [55,56]. In Table 1, the misclassification percentage and the RMSE of the estimated images are provided. In terms of RMSE, the GVBKF-NN and the KF-NN show close performance on average. The GVBKF and the KF, on the other hand, show higher RMSE. Furthermore, we have the conventional method ESTARFM and an NN-based method EDCSTFN, which have even higher RMSE because they rely on only one or two pairs of observed images, instead of all observed images from the past to the present like the recursive algorithms listed above. Note that the PSRFM has the smallest RMSE, and that is because this method takes advantage of both observed images before and after the current time step. In other words, the PSRFM is an offline algorithm, and we can still say that the proposed algorithm performs the best among the online algorithms applied. Furthermore, in terms of the uncertainty estimates, which is the posterior standard deviation of the estimated solution in the KF-based algorithms, we can see that the ones with an NN (KF-NN and GVBKF-NN) have a much tighter bound than the ones without an NN (KF and GVBKF), which indicates the contribution of NNs to precise model description clearly. In terms of misclassification percentage, we can see that the GVBKF-NN holds the smallest misclassification percentage on average among methods, except the offline PSRFM method, and the second best method is the KF-NN, followed by the KF, the GVBKF, the EDCSTFN, and the ESTARFM. This shows that the NN model is important for preventing performance losses in the proposed robust framework in the absence of outliers. Specifically, if we compare the proposed GVBKF-NN with the KF, the advanced Bayesian filtering method with its prototype, the RMSE is reduced by $14.56\%$ on average, and the misclassification percentage is reduced by $21.12\%$ on average.

Table 1. Misclassification and RMSE performance for the Elephant Butte site in the cloudless scenario.

	Online Algorithms						Offline Algorithm
	KF	KF-NN	GVBKF	GVBKF-NN	ESTARFM	EDCSTFN	PSRFM
Misclassification Percentage %
14 June (7 June)	6.1204	5.2231	6.2710	5.1917	9.2678	7.8951	1.9109
27 June (23 June)	7.1397	5.3612	7.3264	5.2879	10.8330	11.0349	1.5899
9 July	7.1007	5.5860	7.2655	5.5803	10.9034	8.7887	—
Average	6.7869	5.3901	6.9543	5.3533	10.3347	9.2396	1.7504
RMSE
14 June (7 June)	0.0679	0.0587	0.0685	0.0585	0.0719	0.0832	0.0355
27 June (23 June)	0.0709	0.0563	0.0721	0.0561	0.0811	0.0766	0.0480
9 July	0.0704	0.0615	0.0709	0.0616	0.0800	0.0784	—
Average	0.0687	0.0588	0.0705	0.0587	0.0777	0.0794	0.0417
Uncertainty	0.1958	0.0526	0.1964	0.0526	—	—	—

In the cloudy scenario, the fusion results for the NIR band and all algorithms are shown in Figure 6, while Figure 7 shows the misclassification mappings. To compare different methods with the proposed one, the Landsat images from 7 June and 23 June act as ground truth to measure the accuracy of fused images for 14 June and 27 June (the MODIS images from 7 June and 23 June were not available due to cloud cover). Note that in the MODIS dataset, on 19 June, part of the pixels in the top part are covered by a cloud. It can be seen that the ESTARFM is heavily influenced by the cloudy image, followed by the EDCSTFN, the KF, and the GVBKF. The cloud in the images estimated by those methods for 19 June is basically not removed at all, while the cloud in the image processed by the PSRFM is removed a bit, but there is still some left. The proposed GVBKF-NN, on the other hand, is able to suppress the outlier, while the KF-NN shows intermediate performance. This indicates that the robustness of the outlier-aware variational inference framework against cloudy pixels depends on the accuracy of the dynamical image evolution model and shows that considering both the robust methodology and the learned NN evolution model is key to obtaining high performance with the GVBKF-NN approach.

Figure 6. Fusion results for the NIR band from MODIS (band 2) and Landsat (band 5) for the Elephant Butte example in the cloudy case. The first two rows show the observed images at MODIS and Landsat bands. At each time index, the fused images by GVBKF-NN, GVBKF, KF-NN, KF, ESTARFM, PSRFM and EDCSTFN are presented in the following rows. Note that some Landsat images were used solely as ground truth with only acquisition date as top label, while images processed by the algorithms are indicated by the top labels, where “M” stands for MODIS and “L” for Landsat.

Figure 7. Absolute error of water map of images for the Elephant Butte example in the cloudy case based on the K-means clustering strategy. In this plot, 0-valued pixels indicate correct classification, and 1-valued pixels indicate misclassification. For comparison, the acquired Landsat images are shown in the first column as the ground truth.

The quantitative results are shown in Table 2, which contains the misclassification percentage and the RMSE of estimated images of the different methods. The GVBKF-NN and the KF-NN show the first- and second-best performance on average in terms of both misclassification percentage and RMSE among the online algorithms. The GVBKF and the KF, which do not have an accurate dynamical evolution model, show significantly worse performance. For the offline method, although the PSRFM method performs the best, it cannot avoid cloud influence on 19 June, leading to higher RMSE and misclassification compared with the GVBKF-NN for the same date. Furthermore, in terms of the uncertainty, the KF-NN and the GVBKF-NN still hold a tighter bound than the KF and the GVBKF, showing that the contribution of the NN to precise model description is not influenced by clouds and shadow. Specifically, comparing the proposed GVBKF-NN with the KF, the RMSE is reduced by $25.17\%$ on average, and the misclassification percentage is reduced by $33.06\%$ on average. In summary, the results for Elephant Butte have showed that the learned NN-based model, by representing the image evolution more accurately, brings about important improvements to both the GVBKF and KF methods.

Table 2. Misclassification and RMSE performance for the Elephant Butte site in the cloudy scenario.

	Online Algorithms						Offline Algorithm
	KF	KF-NN	GVBKF	GVBKF-NN	ESTARFM	EDCSTFN	PSRFM
Misclassification Percentage %
14 June (7 June)	6.1204	5.2231	6.1395	5.1917	9.2678	7.8951	1.9004
19 June (23 June)	11.3521	6.1719	11.3807	5.7785	13.0477	14.0137	7.3217
9 July	7.4703	5.6899	7.5227	5.7261	10.9034	8.7887	—
Average	8.3143	5.6950	8.3476	5.5654	11.0730	10.2325	4.6111
RMSE
14 June (7 June)	0.0679	0.0587	0.0679	0.0585	0.0719	0.0832	0.0362
19 June (23 June)	0.1147	0.0736	0.1136	0.0676	0.0934	0.1392	0.0883
9 July	0.0774	0.0672	0.0763	0.0683	0.0800	0.0784	—
Average	0.0866	0.0665	0.0859	0.0648	0.0818	0.1003	0.0622
Uncertainty	0.1941	0.0526	0.1946	0.0523	—	—	—

7.2. Results and Discussion for the Oroville Dam Site

We now compare the four algorithms on the Oroville Dam case with and without clouds. The setup is similar to the one for the Elephant Butte example. Considering space limitations, only quantitative results are provided. Please refer to the supplemental material for the reflectance and water mapping results. For the cloudless scenario, Table 3 shows the misclassification percentage and the RMSE of the images estimated by the different methods when they were compared with the ground truth. It can be seen that among the online methods, the GVBKF-NN has the smallest RMSE and misclassification percentage on average, while the second-best method is the KF-NN, followed by the GVBKF and the KF. The PSRFM method has the smallest RMSE and misclassification percentage, but it is not fair to compare an offline method with online methods, which take advantage of images observed in both the past and the future. Furthermore, in terms of the uncertainty estimates, we can see that the KF-NN and the GVBKF-NN have a much tighter bound than the KF and the GVBKF, following the pattern we observed in Elephant Butte. Specifically, comparing the proposed GVBKF-NN with the KF, the RMSE is reduced by $17.60\%$ on average, and the misclassification percentage is reduced by $12.25\%$ on average. Like in the Elephant Butte example, this illustrates the performance improvements brought about by the use of the proposed temporal image evolution model based on NNs when no outliers are present.

Table 3. Misclassification and RMSE performance for the Oroville Dam site in the cloudless scenario.

	Online Algorithms						Offline Algorithm
	KF	KF-NN	GVBKF	GVBKF-NN	ESTARFM	EDCSTFN	PSRFM
Misclassification Percentage %
19 July	5.8528	4.0390	3.9857	3.4560	2.4387	8.9963	1.6766
20 August	8.6267	7.4608	7.9675	7.6627	7.9904	8.3714	4.0238
5 September	10.2157	8.4438	9.4917	9.2326	11.6560	13.5688	2.2977
21 September	10.4557	9.3698	9.7165	10.4938	13.1362	13.8736	—
Average	8.7877	7.3283	7.7904	7.7113	8.8058	11.2026	2.6660
RMSE
19 July	0.0540	0.0438	0.0455	0.0394	0.0318	0.0612	0.0333
20 August	0.0734	0.0640	0.0688	0.0618	0.0665	0.0776	0.0488
5 September	0.0705	0.0572	0.0672	0.0590	0.0725	0.0880	0.0244
21 September	0.0748	0.0630	0.0706	0.0644	0.0830	0.0926	—
Average	0.0682	0.0570	0.0630	0.0562	0.0634	0.0798	0.0341
Uncertainty	0.1528	0.0439	0.1544	0.0439	—	—	—

We also evaluate the methods in the cloudy scenario, when the Landsat image from 16 May is completely covered by a large cloud present over the observed area, constituting a more extreme case of outlier contamination. Note that since there is no cloudless Landsat image around 16 May available for use as ground truth, we used the interpolation method to estimate a surrogate ground truth of the Landsat image from 16 May from the Landsat observations from 14 April and 1 June. The quantitative results are shown in Table 4, where the results at 16 May are based on the ground truth obtained by the interpolation method. It can be seen that the results estimated by the KF, the ESTARFM, the EDCSTFN, and even the PSRFM method are heavily influenced by the large cloud cover in the Landsat observation from 16 May. On the other hand, the GVBKF, KF-NN and GVBKF-NN methods exhibit comparatively stable performance, keeping estimated images robust against large cloud cover. Specifically, the GVBKF-NN outperforms the other methods on average, followed by the GVBKF and the KF-NN, which showed similar performance. The KF had the worst performance by far. Additionally, the KF-NN and the GVBKF-NN still have a much tighter uncertainty bound than the KF and the GVBKF.

Table 4. Misclassification and RMSE performance for the Oroville Dam site in the cloudy scenario.

	Online Algorithms						Offline Algorithm
	KF	KF-NN	GVBKF	GVBKF-NN	ESTARFM	EDCSTFN	PSRFM
Misclassification Percentage %
16 May	43.3318	16.2780	10.6996	8.1847	77.6863	48.0262	16.1561
1 June	10.8215	8.0323	8.4743	9.2669	37.3114	21.7802	5.8680
3 July	8.2000	5.6851	7.5293	4.9668	45.4047	27.2977	4.0695
19 July	8.7334	7.8342	7.8647	2.1338	34.0497	28.0902	—
Average	17.7717	9.4574	8.6420	6.1385	48.6130	31.2986	8.6979
RMSE
16 May	0.3208	0.1273	0.1139	0.0850	0.3846	0.7624	0.2464
1 June	0.0893	0.0901	0.0922	0.0883	0.5028	0.4041	0.0409
3 July	0.0730	0.0663	0.0887	0.0648	0.3331	0.3982	0.0300
19 July	0.0833	0.0802	0.0970	0.0683	0.3261	0.4363	—
Average	0.1416	0.0910	0.0980	0.0766	0.3867	0.5003	0.1058
Uncertainty	0.1968	0.0857	0.2386	0.0887	—	—	—

This illustrates the advantages of combining both the robust framework and the learned NN-based model. Specifically, in the presence of an extreme number of outliers in the image, the robust methodology of the GVBKF is able to provide more competitive performance and mitigate the presence of outliers with the KF-NN even without an accurate evolution model. To be detailed, comparing the proposed GVBKF-NN with the KF, the RMSE is reduced by $45.90\%$ on average, and the misclassification percentage is reduced by $65.46\%$ on average.

8. Discussion

This paper develops a robust multimodal multispectral satellite image fusion algorithm, aiming to mitigate the influence of clouds and shadow. The proposed algorithm is based on Bayesian filtering, utilizing information of images observed from past to the current time. For this reason, all Bayesian filtering methods (KF, KF-NN, GVBKF, and GVBKF-NN) outperform conventional methods (e.g., ESTARFM), as shown in Table 1, Table 2, Table 3 and Table 4, in terms of both RMSE and misclassification percentage. Furthermore, to improve the accuracy of model description, an NN is designed as a dynamical model to estimate the mean and covariance of the state vector. This is validated by the fact that the KF-NN and the GVBKF-NN show smaller RMSE and misclassification percentage than the KF and the GVBKF, respectively, as shown in Table 1, Table 2, Table 3 and Table 4. Furthermore, according to the tables, the GVBKF-NN and the KF-NN provide a tighter bound in terms of the uncertainty calculation, indicating the NN’s higher dynamical model description accuracy compared with the conventional one.

The PSRFM method provide the smallest RMSE and misclassification percentage in Table 1 and Table 3 in the cloudless scenario, because it utilizes information from images observed in the past, current time, and future. This means that the PSRFM method is an offline method and cannot applied to real-time satellite image fusion, while our proposed algorithm does not have this limitation. Additionally, according to Table 2 and Table 4, the PSRFM has no robustness against potential cloud and shadow influence, which is another advantage of our proposed algorithm.

Furthermore, a distributed implementation is applied, as introduced in Section 5, to control the computational cost of Bayesian filtering-based algorithms. It allows for a scalable operation of the proposed algorithm without worrying about computational efficiency. The proposed NN is simple and needs a small dataset, meaning that the algorithm can be applied even though the historical dataset of the targeted area is limited, which further improves the practical use of the proposed algorithm.

9. Conclusions

In this paper, we proposed a recursive image fusion method based on location-aware NNs that is robust to outliers such as clouds and shadows. To achieve this goal we proposed an imaging model where the acquisitions are contaminated by discrete outliers. The stochastic time evolution of high-resolution images is represented by an NN learned from a small set of historical images. To estimate the high-spatial–temporal-resolution image sequence, we resorted to a variational Bayesian filtering framework. A distributed approximate solution that is scalable to large datasets was also proposed. Experimental results show that the proposed algorithm accounting for both outliers and learned dynamical model is more robust against cloud cover without losing performance when no clouds are present. Specifically, comparing the GVBKF-NN and the KF on the Elephant Butte case, the RMSE and misclassification percentage are reduced by $14.56\%$ and $21.12\%$ on average in the absence of clouds and by $25.17\%$ and $33.06\%$ on average in the presence of clouds. Furthermore, comparing the GVBKF-NN and the KF on the Oroville Dam case, the RMSE and misclassification percentage are reduced by $17.56\%$ and $12.25\%$ on average without clouds and by $45.90\%$ and $65.46\%$ on average with clouds.

Note that the superior performance of the combination of NNs and the GVBKF is based on the fact that the trained NN is tested on the same location over different time dates and the fact that the hyperparameters ($e_0$ and $f_0$, for example) are tuned meaningfully according to the local cloud and shadow characteristics. Depending on how the cloud and shadow outliers behave, the hyperparameters can be sensitive in the case of outliers values being close to clean pixel values (e.g., Elephant Butte in Figure A2). If the tested location changes, the performance will degrade, because the NN cannot be generalized and the hyperparameters need to be adjusted depending on the local cloud and shadow characteristics. In the future, we will work on the domain generalization of NNs and the self-adaption of the hyperparameters, to further improve the practical capability of the proposed algorithm.

Appendix A.1. Practical Aspects of GVBKF Application

The practical aspects of the algorithm considered in this paper are the same as in [28]. In this paper, we have two criteria. The first one is a threshold, measuring the change between two consecutive state estimates during iterations ($10\%$ relative difference in the experiments in this paper), while the second one is the maximum of the iteration number (20 iterations in the experiments in this paper).

Appendix A.2. NN Structure and Parameters

This subsection aims to show detailed structure and parameters used in the NNs and their cost functions. In terms of ${\mathrm{NN}}_{\varphi }^Q$ in (9), we selected its architecture as

$${\mathrm{NN}}_{\varphi }^Q={\mathrm{Conv}1\mathrm{d}}_1(\mathrm{flatten}\left({\mathrm{Conv}2\mathrm{d}}_2\left({\mathrm{Conv}2\mathrm{d}}_1\left(\mathtt{input}\right)\right)\right))$$

Above, $\mathtt{input}$ contains ${\left[{\mathit{s}}_{k-1},{\mathit{pos}}^{\top },{\mathit{q}}_0^{\top },dat{e}_k\right]}^{\top }$, ordered as a tensor with spatial size $81\times 81$ and seven channels, where ${\mathit{s}}_{k-1}$, containing two bands, covers two channels; $\mathit{pos}$ covers two channels, as it contains the $(x,y)$ relative spatial position of each pixel with respect to the top-left corner; ${\mathit{q}}_0$, representing the diagonal elements of the covariance of ${\mathit{s}}_k$ in the historical dataset for each band, covers two channels; and $dat{e}_k$ covers the final channel. ${\mathrm{Conv}2\mathrm{d}}_1$ has 7 inchannels (corresponding to the 7 input channels) and 12 outchannels, where kernelsize$=9$, stride$=1$, padding$=4$, paddingmode as ‘zero’, dilation$=1$, groups$=4$, bias is ‘True’ and the size of MaxPool1d is 1. ${\mathrm{Conv}2\mathrm{d}}_2$ has 12 inchannels and 2 outchannels, paddingmode as ‘replicate’ and all other parameters the same as ${\mathrm{Conv}2\mathrm{d}}_1$. Note that the ReLU activation function is included after the outputs of ${\mathrm{Conv}2\mathrm{d}}_1$ and ${\mathrm{Conv}2\mathrm{d}}_2$, which are omitted in the equation above for simplicity. ${\mathrm{Conv}1\mathrm{d}}_1$ has one inchannel and one outchannel, with kernelsize$=1$, padding$=0$, and all other parameters the same as ${\mathrm{Conv}2\mathrm{d}}_2$.

In terms of ${\mathrm{NN}}_{\varphi }^s$ in (8), we used as architecture

$${\mathrm{NN}}_{\varphi }^s={\mathrm{Conv}1\mathrm{d}}_2\left(\mathrm{flatten}\left({\mathrm{Conv}2\mathrm{d}}_4\left({\mathrm{Conv}2\mathrm{d}}_3\left(\mathtt{input}\right)\right)\right)\right)$$

where the $\mathtt{input}$ contains ${\left[{\mathit{s}}_{k-1},{\mathit{pos}}^{\top },{\mathit{q}}_0^{\top },dat{e}_k\right]}^{\top }$, ordered as a tensor in the same way as described above for the input of ${\mathrm{NN}}_{\varphi }^Q$. ${\mathrm{Conv}2\mathrm{d}}_3$ has 7 inchannels and 12 outchannels, with all other parameters the same as ${\mathrm{Conv}2\mathrm{d}}_2$ described above, while ${\mathrm{Conv}2\mathrm{d}}_4$ has all the same parameters as ${\mathrm{Conv}2\mathrm{d}}_2$. In addition, ${\mathrm{Conv}1\mathrm{d}}_2$ has the same parameters as ${\mathrm{Conv}1\mathrm{d}}_1$. Additionally, the ReLU activation function is also included in the outputs of ${\mathrm{Conv}2\mathrm{d}}_3$ and ${\mathrm{Conv}2\mathrm{d}}_4$, similar to ${\mathrm{Conv}2\mathrm{d}}_1$ and ${\mathrm{Conv}2\mathrm{d}}_2$.

For the cost function defined in (11), we set the regularization parameters as ${\lambda }_1=0.1$ and ${\lambda }_2=0.001$.

Appendix A.3. Complexity of Distributed GVBKF

According to [57], a challenge with the Kalman filter is the need to compute and store the state covariance matrix, ${\mathit{P}}_{k|k}$. This incurs in storage and operation asymptotic complexity in the order of $\mathcal{O}\left(n_s^2\right)$ and $\mathcal{O}\left(n_s^3\right)$, respectively. Take the complexity $\mathcal{O}\left(n_s^3\right)$, for example; we will present how the proposed distributed system decreases the complexity of the GVBKF.

Following [28], we know that the measurement update phase needs to run for $n_y$ times to have the whole elements in an outlier indicator vector ${\mathit{z}}_k^m$ updated. For this reason, the complexity for the standard GVBKF is $\mathcal{O}\left(n_y{n}_s^3\right)$. Fortunately, based on the block diagonal assumption of the state covariance, after partitioning, the complexity for each iteration decreases from $\mathcal{O}\left(n_y{n}_s^3\right)$ to $\mathcal{O}\left(n_y{(n_s/G)}^{3}G\right)=\mathcal{O}(n_y{n}_s^3/G^2)<<\mathcal{O}\left(n_y{n}_s^3\right)$. Therefore, the distributed GVBKF can be more computationally efficient than the standard GVBKF.