A Robust Non-Gaussian Data Assimilation Method for Highly Non-Linear Models

Abstract

In this paper, we propose an efficient EnKF implementation for non-Gaussian data assimilation based on Gaussian Mixture Models and Markov-Chain-Monte-Carlo (MCMC) methods. The proposed method works as follows: based on an ensemble of model realizations, prior errors are estimated via a Gaussian Mixture density whose parameters are approximated by means of an Expectation Maximization method. Then, by using an iterative method, observation operators are linearized about current solutions and posterior modes are estimated via a MCMC implementation. The acceptance/rejection criterion is similar to that of the Metropolis-Hastings rule. Experimental tests are performed on the Lorenz 96 model. The results show that the proposed method can decrease prior errors by several order of magnitudes in a root-mean-square-error sense for nearly sparse or dense observational networks.

1. Introduction

Data Assimilation is the process of optimally combining the imperfect numerical forecast states and imperfect observations to better estimate the state ${\mathbf{x}}^{*}\in {\mathbb{R}}^{n\times 1}$ of a system that evolves according to some model operator [1,2,3],

$${\displaystyle {\mathbf{x}}_{next}^{*}={\mathcal{M}}_{t_{current}\to t_{next}}\left({\mathbf{x}}_{current}^{*}\right),}$$

(1)

where, $\mathcal{M}:{\mathbb{R}}^{n\times 1}\to {\mathbb{R}}^{n\times 1}$ is an imperfect numerical model that evolves the states, n is the number of model components. One example of $\mathcal{M}$ could be a model that mimics the ocean and/or the atmosphere dynamics. In traditional DA settings, prior errors are described by Gaussian distributions,

$$\begin{array}{c}\hfill \mathbf{x}\sim \mathcal{N}\left({\mathbf{x}}^b,\mathbf{B}\right)\end{array}$$

where ${\mathbf{x}}^b\in {\mathbb{R}}^{n\times 1}$ and $\mathbf{B}\in {\mathbb{R}}^{n\times n}$ is the background state and the background error covariance matrix, respectively. This assumption has plenty of computational benefits, for instance, by assuming Gaussian errors in the prior and the observations, Kalman-like updates can be performed in order to compute posterior (error) moments. However, in the context of geophysics, model dynamics can be highly non-linear and therefore, Gaussian Mixture Models (GMM) [4,5] can be used to capture our prior knowledge about the error dynamics:

$${\displaystyle \mathbf{x}\sim \sum _{k=1}^K{\alpha }_k^b·\mathcal{N}\left({\mathbf{x}}_k^b,{\mathbf{B}}_k\right),\mathrm{with}\sum _{k=1}^K{\alpha }_k^b=1}$$

(2)

where, for the k-th mixture component, ${\mathbf{x}}_k^b\in {\mathbb{R}}^{n\times 1}$ is the mean, ${\mathbf{B}}_k\in {\mathbb{R}}^{n\times n}$ is the background error covariance, and ${\alpha }_k^b$ is the prior weight, for $1\le k\le K$. These weights can be estimated, for instance, using the Expectation Maximization algorithm [6,7,8]. In sequential methods, it is common to assume Gaussian errors over observations $\mathbf{y}\in {\mathbb{R}}^{m\times 1}$,

$${\displaystyle \mathbf{y}\sim \mathcal{N}\left(\mathcal{H}\left(\mathbf{x}\right),\mathbf{R}\right)}$$

(3)

where m is the number of observed components from the model state, $\mathbf{R}\in {\mathbb{R}}^{m\times m}$ is the data error covariance matrix, and $\mathcal{H}:{\mathbb{R}}^{n\times 1}\to {\mathbb{R}}^{m\times 1}$ is the observation mapping operator. When the operator $\mathcal{H}$ is linear, the posterior error distribution can be described by a GMM [9] as well:

$${\displaystyle \mathbf{x}|\mathbf{y}\sim \sum _{k=1}^K{\alpha }_k^a·\mathcal{N}\left({\mathbf{x}}_k^a,{\mathbf{A}}_k\right)}$$

(4)

with updated weights ${\alpha }_k^a$, centroids ${\mathbf{x}}_k^a$, and covariances ${\mathbf{A}}_k$, for $1\le k\le K$, which account for the observation (3). It can be noted that, for the k-th mixture component, the posterior mode ${\mathbf{x}}_k^a\in {\mathbb{R}}^{n\times 1}$ can be obtained by means of a 3D-Var optimization problem:

$${\mathbf{x}}_k^a=\underset{\mathbf{x}}{argmin}{\mathcal{J}}^k\left(\mathbf{x}\right)$$

(5)

where

$${\displaystyle {\mathcal{J}}^k\left(\mathbf{x}\right)=\frac{1}{2}·{∥\mathbf{x}-{\mathbf{x}}_k^b∥}_{{\mathbf{B}}_k^{-1}}^2+\frac{1}{2}·{∥\mathbf{y}-\mathcal{H}\left(\mathbf{x}\right)∥}_{{\mathbf{R}}^{-1}}^2}$$

(6)

Under linear assumptions, ${\mathbf{x}}_k^a$ can be estimated by means of, for instance, an EnKF updating formula. However, for non-linear observation operators, such expression can fail to obtain reasonable estimates of posterior modes and therefore, other alternatives such as sampling methods are employed. For instance, Monte-Carlo based methods are commonly utilized in order to relax the Gaussian assumption in the observational errors. Thus, we propose an efficient sampling method to draw samples from the posterior distribution (4) using cost functions of the form (6). In general, the method works as follows: for a fixed number of clusters, a GMM is fitted with the EM method, then, for each prior component, samples are drawn along steepest descent approximations of the 3D-Var cost function; these samples are accepted/rejected based on the Metropolis–Hastings rule. Besides, background error correlations are estimated based on a modified Cholesky decomposition in order to perform implicit localization and to reduce the impact of sampling errors, a major concern in the context of DA.

This paper is organized as follows: Section 2 discusses EnKF formulations and sampling methods for relaxing Gaussian assumptions in prior and observation errors. Section 3 presents an EnKF formulation based on GMM and MCMC (Markov-Chain-Monte-Carlo) for computing posterior modes together with error statistics; the computational cost of the method is estimated as well. In Section 4, experimental tests are performed on the Lorenz-96 model and non-linear observation operators; conclusions are finally stated in Section 5.

2. Preliminaries

2.1. Ensemble Kalman Filters Based on Modified Cholesky Decomposition

In sequential Data Assimilation (DA), under prior Gaussian assumptions, a well-known filter is the ensemble Kalman filter (EnKF) [10,11]. In EnKF, using an ensemble of model realizations,

$${\mathbf{X}}^b=\left[{\mathbf{x}}^{b\left[1\right]},{\mathbf{x}}^{b\left[2\right]},\dots ,{\mathbf{x}}^{b\left[N\right]}\right]\in {\mathbb{R}}^{n\times N}$$

(7a)

the hyper-parameters of the error distribution,

$${\displaystyle \mathbf{x}\sim \mathcal{N}\left({\mathbf{x}}^b,\mathbf{B}\right)}$$

(7b)

are estimated as:

$${\displaystyle {\mathbf{x}}^b\approx {\overline{\mathbf{x}}}^b=\frac{1}{N}·\sum _{e=1}^N{\mathbf{x}}^{b\left[e\right]}\in {\mathbb{R}}^{n\times 1}}$$

(7c)

and

$${\displaystyle \mathbf{B}\approx {\mathbf{P}}^b=\frac{1}{N-1}·\Delta \mathbf{X}·\Delta {\mathbf{X}}^T\in {\mathbb{R}}^{n\times n}}$$

(7d)

where ${\mathbf{x}}^b\in {\mathbb{R}}^{n\times 1}$ is the background state, $\mathbf{B}\in {\mathbb{R}}^{n\times n}$ is the background error covariance matrix, N is the ensemble size, ${\mathbf{x}}^{b\left[e\right]}$ denotes the e-th ensemble member, for $1\le e\le N$. ${\overline{\mathbf{x}}}^b$ is known as the ensemble mean, and ${\mathbf{P}}^b$ is the ensemble covariance matrix. The matrix of member deviations $\Delta \mathbf{X}\in {\mathbb{R}}^{n\times N}$ reads

$${\displaystyle \Delta \mathbf{X}={\mathbf{X}}^b-{\overline{\mathbf{x}}}^b·{\mathbf{1}}^T}$$

(7e)

and $\mathbf{1}$ is a vector of consistent dimension whose components are all ones. When an observation $\mathbf{y}$ becomes available, in the stochastic EnKF, the analysis ensemble can be estimated as follows:

$${\displaystyle {\mathbf{X}}^a={\mathbf{X}}^b+{\mathbf{P}}^a·\Delta \mathbf{Y}\in {\mathbb{R}}^{n\times N}}$$

(7f)

where $\Delta \mathbf{Y}={\mathbf{H}}^T·{\mathbf{R}}^{-1}·\left[\mathbf{y}·{\mathbf{1}}^T-\mathbf{H}·{\mathbf{X}}^b+\mathbf{E}\right]\in {\mathbb{R}}^{n\times N}$ is the matrix of scaled innovations on the observations. The columns of $\mathbf{E}\in {\mathbb{R}}^{n\times N}$ are formed by samples from a m-th dimensional standard Normal distribution. $\mathbf{H}\in {\mathbb{R}}^{m\times n}$ is the linearized observation operator (with the linearization performed about the background state) also known as the Jacobian matrix of $\mathcal{H}$ at the background state, and ${\mathbf{P}}^a\in {\mathbb{R}}^{n\times n}$ is the analysis covariance matrix:

$${\displaystyle {\mathbf{P}}^a={\left[{\left[{\mathbf{P}}^b\right]}^{-1}+{\mathbf{H}}^T·{\mathbf{R}}^{-1}·\mathbf{H}\right]}^{-1}}$$

(7g)

In operational DA, ensemble sizes are several orders of magnitude smaller than the model dimensions ($N\ll n$) and as a consequence, the covariance matrix ${\mathbf{P}}^b$ is commonly rank-deficient. This implies that (7g) cannot be directly computed and even though equivalent updating formulas which avoid the ${[{\mathbf{P}}^b]}^{-1}$ calculation can be found in the literature, sampling errors can still impact the quality of the analysis corrections. In practice, localization methods are often used to artificially increase the degrees of freedom of ${\mathbf{P}}^b$ and to mitigate the impact of sampling errors [12,13,14,15]. An efficient EnKF implementation which accounts for implicit localization during the assimilation step is the EnKF method based on a modified Cholesky decomposition (EnKF-MC) [16]. In this filter, the vanilla covariance (7d) is replaced by the Bickel and Levina estimator [17]:

$${\displaystyle {\mathbf{B}}^{-1}\approx {\widehat{\mathbf{B}}}^{-1}={\mathbf{L}}^T·\mathbf{D}·\mathbf{L}\in {\mathbb{R}}^{n\times n}}$$

(8a)

where $\mathbf{L}\in {\mathbb{R}}^{n\times n}$ is a lower triangular matrix whose diagonal elements are all ones. Its non-zero sub-diagonal elements are computed by fitting models of the form:

$${\mathbf{x}}_{\left[i\right]}^T=\sum _{j\in P(i,r)}{\mathbf{x}}_{\left[j\right]}^T·{\{-\mathbf{L}\}}_{i,j}+{\mathbf{\varsigma }}_i\in {\mathbb{R}}^{N\times 1},\mathrm{for}1\le i\le n$$

(8b)

In which $P(i,r)$ denotes the predecessors of the i-th grid component for some labelling of model components, ${\mathbf{x}}_{\left[i\right]}\in {\mathbb{R}}^{1\times N}$ stands for the i-th row of the matrix (7e), and, as an assumption, ${\mathbf{\varsigma }}_i\in {\mathbb{R}}^{N\times 1}$ follows a zero-mean Normal distribution with uncorrelated errors of unknown variance ${\sigma }^2$. Likewise, $\mathbf{D}\in {\mathbb{R}}^{n\times n}$ is a diagonal matrix whose diagonal elements are given by the reciprocal variances of the residuals in (8b):

$${\left\{\mathbf{D}\right\}}_{i,i}=\mathbf{var}{\left({\mathbf{x}}_{\left[i\right]}^T-\sum _{j\in P(i,r)}{\mathbf{x}}_{\left[j\right]}^T·{\{-\mathbf{L}\}}_{i,j}\right)}^{-1}\approx \frac{1}{{\sigma }^2}$$

(8c)

Similar to (7f), the analysis ensemble can be built as,

$${\displaystyle {\mathbf{X}}^a={\mathbf{X}}^b+\widehat{\mathbf{A}}·\Delta \mathbf{Y}}$$

(8d)

where an estimate of the analysis covariance is:

$${\displaystyle \widehat{\mathbf{A}}={\left[{\widehat{\mathbf{B}}}^{-1}+{\mathbf{H}}^T·{\mathbf{R}}^{-1}·\mathbf{H}\right]}^{-1}\in {\mathbb{R}}^{n\times n}}$$

(8e)

Matrix-free implementations of the EnKF-MC are currently proposed in the literature, for instance, the Posterior EnKF (P-EnKF) [18,19] exploits the special structure of (8a) in order to estimate Cholesky factors of the posterior precision covariance (8e). Consider

$${\widehat{\mathbf{A}}}^{-1}={\widehat{\mathbf{B}}}^{-1}+{\mathbf{H}}^T·{\mathbf{R}}^{-1}·\mathbf{H}={\widehat{\mathbf{B}}}^{-1}+\mathbf{Z}·{\mathbf{Z}}^T={\widehat{\mathbf{B}}}^{-1}+\sum _{o=1}^m{\mathbf{z}}^{\left[o\right]}·{\left[{\mathbf{z}}^{\left[o\right]}\right]}^T\in {\mathbb{R}}^{n\times n}$$

(9)

where ${\mathbf{z}}^{\left[o\right]}\in {\mathbb{R}}^{n\times 1}$, for $1\le o\le m$, is the o-th column of matrix $\mathbf{Z}={\mathbf{H}}^T·{\mathbf{R}}^{-1/2}\in {\mathbb{R}}^{n\times m}$, the updating process can be done via a sequence of rank-one updates over the prior Cholesky factors:

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{A}}}^{\left(0\right)}& =& {\left[{\mathbf{L}}^{\left(0\right)}\right]}^T·{\mathbf{D}}^{\left(0\right)}·\left[{\mathbf{L}}^{\left(0\right)}\right]={\mathbf{L}}^T·\mathbf{D}·\mathbf{L}={\widehat{\mathbf{B}}}^{-1}\hfill \\ \hfill {\widehat{\mathbf{A}}}^{\left(1\right)}& =& {\widehat{\mathbf{A}}}^{\left(0\right)}+{\mathbf{z}}^{\left[1\right]}·{\left[{\mathbf{z}}^{\left[1\right]}\right]}^T={\left[{\mathbf{L}}^{\left(1\right)}\right]}^T·{\mathbf{D}}^{\left(1\right)}·\left[{\mathbf{L}}^{\left(1\right)}\right]\hfill \\ \hfill {\widehat{\mathbf{A}}}^{\left(2\right)}& =& {\widehat{\mathbf{A}}}^{\left(1\right)}+{\mathbf{z}}^{\left[2\right]}·{\left[{\mathbf{z}}^{\left[2\right]}\right]}^T={\left[{\mathbf{L}}^{\left(2\right)}\right]}^T·{\mathbf{D}}^{\left(2\right)}·\left[{\mathbf{L}}^{\left(2\right)}\right]\hfill \\ ⋮\\ \hfill {\widehat{\mathbf{A}}}^{\left(m\right)}& =& {\widehat{\mathbf{A}}}^{(m-1)}+{\mathbf{z}}^{\left[m\right]}·{\left[{\mathbf{z}}^{\left[m\right]}\right]}^T\hfill \\ & =& {\left[{\mathbf{L}}^{\left(m\right)}\right]}^T·{\mathbf{D}}^{\left(m\right)}·\left[{\mathbf{L}}^{\left(m\right)}\right]={\widehat{\mathbf{L}}}^T·\widehat{\mathbf{D}}·\widehat{\mathbf{L}}={\widehat{\mathbf{A}}}^{-1}\hfill \end{array}$$

where ${\mathbf{L}}^{\left(0\right)}\in {\mathbb{R}}^{n\times n}$ and ${\mathbf{D}}^{\left(0\right)}\in {\mathbb{R}}^{n\times n}$ are the Cholesky factors of ${\widehat{\mathbf{B}}}^{-1}$. Having a posterior precision in the form ${\widehat{\mathbf{A}}}^{-1}={\widehat{\mathbf{L}}}^T·\widehat{\mathbf{D}}·\widehat{\mathbf{L}}$, an estimate of the posterior ensemble can be easily built:

$${\displaystyle {\mathbf{X}}^a={\mathbf{X}}^b+\mathbf{Q}\in {\mathbb{R}}^{n\times N}}$$

(10a)

where $\mathbf{Q}\in {\mathbb{R}}^{n\times N}$ is given by the solution of a lower triangular linear system:

$${\displaystyle \left[{\widehat{\mathbf{L}}}^T·{\widehat{\mathbf{D}}}^{1/2}\right]·\mathbf{Q}=\Delta \mathbf{Y}}$$

(10b)

In spite of EnKF implementations well-recognized by the DA community, the Gaussian assumption (7b) can be easily broken when the numerical model dynamics (1) are highly non-linear. For this reason, one prefers to use a different model to describe the prior error distribution. A Gaussian Mixture Model is frequently used to relax the Gaussian assumption over the forecast distribution [20,21,22,23].

2.2. Gaussian Mixture Models Based Filters

In EnKF formulations based on Gaussian Mixture Models (GMM), the prior error distribution (7b) is replaced by a mixture of Gaussian distributions (2). Based on this general idea, many methods have been proposed in the current literature in order to deal with highly non-linear dynamics of, for instance, operational DA models. These methods rely on the assumption that GMM models can approximate arbitrary density functions. GMM components are commonly fitted by using the Expectation Maximization (EM) algorithm [24]. As the name implies, this iterative method works in two recursive steps: Expectation (E) and Maximization (M). For a fixed number of clusters K, at iteration p, during the E-step, the probability of representativeness of each ensemble member is computed with regard to clusters:

$${\displaystyle w_{e,k}^{\left(p\right)}\propto exp\left(-\frac{1}{2}·{∥{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}∥}_{{\left[{\mathbf{B}}_k^{\left(p\right)}\right]}^{-1}}^2\right),\mathrm{for}1\le e\le N\mathrm{and}1\le k\le K}$$

(11)

which allows us to approximate the k-th weight for the prior mixture component:

$$\begin{array}{c}\hfill {\displaystyle {\alpha }_k^{b\left(p\right)}={\left(\sum _{r=1}^K\sum _{e=1}^N{w}_{e,r}^{\left(p\right)}\right)}^{-1}·\left(\sum _{e=1}^N{w}_{e,k}^{\left(p\right)}\right)}\end{array}$$

GMM components are updated during the M-step:

$$\begin{array}{c}\hfill {\displaystyle {\mathbf{x}}_k^{b(p+1)}={\eta }_k^{\left(p\right)}·\sum _{e=1}^N{w}_{e,k}^{\left(p\right)}·{\mathbf{x}}^{b\left[e\right]}\in {\mathbb{R}}^{n\times 1},\mathrm{and}{\mathbf{B}}_k^{(p+1)}={\eta }_k^{\left(p\right)}·\left[\mathbf{\Delta }{\mathbf{X}}_k^{\left(p\right)}·{\mathbf{\Gamma }}_k^{\left(p\right)}\right]·{\left[\mathbf{\Delta }{\mathbf{X}}_k^{\left(p\right)}·{\mathbf{\Gamma }}_k^{\left(p\right)}\right]}^T\in {\mathbb{R}}^{n\times n}}\end{array}$$

where ${\eta }_k^{\left(p\right)}={\left[{\sum }_{e=1}^N{w}_{e,k}^{\left(p\right)}\right]}^{-1}\in \mathbb{R}$, the matrix of intra-cluster deviations reads

$$\begin{array}{c}\hfill {\displaystyle \Delta {\mathbf{X}}_k^{\left(p\right)}={\mathbf{X}}^b-{\mathbf{x}}_k^{b\left(p\right)}·{\mathbf{1}}^T\in {\mathbb{R}}^{n\times N}\mathrm{and}}\end{array}$$

$$\begin{array}{c}\hfill {\displaystyle {\mathbf{\Gamma }}_k^{\left(p\right)}=\mathbf{diag}\left\{\sqrt{w_{1,k}^{\left(p\right)}},\sqrt{w_{2,k}^{\left(p\right)}},\dots ,\sqrt{w_{N,k}^{\left(p\right)}}\right\}\in {\mathbb{R}}^{N\times N}}\end{array}$$

Under linear assumptions, GMM-EnKF formulations exploit the fact that a Gaussian distribution is conjugate of a GMM. Thus, posterior components can be computed as results of performing Kalman-like updates over prior weights, centroids, and covariances as follows:

$${\alpha }_k^a\propto {\alpha }_k^b·exp\left(-\frac{1}{2}·{∥\mathbf{y}-\mathcal{H}\left({\mathbf{x}}_k^b\right)∥}_{{\mathbf{Q}}_k^{-1}}^2\right),\mathrm{for}1\le k\le K$$

(12a)

$${\mathbf{x}}_k^a={\mathbf{x}}_k^b+{\mathbf{A}}_k·{\mathbf{H}}^T·\left[\mathbf{y}-\mathcal{H}\left({\mathbf{x}}_k^b\right)\right]\in {\mathbb{R}}^{n\times 1}$$

(12b)

and

$${\mathbf{A}}_k={\left[{\mathbf{B}}_k^{-1}+{\mathbf{H}}^T·{\mathbf{R}}^{-1}·\mathbf{H}\right]}^{-1}\in {\mathbb{R}}^{n\times n}$$

(12c)

where ${\mathbf{Q}}_k=\mathbf{H}·{\mathbf{B}}_k·{\mathbf{H}}^T+\mathbf{R}\in {\mathbb{R}}^{m\times m}$, and ${\sum }_{k=1}^K{\alpha }_k^a=1$. This computationally friendly property is exploited by filters such as the Non-linear Bayesian Estimator (N-BE) [25] and the Gaussian Mixture Ensemble Filter [26,27]. In [28] the N-BE is enhanced by using the Bayesian Information Criteria (BIC)

$$\begin{array}{c}\hfill {\displaystyle BIC=-2·\left(\sum _{e=1}^{N}log\left(\sum _{k=1}^K{\alpha }_k^b·\varphi \left({\mathbf{x}}^{b\left[e\right]};{\mathbf{x}}_k^b,{\mathbf{B}}_k\right)\right)\right)+2·(3·N-1)}\end{array}$$

in order to choose the number of components for the GMM, where $\varphi \left(\mathbf{x};{\mathbf{x}}_k^b,{\mathbf{B}}_k\right)$ denotes a Normal probability density function with parameters ${\mathbf{x}}_k^b$ and ${\mathbf{B}}_k$, respectively. This is possible, as well, by other means, for instance in the GMM-EnKF filter proposed by Smith in [29], the number of mixture components relies on the Akaie’s Information Criteria (AIC)

$$\begin{array}{c}\hfill {\displaystyle AIC=-2·\left(\sum _{e=1}^{N}log\left(\sum _{k=1}^K{\alpha }_k^b·\varphi \left({\mathbf{x}}^{b\left[e\right]};{\mathbf{x}}_k^b,{\mathbf{B}}_k\right)\right)\right)+log\left(N\right)·(3·N-1)}\end{array}$$

The BIC and the AIC are common methods for choosing the number of parameters in GMM but, information criteria based methods are poorly suited for selecting a model with a good out-of-sample fit in model-rich environments [30,31]. Besides, these methods are only valid for sample size N much larger than the number K of parameters, which can be difficult in the DA context.

Other methods such as Particle Filters (PFs) [32] are available in the literature in order to attack non-Gaussian DA problems. These methods are a good-choice from a theoretical point of view. Unfortunately, in practice, PFs do suffer from a degeneracy problem [33] (Section 1.4) and even more, many challenges have to be overcome before they can be considered under operational DA scenarios [34,35]. For these reasons, PFs are not considered any further in this paper.

3. Proposed Method

In this section, we develop an efficient Gaussian mixture ensemble Kalman filter implementation based on Markov-Chain-Monte-Carlo (GM-EnKF-MCMC) for non-Gaussian data assimilation. We describe in detail how hyper-parameters are computed as well as how observations are digested for the GM-EnKF-MCMC. Lastly, we briefly discuss how posterior ensembles are built.

3.1. Estimation of Hyper-Parameters—EM Method

Consider an ensemble of model realizations

$${\displaystyle {\mathbf{X}}^b=\left[{\mathbf{x}}^{b\left[1\right]},{\mathbf{x}}^{b\left[2\right]},\dots ,{\mathbf{x}}^{b\left[N\right]}\right]\in {\mathbb{R}}^{n\times N}}$$

(13)

where ${\mathbf{x}}^{b\left[e\right]}\in {\mathbb{R}}^{n\times 1}$ stands for the e-th ensemble member, for $1\le e\le N$. To estimate the hyper-parameters of the prior error distribution, we use the EM method to fit a GMM with K components. Instead of estimating background error covariances ${\mathbf{B}}_k$, for $1\le k\le K$, for the mixture components, we do prefer to estimate precision covariances ${\mathbf{B}}_k^{-1}$ due to their computational benefits. For instance, in the context of Normal distributions, probabilities are computed based on precision covariances which would require the inversion of large matrices when background error covariances are fitted during EM steps. Besides, by using the modified Cholesky decomposition for computing the precision covariances, we can exploit their special structures in order to obtain huge savings in terms of memory usage and to reduce the computational cost of matrix-vector products among iterations.

The proposed EM method works as follows: for a given number of clusters K, we choose K random members from the ensemble (13) in order to set the initial mixture centroids while the initial precision covariances are all equal to the precision covariance of the ensemble (8a). During the E-step, we compute the representativeness of each ensemble member regarding each cluster

$${\widehat{w}}_{e,k}^{\left(p\right)}\propto exp\left(-\frac{1}{2}·{∥{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}∥}_{{\left[{\widehat{\mathbf{B}}}_k^{\left(p\right)}\right]}^{-1}}^2\right),\mathrm{for}1\le e\le N\mathrm{and}1\le k\le K$$

(14a)

where

$${\displaystyle {\left[{\widehat{\mathbf{B}}}_k^{\left(p\right)}\right]}^{-1}={\left[{\mathbf{L}}_k^{\left(p\right)}\right]}^T·{\mathbf{D}}_k^{\left(p\right)}·\left[{\mathbf{L}}_k^{\left(p\right)}\right]\in {\mathbb{R}}^{n\times n}}$$

(14b)

The M-step updates the precision covariances as well as the background error states for each cluster. Consider the diagonal matrix of fuzzy weights,

$${\displaystyle {\widehat{\mathbf{\Gamma }}}_k^{\left(p\right)}=\mathbf{diag}\left\{{\widehat{w}}_{1,k}^{\left(p\right)},{\widehat{w}}_{2,k}^{\left(p\right)},\dots ,{\widehat{w}}_{N,k}^{\left(p\right)}\right\}\in {\mathbb{R}}^{N\times N}}$$

(15a)

the GMM centroids are updated as follows:

$${\overline{\mathbf{x}}}_k^{b\left(p\right)}={\widehat{\eta }}_k^{\left(p\right)}·\left[{\mathbf{X}}^b·{\widehat{\mathbf{\Gamma }}}_k^{\left(p\right)}\right]\in {\mathbb{R}}^{n\times 1}$$

(15b)

where ${\widehat{\eta }}_k^{\left(p\right)}={\left[{\sum }_{e=1}^N{\widehat{w}}_{e,k}^{\left(p\right)}\right]}^{-1}\in \mathbb{R}$. Likewise, the non-zero elements of ${\mathbf{L}}_k^{(p+1)}\in {\mathbb{R}}^{n\times n}$ (14b) are revised by fitting models of the form:

$${\displaystyle {\left[{{\mathbf{x}}_{\left[i\right]}}_k^{\left(p\right)}\right]}^T=\sum _{q\in P(i,r)}{\left[{{\mathbf{x}}_{\left[q\right]}^{\left(p\right)}}_k\right]}^T·{\left\{{\mathbf{L}}_k^{(p+1)}\right\}}_{i,q}+{\mathbf{\gamma }}_{i,k},\mathrm{for}1\le i\le n\mathrm{and}1\le k\le K}$$

(15c)

where ${{\mathbf{x}}_{\left[i\right]}}_k^{\left(p\right)}\in {\mathbb{R}}^{1\times N}$ is the i-th row of matrix $\Delta {\mathbf{X}}_k^{\left(p\right)}\in {\mathbb{R}}^{n\times N}$

$$\Delta {\mathbf{X}}_k^{\left(p\right)}={\mathbf{X}}^b-{\overline{\mathbf{x}}}_k^{\left(p\right)}·{\mathbf{1}}^T$$

(15d)

errors ${\mathbf{\gamma }}_{i,k}\in {\mathbb{R}}^{N\times 1}$ are assumed Normal distribution with zero mean and uncorrelated components of variance ${\sigma }^2$. Moreover, the diagonal entries of ${\mathbf{D}}_k^{\left(p\right)}\in {\mathbb{R}}^{n\times n}$ are estimated via the residuals of model (15c):

$${\displaystyle {\left\{{\mathbf{D}}_k^{\left(p\right)}\right\}}_{i,i}=\mathbf{var}{\left({\left[{{\mathbf{x}}_{\left[i\right]}}_k^{\left(p\right)}\right]}^T-\sum _{q\in P(i,r)}{\left[{{\mathbf{x}}_{\left[q\right]}}_k^{\left(p\right)}\right]}^T·{\left\{{\mathbf{L}}_k^{(p+1)}\right\}}_{i,q}\right)}^{-1},\mathrm{for}2\le i\le n,\mathrm{and}1\le k\le K}$$

(15e)

with ${\left\{{\mathbf{D}}_k^{\left(p\right)}\right\}}_{1,1}=\mathbf{var}{\left({\mathbf{x}}_{\left[1\right]}^{\left(p\right)}\right)}^{-1}$. Note that whenever covariance inflation is desired before assimilation steps, the inflation factors can be applied to the matrix of member deviations with regard to centroids (15d). In this manner, the estimated precision covariances come already inflated. Once the EM steps are concluded (i.e., when a maximum number of iterations is reached), the prior error distribution is described by the GMM:

$${\displaystyle \mathbf{x}\sim \sum _{k=1}^K{\alpha }_k^b·\left({\overline{\mathbf{x}}}_k^b,{\left[{\mathbf{L}}_k^T·{\mathbf{D}}_k·{\mathbf{L}}_k\right]}^{-1}\right)}$$

(16)

The special structure of the resulting precision covariances ${\widehat{\mathbf{B}}}_k^{-1}={\mathbf{L}}_k^T·{\mathbf{D}}_k·{\mathbf{L}}_k$ can be exploited in order to reduce the computational effort of drawing samples from the posterior error distribution as is discussed in the next section.

3.2. Sampling Method—Approaching the Posterior

In order to approximate samples from the posterior distribution (4), for each mixture component $1\le k\le K$ in (16), we use a Markov-Chain-Monte-Carlo (MCMC) approximation. Traditionally, Normal distributions are good candidates for proposing states, for instance, in our context, starting with $u=0$ and ${\mathbf{x}}^{\left(u\right)}={\overline{\mathbf{x}}}_k^b\in {\mathbb{R}}^{n\times 1}$, at iteration $0\le u\le v$, where v is a user-defined number of iterations, a state can be proposed as follows:

$${\displaystyle {\mathbf{z}}^{\left(u\right)}\sim \mathcal{N}\left({\mathbf{x}}^{\left(u\right)},{\left[{\widehat{\mathbf{B}}}_k^{-1}\right]}^{-1}\right)}$$

Or simply,

$${\displaystyle {\mathbf{z}}^{\left(u\right)}={\mathbf{x}}^{\left(u\right)}+\mathbf{\nu }\in {\mathbb{R}}^{n\times 1}}$$

(17)

where $\mathbf{\nu }\in {\mathbb{R}}^{n\times 1}$ is given by the solution of an upper triangular linear system:

$$\left[{\mathbf{L}}^T·{\mathbf{D}}^{1/2}\right]·\mathbf{\nu }=\mathbf{\epsilon }\in {\mathbb{R}}^{n\times 1}$$

and $\mathbf{\epsilon }\in {\mathbb{R}}^{n\times 1}$ follows a standard Normal distribution. Nevertheless, in high-dimensional spaces such as those found in the context of operational DA, high-probability zones of posterior error distributions can be reached after a huge number of iterations. Thus, to overcome this situation, we proceed as follows: we linearize the observation operator about the current state ${\mathbf{x}}^{\left(u\right)}$,

$$\mathcal{H}\left(\mathbf{x}\right)\approx {\mathcal{G}}_u\left(\mathbf{x}\right)=\mathcal{H}\left({\mathbf{x}}^{\left(u\right)}\right)+{\mathbf{H}}_{{\mathbf{x}}^{\left(u\right)}}·\left[\mathbf{x}-{\mathbf{x}}^{\left(u\right)}\right]$$

where ${\mathbf{H}}_{{\mathbf{x}}^{\left(u\right)}}\in {\mathbb{R}}^{m\times n}$ is the Jacobian matrix of $\mathcal{H}\left(\mathbf{x}\right)$ at ${\mathbf{x}}^{\left(u\right)}$, the cost function (6) can then be approximated by the quadratic cost function:

$${\displaystyle {\widehat{\mathcal{J}}}_k\left(\mathbf{x}\right)=\frac{1}{2}·{∥\mathbf{x}-{\overline{\mathbf{x}}}_k^b∥}_{{\mathbf{B}}_k^{-1}}^2+\frac{1}{2}·{∥\mathbf{y}-{\mathcal{G}}_u\left(\mathbf{x}\right)∥}_{{\mathbf{R}}^{-1}}^2}$$

(18a)

whose gradient reads,

$${\displaystyle \nabla {\widehat{\mathcal{J}}}_k\left(\mathbf{x}\right)={\widehat{\mathbf{B}}}_k^{-1}·\left[\mathbf{x}-{\overline{\mathbf{x}}}_k^b\right]-{\mathbf{H}}_{{\mathbf{x}}^{\left(u\right)}}^T·{\mathbf{R}}^{-1}·\left[\mathbf{y}-{\mathcal{G}}_u\left(\mathbf{x}\right)\right]\in {\mathbb{R}}^{n\times 1}}$$

(18b)

By using this gradient, the proposal distribution (17) can be modified in such a manner that, samples along the (approximated) steepest descent direction $-\nabla {\widehat{\mathcal{J}}}_k\left({\mathbf{x}}^{\left(u\right)}\right)$ of (6) have high probability of occurrence, that is,

$${\displaystyle {\mathbf{z}}^{\left(u\right)}={\mathbf{x}}^{\left(u\right)}+\psi ·\left[-\frac{\nabla {\widehat{\mathcal{J}}}_k\left({\mathbf{x}}^{\left(u\right)}\right)}{∥\nabla {\widehat{\mathcal{J}}}_k\left({\mathbf{x}}^{\left(u\right)}\right)∥}\right],\mathrm{with}\psi \sim \mathcal{U}\left(0,\beta \right)}$$

(19)

where $\mathcal{U}\left(0,\beta \right)$ stands for an Uniform distribution on the interval $(0,\beta )$. The value of $\beta$ can be tuned according to the degree of the observation operator. For instance, for a linear observation operator, $\beta$ can be set as $∥\nabla \widehat{\mathcal{J}}\left({\mathbf{x}}^{\left(u\right)}\right)∥$ since the gradient of (6) and (18a) are the same. When observation operators are highly non-linear, since Taylor based approximations suffer from myopia, a small step of such gradient must be taken and therefore, a good choice under this consideration is 1. Thus, an intuitive range for $\beta$ can be

$$\begin{array}{c}\hfill \beta \in \left[1,∥\nabla \widehat{\mathcal{J}}\left({\mathbf{x}}^{\left(u\right)}\right)∥\right]\end{array}$$

In the absence of prior information about $\beta$, one can choose 1. The main motivation for using gradient approximations is that, the subset of samples along the descent direction (19) can provide states which potentially maximize the posterior probability. Computationally speaking, no matrix inversion is needed in this context in order to propose states. The acceptance/rejection rule for the states (19) rely on the Metropolis-Hastings criterion which can be adapted as follows:

$${\displaystyle {\mathbf{x}}^{(u+1)}=\left\{\begin{array}{cc}{\mathbf{x}}^{\left(u\right)}\hfill & ,y>\frac{{\mathcal{J}}_k\left({\mathbf{x}}^{\left(u\right)}\right)}{{\mathcal{J}}_k\left({\mathbf{z}}^{\left(u\right)}\right)},\mathrm{for}y\sim \mathcal{U}(0,1)\hfill \\ {\mathbf{z}}^{\left(u\right)}\hfill & ,\mathrm{otherwise}\hfill \end{array}\right.}$$

(20)

The observation operator is then linearized about ${\mathbf{x}}^{(u+1)}$ and the overall process is repeated until some number of iterations is satisfied (or any other numerical condition). Putting it all together, the sampling procedure to compute the posterior modes is in the following:

Step 1

Let $k=1$, set $u=0$, go to step 2.

Step 2

Set ${\mathbf{x}}^{\left(u\right)}={\overline{\mathbf{x}}}_k^b$.

Step 3

Linearize $\mathcal{H}$ about ${\mathbf{x}}^{\left(u\right)}$ and compute the direction (18b).

Step 4

Compute ${\mathbf{z}}^{\left(u\right)}$ via Equation (19).

Step 5

Set ${\mathbf{x}}^{(u+1)}$ according to Equation (20).

Step 6

If $u\le v$ set $u=u+1$ and go to step 3, set ${\overline{\mathbf{x}}}_k^a={\mathbf{x}}^{\left(v\right)}$ and go to step 7 otherwise.

Step 7

If $k\le K$ go to step 1, go to step 8 otherwise.

Step 8

The posterior mode approximations read ${\left\{{\overline{\mathbf{x}}}_k^a\right\}}_{k=1}^K$.

Note that, unlike MCMC methods based on Random-Walk, for each prior mode, this procedure does not return a chain but only the last state in ${\mathbf{x}}^{\left(v\right)}$. This sampling step can be replaced by an optimization method such as Trust Region [36,37,38] or Line Search [39,40]. However, in order for it to work, some regularities must be satisfied by the gradient approximation (18b) [41,42,43,44,45], for instance, smoothness, which in practice is not necessarily the case.

3.3. Building the Posterior Ensemble

Once the posterior modes are computed, posterior covariances within clusters can be estimated as follows:

$${\widehat{\mathbf{A}}}_k={\left[{\widehat{\mathbf{B}}}_k^{-1}+{\mathbf{H}}_{{\overline{\mathbf{x}}}_k^a}·{\mathbf{R}}^{-1}·{\left[{\mathbf{H}}_{{\overline{\mathbf{x}}}_k^a}\right]}^T\right]}^{-1}\in {\mathbb{R}}^{n\times n}$$

(21a)

Therefore, the analysis members can be computed as:

$${\mathbf{x}}^{a\left[e\right]}\sim \mathcal{N}\left({\overline{\mathbf{x}}}_k^a,{\widehat{\mathbf{A}}}_k\right),\mathrm{with}\mathrm{probability}{\alpha }_k^a,\mathrm{for}1\le e\le N$$

(21b)

where ${\alpha }_k^a$ is estimated via the likelihood ratio:

$${\displaystyle {\alpha }_k^a={\left[\sum _{j=1}^K\varphi \left(\mathbf{y};\mathcal{H}\left({\mathbf{x}}_j^a\right),\mathbf{R}\right)\right]}^{-1}·\varphi \left(\mathbf{y};\mathcal{H}\left({\mathbf{x}}_k^a\right),\mathbf{R}\right)}$$

3.4. Computational Complexity

In this section, we estimate the number of long computations of the proposed method in order to assess its computational effort. We detail the number of computations of the GMM-EnKF-MCMC below, we avoid the use of iteration indexes for ease of reading:

• During the E-Step, the computations of weights (14a) depend on the calculation:

  $$\begin{array}{ccc}\hfill {\displaystyle -\frac{1}{2}·{∥{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}∥}_{{\left[{\widehat{\mathbf{B}}}_k^{\left(p\right)}\right]}^{-1}}^2}& =& -\frac{1}{2}·{\left[{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}\right]}^T·{\widehat{\mathbf{B}}}_k^{-1}\left[{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}\right]\hfill \\ & =& -\frac{1}{2}·{\left[{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}\right]}^T·{\mathbf{L}}_k^T·{\mathbf{D}}_k·{\mathbf{L}}_k·\left[{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}\right]\hfill \\ & =& -\frac{1}{2}·\underset{{\mathbf{s}}_k}{\overset{T}{\underbrace{\left[{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}\right]}}}·{\mathbf{L}}_k^T·{\mathbf{D}}_k^{T/2}·{\mathbf{D}}_k^{1/2}·{\mathbf{L}}_k·\underset{{\mathbf{s}}_k}{\underbrace{\left[{\mathbf{x}}^{b\left[e\right]}-{\mathbf{x}}_k^{b\left(p\right)}\right]}}\hfill \\ & =& -\frac{1}{2}·{\mathbf{s}}_k^T·{\mathbf{L}}_k^T·{\mathbf{D}}_k^{T/2}·{\mathbf{D}}_k^{1/2}·{\mathbf{L}}_k·{\mathbf{s}}_k\hfill \\ & =& {\left[{\mathbf{D}}_k^{1/2}·\underset{{\widehat{\mathbf{s}}}_k}{\underbrace{{\mathbf{L}}_k·{\mathbf{s}}_k}}\right]}^T·{\mathbf{D}}_k^{1/2}·\underset{{\widehat{\mathbf{s}}}_k}{\underbrace{{\mathbf{L}}_k·{\mathbf{s}}_k}}\hfill \\ & =& {\left[\underset{{\tilde{\mathbf{s}}}_k}{\underbrace{{\mathbf{D}}_k^{1/2}·{\widehat{\mathbf{s}}}_k}}\right]}^T·\underset{{\tilde{\mathbf{s}}}_k}{\underbrace{{\mathbf{D}}_k^{1/2}·{\widehat{\mathbf{s}}}_k}}={\tilde{\mathbf{s}}}_k^T·{\tilde{\mathbf{s}}}_k={∥{\tilde{\mathbf{s}}}_k∥}_2^2\hfill \end{array}$$
  From this step, given the special structure of ${\mathbf{L}}_k$, ${\tilde{\mathbf{s}}}_k\in {\mathbb{R}}^{n\times 1}$ can be computed with no more than $\mathcal{O}\left({\theta }^2·n\right)$ long computations where $\theta$ denotes the maximum number of non-zero elements across all rows in ${\mathbf{L}}_k$ with $\theta \ll n$. Likewise, the number of long computations in order to obtain $\tilde{\mathbf{s}}\in {\mathbb{R}}^{n\times 1}$ is bounded by $\mathcal{O}\left(n\right)$ since ${\mathbf{D}}_k$ is diagonal. Thus, since there are K clusters, each E-step has the following operation counting:

  $${\displaystyle \mathcal{O}\left(K·\left[n+{\theta }^2·n\right]\right)}$$

  (22a)
• During the M-step, updating the centroids (15b) can be performed with no more than $\mathcal{O}\left(N^2·n\right)$ since ${\mathbf{D}}_k$ has only n components different from zero (the diagonal ones), the least square solution of (15c) is bounded by $\mathcal{O}\left({\theta }^2·n\right)$ calculations since there are n model components, and the cost of (15e) is bounded by $\mathcal{O}\left(n·{\theta }^2\right)$ since the multiplication of coefficients and model components is constrained to the neighbourhood of each model component. The computational effort of this method is then estimated as follows:

  $${\displaystyle \mathcal{O}\left(K·\left[{\theta }^2·n+N^2·n\right]\right)}$$

  (22b)
• During the sampling procedure, the gradient (18b) can be efficiently computed as follows:

  $$\begin{array}{ccc}\hfill {\displaystyle \nabla {\widehat{\mathcal{J}}}_k\left(\mathbf{x}\right)}& =& {\widehat{\mathbf{B}}}_k^{-1}·\left[\underset{{\mathbf{g}}_k}{\underbrace{\mathbf{x}-{\overline{\mathbf{x}}}_k^b}}\right]-{\mathbf{H}}_{{\mathbf{x}}^{\left(u\right)}}^T·{\mathbf{R}}^{-1}·\left[\underset{{\mathbf{f}}_k}{\underbrace{\mathbf{y}-{\mathcal{G}}_u\left(\mathbf{x}\right)}}\right]={\mathbf{L}}_k^T·{\mathbf{D}}_k·\underset{\widehat{{\mathbf{g}}_k}}{\underbrace{{\mathbf{L}}_k·{\mathbf{g}}_k}}-{\mathbf{H}}_{{\mathbf{x}}^{\left(u\right)}}^T·\underset{\widehat{{\mathbf{f}}_k}}{\underbrace{{\mathbf{R}}^{-1}·{\mathbf{f}}_k}}\hfill \\ & =& {\mathbf{L}}_k^T·\underset{\widetilde{{\mathbf{g}}_k}}{\underbrace{{\mathbf{D}}_k·{\widehat{\mathbf{g}}}_k}}-\underset{\widetilde{{\mathbf{f}}_k}}{\underbrace{{\mathbf{H}}_{{\mathbf{x}}^{\left(u\right)}}^T·\widehat{{\mathbf{f}}_k}}}=\underset{\overline{{\mathbf{g}}_k}}{\underbrace{{\mathbf{L}}_k^T·{\tilde{\mathbf{g}}}_k}}-\widetilde{{\mathbf{f}}_k}=\overline{{\mathbf{g}}_k}-{\mathbf{f}}_k\hfill \end{array}$$

  Given the special structure of ${\mathbf{L}}_k$, ${\mathbf{g}}_k\in {\mathbb{R}}^{n\times 1}$ and $\overline{{\mathbf{g}}_k}\in {\mathbb{R}}^{n\times 1}$ can be computed with no more than $\mathcal{O}\left({\theta }^2·n\right)$, the computations in $\widetilde{{\mathbf{g}}_k}\in {\mathbb{R}}^{n\times 1}$ are bounded by $\mathcal{O}\left(n\right)$ since ${\mathbf{D}}_k$ is diagonal. Thus, since this sampling method is performed v times, the computational effort of the sampling procedure reads,

  $${\displaystyle \mathcal{O}\left(v·\left[n+{\theta }^2·n\right]\right)}$$

  (22c)
• The posterior ensemble can be built (Section 3.3 [19]) with no more than

  $${\displaystyle \mathcal{O}\left(n·{\theta }^2+m·N\right),}$$

  (22d)

  long computations.

Assuming that the number of clusters and the number of iterations in the sampling process are much lower than the model dimension, based on Equation (22) the computational effort of the GMM-EnKF-MCMC reads,

$$\begin{array}{c}\hfill {\displaystyle \mathcal{O}\left(n·{\theta }^2+m·N\right)}\end{array}$$

which is linear with regard to the model resolution n.

3.5. Comparison of GM-EnKF-MCMC with Other Sampling Methods

In this section, we briefly compare the GM-EnKG-MCMC method with well-known filters from the literature: the Cluster Sampling Filter (CSF) [21], the Cluster Monte Carlo Implementation (CMCI) [46], and the Cluster Ensemble Kalman Filter (CEnKF) [29].

The CSF exploits the evolution of a system under Hamiltonian dynamics, for instance, based on Newton’s law it is possible to describe the dynamics of particles. Each particle is fully described by two components: the position and the velocity coordinates which are associated with model states $\mathbf{x}\in {\mathbb{R}}^{n\times 1}$ and momentums $\mathbf{p}\in {\mathbb{R}}^{n\times 1}$ (auxiliary variables), respectively. The performance of CSF relies on the proper choice of a mass matrix $\mathbf{M}\in {\mathbb{R}}^{n\times n}$ in order to describe the probability distribution of $\mathbf{p}$ during the sampling process. Besides, numerical integrators which preserve Liouville’s Theorem are a must in order to obtain a proposal state $({\mathbf{p}}^{*},{\mathbf{x}}^{*})$. Some of those are the Verlet integrator [47] and the Leap-frog scheme [48]. A main difference between the GMM-EnKF-MCMC and the CSF is that, no auxiliary n-th dimensional vectors are required in our method during the sampling process. This can be convenient, for instance, under current operational DA systems. Besides, during the sampling process, the only parameter to be tuned in the GMM-EnKF-MCMC is $\beta$ in (17) while in the CSF method, as we mentioned before, proper choices of $\mathbf{M}$ are required in order to speed-up the convergence of MCMC towards high-probability zones of the posterior. Besides, extra parameters may need to be tuned depending on the numerical integrator chosen in order to propose states during the sampling process.

In the CMCI, the background error distribution is approximated by a summation of Gaussian Kernels. These are selected using the method of Fukunaga [49] to form a continuous approximation to the random ensemble. By assuming linear observation operators, the posterior ensemble is Gaussian as well. Posterior members are drawn by sampling each of the components/kernels based on a set of calculated weights which account for the observation. As is pointed out by the authors, the CMCI is a very promising filter but, additional work is needed in order to extend its capabilities to more realistic scenarios. Moreover, this filter has been developed under Gaussian assumptions on the observations; such assumption is not required in the GMM-EnKF-MCMC.

The CEnKF is developed by assuming linear observation operators. Prior error distributions are described by GMM and its components are fitted by using the EM-method. This filter can fail to obtain reasonable estimates of posterior ensembles when non-linear observation operators are present during the assimilation of observations, as is typical in practice.

4. Experimental Settings

In this section, numerical tests are performed to assess the accuracy of the proposed filter. We use the Lorenz-96 model [50] as our surrogate model. The Lorenz-96 model is described by the following set of ordinary differential equations [51]:

$${\displaystyle \frac{d{x}_j}{dt}=\left\{\begin{array}{cc}\left(x_2-x_{n-1}\right)·x_n-x_1+F\hfill & \mathrm{for}j=1\hfill \\ \left(x_{j+1}-x_{j-2}\right)·x_{j-1}-x_j+F\hfill & \mathrm{for}2\le j\le n-1\hfill \\ \left(x_1-x_{n-2}\right)·x_{n-1}-x_n+F\hfill & \mathrm{for}j=n\hfill \end{array}\right.}$$

(23)

where F is external force and $n=40$ is the number of model components. Periodic boundary conditions are assumed. When $F=8$ the model exhibits chaotic behavior, which makes it a relevant surrogate problem for atmospheric dynamics [52,53]. One time unit in the Lorenz-96 represents 7 days in real case. The experimental settings are described below:

• Starting with an initial random solution, a 4th order Runge Kutta method is employed in order to integrate it over a long time period from which initial condition ${\mathbf{x}}_{-2}^{*}\in {\mathbb{R}}^{n\times 1}$ is obtained.
• A perturbed background solution ${\tilde{\mathbf{x}}}_{-2}^b$ is formed at time $t_{-2}$ by drawing a sample from the Normal distribution,

  $$\begin{array}{c}\hfill {\displaystyle {\tilde{\mathbf{x}}}_{-2}^b\sim \mathcal{N}\left({\mathbf{x}}_{-2}^{*},{0.05}^2·\mathbf{I}\right)}\end{array}$$

  This solution is then integrated for 10 time units (equivalent to 70 days) in order to obtain a background solution ${\mathbf{x}}_{-1}^b$ consistent with the dynamics of the numerical model.
• An initial perturbed ensemble is built about the background state by taking samples from the distribution,

  $$\begin{array}{c}\hfill {\displaystyle {\tilde{\mathbf{x}}}_{-1}^{b\left[\widehat{e}\right]}\sim \mathcal{N}\left({\mathbf{x}}_{-1}^b,{0.05}^2·\mathbf{I}\right),\mathrm{for}1\le \widehat{e}\le \widehat{N}}\end{array}$$

  In order to make them consistent with the model dynamics, the ensemble members are propagated for 10 time units, from which the initial ensemble members are obtained. We create the initial pool ${\widehat{{\mathbf{X}}^b}}_0$ of $\widehat{N}={10}^4$ members. The actual solution is integrated over 20 more time units in order to place it at the beginning of the assimilation window.
• Two assimilation windows are proposed for the tests, both of them consist of $M=15$ observations. In the first assimilation window, observations are taken every 16 h (time step of 0.1 time units) while in the last one, observations are available every 50 h (time step of 0.3 time units). We denote by $\delta t\in \{16,50\}$ the elapsed time between two observations.
• The observational errors are described by the probability distribution,

  $$\begin{array}{c}\hfill {\displaystyle {\mathbf{y}}_{\ell }\sim \mathcal{N}\left({\mathcal{H}}_k\left({\mathbf{x}}_{\ell }^{*}\right),{\left[{ϵ}^o\right]}^2·\mathbf{I}\right),\mathrm{for}1\le \ell \le M}\end{array}$$

  where the standard deviations of observational errors ${ϵ}^o={10}^{-2}$, and ℓ should be interpreted as time index. Random observation networks are formed at the different assimilation cycles. The space between observations will depend on the step size, for instance, in the first step size observations are available every 0.1 time units (16 h) while in the last one, observations are taken every 0.3 time units (50 h).
• We consider the non-linear observation operator [32]:

  $${\displaystyle {\left\{\mathcal{H}\left(\mathbf{x}\right)\right\}}_j=\frac{{\left\{\mathbf{x}\right\}}_j}{2}·\left[{\left(\frac{\left|{\left\{\mathbf{x}\right\}}_j\right|}{2}\right)}^{\gamma -1}+1\right]}$$

  (24)

  where j denotes the j-th observed component from the model state. $\gamma \in \{1,3,5\}$.
• We consider two percentages of observed components s from the model state $s\in \{70\%,100\%\}$.
• The radius of influence is set to $r=1$ while the inflation factor is set to 1.02 (a typical value).
• We propose two ensemble sizes for the benchmark $N\in \{20,80\}$. These members are randomly chosen from the pool ${\widehat{{\mathbf{X}}^b}}_0$ for different experiments in order to form the initial ensemble ${\mathbf{X}}_0^b$ for the assimilation window. Evidently, ${\mathbf{X}}_0^b\subset {\widehat{{\mathbf{X}}^b}}_0$.
• The $L-2$ norm of errors are utilized as a measure of accuracy at the assimilation step ℓ,

  $${\lambda }_{\ell }=\sqrt{{\left[{\mathbf{x}}_{\ell }^{*}-{\mathbf{x}}_{\ell }^a\right]}^T·\left[{\mathbf{x}}_{\ell }^{*}-{\mathbf{x}}_{\ell }^a\right]}$$

  (25)

  where ${\mathbf{x}}_k^{*}$ and ${\mathbf{x}}_k^a$ are the reference and the analysis solutions, respectively. The analysis state is obtained by a weighted combination of posterior centroids via the likelihood ratio (21b) in lieu of the posterior mean.
• The Root-Mean-Square-Error (RMSE) is used as a measure of performance. On average, on a given assimilation window,

  $${\displaystyle \lambda =\sqrt{\frac{1}{M}·\sum _{\ell =1}^M{\lambda }_{\ell }^2}}$$

  (26)

Results for $\delta t=16$ h, different ensemble sizes, and different values of $\gamma$ are shown in Figure 1 in terms of L-2 error norms ${\lambda }_{\ell }$ per assimilation step (25). Notoriously, the performance of the filter is improved as long as prior errors are described by a GMM ($K>1$). This can be expected owing to the non-linear dynamics of the Lorenz-96 model. In addition, the performance of the GMM-EnKF-MCMC is not impacted by the degree $\gamma$ of the observation operator. Both features make the proposed formulation attractive for DA systems where non-linear observation operators are the bridge for digesting observations and besides, non-linear dynamics are encapsulated in the numerical model. Moreover, for $K>1$, there are less posterior errors than in prior and observations; this of course must be expected for full observational networks. In some cases, filter divergence is possible for $K=1$ (a single mode prior distribution). Assuming that spurious correlations are not impacting the analysis corrections, in this set of experiments, filter divergence can be seen as a consequence of one of two things: prior errors are not well fitted by a Gaussian distribution (the actual error distribution is multimodal) or inflation factors are not properly tuned. Regardless of the main cause, filter convergence is evident when a GMM describes the background error distribution. In Figure 2, results for $\delta t=50$, different ensemble sizes, and different values of $\gamma$ are shown as well. Under this configuration, the accuracy of the GMM-EnKF-MCMC is slightly impacted by increments in $\gamma$. Filter convergence is rapidly achieved when the number of prior modes is larger than one. Despite the fact that filter divergence can be still for $K=1$, interestingly, the proposed method does a reasonable estimation of the actual state of the system. In the linear case, this is not surprising, EnKF formulations are widely utilized in practice wherein Gaussian assumptions are commonly broken during assimilation steps. For non-linear observation operators $\gamma >1$, and Gaussian assumptions on the prior $K=1$, the accuracy of the filter relies on the sampling procedure. As can be seen, under such assumptions, the proposed method is capable of obtaining good estimates of posterior error modes.

RMSE values ($\lambda$) (26) in the log-scale are shown in the

Table 1. Experimental results with the Lorenz-96 model. Mean of RMSE values are reported across the 10 different experimental configurations for different configuration of parameters $\delta t$, s, K, and $\gamma$. The number of ensemble members equals $N=20$.

$\mathit{s}$	$\mathit{\delta }\mathit{t}$	$\mathit{\gamma }=\mathbf{1}$	$\mathit{\gamma }=\mathbf{3}$	$\mathit{\gamma }=\mathbf{5}$
70%	16 h
	50 h
100%	16 h
	50 h

and

Table 2. Experimental results with the Lorenz-96 model. Mean of RMSE values are reported across the 10 different experimental configurations for different configuration of parameters $\delta t$, s, K, and $\gamma$. The number of ensemble members equals $N=80$.

s	$\mathit{\delta }\mathit{t}$	$\mathit{\gamma }=\mathbf{1}$	$\mathit{\gamma }=\mathbf{3}$	$\mathbf{\gamma }=\mathbf{5}$
70%	16 h
	50 h
100%	16 h
	50 h

for $N=20$ and $N=80$, respectively. We group RMSE values by elapsed time between observations $\delta t$ and number of observed components s. Notice that the performance of the filter can be improved when the number of prior modes is larger than one. This aligns with the highly non-linear (and chaotic) dynamics exhibited by the Lorenz 96 model which makes a single mode distribution insufficient to encapsulate all prior information. However, the GMM-EnKF-MCMC can be impacted when a large number of mixture components is utilized and in some cases, overfitting is possible, for instance, in Table 1, for $\gamma =5$ and $N=20$.

Yet another interesting analysis is the behaviour of the sampling process. In Figure 3, we show a two dimensional projection of our sampling steps for some particular choices of parameters $\gamma$, N, s, and K. We use the two leading components of the space generated by the accepted samples among iterations. Note that, in all cases, the sampling method obtains good estimates of the actual state of the system ${\mathbf{x}}^{*}$, even more, assuming $N=K$ (which is equivalent to assume Gaussian errors for each ensemble member), the sampling steps converges to ${\mathbf{x}}^{*}$.

5. Conclusions

In this paper we propose an EnKF implementation based on a modified Cholesky decomposition and a Markov-Chain-Monte-Carlo (MCMC) Method, the GMM-EnKF-MCMC. During the assimilation of observations, the method proceeds as follows: prior error distributions are fitted by using Gaussian Mixture Models. Prior components are estimated making use of the Expectation Maximization algorithm. Based on MCMC, posterior components are individually estimated. This is done by sampling states along an approximated steepest descent direction of the well-known Three Dimensional Variational cost function. Experimental tests are performed on the Lorenz-96 model. The results reveal that the proposed method is able to handle non-linearities produced by the model dynamics as well as those generated by the non-linear observation operator and even more, in a root-mean-square-error sense, the accuracy of the filter is not highly impacted as the degree of the observation operator is increased.