Adaptive Conditional Bias-Penalized Kalman Filter for Improved Estimation of Extremes and Its Approximation for Reduced Computation

Abstract

Kalman filter (KF) and its variants and extensions are wildly used for hydrologic prediction in environmental science and engineering. In many data assimilation applications of Kalman filter (KF) and its variants and extensions, accurate estimation of extreme states is often of great importance. When the observations used are uncertain, however, KF suffers from conditional bias (CB) which results in consistent under- and overestimation of extremes in the right and left tails, respectively. Recently, CB-penalized KF, or CBPKF, has been developed to address CB. In this paper, we present an alternative formulation based on variance-inflated KF to reduce computation and algorithmic complexity, and describe adaptive implementation to improve unconditional performance. For theoretical basis and context, we also provide a complete self-contained description of CB-penalized Fisher-like estimation and CBPKF. The results from one-dimensional synthetic experiments for a linear system with varying degrees of nonstationarity show that adaptive CBPKF reduces the root-mean-square error at the extreme tail ends by 20 to 30% over KF while performing comparably to KF in the unconditional sense. The alternative formulation is found to approximate the original formulation very closely while reducing computing time to 1.5 to 3.5 times of that for KF depending on the dimensionality of the problem. Hence, adaptive CBPKF offers a significant addition to the dynamic filtering methods for general application in data assimilation when the accurate estimation of extremes is of importance.

1. Introduction

Streamflow prediction is subject to uncertainties from multiple sources. These include uncertainties in the input (e.g., mean areal precipitation (MAP) and mean areal potential evapotranspiration (MAPE)), hydrologic model uncertainties, and the uncertainties in the initial conditions. With climate change and rapid urbanization, the uncertainty in the input and parametric uncertainties will increase; thus, reducing them has become increasingly challenging. Recently, data assimilation is widely used to reduce the uncertainty in initial conditions [1,2].

In many data assimilation applications in hydrologic predictions, Kalman filter (KF) and its variants and extensions are widely used to fuse observations with model predictions in a wide range of applications [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]. In geophysics and environmental science and engineering, often the main objective of data assimilation is to improve the estimation and prediction of states in their extremes rather than in normal ranges. In hydrologic forecasting, for example, the accurate prediction of floods and droughts is far more important than that of streamflow and soil moisture in normal conditions. Because KF minimizes unconditional error variance, its solution tends to improve the estimation near median where the state of the dynamic system resides most of the times while often leaving significant biases in the extremes. Such conditional biases (CB) [24] generally result in consistent under- and overestimation of the true states in the upper and lower tails of the distribution, respectively. To address CB, CB-penalized Fisher-like estimation and CB-penalized KF (CBPKF) [25] have recently been developed which jointly minimize error variance and expectation of the Type-II CB squared for the improved estimation and prediction of extremes. The Type-II CB, defined as $E\left[\widehat{X}|X=x\right]-x$, is associated with failure to detect the event where x denotes the realization of $X$ where $X$, $\widehat{X}$ and $\widehat{x}$ denote the unknown truth, the estimate, and the realization of $\widehat{X}$, respectively [26]. The original formulation of CBPKF, however, is computationally extremely expensive for high-dimensional problems. Additionally, whereas CBPKF improves performance in the tails, it deteriorates performance in the normal ranges. In this work, we approximate CBPKF with forecast error covariance-inflated KF, referred to hereafter as the variance-inflated KF (VIKF) formulation, as a computationally less expensive and algorithmically simpler alternative, and implement adaptive CBPKF to improve unconditional performance.

Elements of CB-penalized Fisher-like estimation have been described in the forms of CB-penalized indicator cokriging for fusion of predicted streamflow from multiple models and observed streamflow [27], CB-penalized kriging for spatial estimation [28] and rainfall estimation [29], and CB-penalized cokriging for fusion of radar rainfall and rain gauge data [30]. The original formulation of CBPKF have been described in [25], respectively. Its ensemble extension, CB-penalized ensemble KF, or CEnKF, is described in [31] in the context of ensemble data assimilation for flood forecasting. In this paper, we provide in the context of data assimilation a complete self-contained description of CBPKF for theoretical background for the alternative formulation and adaptive implementation. Whereas CBPKF was initially motivated for environmental and geophysical state estimation and prediction, it is broadly applicable to a wide range of applications for which improved performance in the extremes is desired. This paper is organized as follows. Section 2 and Section 3 describe CB-penalized Fisher-like solution and CBPKF, respectively. Section 4 describes approximation of CBPKF. Section 5 describe the evaluation experiments and results, respectively. Section 6 describes adaptive CBPKF. Section 7 provides the conclusions.

2. Conditional Bias-Penalized Fisher-like Solution

As in Fisher estimation [32], the estimator sought for CB-penalized Fisher-like estimation is $X^{*}=\mathit{W}\mathit{Z}$ where $X^{*}$ denotes the (m × 1) vector of the estimated states, W denotes the (m × (n + m)) weight matrix, and Z denotes the ((n + m) × 1) augmented observation vector. In the above, n denotes the number of observations, m denotes the number of state variables, and (n + m) represents the dimensionality of the augmented vector of the observations and the model-predicted states to be fused for the estimation of the true state $X$. The purpose of augmentation is to relate directly to CBPKF in Section 3 without introducing additional notations. Throughout this paper, we use regular and bold letters to differentiate the non-augmented and augmented variables, respectively. The linear observation equation is given by:

$$Z=HX+V$$

(1)

where X denotes the (m × 1) vector of the true state with E[X] = M_X and Cov[X,X^T] = Ψ_XX, H denotes the ((n + m) × m) augmented linear observation equation matrix, and V denotes the ((n + m) × 1) augmented zero-mean observation error vector with $\mathrm{Cov}\left[\mathrm{V},{\mathrm{V}}^{\mathrm{T}}\right]=\mathrm{R}$. Assuming independence between X and V, we write the Bayesian estimator [30] for $X$, or $X^{*}$, as:

$$X^{*}=M_X+\mathit{W}\left(\mathit{Z}-\mathit{H}{M}_x\right)$$

(2)

The error covariance matrix for $X^{*}$, $E\left[\left(X-X^{*}\right){\left(X-X^{*}\right)}^T\right]$, is given by:

$${\mathsf{\Sigma }}_{EV}=\left(\mathit{I}-\mathit{W}\mathit{H}\right){\mathsf{\Psi }}_{XX}{\left(\mathit{I}-\mathit{W}\mathit{H}\right)}^{\mathit{T}}+\mathit{W}\mathit{R}{\mathit{W}}^{\mathit{T}}$$

(3)

With Equation (2), we may write Type-II CB as:

$$X-E[X^{*}|X]=(X-M_X)-\mathit{W}E[\left(\mathit{Z}-\mathit{H}{M}_x\right)|X]$$

(4)

The observation equation for Z is obtained by inverting Equation (1):

$$X=G^{T}Z-G^{T}V$$

(5)

The (mxn) matrix, G^T, in Equation (5) is given by:

$${\mathit{G}}^{\mathit{T}}={\left({\mathit{U}}^{\mathit{T}}\mathit{H}\right)}^{-1}{\mathit{U}}^{\mathit{T}}$$

(6)

where U^T is some (m × (n + m)) nonzero matrix. Using Equation (5) and the identity, ${\Psi }_{ZZ}=\mathit{H}{\mathsf{\Psi }}_{XX}{\mathit{H}}^{\mathit{T}}+\mathit{R}$, we may write the Bayesian estimate for $\mathrm{E}\left[\mathit{Z}|X\right]$ in Equation (4) as:

$$\widehat{E}\left[\mathit{Z}|X\right]=\mathit{H}{M}_X+\mathit{C}\left(X-M_X\right)$$

(7)

where

$$\mathit{C}=\left(\mathit{H}{\mathsf{\Psi }}_{XX}{\mathit{H}}^{\mathit{T}}+\mathit{R}\right){\mathit{G}}^{\mathit{T}}{\left[\mathit{G}\left(\mathit{H}{\mathsf{\Psi }}_{XX}{\mathit{H}}^{\mathit{T}}+2\mathit{R}\right){\mathit{G}}^{\mathit{T}}\right]}^{-\mathbf{1}}$$

(8)

Equations (7) and (8) state that the Bayesian estimate of Z given X is given by HX if the a priori state error covariance ${\mathsf{\Psi }}_{XX}$ is noninformative or there are no observation errors, but by the average of the a priori mean $M_X$ and the observed true state X if the a priori ${\mathsf{\Psi }}_{XX}$ is perfectly informative or observations are information-less.

With Equation (4), we may write the quadratic penalty due to Type-II CB as:

$${\mathsf{\Sigma }}_{CB}=E[(X-E_{X^{*}}[X^{*}|X]){(X-E_{X^{*}}[X^{*}|X])}^T]=\left(I-\mathit{W}\mathit{C}\right){\mathsf{\Psi }}_{XX}{\left(I-\mathit{W}\mathit{C}\right)}^T$$

(9)

where I denotes the (m × m) identity matrix. Combining ${\mathsf{\Sigma }}_{EV}$ in Equation (3) and ${\mathsf{\Sigma }}_{CB}$ in Equation (9), we have the apparent error covariance, ${\mathsf{\Sigma }}_a$, which reflects both the error covariance and Type-II CB:

$${\mathsf{\Sigma }}_a=\left(I-\mathit{W}\mathit{H}\right){\Psi }_{\mathit{X}\mathit{X}}{\left(I-\mathit{W}\mathit{H}\right)}^{\mathit{T}}+\mathit{W}\mathit{R}{\mathit{W}}^{\mathit{T}}+\alpha \left(I-\mathit{W}\mathit{C}\right){\mathsf{\Psi }}_{XX}{\left(I-\mathit{W}\mathit{C}\right)}^{\mathit{T}}$$

(10)

where α denotes the scaler weight given to the CB penalty term. Minimizing Equation (10) with respect to W, or by direct analogy with the Bayesian solution [32], we have:

$$W={\mathsf{\Psi }}_{XX}{\widehat{\mathit{H}}}^{\mathit{T}}{\left[\widehat{\mathit{H}}{\mathsf{\Psi }}_{XX}{\widehat{\mathit{H}}}^{\mathit{T}}+\Lambda \right]}^{-\mathbf{1}}$$

(11)

The modified structure matrix ${\widehat{\mathit{H}}}^{\mathit{T}}$ and observation error covariance matrix $\Lambda$ in Equation (11) are given by:

$${\widehat{\mathit{H}}}^{\mathit{T}}={\mathit{H}}^{\mathit{T}}+\alpha {\mathit{C}}^{\mathit{T}}$$

(12)

$$\Lambda =\mathit{R}+\alpha \left(1-\alpha \right)\mathit{C}{\mathsf{\Psi }}_{XX}{\mathit{C}}^{\mathit{T}}-\alpha \mathit{H}{\mathsf{\Psi }}_{XX}{\mathit{C}}^{\mathit{T}}-\alpha \mathit{C}{\mathsf{\Psi }}_{XX}{\mathit{H}}^{\mathit{T}}$$

(13)

Using Equation (11) and the matrix inversion lemma [33], we have for ${\mathsf{\Sigma }}_a$ and $X^{*}$ in Equations (10) and (2), respectively:

$${\mathsf{\Sigma }}_a=\alpha {\mathsf{\Psi }}_{XX}+{\left[\widehat{\mathit{H}}{\Lambda }^{-\mathbf{1}}{\widehat{\mathit{H}}}^{\mathit{T}}+{\mathsf{\Psi }}_{XX}^{-1}\right]}^{-\mathbf{1}}$$

(14)

$$X^{*}={\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\widehat{\mathit{H}}+{\mathsf{\Psi }}_{XX}^{-1}\right]}^{-1}\left\{{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{Z}+{\mathsf{\Psi }}_{XX}^{-1}{M}_X\right\}+\mathsf{\Delta }$$

(15)

where $\mathsf{\Delta }={\mathsf{\alpha }\mathsf{\Psi }}_{XX}{\widehat{\mathit{H}}}^{\mathit{T}}{\left[\widehat{\mathit{H}}{\mathsf{\Psi }}_{XX}{\widehat{\mathit{H}}}^{\mathit{T}}+\Lambda \right]}^{-\mathbf{1}}\mathit{C}{M}_X$. To render the above Bayesian solution to a Fisher-like solution, we assume no a priori information in X and let ${\mathsf{\Psi }}_{XX}^{-1}$ in the brackets in Equations (14) and (15) vanish:

$${\mathsf{\Sigma }}_a=B{\left[\widehat{\mathit{H}}{\Lambda }^{-\mathbf{1}}{\widehat{\mathit{H}}}^{\mathit{T}}\right]}^{-\mathbf{1}}$$

(16)

$$X^{*}={\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\widehat{\mathit{H}}\right]}^{-\mathbf{1}}{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-1}\mathit{Z}+\mathsf{\Delta }$$

(17)

where the scaling matrix B is given by $B=\alpha {\mathsf{\Psi }}_{XX}{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\widehat{\mathit{H}}+I$. To obtain the estimator of the form, $X^{*}=\mathit{W}\mathit{Z}$, we impose the unbiasedness condition, $\mathrm{E}\left[X^{*}\right]=\mathrm{X}$, or equivalently, $\mathit{W}\mathit{H}=I$. The above condition is satisfied by replacing ${\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\widehat{\mathit{H}}\right]}^{-\mathbf{1}}$ with ${\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{H}\right]}^{-\mathbf{1}}$ and dropping ∆ in Equation (17):

$${\mathsf{\Sigma }}_a=\mathit{B}{\left[\widehat{\mathit{H}}{\Lambda }^{-\mathbf{1}}{\mathit{H}}^{\mathit{T}}\right]}^{-\mathbf{1}}$$

(18)

$$X^{*}={\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{H}\right]}^{-\mathbf{1}}{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{Z}$$

(19)

Finally, we obtain from Equation (3) the error covariance, ${\mathsf{\Sigma }}_{EV}$, associated with $X^{*}$ in Equation (19):

$${\mathsf{\Sigma }}_{EV}=\mathit{W}\mathit{R}{\mathit{W}}^{\mathit{T}}={\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{H}\right]}^{-\mathbf{1}}{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{R}{\Lambda }^{-\mathbf{1}}\widehat{\mathit{H}}{\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{H}\right]}^{-\mathbf{1}}$$

(20)

Note that, if α = 0, we have ${\widehat{\mathit{H}}}^{\mathit{T}}=\mathit{H}$ and $\Lambda =\mathit{R}$, and hence the CB-penalized Fisher-like solution, Equations (19) and (20), is reduced to the Fisher solution [32].

3. Conditional Bias-Penalized Kalman Filter

CBPKF results directly from decomposing the augmented matrices and vectors in Equations (19) and (20) as KF does from the Fisher solution [32]. The CBPKF solution, however, is not very simple because the modified observation error covariance matrix, Λ, is no longer diagonal. An important consideration in casting the CB-penalized Fisher-like solution into CBPKF is to recognize that CB arises from the error-in-variable effects associated with uncertain observations [34], and that the a priori state, represented by the dynamical model forecast, is not subject to CB. We therefore apply the CB penalty to the observations only and reduce C in Equation (8) to ${\mathit{C}}^{\mathit{T}}=\left(C_{1,k}^T C_{2,k}^T\right)=\left(C_{1,k}^T 0\right)$. Separating the observation and dynamical model components in ${\widehat{\mathit{H}}}^{\mathit{T}}$ and $\Lambda$ via the matrix inversion lemma, we have:

$${\widehat{\mathit{H}}}^{\mathit{T}}=\left({\widehat{H}}_{1,k}^T I\right)$$

(21)

$$\Lambda =\left[\begin{array}{cc}{\mathsf{\Lambda }}_{11,k}& {\mathsf{\Lambda }}_{12,k}\\ {\mathsf{\Lambda }}_{21,k}& {\mathsf{\Lambda }}_{22,k}\end{array}\right]$$

(22)

where

$${\widehat{H}}_{1,k}^T=H_k^T+\alpha C_{1,k}^T$$

(23)

$${\mathsf{\Lambda }}_{11,k}=R_k+\alpha \left(1-\alpha \right)C_{1,k}{\mathsf{\Psi }}_{XX}{C}_{1,k}^T-\alpha H_k{\mathsf{\Psi }}_{XX}{C}_{1,k}^T-\alpha C_{1,k}{\mathsf{\Psi }}_{XX}{H}_k^T$$

(24)

$${\mathsf{\Lambda }}_{12,k}=-\alpha C_{1,k}{\mathsf{\Psi }}_{XX}$$

(25)

$${\mathsf{\Lambda }}_{21,k}={\mathsf{\Lambda }}_{12,k}^T$$

(26)

$${\mathsf{\Lambda }}_{22,k}={\mathsf{\Sigma }}_{k|k-1}$$

(27)

In the above, $H_k$ denotes the (n × m) observation matrix, and $R_k$ denotes the (n × n) observation error covariance matrix. To evaluate the (m × n) matrix, $C_{1,k}$, it is necessary to specify ${\mathit{U}}^{\mathit{T}}$ in Equation (6). We use ${\mathit{U}}^{\mathit{T}}={\mathit{H}}^{\mathit{T}}$ which ensures invertibility of ${\mathit{U}}^{\mathit{T}}\mathit{H}$, but other choices are also possible. We then have for $C_{1,k}^T$:

$$C_{1,k}=\left[\left(H_k{\mathsf{\Psi }}_{XX}{H}_k^T+R_k\right)G_{1,k}+H_k{\mathsf{\Psi }}_{XX}{G}_{2,k}\right]L_k^{-1}$$

(28)

where

$$G_{2,k}^T={\left(H_k^T{H}_k+I\right)}^{-1}$$

(29)

$$G_{1,k}^T=G_{2,k}^T{H}_k^T$$

(30)

$$L_k=G_{2,k}^T\left[H_k^T\left(H_k{\mathsf{\Psi }}_{XX}{H}_k^T+2R_k\right)H_k+H_k^T{H}_k{\mathsf{\Psi }}_{XX}+{\mathsf{\Psi }}_{XX}{H}_k^T{H}_k+{\mathsf{\Psi }}_{XX}+2{\mathsf{\Sigma }}_{k|k-1}\right]G_{2,k}.$$

(31)

Expanding W in Equation (11) with ${\Lambda }^{-\mathbf{1}}=\Gamma =\left[\begin{array}{cc}{\mathsf{\Gamma }}_{11,k}& {\mathsf{\Gamma }}_{12,k}\\ {\mathsf{\Gamma }}_{21,k}& {\mathsf{\Gamma }}_{22,k}\end{array}\right]$, we have:

$$\mathit{W}={\left[{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}\mathit{H}\right]}^{-\mathbf{1}}{\widehat{\mathit{H}}}^{\mathit{T}}{\Lambda }^{-\mathbf{1}}={\left({\varpi }_{1,k}{H}_k+{\varpi }_{2,k}\right)}^{-1}\left({\varpi }_{1,k} {\varpi }_{2,k}\right)$$

(32)

In Equation (32), the (m × n) and (m × m) weight matrices for the observation and model prediction, ω_1,k and ω_2,k, respectively, are given by:

$${\varpi }_{1,k}={\widehat{H}}_{1,k}^T{\mathsf{\Gamma }}_{11,k}+{\mathsf{\Gamma }}_{21,k}$$

(33)

$${\varpi }_{2,k}={\widehat{H}}_{1,k}^T{\mathsf{\Gamma }}_{12,k}+{\mathsf{\Gamma }}_{22,k}$$

(34)

where

$${\mathsf{\Gamma }}_{22,k}={\left[{\mathsf{\Lambda }}_{22,k}-{\mathsf{\Lambda }}_{21,k}{\mathsf{\Lambda }}_{11,k}^{-1}{\mathsf{\Lambda }}_{12,k}\right]}^{-1}$$

(35)

$${\mathsf{\Gamma }}_{11,k}={\mathsf{\Lambda }}_{11,k}^{-1}+{\mathsf{\Lambda }}_{11,k}^{-1}{\mathsf{\Lambda }}_{12,k}{\mathsf{\Gamma }}_{22,k}{\mathsf{\Lambda }}_{21,k}{\mathsf{\Lambda }}_{11,k}^{-1}$$

(36)

$${\mathsf{\Gamma }}_{12,k}=-{\mathsf{\Lambda }}_{11,k}^{-1}{\mathsf{\Lambda }}_{12,k}{\mathsf{\Gamma }}_{22,k}$$

(37)

The apparent CBPKF error covariance, which reflects both ${\mathsf{\Sigma }}_{EV}$ and ${\mathsf{\Sigma }}_{CB}$, is given by Equation (18) as:

$${\mathsf{\Sigma }}_{a,k|k}=\alpha {\mathsf{\Sigma }}_{k|k-1}+{\left[{\varpi }_{1,k}{H}_k+{\varpi }_{2,k}\right]}^{-1}$$

(38)

The CBPKF error covariance, which reflects ${\mathsf{\Sigma }}_{EV}$ only, is given by Equation (20) as:

$${\mathsf{\Sigma }}_{k|k}={\left[{\varpi }_{1,k}{H}_k+{\varpi }_{2,k}\right]}^{-1}\left({\varpi }_{1,k}{R}_k{\varpi }_{1,k}^T+{\varpi }_{2,k}{\mathsf{\Sigma }}_{k|k-1}{\varpi }_{2,k}^T\right){\left[{\varpi }_{1,k}{H}_k+{\varpi }_{2,k}\right]}^{-1}$$

(39)

Because CBPKF minimizes ${\mathsf{\Sigma }}_{a,k|k}$ rather than ${\mathsf{\Sigma }}_{k|k}$, it is not guaranteed that Equation (39) satisfies ${\mathsf{\Sigma }}_{k|k}\le {\mathsf{\Sigma }}_{k|k-1}$ a priori. If the above condition is not met, it is necessary to reduce α and repeat the calculations. If α is reduced all the way to zero, CBPKF collapses to KF. The CBPKF estimate may be rewritten into a more familiar form:

$${\widehat{X}}_{k|k}={\left[{\varpi }_{1,k}{H}_k+{\varpi }_{2,k}\right]}^{-1}\left[{\varpi }_{1,k}{Z}_k+{\varpi }_{2,k}{\widehat{X}}_{k|k-1}\right]={\widehat{\mathrm{X}}}_{k|k-1}+K_k\left[Z_k-H_k{\widehat{\mathrm{X}}}_{k|k-1}\right]$$

(40)

In Equation (40), Z_k denotes the (n × 1) observation vector, and the (m × n) CB-penalized Kalman gain, $K_k$, is given by:

$$K_k={\left[{\varpi }_{1,k}{H}_k+{\varpi }_{2,k}\right]}^{-1}{\varpi }_{1,k}$$

(41)

To operate the above as a sequential filter, it is necessary to prescribe ${\mathsf{\Psi }}_{XX}$ and α. An obvious choice for ${\mathsf{\Psi }}_{XX}$, i.e., the a priori error covariance of the state, is ${\mathsf{\Sigma }}_{k|k-1}$. Specifying α requires some care. In general, a larger α improves accuracy over the tails but at the expense of increasing unconditional error. Too small an α may not affect large enough CB penalty in which case the CBPKF and KF solutions would differ a little. Too large an α, on the other hand, may severely violate the ${\mathsf{\Sigma }}_{k|k}\le {\mathsf{\Sigma }}_{k|k-1}$ condition in which case the filter may have to be iterated at an additional computational expense with successively reduced $\alpha$. A reasonable strategy for reducing $\alpha$ is ${\alpha }_i=c{\alpha }_{i-1}, i=1,2,3,\dots$, with 0 < c < 1 where ${\alpha }_i$ denotes the value of α at the i-th iteration [24,29]. For high-dimensional problems, CBPKF can be computationally very expensive. Whereas KF requires solving an (m × n) linear system only once per updating or fusion cycle, CBPKF additionally requires solving two (m × m) linear systems (for $C_{1,k}$. and ${\mathsf{\Gamma }}_{22}$), and a (n × n) system (for ${\mathsf{\Lambda }}_{11}$), assuming that the structure of the observation equation does not change in time (in which case $G_{2,k}^T$ in Equation (29) may be evaluated only once). To reduce computation, below we approximate CBPKF with KF by inflating the forecast error covariance.

4. VIKF Approximation of CBPKF

The main idea behind this simplification is that, if the gain for the CB penalty, C, in Equation (10) can be linearly approximated with H, the apparent error covariance ${\mathsf{\Sigma }}_a$ becomes identical to ${\mathsf{\Sigma }}_{EV}$ in Equation (3) but with ${\mathsf{\Psi }}_{XX}$ inflated by a factor of 1 + α:

$${\mathsf{\Sigma }}_{\left(1+\alpha \right)}=\left(I-\mathit{W}\mathit{H}\right)\left(1+\alpha \right){\mathsf{\Psi }}_{XX}{\left(I-\mathit{W}\mathit{H}\right)}^T+\mathit{W}{\mathit{R}}_{\left(\mathbf{1}+\mathit{\alpha }\right)}{\mathit{W}}^{\mathit{T}}$$

(42)

where ${\mathit{R}}_{\left(\mathbf{1}+\mathit{\alpha }\right)}=\left[\begin{array}{cc}R& 0\\ 0& \left(1+\alpha \right){\mathsf{\Psi }}_{XX}\end{array}\right]$. The KF solution for Equation (42) is identical to the standard KF solution but with ${\mathsf{\Sigma }}_{k|k-1}$ replaced by $\left(1+\mathsf{\alpha }\right){\mathsf{\Sigma }}_{k|k-1}$:

$${\widehat{\mathrm{X}}}_{k|k}={\left[H_k^T{R}_k^{-1}{H}_k+{\left\{\left(1+\alpha \right){\mathsf{\Sigma }}_{k|k-1}\right\}}^{-1}\right]}^{-1}\left[H_k^T{R}_k^{-1}{Z}_k+{\left\{\left(1+\alpha \right){\mathsf{\Sigma }}_{k|k-1}\right\}}^{-1}{\widehat{\mathrm{X}}}_{k|k-1}\right]$$

(43)

With WH=I in Equation (43) for the VIKF solution, we have ${\mathsf{\Sigma }}_{\left(1+\alpha \right)}=\mathit{W}{\mathit{R}}_{\left(1+\mathit{\alpha }\right)}{\mathit{W}}^{\mathit{T}}$ for the apparent filtered error variance of ${\widehat{\mathrm{X}}}_{k|k}$ in Equation (42). The error covariance of ${\widehat{\mathrm{X}}}_{k|k}$, ${\mathsf{\Sigma }}_{k|k}$, is given by Equation (3) as:

$$\begin{gathered} {\mathsf{\Sigma }}_{k|k}=\mathit{W}\mathit{R}{\mathit{W}}^{\mathit{T}}={\left[{\mathit{H}}^{\mathit{T}}{\mathit{R}}_{\left(\mathbf{1}+\mathit{\alpha }\right)}^{-\mathbf{1}}\mathit{H}\right]}^{-\mathbf{1}}{\mathit{H}}^{\mathit{T}}{\mathit{R}}_{\left(1+\mathit{\alpha }\right)}^{-\mathbf{1}}\mathit{R}{\mathit{R}}_{\left(1+\mathit{\alpha }\right)}^{-\mathbf{1}}\mathit{H}{\left[{\mathit{H}}^{\mathit{T}}{\mathit{R}}_{\left(\mathbf{1}+\mathit{\alpha }\right)}^{-\mathbf{1}}\mathit{H}\right]}^{-\mathbf{1}} \\ ={\mathsf{\Sigma }}_{\left(1+\alpha \right),k|k}{\mathsf{\Sigma }}_{{\left(1+\alpha \right)}^2,k|k}^{-1}{\mathsf{\Sigma }}_{\left(1+\alpha \right),k|k} \end{gathered}$$

(44)

In Equation (44), the inflated filtered error covariance, ${\mathsf{\Sigma }}_{\beta ,k|k}$, where $\beta$ denotes the multiplicative inflation factor, is given by:

$${\mathsf{\Sigma }}_{\beta ,k|k}=\beta {\mathsf{\Sigma }}_{k|k-1}-{\mathsf{\beta }\mathsf{\Sigma }}_{k|k-1}{H}_k^T{\left[H_k{\mathsf{\beta }\mathsf{\Sigma }}_{k|k-1}{H}_k^T+R_k\right]}^{-1}{H}_k{\mathsf{\beta }\mathsf{\Sigma }}_{k|k-1}={\left[H_k^T{R}_k^{-1}{H}_k+{\left(\beta {\mathsf{\Sigma }}_{k|k-1}\right)}^{-1}\right]}^{-1}$$

(45)

Computationally, the evaluation of Equations (43) and (44) requires solving two (m × n) and a (m × m) linear systems. As in the original formulation of CBPKF, iterative reduction of $\alpha$ is necessary to ensure ${\mathsf{\Sigma }}_{k|k}\le {\mathsf{\Sigma }}_{k|k-1}$.

The above approximation assumes that the CB penalty, ${\mathsf{\Sigma }}_{CB}$, is proportional to the error covariance, ${\mathsf{\Sigma }}_{EV}$. To help ascertain how KF, CBPKF and the VIKF approximation may differ, we compare in Table 1 their analytical solutions for gain ${\kappa }_k$, and filtered error variance ${\sigma }_{k|k}^2$ for the 1D case of m = n = 1. The table shows that the VIKF approximation and CBPKF are identical for the 1D problem except that the CB penalty for CBPKF is twice as large as that for the VIKF approximation. To visualize the differences, Figure 1 shows ${\kappa }_k$ and ${\sigma }_{k|k}^2$ for KF, the VIKF approximation and CBPKF for the three cases of ${\sigma }_{k|k-1}^2=1$ and ${\sigma }_Z^2=1$ (left), ${\sigma }_{k|k-1}^2=1$ and ${\sigma }_Z^2=4$ (middle), and ${\sigma }_{k|k-1}^2=4$ and ${\sigma }_Z^2=1$ (right). For all cases, we set h to unity and varied $\alpha$ from 0 to 1. The figure indicates that, compared to KF, the VIKF approximation and CBPKF prescribe appreciably larger gains, that the increase in gain is larger for larger α, and that the CBPKF gain is larger than the gain in the VIKF approximation for the same value of α. The figure also indicates that, compared to KF error variance, CBPKF error variance is larger, and that the increase in error variance is larger for larger α. Note that the differences between the KF and CBPKF solutions are the smallest for ${\sigma }_{k|k-1}^2>{\sigma }_Z^2$, a reflection of the diminished impact of CB owing to the comparatively smaller uncertainty in the observations. The above development suggests that one may be able to approximate CBPKF very closely with the VIKF-based formulation by adjusting α in the latter. Below, we evaluate the performance of CBPKF relative to KF and the VIKF-based approximation of CBPKF.

Table 1. Comparison of gain and filtered error variance among KF, the VIKF approximation, and CBPKF.

	$\mathbf{Gain},{\mathit{\kappa }}_{\mathit{k}}$	$\mathbf{Filtered}\mathbf{Error}\mathbf{Variance},{\mathit{\sigma }}_{\mathit{k}|\mathit{k}}^2$
KF	$\frac{h{\sigma }_{k|k-1}^2}{h^2{\sigma }_{k|k-1}^2+{\sigma }_Z^2}$	$\frac{{\sigma }_Z^2}{h^2{\sigma }_{k|k-1}^2+{\sigma }_Z^2}{\sigma }_{k|k-1}^2$
VIKF approx.	$\frac{h\left(1+\alpha \right){\sigma }_{k|k-1}^2}{h^2\left(1+\alpha \right){\sigma }_{k|k-1}^2+{\sigma }_Z^2}$	$\frac{\left\{{\left(1+\alpha \right)}^2{h}^2{\sigma }_{k|k-1}^2+{\sigma }_Z^2\right\}{\sigma }_Z^2}{{\left\{\left(1+\alpha \right)h^2{\sigma }_{k|k-1}^2+{\sigma }_Z^2\right\}}^2}{\sigma }_{k|k-1}^2$
CBPKF	$\frac{h\left(1+2\alpha \right){\sigma }_{k|k-1}^2}{h^2\left(1+2\alpha \right){\sigma }_{k|k-1}^2+{\sigma }_Z^2}$	$\frac{\left\{{\left(1+2\alpha \right)}^2{h}^2{\sigma }_{k|k-1}^2+{\sigma }_Z^2\right\}{\sigma }_Z^2}{{\left\{\left(1+2\alpha \right)h^2{\sigma }_{k|k-1}^2+{\sigma }_Z^2\right\}}^2}{\sigma }_{k|k-1}^2$

Figure 1. Comparison of ${\kappa }_k$ and ${\sigma }_{k|k}^2$ for KF, the VIKF approximation and CBPKF for three different cases: ${\sigma }_{k|k-1}^2=1$ and ${\sigma }_Z^2=1$ (left), ${\sigma }_{k|k-1}^2=1$ and ${\sigma }_Z^2=4$ (middle), ${\sigma }_{k|k-1}^2=4$ and ${\sigma }_Z^2=1$ (right).

5. Evaluation and Results

For comparative evaluation, we carried out the synthetic experiments of [25]. We assume the following linear dynamical and observation models with perfectly known statistical parameters:

$$X_k={\mathsf{\Phi }}_{k-1}{X}_{k-1}+W_{k-1}$$

(46)

$$Z_k=H_k{X}_k+V_k$$

(47)

where X_k and X_k−1 denote the state vectors at time steps k and k − 1, respectively, Φ_k−1 denotes the state transition matrix at time step k − 1 assumed as ${\mathsf{\Phi }}_{k-1}={\phi }_{k-1}I$, $W_{k-1}$ denotes the white noise vector, $w_{j,k-1}~N\left(0,{\sigma }_{w_{k-1}}^2\right)$, j = 1,…,m, with $Q_{k-1}=E\left[W_{k-1}{W}_{k-1}^T\right]$, and V_k denotes the observation error vector, $v_{i,k}~N\left(0,{\sigma }_{v_k}^2\right)$, I = 1, …, n. The number of observations, n, is assumed to be time-invariant. The observation errors are assumed to be independent among themselves and of the true state. To assess comparative performance under widely varying conditions, we randomly perturbed φ_k−1, σ_w,k−1 and σ_v,k above according to Equations (48) through (50) below, and used only those deviates that satisfy the bounds:

$${\phi }_{k-1}^p={\phi }_{k-1}+{\gamma }_{\phi }{\epsilon }_{\phi } 0.5\le {\phi }_{k-1}^p\le 0.95$$

(48)

$${\sigma }_{w,k-1}^p={\sigma }_{w,k-1}+{\gamma }_w{\epsilon }_{w } {\sigma }_{w,k-1}^p\ge 0.01$$

(49)

$${\sigma }_{v,k}^p={\sigma }_{v,k}+{\gamma }_v{\epsilon }_{v } {\sigma }_{v,k}^p\ge 0.01$$

(50)

In the above, the superscript p signifies that the variable is a perturbation, ${\epsilon }_{\phi }$ ${\epsilon }_{w }$ and ${\epsilon }_{v }$ denote the normally distributed white noise for the respective variables, and ${\gamma }_{\phi }$ ${\gamma }_w$ and ${\gamma }_v$ denote the standard deviations of the white noise added to ${\phi }_{k-1}$, ${\mathsf{\sigma }}_{w,k-1}$ and ${\sigma }_{v,k}$, respectively. The parameter settings (see Table 1) are chosen to encompass less predictable (small φ_k−1) to more predictable (large φ_k−1) processes, certain (small σ_w,k−1) to uncertain (large σ_w,k−1) model dynamics, and more informative (small σ_v,k) to less informative (large σ_v,k) observations. The bounds for ${\phi }_{k-1}^p$ in Equation (48) are based on the range of lag-1 serial correlation representing moderate to high predictability where CBPKF and KF are likely to differ the most. The bounding of the perturbed values ${\sigma }_{w,k-1}^p$ and ${\sigma }_{v,k}^p$ in Equations (49) and (50), respectively, is necessary to avoid the observational or model prediction uncertainty becoming unrealistically small. Very small ${\sigma }_{w,k-1}^p$ and ${\sigma }_{v,k}^p$ render the information content of the model prediction, ${\mathsf{\Sigma }}_{k|k-1}$, and the observation, Z_k, respectively, very large, and hence keep the filters operating in unrealistically favorable conditions for extended periods of time. We then apply KF, CBPKF, and the VIKF approximation to obtain ${\widehat{\mathrm{X}}}_{k|k}$ and ${\mathsf{\Sigma }}_{k|k}$, and verify them against the assumed truth. To evaluate the performance of CBPKF relative to KF, we calculate percent reduction in root-mean-square error (RMSE) by CBPKF over KF conditional on the true state exceeding some threshold between 0 and the largest truth.

Figure 2 shows the percent reduction in RMSE by CBPKF over KF for Cases 1 (left), 5 (middle) and 9 (right) representing Groups 1, 2 and 3 in Table 1, respectively. The three groups differ most significantly in the variability of the dynamical model error, ${\gamma }_w$, and may be characterized as nearly stationary (Group 1), nonstationary (Group 2), and highly nonstationary (Group 3). The range of $\alpha$ values used is [0.1, 1.2] with an increment of 0.1. The numbers of state variables, observations, and updating cycles used in Figure 2 are 1, 10, and 100,000 for all cases. The dotted line at 10% reduction in the figure serves as a reference for significant improvement. The figure shows that, at the extreme end of the tail, CBPKF with $\alpha$ of 0.7, 0.6, and 0.5 reduces RMSE by about 15, 25, and 30% for Cases 1, 5 and 9, respectively, but at the expense of increasing unconditional RMSE by about 5%. The general pattern of reduction in RMSE for other cases in Table 1 is similar within each group and is not shown. We only note here that larger variability in observational uncertainty (i.e., larger ${\gamma }_v$) reduces the relative performance of CBPKF somewhat, and that the magnitude of variability in predictability (i.e., ${\gamma }_{\phi }$) has a relatively small impact on the relative performance.

Figure 2. Percent reduction in RMSE by CBPKF over KF for a range of values of $\alpha$ for Cases 1 (left), 5 (middle), and 9 (right).

It was seen in Table 1 that the VIKF approximation is identical to CBPKF for m = n = 1 but for the multiplicative scaler weight for the CB penalty. Numerical experiments indicate that, whereas the above relationship does not hold for other m or n, one may very closely approximate CBPKF with the VIKF-based formulation by adjusting $\alpha$. For example, the VIKF approximation with $\alpha$ increased by a factor of 1.25 to 1.90 differ from CBPKF only by 1% or less for all 12 cases in Table 2 with m = 1 and n = 10. The above findings indicate that the VIKF approximation may be used as a computationally less expensive alternative for CBPKF. Table 3 compares the CPU time among KF, CBPKF, and the VIKF approximation for six different combinations of m and n based using Intel(R) Xeon(R) Gold 6152 CPU @ 2.10 GHz. The computing time is reported in multiples of the KF’s. Note that the original formulation of CBPKF quickly becomes extremely expensive as the dimensionality of the problem increases, whereas the CPU time of the VIKF approximation stays under 3.5 times that of KF for the size of the problems considered.

Table 2. Parameter settings for the 12 cases considered.

Group	Case	${\mathit{\sigma }}_{\mathit{w},\mathit{k}-\mathbf{1}}$	${\mathit{\gamma }}_{\mathit{w}}$	${\mathit{\sigma }}_{\mathit{v},\mathit{k}}$	${\mathit{\gamma }}_{\mathit{v}}$	${\mathit{\phi }}_{\mathit{k}-\mathbf{1}}$	${\mathit{\gamma }}_{\mathit{\phi }}$
1	1	0.1	0.01	1.5	0.4	0.7	0.1
	2	0.1	0.01	1.5	0.4	0.7	0.8
	3	0.1	0.01	1.5	1.2	0.7	0.1
	4	0.1	0.01	1.5	1.2	0.7	0.8
2	5	0.1	0.1	1.5	0.4	0.7	0.1
	6	0.1	0.1	1.5	0.4	0.7	0.8
	7	0.1	0.1	1.5	1.2	0.7	0.1
	8	0.1	0.1	1.5	1.2	0.7	0.8
3	9	0.1	0.2	1.5	0.4	0.7	0.1
	10	0.1	0.2	1.5	0.4	0.7	0.8
	11	0.1	0.2	1.5	1.2	0.7	0.1
	12	0.1	0.2	1.5	1.2	0.7	0.8

Table 3. Comparison of computing time among KF, CBPKF, and VIKF approximation.

Dimensionality		Normalized Computing Time
m	n	KF	CBPKF	VIKF approx.
1	10	1	5.23	1.51
1	40	1	18.41	2.74
5	10	1	6.44	1.67
5	40	1	24.03	2.88
10	10	1	14.27	2.03
10	40	1	27.96	3.46

If the filtered error variance is unbiased, one would expect the mean of the actual error squared associated with the variance to be approximately the same as the variance itself. To verify this, we show in Figure 3 the filtered error variance vs. the actual error squared for KF (left), the VIKF approximation (middle), and CBPKF (right) for all ranges of filtered error variance. For reference, we plot the one-to-one line representing the unbiased error variance conditional on the magnitude of the filtered error variance and overlay the local regression fit through the actual data points using the R package locfit [35]. The figure shows that all three provide conditionally unbiased estimates of filtered error variance as theoretically expected, and that the VIKF approximation and CBPKF results are extremely similar to each other.

Figure 3. Filtered error variance vs. error squared for KF (left), the VIKF approximation (middle), and CBPKF (right). The one-to-one line is shown in black and the local regression fit is shown in green.

6. Adaptive CBPKF

Whereas CBPKF or the VIKF approximation significantly improves the accuracy of the estimates over the tails, it deteriorates performance near the median. Figure 2 suggests that if $\alpha$ can be prescribed adaptively such that a small/large CB penalty is affected when the system is in the normal/extreme state, the unconditional performance of CBPKF would improve. Because the true state of the system is not known, adaptively specifying $\alpha$ is necessarily an uncertain proposition. There are, however, certain applications in which the normal vs. extreme state of the system may be ascertained with higher accuracy than others. For example, the soil moisture state of a catchment may be estimated from assimilating precipitation and streamflow data into hydrologic models [36,37,38,39,40,41]. If $\alpha$ is prescribed adaptively based on the best available estimate of the state of the catchment, one may expect improved performance in hydrologic forecasting. In this section, we apply adaptive CBPKF in the synthetic experiment and assess its performance. An obvious strategy for adaptively filtering is to parameterize $\alpha$ in terms of the KF estimate (i.e., the CBPKF estimate with $\alpha =0$) as the best guess for the true state. The premise of this strategy is that, though it may be conditionally biased, the KF estimate fuses the information available from both the observations and the dynamical model, and hence best captures the relationship between $\alpha$ and the departure of the state of the system from median. A similar approach has been used in fusing radar rainfall data and rain gauge observations for multisensor precipitation estimation in which an ordinary cokriging estimate was used to prescribe $\alpha$ in CB-penalized cokriging [30].

Necessarily, the effectiveness of the above strategy depends on the skill of the KF estimate; if the skill is very low, one may not expect significant improvement. Figure 2 suggests that, qualitatively, α should increase as the state becomes more extreme. To that end, we employed the following model for time-varying $\alpha$:

$${\alpha }_k=\gamma \Vert {\widehat{X}}_{k|k}^{KF}\Vert$$

(51)

where ${\alpha }_k$ denotes the multiplicative CB penalty factor for CBPKF at time step k, $\Vert {\widehat{X}}_{k|k}^{KF}\Vert$ denotes some norm of the KF estimate at time step k, and $\gamma$ denotes the proportionality constant. Figure 4(left) shows the RMSE reduction by adaptive CBPKF over KF with ${\alpha }_k=\gamma \left|{\widehat{X}}_{k|k}^{KF}\right|$ for the 12 cases in Table 2 m = 1 and n = 10. The $\gamma$ values used were 3.0, 1.0, and 0.5 for Groups 1, 2, and 3 in Table 2, respectively. The figure shows that adaptive CBPKF performs comparably to KF in the unconditional sense while substantially improving performance in the tails. The rate of reduction in RMSE with respect to the increasing conditioning truth, however, is now slower than that seen in Figure 2 due to the occurrences of incorrectly specified α. To assess the uppermost bound of the feasible performance of adaptive CBPKF, we also specified $\alpha$ with perfect accuracy under Equation (51) via ${\alpha }_k=\gamma \left|X_k\right|$ where $X_k$ denotes the true state. The results are shown in Figure 4(right) for which the $\gamma$ values used were 3.0, 1.5, and 1.0 for Groups 1, 2, and 3 in Table 2, respectively. The figure indicates that adaptive CBPKF with perfectly prescribed $\alpha$ greatly improves performance, even outperforming KF in the unconditional sense. Figure 4 suggests that, if $\alpha$ can be prescribed more accurately with additional sources of information, the performance of adaptive CBPKF may be improved beyond the level seen in Figure 4(left). Finally, we show in Figure 5 the example scatter plots of the KF (black) and adaptive CBPKF (red) estimates vs. truth. They are for Cases 1 and 9 in Table 2 representing Groups 1 and 3, respectively. It is readily seen that the CBPKF significantly reduces CB in the tails while keeping its estimates close to the KF estimates in normal ranges.

Figure 4. Percent reduction in RMSE by adaptive CBPKF over KF in which $\alpha$ is prescribed using the KF estimate (left) and the truth (right).

Figure 5. Example scatter plots of KF (black) and adaptive CBPKF (red) estimates vs. truth for Cases 1 (left) and 9 (right) in Table 2.

7. Conclusions

Conditional bias-penalized Kalman filter (CBPKF) has recently been developed to improve the estimation and prediction of extremes. The original formulation, however, is computationally very expensive, and deteriorates performance in the normal ranges relative to KF. In this work, we present a computationally less expensive alternative based on the variance-inflated KF (VIKF) approximation, and improve unconditional performance by adaptively prescribing the weight for the CB penalty. For evaluation, we carried out synthetic experiments using linear systems with varying degrees of dynamical model uncertainty, observational uncertainty, and predictability. The results indicate that the VIKF-based approximation of CBPKF provides a computationally much less expensive alternative to the original formulation, and that adaptive CBPKF performs comparably to KF in the unconditional sense while improving the estimation of extremes by about 20 to 30% over KF. It is also shown that additional improvement may be possible by improving adaptive prescription of the weight to the CB penalty using additional sources of information. The findings indicate that adaptive CBPKF offers a significant addition to the dynamic filtering methods for general application in data assimilation and, in particular, when or where the estimation of extremes is of importance. The findings in this work are based on idealized synthetic experiments that satisfy linearity and normality. Additional research is needed to assess performance for non-normal problems and for nonlinear problems using the ensemble extension [31], and to prescribe the weight for the CB penalty more skillfully.