A Linear Bayesian Updating Model for Probabilistic Spatial Classification

Abstract

Categorical variables are common in spatial data analysis. Traditional analytical methods for deriving probabilities of class occurrence, such as kriging-family algorithms, have been hindered by the discrete characteristics of categorical fields. To solve the challenge, this study introduces the theoretical backgrounds of the linear Bayesian updating (LBU) model for spatial classification through an expert system. The main purpose of this paper is to present the solid theoretical foundations of the LBU approach. Since the LBU idea is originated from aggregating expert opinions and is not restricted to conditional independent assumption (CIA), it may prove to be reasonably adequate for analyzing complex geospatial data sets, such as remote sensing images or area-class maps.

1. Introduction

Categorical spatial data, such as lithofacies, land-use/land-cover classifications, and mineralization phases, are widely investigated geographical and geological information sources. They are typically represented by mutually exclusive and collectively exhaustive classes and visualized as area-class maps [1]. In the Geo-information context, Rao’s quadratic diversity was used in [2] to measure the scale-dependent landscape structure. In the geological counterpart, a spatial hidden Markov chain model was employed in [3] for estimation of petroleum reservoir categorical variables. As a geostatistical model, the Markov chain random field (MCRF) theory [4] and Markov chain sequential simulation (MCSS) algorithm [5] are common choices for the prediction of categorical spatial data. They have been widely used in spatial-related fields and gratifying results have been achieved. However, the MCRF approach is based on a conditional independent assumption (CIA), which may be inappropriate due to complex data interaction in a spatial context [6]. The Tau model [7] and Nu expression [8] introduce additional weights to relax the assumption of conditional independence. It is obvious that these power or multiplication relationships between multi-point pre-posterior probabilities and two-point conditional probabilities involve some subjective guesswork and may not be suitable in real-world spatial analysis. As for the generalized linear mixed model (GLMM) [6], where intermediate, latent, spatially correlated, normal variables are assumed for the observable non-normal responses to account for spatial dependence information. The random effects are always assumed to follow a normal distribution in the GLMM. Our concern here is whether the latent variables for different categories can be assumed to be independent of each other at the same location.

Generally speaking, the spatial classification problem can be regarded as combing two-point transition probabilities into a multi-point conditional probability. A formal introduction of most of the available approaches to aggregate probability distributions in geosciences can be found in [9]. Our task is to use the probability pooling method for spatial classification based on the pioneering work of [10,11]. We profit from the predecessor’s studies and interpret transition probabilities as expert opinions. The transition probabilities are obtained by the transiogram [12] spatial measure. The remainder of this paper is organized as follows. We begin by introducing the basic forms of the linear Bayesian updating (LBU) method in Section 2. We then make detailed proofs in Section 3 for some propositions introduced by [10,11] which have not yet been proven. A real-world case study is given in Section 4. Finally, conclusions and future challenges are discussed in Section 5.

2. Linear Bayesian Updating

Consider the spatial locations $x_0,x_1,\cdots ,x_n$ in the remote sensing images or area-class maps. We use $A$ and $D_1,\cdots ,D_n$ to represent the events in sample spaces of categorical random variable $C\left(x_0\right)$ and $C\left(x_1\right),\cdots ,C\left(x_n\right)$ respectively and $\overline{A}$ denotes the complementary event of $A$. In the case of categorical data, let $A$ be the finite set of events in the sample space $\mathrm{\Omega }$ such that the events $A_1,A_2\cdots ,A_K$ of $A$ are mutually exclusive and collectively exhaustive. Obviously,

$$A_j=\{\begin{array}{l}1\mathrm{if}C\left(x_0\right)=j,j=1,2,\cdots ,K\\ 0\mathrm{otherwise}.\end{array}$$

(1)

In the subsequent discussions, $A$ will be used as a general notation for $A_j$. We treat the $n$ neighboring events $D_1,D_2,\cdots ,D_n$ as experts, and consider the conditional probabilities $P\left(A|D_i\right)$ as expert opinions $Q_i$ for the occurrence of $A$, an event of interest. The experts’ opinions are regarded as random variables $Q_i$ whose values $q_i,1\le i\le n$, are to be revealed to the decision maker (DM). The posterior probability of $A$ given $Q=q$ is then $p^{\ast }\left(q\right)$.

The original LBU method was firstly proposed by [10] in statistical science in the form

$$p*\left(q\right)=p+{\displaystyle {\sum }_{i=1}^n{\lambda }_i\left(q_i-{\mu }_i\right)}$$

(2)

with possibly negative weights, ${\lambda }_i$, expressing the amounts of correlation between each $Q_i$ and $A$. ${\mu }_i$ denotes the mathematical expectation of $Q_i$. When ${\mu }_1=\cdots ={\mu }_n=p$ and ${\lambda }_i\ge 0$, Equation (2) yields to the linear opinion pool [13]

$$p*\left(q\right)={\lambda }_{0}p+{\displaystyle {\sum }_{i=1}^n{\lambda }_i{q}_i}\mathrm{subject}\mathrm{to}{\displaystyle {\sum }_{i=0}^n{\lambda }_i}=1$$

(3)

where the DM is considered as one of the experts. Our LBU method follows closely to that of [10], which is proved to be the only formula satisfying ${\displaystyle \int p^{\ast }\left(q\right)dF\left(q\right)=p}$ for all distribution $dF$ with mean vector $\mu$. The highlight of our LBU model lies in the fact that the random variable $Q_i$ has been replaced by transition probability, a measure for spatial continuity.

3. Theoretical Foundations of Linear Bayesian Updating

3.1. Parameter Ranges for Linear Bayesian Updating

Although the LBU model has been used in [10,11], it is our conviction that many theoretical challenges need to be solved to better develop this method for further use. A legitimate posterior probability can be obtained only when ${\lambda }_i$ obeys a number of inequalities [10]. Since $p^{\ast }\left(q\right)$ is a probability, it must satisfy $0\le p^{\ast }\left(q\right)\le 1$, that is to say,

$$\{\begin{array}{l}p+{\displaystyle {\sum }_{i=1}^n{\lambda }_i\left(q_i-{\mu }_i\right)}\ge 0\\ p+{\displaystyle {\sum }_{i=1}^n{\lambda }_i\left(q_i-{\mu }_i\right)}\le 1.\end{array}$$

(4)

Through algebra transformation, (4) can be simplified as

$$\{\begin{array}{l}1+\frac{{\displaystyle {\sum }_{i=1}^n{\lambda }_i{q}_i}}{p}\ge \frac{{\displaystyle {\sum }_{i=1}^n{\lambda }_i{\mu }_i}}{p}\\ \frac{{\displaystyle {\sum }_{i=1}^n{\lambda }_i{q}_i}-{\displaystyle {\sum }_{i=1}^n{\lambda }_i{\mu }_i}}{1-p}\le 1.\end{array}$$

(5)

Suppose that all ${\lambda }_i$ are positive, since $0\le q_i\le 1$, as long as

$$\{\begin{array}{l}\frac{{\displaystyle {\sum }_{i=1}^n{\lambda }_i{\mu }_i}}{p}\le 1\\ \frac{{\displaystyle {\sum }_{i=1}^n{\lambda }_i\left(1-{\mu }_i\right)}}{1-p}\le 1,\end{array}$$

(6)

(5) can be satisfied. Therefore, if the DM considers that all ${\lambda }_i$ are positive, the most common case, then they must be chosen so that

$$\max \left\{{\displaystyle {\sum }_{i=1}^n{\lambda }_i{\mu }_i/p},{\displaystyle {\sum }_{i=1}^n{\lambda }_i\left(1-{\mu }_i\right)/\left(1-p\right)}\right\}\le 1,$$

(7)

which can be regarded as a sufficient but not necessary condition of the LBU method. Only when (6), or equivalently (7), is satisfied, can Equation (2) be a valid probabilistic model.

3.2. Interpreting Parameters as Regression Coefficients for Linear Bayesian Updating

The LBU model given above has some parameters, which need to be learned or estimated by the DM. Let ${\mathrm{\Sigma }}_Q$ denote the covariance matrix of $Q$, and ${\sigma }_{p^{\ast }Q}$ be the vector of covariances between $p^{\ast }$ and $Q$. Let $t$ denote matrix transposition and

$$Q=\left(Q_1,Q_2,\cdots ,Q_n\right)\mu =\left({\mu }_1,{\mu }_2,\cdots ,{\mu }_n\right)\mathsf{\lambda }={\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t,$$

using the definition of expectation, it is

$$E\left(A|Q\right)=p^{*},E\left(E\left(A|Q\right)\right)=E\left(A\right)=E\left(p^{*}\right)=p.$$

In addition, Equation (2) can be given as

$$p^{\ast }=p+\left(Q-\mu \right)\cdot \lambda \text{ and }\mu =E\left(Q\right)$$

i.e.,

$$p^{\ast }=p+\left(Q_1-{\mu }_1,Q_2-{\mu }_2,\cdots ,Q_n-{\mu }_n\right)\cdot {\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t.$$

Taking the expectations on both sides of the equation after transformation yields

$$\begin{array}{ll}E\left[{\left(Q-\mu \right)}^t\cdot \left(p^{\ast }-p\right)\right]& =E\left[{\left(Q_1-{\mu }_1,Q_2-{\mu }_2,\cdots ,Q_n-{\mu }_n\right)}^t\cdot \left(Q_1-{\mu }_1,Q_2-{\mu }_2,\cdots ,Q_n-{\mu }_n\right)\right]\cdot {\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t\\ & =\mathrm{cov}\left(Q_1,Q_2,\cdots ,Q_n\right)\cdot {\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t\\ & ={\mathrm{\Sigma }}_Q\cdot \lambda \end{array}$$

we have

$$\mathsf{\lambda }={{\mathrm{\Sigma }}_Q}^{-1}{\sigma }_{p^{\ast }q}$$

provided that the covariance matrix ${\mathrm{\Sigma }}_Q$ is invertible.

Suppose we have $m$ samples in the training set, consider the regression model

$$Y=X\cdot B+\epsilon$$

where

$$Y=\left(\begin{array}{c}{p}_1^{\ast }-p\\ p_2^{\ast }-p\\ ⋮\\ p_m^{\ast }-p\end{array}\right)X=\left(\begin{array}{l}1q_{11}-{\mu }_1\cdots q_{1n}-{\mu }_n\\ 1q_{21}-{\mu }_1\cdots q_{2n}-{\mu }_n\\ ⋮\\ 1q_{m1}-{\mu }_1\cdots q_{mn}-{\mu }_n\end{array}\right)B=\left(\begin{array}{l}{b}_0\\ b_1\\ ⋮\\ b_n\end{array}\right)\epsilon =\left(\begin{array}{l}{\epsilon }_1\\ {\epsilon }_2\\ ⋮\\ {\epsilon }_m\end{array}\right),$$

$\epsilon ={\left({\epsilon }_1,{\epsilon }_1,\cdots ,{\epsilon }_m\right)}^t$ denotes the random errors. Provided that the experts do not have a linearly dependent relationship, the least squares estimation of the regression coefficients yields

$$\widehat{B}={\left(X^{t}X\right)}^{-1}\left(X^{t}Y\right),$$

which can be written in its matrix form

$$\begin{array}{cc}\widehat{B}& ={\left[\left(\begin{array}{cccc}1& 1& \cdots & 1\\ q_{11}-{\mu }_1& q_{21}-{\mu }_1& \cdots & q_{m1}-{\mu }_1\\ q_{12}-{\mu }_2& q_{22}-{\mu }_2& \cdots & q_{m2}-{\mu }_2\\ ⋮& ⋮& \cdots & ⋮\\ q_{1n}-{\mu }_n& q_{2n}-{\mu }_n& \cdots & q_{mn}-{\mu }_n\end{array}\right)\cdot \left(\begin{array}{llll}1& q_{11}-{\mu }_1& \cdots & q_{1n}-{\mu }_n\\ 1& q_{21}-{\mu }_1& \cdots & q_{2n}-{\mu }_n\\ ⋮& & & \\ 1& q_{m1}-{\mu }_1& \cdots & q_{mn}-{\mu }_n\end{array}\right)\right]}^{-1}\cdot \left(\begin{array}{ccc}1& \cdots & 1\\ q_{11}-{\mu }_1& \cdots & q_{m1}-{\mu }_1\\ q_{12}-{\mu }_2& \cdots & q_{m2}-{\mu }_2\\ ⋮& \cdots & ⋮\\ q_{1n}-{\mu }_n& \cdots & q_{mn}-{\mu }_n\end{array}\right)\cdot \left(\begin{array}{c}{p}_1^{\ast }-p\\ p_2^{\ast }-p\\ ⋮\\ p_m^{\ast }-p\end{array}\right)\\ & ={\left(\begin{array}{cccc}m& m\left(\overline{Q_1}-{\mu }_1\right)\hfill & \cdots & m\left(\overline{Q_n}-{\mu }_n\right)\hfill \\ m\left(\overline{Q_1}-{\mu }_1\right)\hfill & {\displaystyle \sum _{i=1}^m}{\left(q_{i1}-{\mu }_1\right)}^2\hfill & \cdots & {\displaystyle \sum _{i=1}^m\left(q_{i1}-{\mu }_1\right)\left(q_{in}-{\mu }_n\right)}\hfill \\ m\left(\overline{Q_2}-{\mu }_2\right)\hfill & {\displaystyle \sum _{i=1}^m}\left(q_{i2}-{\mu }_2\right)\left(q_{i1}-{\mu }_1\right)\hfill & \cdots & {\displaystyle \sum _{i=1}^m\left(q_{i2}-{\mu }_2\right)\left(q_{in}-{\mu }_n\right)}\hfill \\ ⋮& & & \\ m\left(\overline{Q_n}-{\mu }_n\right)\hfill & {\displaystyle \sum _{i=1}^m\left(q_{in}-{\mu }_n\right)\left(q_{i1}-{\mu }_1\right)}\hfill & \cdots & {\displaystyle \sum _{i=1}^m{\left(q_{in}-{\mu }_n\right)}^2}\hfill \end{array}\right)}^{-1}\cdot \left(\begin{array}{c}m\overline{p^{\ast }}-mp\\ {\displaystyle \sum _{i=1}^m\left(p_i^{*}-p\right)\left(q_{i1}-{\mu }_1\right)}\\ {\displaystyle \sum _{i=1}^m\left(p_i^{*}-p\right)\left(q_{i2}-{\mu }_2\right)}\\ ⋮\\ {\displaystyle \sum _{i=1}^m\left(p_i^{*}-p\right)\left(q_{in}-{\mu }_n\right)}\end{array}\right).\hfill \end{array}$$

When the sample size is large enough, the sample mean is approximately equal to the total expectation, we have

$$\begin{array}{ll}\widehat{B}& =\frac{1}{m}\cdot {\left(\begin{array}{llll}1& E\left(Q_1\right)-{\mu }_{1}E\left(Q_2\right)-{\mu }_2& \cdots & E\left(Q_n\right)-{\mu }_n\\ E\left(Q_1\right)-{\mu }_1& E\left[{\left(Q_1-{\mu }_1\right)}^2\right]& \cdots & E\left[\left(Q_1-{\mu }_1\right)\left(Q_n-{\mu }_n\right)\right]\\ E\left(Q_2\right)-{\mu }_2& E\left[\left(Q_2-{\mu }_2\right)\left(Q_1-{\mu }_1\right)\right]& \cdots & E\left[\left(Q_2-{\mu }_2\right)\left(Q_n-{\mu }_n\right)\right]\\ ⋮& & & \\ E\left(Q_n\right)-{\mu }_n& E\left[\left(Q_n-{\mu }_n\right)\left(Q_1-{\mu }_1\right)\right]& \cdots & E\left[{\left(Q_n-{\mu }_n\right)}^2\right]\end{array}\right)}^{-1}\cdot m\cdot \left(\begin{array}{c}E\left(p^{\ast }\right)-p\\ E\left[\left(p^{\ast }-p\right)\left(Q_1-{\mu }_1\right)\right]\\ E\left[\left(p^{\ast }-p\right)\left(Q_2-{\mu }_2\right)\right]\\ ⋮\\ E\left[\left(p^{\ast }-p\right)\left(Q_n-{\mu }_n\right)\right]\end{array}\right)\\ & =\frac{1}{m}\cdot {\left(\begin{array}{cc}1\hfill & E\left(Q_1\right)-{\mu }_{1}E\left(Q_2\right)-{\mu }_2\cdots E\left(Q_n\right)-{\mu }_n\\ E\left(Q_1\right)-{\mu }_1\hfill & \\ E\left(Q_2\right)-{\mu }_2\hfill & \mathrm{cov}\left(Q_1,Q_2,\cdots Q_n\right)\\ ⋮& \\ E\left(Q_n\right)-{\mu }_n\hfill & \end{array}\right)}^{-1}\cdot m\cdot \left(\begin{array}{c}E\left(p^{\ast }\right)-p\\ {\sigma }_{p^{\ast }Q}\end{array}\right)\\ & ={\left(\begin{array}{c}0\\ {{\mathrm{\Sigma }}_Q}^{-1}\cdot {\sigma }_{p^{\ast }Q}\end{array}\right)}_{\left(n+1\right)\times 1}.\end{array}$$

Therefore, our derivation gives an explanation of the parameters ${\lambda }_i$ in the LBU model as the linear regression coefficients of $p^{\ast }-p$ with respect to $Q-\mu$ when the neighboring events are not linearly dependent. As in multiple regression, each ${\lambda }_i$ can thus be thought of as a measure of the additional information that the $i$th expert provides over and above the other experts and what the DM already knows.

3.3. Invertible Conditions of Linear Bayesian Updating

We now discuss what happened when the linear systems of Equation (2) become invertible. Equation (2) can be rewritten as

$$p*\left(q\right)=p-{\displaystyle {\sum }_{i=1}^n{\lambda }_i{\mu }_i+\left(q_1,q_2,\cdots ,q_n\right)}{\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t.$$

Therefore,

$$\left(q_1,q_2,\cdots ,q_n\right){\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t=p^{\ast }\left(q\right)-p+\left({\mu }_1,{\mu }_2\cdots ,{\mu }_n\right){\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t,$$

we have

$$\left(q_1,q_2,\cdots ,q_n\right)\mathrm{\Lambda }=\left[p^{\ast }\left(q\right)-p\right]\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)+\left({\mu }_1,{\mu }_2\cdots ,{\mu }_n\right)\mathrm{\Lambda },$$

where

$$\begin{array}{cc}\mathrm{\Lambda }& ={\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)}^t\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)\\ & =\left(\begin{array}{cccc}{{\lambda }_1}^2& {\lambda }_1{\lambda }_2& \cdots & {\lambda }_1{\lambda }_n\\ {\lambda }_2{\lambda }_1& {{\lambda }_2}^2& \cdots & {\lambda }_2{\lambda }_n\\ ⋮& ⋮& \ddots & ⋮\\ {\lambda }_n{\lambda }_1& {\lambda }_n{\lambda }_2& \cdots & {{\lambda }_n}^2\end{array}\right).\end{array}$$

Since the determinant $\left|\mathrm{\Lambda }\right|=0$, $\mathrm{\Lambda }$ is irreversible. Therefore, $q_1,q_2,\cdots ,q_n$ cannot be uniquely determined. Thus, the linear systems of Equation (2) are not invertible under these circumstances.

The case where the linear system of Equation (2) is invertible happens only when there is one expert to be consulted, i.e.,

$$p^{\ast }\left(q\right)=p+\lambda \left(q-\mu \right).$$

In this case, the left inverse of the system is

$$q=\frac{p^{\ast }-p}{\lambda }+\mu ,$$

where $p^{\ast }$ is the input. The right inverse can be given as

$$q_i=\frac{p^{\ast }\left(q_i\right)-p}{{\lambda }_i}+{\mu }_i,$$

where $p^{\ast }\left(q_i\right)$ is the desired output after consultation. The necessary $q_i$ can be compared with the corresponding transition probability $P\left(A|D_i\right)$. If large deviation emerges, the expert $Q_i$ may seem not to be convincing.

4. Case Study

We now present a case study to demonstrate the use of the method. The Swiss Jura data set [14] is used, where four lithology types are sampled in a 14.5 km² region. These rock types are Argovian, Kimmeridgian, Sequanian and Quaternary; corresponding class proportions of these four categories are 20.46%, 32.82%, 24.32%, 22.39% respectively. We have 259 samples in total for prediction (Figure 1).

The first task we need to do is to obtain the expert opinions (i.e., transition probabilities) in spatial scenarios. We use the 10 nearest samples for prediction, thus we always get 10 experts for consultation. The detailed procedures for estimating the transition probability are beyond the scope of this work. One can find the discussions with respect to transiogram fitting in [1,11]. We only show the descriptive statistics of transition probabilities in

Table 1. Descriptive statistics of transition probabilities (expert opinions).

Transition Probability	Mean	Median	Maximum	Minimum	Standard Deviation
$P_{11}$	0.270	0.201	1.000	0.000	0.250
$P_{12}$	0.290	0.292	0.632	0.000	0.173
$P_{13}$	0.254	0.239	0.769	0.000	0.140
$P_{14}$	0.186	0.193	0.436	0.000	0.123
$P_{21}$	0.227	0.239	0.563	0.000	0.134
$P_{22}$	0.332	0.282	1.000	0.091	0.198
$P_{23}$	0.254	0.256	0.436	0.000	0.090
$P_{24}$	0.187	0.192	0.387	0.000	0.088
$P_{31}$	0.220	0.200	0.636	0.000	0.141
$P_{32}$	0.313	0.309	1.000	0.000	0.155
$P_{33}$	0.294	0.262	1.000	0.000	0.203
$P_{34}$	0.173	0.185	0.469	0.000	0.119
$P_{41}$	0.219	0.182	0.720	0.000	0.154
$P_{42}$	0.312	0.318	0.562	0.000	0.142
$P_{43}$	0.256	0.238	0.586	0.000	0.123
$P_{44}$	0.213	0.184	1.000	0.000	0.208

.

After obtaining the expert opinions, we can use the regression model represented by Equation (2) to estimate the linear weights in spatial classification. Given that multiple neighbors will be involved in spatial scenarios most of the time, the LBU should often be a multivariable linear regression model. With the estimated regression coefficients, we can use the maximum a posteriori (MAP) probability criterion for classification [11].

The final prediction results have been shown in Figure 2. We get an overall classification accuracy of 82.63% (214 out of259). To better reflect the prediction accuracy, the precision indicator for each lithoface has also been illustrated in Figure 3.

5. Conclusions

In this work, we consummate the theoretical foundations of the LBU model for the prediction of categorical spatial data. We have enriched our previous findings [11] by adding some rigorous theoretical proofs of the LBU method. To show how the LBU model can work in spatial settings, a real-world case study has also been carried out. As pointed out by [11], our method can also be generalized to nonlinear systems, where more confident probability forecasting results can be obtained.

In the proposed model, the choice of the size of a neighborhood can be regarded as a variable selection problem. The involvement of more neighboring samples is likely to boost the prediction accuracy for the training set, while it may be computation-intensive and accompanied by a higher generalization error, the so-called overfitting. Challenges to determine the optimal number of neighbors will be the focus of our future works and may be addressed in our upcoming papers.