Optimal Nonlinear Estimation in Statistical Manifolds with Application to Sensor Network Localization

Abstract

Information geometry enables a deeper understanding of the methods of statistical inference. In this paper, the problem of nonlinear parameter estimation is considered from a geometric viewpoint using a natural gradient descent on statistical manifolds. It is demonstrated that the nonlinear estimation for curved exponential families can be simply viewed as a deterministic optimization problem with respect to the structure of a statistical manifold. In this way, information geometry offers an elegant geometric interpretation for the solution to the estimator, as well as the convergence of the gradient-based methods. The theory is illustrated via the analysis of a distributed mote network localization problem where the Radio Interferometric Positioning System (RIPS) measurements are used for free mote location estimation. The analysis results demonstrate the advanced computational philosophy of the presented methodology.

1. Introduction

Information geometry, pioneered by Rao in the 1940s [1] and further developed by Chentsov [2], Efron [3,4] and Amari [5,6], considers the statistical relationships between families of probability densities in terms of the geometric properties of Riemann manifolds. It is the study of intrinsic properties of manifolds of probability distributions [7], where the ability of the data to discriminate those distributions is translated into a Riemannian metric. Specifically, the Fisher information gives a local measure of discrimination of the distributions which immediately provides a Riemannian metric on the parameter manifold of the distributions [1]. In particular, the collection of probability density functions called curved exponential families, which encapsulate important distributions in many real world problems, have been treated using this framework [5].

The main tenet of information geometry is that many important notions in probability theory, information theory and statistics can be treated as structures in differential geometry by regarding a space of probabilities as a differentiable manifold endowed with a Riemannian metric and a family of affine connections, including but not exclusively, the canonical Levi-Civita affine connection [6]. By providing a means to analyse the Riemannian geometric properties of various families of probability density functions, information geometry offers comprehensive results about statistical problems simply by considering them as geometrical objects. Information geometry opens new prospect to study the intrinsic geometrical nature of information theory and provides a new way to deal with statistical problems on manifolds. For example, Smith [8] studied the intrinsic Cramér-Rao bounds on estimation accuracy for the estimation problems on arbitrary manifolds where the set of intrinsic coordinates is not apparent, and derived the intrinsic bounds in the examples of covariance matrix and subspace estimation. Srivastava et al. [9,10] addressed the geometric subspace estimation and target tracking problems under a Bayesian framework. Bhattacharya and Patrangenaru [11] treated the general problem of estimation on Riemannian manifolds.

As this new general theory reveals the capability of defining a new perspective on existing questions, many researchers are extending their work on this geometric theory of information to new areas of application and interpretation. For example, a most important milestone in the area of signal processing is the work of Richardson where the geometric perspective clearly indicates the relationship between turbo-decoding and maximum-likelihood decoding [12]. The results of Amari et al. on the information geometry of turbo and low-density parity-check codes extend the geometrical framework initiated by Richardson to the information geometrical framework of dual affine connections [13]. Other investigations include the geometrical interpretation of fading in wireless networks [14]; the geometrical interpretation of the solution to the multiple hypothesis testing problem in the asymptotic limit developed by Westover [15]; and a geometric characterization of multiuser detection for synchronous DS/CDMA channels [16]. Recently, the framework of information geometry has been applied to address issues in the application of sensor networks such as target resolvability [17], radar information resolution [18] and passive sensor scheduling [19,20].

In this paper, we mainly focus on the nonlinear estimation problem and illustrate how it can benefit from the powerful methodologies of information geometry. The geometric interpretation for the solution to the maximum likelihood estimation for curved exponential families and the convergence of the gradient-based methods (such as Newton’s method and the Fisher scoring algorithm) are demonstrated via the framework developed by Efron and Amari et al. Our essential motivation of this work is to provide some insights into the nonlinear parameter estimation problems in signal processing using the theory of information geometry. By gaining a better understanding of the existing algorithms through the use of information geometric method, we are, hopefully, enabled to derive better algorithms for solving non-linear problems.

The work described in this paper consists of the following aspects. Firstly, an iterative maximum likelihood estimator (MLE) for estimating non-random parameters with measurement of the curved exponential distributions is presented. The estimator belongs to the gradient-based methods that operate on statistical manifolds and can be seen as a generalization of Newton’s method to families of probability density functions and their relevant statistics. Its interpretation in terms of differential and information geometry provides insight into its optimality and convergence. Then, by utilizing the properties of exponential families and thus identifying the parameters on statistical manifolds, the implementation of the presented MLE algorithm is simplified via reparametrization. Furthermore, it is shown that the associated stochastic estimation problem reduces to a deterministic optimization problem with respect to the measures (statistics) defined over the distributions. Finally, an example of a one-dimensional curved Gaussian is presented to demonstrate the method in the manifold. A practical example of distributed mote network localization using the Radio Interferometric Positioning System (RIPS) measurements is given to demonstrate the issues addressed in this paper. The performance of the estimator is discussed.

In the next section, classical nonlinear estimation via natural gradient MLE for curved exponential families is derived. The reparametrization from local parameters to natural parameters and expectation parameters is highlighted. In Section 3, the principles of information geometry are introduced. Further, the geometric interpretation for the presented iterative maximum likelihood estimator and the convergence of the developed algorithm is demonstrated via the properties of statistical manifolds. In Section 4, a one parameter estimation example is presented to illustrate the geometric operation of the algorithm. The performance and efficiency of the algorithm are further demonstrated via a mote localization example using RIPS measurements. Finally, conclusions are made in Section 5.

2. Nonlinear Estimation via Natural Gradient MLE

In probability and statistics, exponential families (including the normal, exponential, Gamma, Chi-squared, Beta, Poisson, Bernoulli, multinomial and many others) are an important class of distributions naturally considered. There is a framework for selecting a possible alternative parameterization of these distributions, in terms of the natural parameters, and for defining useful statistics, called the natural statistics of the families. When the natural parameters in exponential families are nonlinear functions of some “local” parameters, the distributions involved are in the curved exponential families. While curved exponential families which encapsulate important distributions are more suitable to describe many real world problems, the estimation of local parameters is often non-trivial because of the nonlinearity in parameters.

In this section, a natural gradient based maximum likelihood estimator is derived to address a nonlinear estimation problem for curved exponential families. Although the estimator has been well-known as the Fisher scoring method in the literature, the interpretation of its iterative operations via the theory of information geometry is interesting and will be presented in the following section.

The general form of a curved exponential family [5] can be expressed as

$$p\left(\mathit{x}\right|\mathit{u})=exp\left\{C\left(\mathit{x}\right)+{\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{F}\left(\mathit{x}\right)-\phi \left(\mathit{\theta }\left(\mathit{u}\right)\right)\right\}=p\left(\mathit{x}\left|\mathit{\theta }\right(\mathit{u})\right)$$

(1)

where $\mathit{x}\in \Omega$ is a vector valued measurement, $\mathit{\theta }=\{{\theta }_1,\cdots ,{\theta }_n\}$ are the natural coordinates or canonical parameters, $\mathit{u}$ denote local parameters and $\mathit{F}\left(\mathit{x}\right)=\{F_1\left(\mathit{x}\right),\cdots ,F_n\left(\mathit{x}\right)\}$ are sufficient statistics for $\mathit{\theta }=\{{\theta }_1,\cdots ,{\theta }_n\}$, and functions on the measurement space with elements denoted by $\mathit{x}$. The function $\phi$ is called the potential function of the exponential family and it is found from the normalisation condition ${\int }_{\Omega }{p}_{\mathit{\theta }}\left(\mathit{x}\right)d\mathit{x}=1$, i.e.,

$$\phi \left(\mathit{\theta }\right)=log{\int }_{\Omega }exp\left\{C\left(\mathit{x}\right)+\sum _i^n\left({\theta }_i{F}_i\left(\mathit{x}\right)\right)\right\}d\mathit{x}$$

(2)

The term “curved” comes from the fact that the distribution in Equation (1) is defined by a smooth embedding $\mathit{u}⟶\mathit{\theta }\left(\mathit{u}\right)$ from the manifold parameterized by $\mathit{u}$ into the canonical exponential family $p\left(\mathit{x}\right|\mathit{\theta })$.

As an example, a curved Gaussian distribution with local parameter $\mathit{u}\in {\mathbb{R}}^m$, mean $\mathit{\mu }\left(\mathit{u}\right)\in {\mathbb{R}}^n$ and covariance $\mathbf{\Sigma }\left(\mathit{u}\right)\in {\mathbb{R}}^{n\times n}$ is expressed as

$$\mathit{x}\sim \mathcal{N}\left(\mathit{\mu }\left(\mathit{u}\right),\mathbf{\Sigma }\left(\mathit{u}\right)\right)$$

(3)

By reparameterization, the standardized natural parameters $(\mathit{\theta },\mathsf{\Theta })$ of a curved Gaussian distribution are found to be

$$\begin{array}{cc}\hfill \mathit{\theta }& ={\mathbf{\Sigma }}^{-1}\left(\mathit{u}\right)\mathit{\mu }\left(\mathit{u}\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill \mathsf{\Theta }& =-\frac{1}{2}{\mathbf{\Sigma }}^{-1}\left(\mathit{u}\right)\hfill \end{array}$$

(5)

The sufficient statistics of the Gaussian distribution in Equation (3) is

$$\mathit{F}\left(\mathit{x}\right)=\{\mathit{x},\mathit{x}{\mathit{x}}^T\}$$

(6)

and the potential function expressed in terms of local parameters $(\mathit{\mu },\mathbf{\Sigma })$ is given by

$$\begin{array}{cc}\hfill \phi (\mathit{\mu },\mathbf{\Sigma })& =\frac{1}{2}{\mathit{\mu }}^T{\mathbf{\Sigma }}^{-1}\mathit{\mu }+\frac{1}{2}log|\mathbf{\Sigma }|+\frac{n}{2}log2\pi .\hfill \end{array}$$

(7)

where the superscript T signifies the transpose operation and n is the cardinality of $\mathit{\mu }\in {\mathbb{R}}^n$.

Let $l(\mathit{\theta },\mathit{x})=logp\left(\mathit{x}\right|\mathit{\theta })$ be the log likelihood of a general family of distributions $p\left(\mathit{x}\right|\mathit{\theta })$ as in Equation (1), and $\nabla \mathit{\theta }$ the Jacobian matrix of the natural parameter $\mathit{\theta }$ as a function of the local parameters $\mathit{u}$. We may write the following equations by using Equation (1)

$$\begin{array}{cc}\nabla l(\mathit{\theta },\mathit{x})& =\nabla \left\{{\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{F}\left(\mathit{x}\right)-\phi \left(\mathit{\theta }\left(\mathit{u}\right)\right)\right\}\hfill \\ & =\nabla {\mathit{\theta }}^T\left(\mathit{u}\right)\left[\mathit{F}\left(\mathit{x}\right)-\mathit{\eta }\left(\mathit{u}\right)\right]\hfill \end{array}$$

(8)

where

$$\mathit{\eta }\left(\mathit{u}\right)\stackrel{\Delta }{=}E\left\{\mathit{F}\left(\mathit{x}\right)\right\}$$

(9)

is called the expectation parameter which connects to $\mathit{\theta }\left(\mathit{u}\right)$ by the well known Legendre transformation [5], and

$$\nabla \phi \left(\mathit{u}\right)=\nabla {\mathit{\theta }}^T\left(\mathit{u}\right){\nabla }_{\mathit{\theta }}\phi \left(\mathit{\theta }\right)=\nabla {\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{\eta }\left(\mathit{u}\right)$$

(10)

Both the expectation parameter $\mathit{\eta }$ and Fisher information matrix $\mathit{G}\left(\mathit{\theta }\right)$ can be obtained by differentiating the potential function $\phi \left(\mathit{\theta }\right)$ with respect to natural parameters [21]

$$\begin{array}{cc}\hfill {\eta }_i=\frac{\partial }{\partial {\theta }_i}\phi \left(\mathit{\theta }\right)& \stackrel{\Delta }{=}E\left\{F_i\left(\mathit{x}\right)\right\}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill g_{ij}\left(\mathit{\theta }\right)=\frac{{\partial }^2}{\partial {\theta }_i\partial {\theta }_j}\phi \left(\mathit{\theta }\right)& \stackrel{\Delta }{=}E\left\{\left[F_i\left(\mathit{x}\right)-E\left(F_i\left(\mathit{x}\right)\right)\right]{\left[F_j\left(\mathit{x}\right)-E\left(F_j\left(\mathit{x}\right)\right)\right]}^T\right\}\hfill \end{array}$$

(12)

The maximum likelihood estimator $\widehat{\mathit{u}}$ of the curved exponential family satisfies the following likelihood equation

$$\nabla l\left(\widehat{\mathit{u}}\right)=\nabla logp\left(\mathit{x}\right|\mathit{u})=\nabla {\mathit{\theta }}^T\left(\widehat{\mathit{u}}\right)\left[\mathit{F}\left(\mathit{x}\right)-\mathit{\eta }\left(\widehat{\mathit{u}}\right)\right]=0$$

(13)

Here $l\left(\widehat{\mathit{u}}\right)$ is an objective function to be maximized with parameter $\mathit{u}$. It was pointed out by Amari in [22] that the geometry of the Riemannian manifold must be taken into account when calculating the steepest learning directions on a manifold. He suggested the use of natural gradient (NAT) updates for the optimization on a Riemannian manifold, i.e.,

$${\mathit{u}}^{new}=\mathit{u}+\lambda {\mathit{G}}^{-1}\left(\mathit{u}\right)\nabla l\left(\mathit{u}\right)$$

(14)

where $\lambda$ is a positive learning rate that determines the step size and $\mathit{G}\left(\mathit{u}\right)$ denotes the Riemannian metric matrix of the manifold.

For a parameterized family of probability distributions on a statistical manifold, the Riemannian metric is defined as the Fisher information matrix (FIM) [1]. For the curved exponential family in Equation (1), the FIM with respect to the local parameter $\mathit{u}$ is

$$\mathit{G}\left(\mathit{u}\right)=\nabla {\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{G}\left(\mathit{\theta }\right)\nabla \mathit{\theta }\left(\mathit{u}\right)$$

(15)

where $\mathit{G}\left(\mathit{\theta }\right)$ is the FIM with respect to the natural parameter $\mathit{\theta }$. A recursive MLE of curved exponential families can then be implemented as follows

$$\begin{array}{ll}{\mathit{u}}^{(k+1)}& ={\mathit{u}}^{\left(k\right)}+\lambda {\mathit{G}}^{-1}\left({\mathit{u}}^{\left(k\right)}\right)\nabla l\left({\mathit{u}}^{\left(k\right)}\right)\\ & ={\mathit{u}}^{\left(k\right)}+\lambda {\mathit{G}}^{-1}\left({\mathit{u}}^{\left(k\right)}\right)\nabla {\mathit{\theta }}^T\left({\mathit{u}}^{\left(k\right)}\right)\left[\mathit{F}\left(\mathit{x}\right)-\mathit{\eta }\left({\mathit{u}}^{\left(k\right)}\right)\right]\end{array}$$

(16)

where $\mathit{\theta }\left(\mathit{u}\right)$ and $\mathit{\eta }\left(\mathit{u}\right)$ denote the natural parameter and expectation parameter of the distribution, respectively. $\mathit{F}\left(\mathit{x}\right)$ is the sufficient statistics for the measurement $\mathit{x}$.

The covariance (CRLB) of the recursive MLE estimator ${\mathit{u}}^{(k+1)}$ is the inverse of Fisher information matrix ${\mathit{G}}^{-1}\left({\mathit{u}}^{(k+1)}\right)$, where

$$\mathit{G}({\mathit{u}}^{(k+1)})=\nabla {\mathit{\theta }}^T({\mathit{u}}^{(k+1)})\mathit{G}\left(\mathit{\theta }({\mathit{u}}^{(k+1)})\right)\nabla \mathit{\theta }({\mathit{u}}^{(k+1)})$$

(17)

The proposed algorithm is summarized in Algorithm 1.

Algorithm 1: The natural gradient based iterative MLE algorithm.

Distribution reparameterization $$p\left(\mathit{x}\right|\mathit{u})=exp\left\{C\left(\mathit{x}\right)+{\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{F}\left(\mathit{x}\right)-\phi \left(\mathit{\theta }\left(\mathit{u}\right)\right)\right\}=p\left(\mathit{x}\left|\mathit{\theta }\right(\mathit{u})\right)$$ Identify the natural parameter $\mathit{\theta }\left(\mathit{u}\right)$, sufficient statistics $\mathit{F}\left(\mathit{x}\right)$ and potential function $\phi \left(\mathit{\theta }\left(\mathit{u}\right)\right)$. Find expectation parameter $\mathit{\eta }$ and Fisher information metric $\mathit{G}$ to construct manifold of the curved exponential family $p\left(\mathit{x}\left|\mathit{\theta }\right(\mathit{u})\right)$, $$\begin{array}{cc}\hfill \mathit{\eta }\left(\mathit{u}\right)& =E_{\mathit{\theta }}\left(\mathit{F}\left(\mathit{x}\right)\right)={\nabla }_{\mathit{\theta }}\phi \left(\mathit{\theta }\right)\hfill \\ \hfill \mathit{G}\left(\mathit{\theta }\right)& ={\mathrm{Cov}}_{\mathit{\theta }}\left(\mathit{F}\left(\mathit{x}\right)\right)={\nabla }_{\mathit{\theta }}{\nabla }_{\mathit{\theta }}^T\phi \left(\mathit{\theta }\right),\mathit{G}\left(\mathit{u}\right)=\nabla {\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{G}\left(\mathit{\theta }\right)\nabla \mathit{\theta }\left(\mathit{u}\right)\hfill \end{array}$$ Input initial conditions $${\mathit{u}}^{\left(0\right)},\mathit{G}\left({\mathit{u}}^{\left(0\right)}\right)=\nabla {\mathit{\theta }}^T\left({\mathit{u}}^{\left(0\right)}\right)\mathit{G}\left[\mathit{\theta }\left({\mathit{u}}^{\left(0\right)}\right)\right]\nabla \mathit{\theta }\left({\mathit{u}}^{\left(0\right)}\right)$$ Set step size $e^{\left(k\right)}>\epsilon >0,k=0$, while $e^{\left(k\right)}>\epsilon$ Loop for the $(k+1)$th iteration $$\begin{array}{cc}\hfill {\mathit{u}}^{(k+1)}& ={\mathit{u}}^{\left(k\right)}+\lambda {\mathit{G}}^{-1}\left({\mathit{u}}^{\left(k\right)}\right)\nabla {\mathit{\theta }}^T\left({\mathit{u}}^{\left(k\right)}\right)\left[\mathit{F}\left(\mathit{x}\right)-\mathit{\eta }\left({\mathit{u}}^{\left(k\right)}\right)\right]\hfill \\ \hfill \mathit{G}\left({\mathit{u}}^{(k+1)}\right)& =\nabla {\mathit{\theta }}^T\left({\mathit{u}}^{(k+1)}\right)\mathit{G}\left(\mathit{\theta }\left({\mathit{u}}^{(k+1)}\right)\right)\nabla \mathit{\theta }\left({\mathit{u}}^{(k+1)}\right)\hfill \end{array}$$ Update step size $$e^{(k+1)}=\left|\right|({\mathit{u}}^{\left(k\right)}-{\mathit{u}}^{(k+1)}\left|\right|$$ $$k+1\to k$$ end

The above natural gradient approach has a similar structure as the common gradient-based algorithms, such as the well-known steepest descent method, Newton’s method and Fisher scoring algorithm. However, it does distinguish itself from the others in the following points:

• The natural gradient estimator updates the underlying manifold metric $\mathit{G}$ (i.e., FIM) at each iteration as well, which evaluates the estimate accuracy.
• Updates in the classical steepest descent types are performed via the standard gradient $\nabla l\left(\mathit{u}\right)$ and are well-matched to the Euclidean distance measure as well as the gradient adaptation. For the cases where the underlying parameter spaces are not Euclidean but are curved, i.e., Riemannian, $-\nabla l\left(\mathit{u}\right)$ does not represent the steepest descent direction in the parameter space, and thus the standard gradient adaptation is no longer appropriate. The natural gradient updates in Equation (14) improve the steepest descent update rule by taking the geometry of the Riemannian manifold into account to calculate the learning directions. In other words, it modifies the standard gradient direction according to the local curvature of the parameter space in terms of the Riemannian metric tensor $\mathit{G}\left(\mathit{u}\right)$, thus offers faster convergence than the steepest descent method.
• The Newton method

  $${\mathit{u}}^{new}=\mathit{u}-\lambda {\left[\frac{{\partial }^{2}l\left(\mathit{u}\right)}{\partial \mathit{u}\partial {\mathit{u}}^T}\right]}^{-1}\nabla l\left(\mathit{u}\right).$$

  (18)

  improves the steepest descent method by using the second-order derivatives of the cost function, i.e., the inverse of the Hessian of $l\left(\mathit{u}\right)$ to adjust the gradient search direction. When $l\left(\mathit{u}\right)$ is a quadratic function of $\mathit{u}$, the inverse of the Hessian is equal to $\mathit{G}\left(\mathit{u}\right)$, and thus Newton’s method and the natural gradient approach are identical [22]. However, in more general contexts, the two techniques are different. Generally, the natural gradient approach increases the stability of the iteration with respect to Newton’s method through replacing the Hessian by its expected value, i.e., the Riemannian metric tensor $\mathit{G}\left(\mathit{u}\right)$.
• The natural gradient approach is identical to the Fisher scoring method in cases where the Fisher information matrix coincides with the Riemannian metric tensor of the underlying parameter space. In such cases, the natural gradient approach is a Riemannian-based version of the Fisher scoring method performed on manifolds, and it is very appropriate when the cost function is related to the Riemannian geometry of the underlying parameter space [23]. Once these methods are entered into the manifold, additional insights into their geometric meaning may be deduced in the framework of differential and information geometry.

It is worth mentioning that a strategic choice of parameterizations of the cost function may result in a faster convergence or a more meaningful implementation of an optimization algorithm, though it is quite-often non-trivial. In the proposed iterative MLE algorithm in Equation (16), an alternative parameterization of the curved exponential family, in terms of the natural and expectation parameters, are employed. Through such a reparameterization, the implementation of the natural gradient updates is facilitated by the relevant statistics of a curved exponential family.

3. Information Geometric Interpretation for Natural Gradient MLE

3.1. Principles of Information Geometry

(1) Definition of a statistical manifold: Information geometry originates from the study of manifolds of probability distributions. Consider the parameterized family of probability distributions $S=\left\{p\right(\mathit{x}\left|\mathit{\theta }\right)\}$, where $\mathit{x}$ is a random variable and $\mathit{\theta }=({\theta }_1,\cdots ,{\theta }_n)$ is a parameter vector specifying the distribution. The family S is regarded as a statistical manifold with $\mathit{\theta }$ as its (possibly local) coordinate system [24].

Figure 1 illustrates the definition of a statistical manifold. For a given state of interest $\mathit{\theta }$ in the parameter space $\mathsf{\Theta }\in {\mathbb{R}}^n$, the measurement $\mathit{x}$ in the sample space $\mathit{X}\in {\mathbb{R}}^m$ is an instantiation of a probability distribution $p\left(\mathit{x}\right|\mathit{\theta })$. Each probability distribution $p\left(\mathit{x}\right|\mathit{\theta })$ is labelled by a point $s\left(\mathit{\theta }\right)$ in the manifold S. The parameterized family of probability distributions $S=\left\{p\right(\mathit{x}\left|\mathit{\theta }\right)\}$ forms an n-dimensional statistical manifold where $\mathit{\theta }$ plays the role of a coordinate system of $\mathit{S}$.

(2) Fisher information metric: The metric is the object specifying the scalar product in a particular point on a manifold in differential geometry. It encodes how to measure distances, angles and area at a particular point on the manifold by specifying the scalar product between tangent vectors at that point [25]. For a parameterized family of probability distributions on a statistical manifold, the FIM plays the role of a Riemannian metric tensor [1]. Denoted by $\mathit{G}\left(\mathit{\theta }\right)=\left[g_{ij}\left(\mathit{\theta }\right)\right]$, where

$$g_{ij}=E\left\{\frac{\partial logp\left(\mathit{x}\right|\mathit{\theta })}{\partial {\theta }_i}·\frac{\partial logp\left(\mathit{x}\right|\mathit{\theta })}{\partial {\theta }_j}\right\},$$

(19)

the FIM measures the ability of the random variable $\mathit{x}$ to discriminate the values of the parameter ${\mathit{\theta }}^{\mathbf{\prime }}$ from $\mathit{\theta }$ for ${\mathit{\theta }}^{\mathbf{\prime }}$ close to $\mathit{\theta }$.

(3) Affine connection and Flatness: The affine connection ∇ (The notation ∇ is also used to denote the Jacobian in this paper. However, there should be no confusion from the context.) on a manifold S defines a linear one-to-one mapping between two neighboring tangent spaces of the manifold. When the connection coefficients of ∇ with respect to a coordinate system of S are all identically 0, then ∇ is said to be flat, or alternatively, S is flat with respect to ∇. The curvature of a flat manifold is zero everywhere. Correspondingly, flatness will result in considerable simplification of the geometry. Intuitively, a flat manifold is one that “locally looks like” a Euclidean space in terms of distances and angles. Consequently, many operations on the flat manifold such as projection and orthogonality become more closely analogous to the case of an Euclidean space.

It is worth mentioning that the flatness of a manifold is closely related to the definition of affine connections as well as the choice of coordinate systems of the manifold. In 1972, Chentsov [2] introduced a one-parameter family of affine connections called $\alpha$-connections which were later popularized by Amari [5]:

$${\stackrel{\alpha }{\mathbf{\Gamma }}}_{jim}\left(\mathit{\theta }\right)=E_{\mathit{\theta }}\left\{{\partial }_j{\partial }_{i}l(\mathit{x},\mathit{\theta }){\partial }_{m}l(\mathit{x},\mathit{\theta })\right\}+\frac{1-\alpha }{2}{E}_{\mathit{\theta }}\left\{{\partial }_{j}l(\mathit{x},\mathit{\theta }){\partial }_{i}l(\mathit{x},\mathit{\theta }){\partial }_{m}l(\mathit{x},\mathit{\theta })\right\}$$

(20)

where

$${\partial }_i=\frac{\partial }{\partial {\theta }_i}\mathrm{and}l(\mathit{\theta },\mathit{x})=logp\left(\mathit{x}\right|\mathit{\theta }).$$

(21)

In Equation (20), $\alpha$ = 0 corresponds to the Levi-Civita connection (information connection). The case $\alpha$ = −1 defines the e-connection (exponential connection) while $\alpha$ = −1 defines the m-connection (mixture connection). An exponential family with natural parameter $\mathit{\theta }$ as the coordinate system (parameterization) is a flat manifold under the e-connection while a mixture family with expectation parameter $\mathit{\eta }$ as the coordinate system is a flat manifold under the m-connection [24]. The e-connection and m-connection play an important role in statistical inference and geometrize the estimation on the flat manifolds.

3.2. Information Geometric Interpretation for Natural Gradient MLE

Based on the principles of information geometry introduced above, the algorithm described in Algorithm 1 can be explained via Figure 2, where the upper figure illustrates the estimation operation in Euclidean space while an alternative view of the estimation operation on statistical manifolds is analogously illustrated in the lower figure. In the upper figure, the local parameter $\mathit{u}$ is to be estimated from measurements (samples) $\mathit{x}$ via the likelihood function $p\left(\mathit{x}\right|\mathit{u})$. When the measurement model is nonlinear in the parameter, the underlying estimation of the local parameter is a nonlinear estimation or filtering problem. Usually, linear signal processing problems can be routinely solved systematically by the astute application of results from linear algebra. However, the nonlinear cases are not easy to solve. The methodology of differential and information geometry are more adaptable and capable of dealing with nonlinear problems.

Given that $\mathcal{M}=\left\{p\right(\mathit{x}\left|\mathit{\theta }\right)\}$ signifies a general set of conditional distributions, the natural parameter space $\left\{\mathit{\theta }\right\}\in {\mathcal{A}}^n$ contains the distributions of all exponential families; to be regarded as an enveloping space. Then the curved exponential family $p\left(\mathit{x}\right|\mathit{u})$ in the upper figure is smoothly embedded in the enveloping space ${\mathcal{A}}^n,m\le n$ by distribution reparameterization $\mathit{u}⟶\mathit{\theta }\left(\mathit{u}\right)$, i.e., the curved exponential family $p\left(\mathit{x}\right|\mathit{u})$ can be represented by a curve $\{\mathit{\theta }=\mathit{\theta }(\mathit{u}\left)\right\}$ embedded in ${\mathcal{A}}^n$. Consequently the nonlinearity in the underlying estimation problem is completely characterized by the red curve inside the natural parameter space.

The circle on the lower right side of Figure 2 is the expectation parameter space $\left\{\mathit{\eta }\right\}\in {\mathcal{B}}^n$ or sampling space, which is dual to the natural parameter space $\mathcal{A}$. The dots in the space $\mathcal{B}$ signify the “realizations” of the sufficient statistics $\mathit{F}\left(\mathit{x}\right)$ of the distribution $p\left(\mathit{x}\right|\mathit{\theta })$ and they are obtained from measurements (samples) $\mathit{x}$. Connected by the Legendre transformation the dual enveloping sub-manifolds $\mathcal{A}$ and $\mathcal{B}$ (i.e., natural and expectation parameter spaces) are in one-to-one correspondence [5]. By viewing the expression of the curved exponential families in Equation (1), we observe that the newly formulated likelihood $p\left(\mathit{F}\left(\mathit{x}\right)|\mathit{\theta }\right)$ is linear, which indicates the possibility for linearly estimating the natural parameter $\mathit{\theta }$ by sufficient statistics $\mathit{F}\left(\mathit{x}\right)$ firstly and then obtaining the estimation of local parameter $\mathit{u}$ by a deterministic mapping from $\mathit{\theta }$ to $\mathit{u}$.

The nonlinear filtering is performed in the expectation parameter space $\mathcal{B}$ by projecting the samples $\mathit{F}\left(\mathit{x}\right)$ on to the sub-manifold represented by the red curve which signifies the embedding of the curved exponential families in $\mathcal{B}$. The process is also called m-projection. As mentioned earlier, under both the e- and m-connections, the dual enveloping manifolds are flat. Therefore, the filtering (m-projection) is analogous to the projection in a deterministic Euclidean space. In consequence, the nonlinear estimation problem is finally realized as a deterministic optimization method.

The fundamental difference between the filter described here and existing nonlinear filters is that the filtering process presented is performed linearly in the dual spaces of natural parameters and expectation parameters under e- and m-connections, respectively. The filtering outcome is then mapped to local parameter space. The estimator is optimal in the MLE sense and thus has no information loss in the filtering process since it attains the CRLB [5].

The convergence of the nonlinear iterative estimator can be geometrically explained in information geometric terms diagrammatically via Figure 3. The curved exponential family in Equation (1) is represented by the curve ${\mathcal{F}}_A\equiv \{\mathit{\theta }\left(\mathit{u}\right):\mathit{u}\in {\mathbb{R}}^m\}$ in $\mathcal{A}$ and also by ${\mathcal{F}}_B\equiv \{\mathit{\eta }\left(\mathit{u}\right):\mathit{u}\in {\mathbb{R}}^m\}$ in the dual space $\mathcal{B}$. Starting from an initial parameter ${\mathit{u}}^{\left(k\right)}$, the algorithm constructs a vector ${\mathcal{L}}_{{\mathit{u}}^{\left(k\right)}}\equiv \{\tilde{\mathit{\eta }}\left({\mathit{u}}^{\left(k\right)}\right)=\mathit{F}\left(\mathit{x}\right)-\mathit{\eta }\left({\mathit{u}}^{k)}\right)\}$ from the current distribution represented by its expectation parameter $\mathit{\eta }\left({\mathit{u}}^{\left(k\right)}\right)$ to the measurement $\mathit{F}\left(\mathit{x}\right)$. The projection of $\tilde{\mathit{\eta }}\left({\mathit{u}}^{\left(k\right)}\right)$ to the tangent vector $\nabla \mathit{\theta }\left({\mathit{u}}^{\left(k\right)}\right)$ of the natural parameter $\mathit{\theta }\left({\mathit{u}}^{\left(k\right)}\right)$ with respect to the metric $\mathit{G}\left({\mathit{u}}^{\left(k\right)}\right)$ gives the steepest descent gradient (natural gradient) to update the current estimates (where $\nabla \mathit{\theta }\left({\mathit{u}}^{\left(k\right)}\right)$ is represented by the dashed arrow in both $\mathcal{A}$ and $\mathcal{B}$, while the natural gradient is represented by the solid arrow in $\mathcal{B}$).

The iterations continue according to Equation (16) until the two vectors $\tilde{\mathit{\eta }}\left({\mathit{u}}^{\left(k\right)}\right)$ and $\nabla \mathit{\theta }\left({\mathit{u}}^{\left(k\right)}\right)$ are (approximately) orthogonal to each other, i.e., $\tilde{\mathit{\eta }}\left({\widehat{\mathit{u}}}_{ML}\right)\perp \nabla \mathit{\theta }\left({\widehat{\mathit{u}}}_{ML}\right)$. The algorithm achieves convergence with the steepest descent gradient ${\mathit{G}}^{-1}\left({\mathit{u}}^{\left(k\right)}\right)\nabla {\mathit{\theta }}^T\left({\mathit{u}}^{\left(k\right)}\right)\tilde{\mathit{\eta }}\left({\mathit{u}}^{\left(k\right)}\right)$ vanishes and a solution to the MLE Equation (13) is obtained by projecting the data $\mathit{F}\left(\mathit{x}\right)$ onto ${\mathcal{F}}_B$ orthogonally to $\nabla \mathit{\theta }\left(\mathit{u}\right)$.

• Statistical problems can be described in manifolds in a number of ways. In the parameter estimation problems as we have discussed here the parameter belongs to a curved manifold, whereas the observations may lie on an enveloping manifold. The filtering process is thus implemented by means of projection in the manifolds.
• The iterative estimator is optimal in the MLE sense as the filtering itself involves no information loss. The stochastic filtering problem becomes an optimization problem defined over a statistical manifold.
• As seen from Algorithm 1, the algorithm implementation is relatively simple and straightforward by distribution reparameterization and operating in the dual flat manifolds. Though a Newton method-based MLE estimator can be derived directly via the likelihood. However, in most cases the operation is not trivial.
• The initial guess is important to facilitate convergence of the estimator to the true value. This can be varied and such initial value sampling may provide more certainty about reaching a global minimum. This has not been examined here.

In the next section, two examples are given to demonstrate the implementation of the developed estimator as well as its geometric interpretation.

4. Examples of Implementation of Natural Gradient MLE

4.1. An One Parameter Estimation Example of Curved Gaussian Distribution

Consider a curved Gaussian distribution

$$x\sim \mathcal{N}(u,u^2{a}^2)$$

(22)

where a is a constant and u is an unknown parameter to be estimated. The collection of distributions specified by the parameter u constitute a one-dimensional curved exponential family, which can be embedded in the natural parameter space $\mathcal{A}$ in terms of natural coordinates

$${\theta }_1=\frac{1}{a^{2}u},{\theta }_2=-\frac{1}{2a^2{u}^2}$$

(23)

which is a parabola (denoted by ${\mathcal{F}}_A$)

$${\theta }_2=-\frac{a^2}{2}{\theta }_1^2$$

(24)

in $\mathcal{A}$. The underlying distribution in Equation (22) can be alternatively embedded in the expectation parameter space $\mathcal{B}$ in terms of expectation coordinates

$${\eta }_1=u,{\eta }_2=(a^2+1)u^2$$

(25)

which is also a parabola (denoted by ${\mathcal{F}}_B$)

$${\eta }_2=(a^2+1){\eta }_1^2$$

(26)

in $\mathcal{B}$.

The tangent vector $\nabla \mathit{\theta }\left(u\right)$ of the curve ${\mathcal{F}}_A$ is

$$\nabla \mathit{\theta }\left(u\right)=\frac{1}{a^2{u}^3}[-u,1]$$

(27)

The metric $\mathit{G}\left(u\right)$ has only one component g in this case, and is

$$g=\frac{2a^2+1}{a^2{u}^2}$$

(28)

The sufficient statistics $\mathit{F}\left(x\right)$ of the underlying distribution are

$$\mathit{F}\left(x\right)=\{x,x^2\}$$

(29)

and they are obtained from measurements (samples) x.

Figure 4 shows the two dual flat spaces ($\mathcal{A}$ and $\mathcal{B}$) and illustrates the estimation process implemented in them, where the blue parabolas in two figures denote the embeddings of the curved Gaussian distribution specified by parameter u. Without loss of generality, $a=1,u=2$ are assumed. The red arrows in two figures show the tangent vector $\nabla \mathit{\theta }\left(u\right)$ of the curve ${\mathcal{F}}_A$ while the two red dots on the parabolas denote the “realizations” of the distribution in Equation (22) specified by the given parameter $u=2$. One hundred observed data (measurements) are shown by the blue dots in the expectation parameter space specified by coordinates $(x,x^2)$. The red asterisk denotes the sufficient statistics $\mathit{F}\left(x\right)$ obtained from the statistical mean of the measurements (samples). By projecting the data $\mathit{F}\left(x\right)$ on to the sub-manifold represented by ${\mathcal{F}}_B$ orthogonally to $\nabla \mathit{\theta }\left(u\right)$, i.e., $(\mathit{F}\left(x\right)-\mathit{\eta }\left({\widehat{u}}_{ML}\right))\perp \nabla \mathit{\theta }\left({\widehat{u}}_{ML}\right)$, the MLE estimation of parameter u is obtained. By viewing Equation (25) and Figure 4b, the estimation ${\widehat{u}}_{ML}={\widehat{\eta }}_1$ is with high accuracy to the true value of u in this example.

4.2. A Mote Localization Example via RIPS Measurements

The Radio Interferometric Positioning System (RIPS) is an efficient sensing technique for sensor networks of small, low cost sensors with various applications. It utilizes radio frequency interference to obtain a sum of distance differences between the locations of a quartet of the motes which is initially reported in [26] and further discussed in [27]. In this paper, we take this mote localization problem via RIPS measurements as an application of the presented natural gradient MLE.

(1) Problem description: The Radio Interferometric Positioning System (RIPS) measurement model is described in [28]. A single RIPS measurement, as described in [26], involves four motes. A mote is a node in a sensor network that is capable of performing some processing, gathering sensory information and communicating with other connected nodes in the network. The main components of a sensor mote are a microcontroller, transceiver, external memory, power source and sensing hardware device. Figure 5 illustrates a collection of four motes $A,B,C,D$, where $A,B$ and C are anchor motes (i.e., their location states are already known) and D is a free mote with unknown location. Two motes act as transmitters, sending a pure sine wave at slightly different frequencies. This results in an interference signal at a low beat frequency that is received by the other two motes (acting as receivers). A sum of range differences between the four motes can be obtained from the phase difference of the received interference signals at the two receiver locations. If motes A and B serve as transmitters and motes C and D form the receiver pair, then the corresponding RIPS measurement, denoted $k_{A,B,C,D}$, measures the distance differences

$$\begin{array}{ccc}\hfill k_{A,B,C,D}& =\hfill & \left|\right|{\mathit{X}}_D-{\mathit{X}}_A\left|\right|-\left|\right|{\mathit{X}}_D-{\mathit{X}}_B\left|\right|\hfill \\ & +\hfill & \left|\right|{\mathit{X}}_B-{\mathit{X}}_C\left|\right|-\left|\right|{\mathit{X}}_A-{\mathit{X}}_C\left|\right|\hfill \end{array}$$

(30)

which may be simply written as

$$k_{A,B,C,D}=d_{AD}-d_{BD}+d_{BC}-d_{AC}.$$

(31)

In the absence of noise, two independent RIPS measurements can be found. Therefore, the other independent measurement is given by

$$k_{A,C,B,D}=d_{AD}-d_{CD}+d_{BC}-d_{AB}$$

(32)

which uses motes A and C as a transmitter pair and B and D as a receiver pair.

The two independent RIPS measurements can be written as

$$\mathit{k}\left(\mathit{u}\right)=\left[\begin{array}{c}{k}_{A,B,C,D}\\ k_{A,C,B,D}\end{array}\right]=\left[\begin{array}{c}{\delta }_{ab}+\sqrt{{(x_a-u_x)}^2+{(y_a-u_y)}^2}-\sqrt{{(x_b-u_x)}^2+{(y_b-u_y)}^2}\\ {\delta }_{ac}+\sqrt{{(x_a-u_x)}^2+{(y_a-u_y)}^2}-\sqrt{{(x_c-u_x)}^2+{(y_c-u_y)}^2}\end{array}\right]$$

(33)

where $\mathit{u}={[u_x,u_y]}^{\prime }$ is the unknown location of the free node D, and

$$\begin{array}{ccc}\hfill {\delta }_{ab}& =& d_{BC}-d_{AC}\hfill \\ & =& \sqrt{{(x_b-x_c)}^2+{(y_b-y_c)}^2}-\sqrt{{(x_a-x_c)}^2+{(y_a-y_c)}^2}\hfill \\ \hfill {\delta }_{ac}& =& d_{BC}-d_{AB}\hfill \\ & =& \sqrt{{(x_b-x_c)}^2+{(y_b-y_c)}^2}-\sqrt{{(x_a-x_b)}^2+{(y_a-y_b)}^2}\hfill \end{array}$$

are both known constants in which $x_i,y_i,i=a,b,c,$ are the location coordinates of the three anchor motes.

Accordingly, a generic RIPS measurement model can be written as

$$\mathit{x}=\mathit{k}\left(\mathit{u}\right)+\mathit{w},\mathit{w}\sim \mathcal{N}(0,\Sigma ),\Sigma ={\sigma }^2{I}_{2\times 2}$$

(34)

The underlying localization problem is to estimate the location of the free mote D based on RIPS measurement in Equation (34), where we assume that the knowledge of anchor node locations are known. The problem of locating the node D from RIPS measurements corrupted with Gaussian noise in Equation (34) reduces to a nonlinear parameter estimation problem. We adopt the natural gradient based MLE estimator described above to address it.

(2) Mote localization via RIPS measurements: Based on the RIPS measurement model in Equation (34), we can write the likelihood function in the form

$$p\left(\mathit{x}\right|\mathit{u})={|2\pi \Sigma |}^{-\frac{1}{2}}exp\left\{-\frac{1}{2}{[\mathit{x}-\mathit{k}\left(\mathit{u}\right)]}^T{\Sigma }^{-1}[\mathit{x}-\mathit{k}\left(\mathit{u}\right)]\right\}$$

(35)

Rearranging Equation (35) to describe in terms of the canonical curved exponential family, we obtain

$$p\left(\mathit{x}\right|\mathit{u})=exp\left\{C\left(\mathit{x}\right)+{\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{F}\left(\mathit{x}\right)-\phi \left[\mathit{\theta }\left(\mathit{u}\right)\right]\right\},$$

(36)

which yields (The required quantities can be obtained directly from the standard relations between Equations (4)–(7) for curved Gaussian distributions.)

$$\begin{array}{cc}\hfill \mathit{\theta }\left(\mathit{u}\right)& ={\Sigma }^{-1}\mathit{k}\left(\mathit{u}\right)\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill \mathit{F}\left(\mathit{x}\right)& =\mathit{x}\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill \phi \left[\mathit{\theta }\right(\mathit{u}\left)\right]& =-\frac{1}{4}\mathrm{tr}\left({\mathsf{\Theta }}^{-1}\mathit{\theta }{\mathit{\theta }}^T\right)-\frac{1}{2}log|-\mathsf{\Theta }|+log\pi ,\mathsf{\Theta }=-\frac{1}{2}{\Sigma }^{-1}\hfill \end{array}$$

(39)

The expectation parameter and FIM on natural parameter are given by

$$\begin{array}{cc}\hfill \mathit{\eta }\left(\mathit{u}\right)& ={\nabla }_{\mathit{\theta }}\phi \left(\mathit{\theta }\right)=-\frac{1}{2}{\mathsf{\Theta }}^{-1}\mathit{\theta }=\mathit{k}\left(\mathit{u}\right)\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill \mathit{G}\left(\mathit{\theta }\right)& ={\nabla }_{\mathit{\theta }}{\nabla }_{\mathit{\theta }}^T\phi \left(\mathit{\theta }\right)=-\frac{1}{2}{\mathsf{\Theta }}^{-1}=\Sigma \hfill \end{array}$$

(41)

The Jacobian matrix of natural parameter $\mathit{\theta }$ with respect to local parameter $\mathit{u}$ is given by

$$\nabla \mathit{\theta }\left(\mathit{u}\right)={\Sigma }^{-1}{\nabla }_{\mathit{u}}\mathit{k}\left(\mathit{u}\right)={\Sigma }^{-1}\left[\begin{array}{cc}\frac{\partial k_{A,B,C,D}}{\partial u_x}& \frac{\partial k_{A,B,C,D}}{\partial u_y}\\ \frac{\partial k_{A,C,B,D}}{\partial u_x}& \frac{\partial k_{A,C,B,D}}{\partial u_y}\end{array}\right]$$

(42)

where

$$\frac{\partial k_{A,B,C,D}}{\partial u_x}=\frac{x_b-u_x}{\sqrt{{(x_b-u_x)}^2+{(y_b-u_y)}^2}}-\frac{x_a-u_x}{\sqrt{{(x_a-u_x)}^2+{(y_a-u_y)}^2}}$$

(43)

$$\frac{\partial k_{A,B,C,D}}{\partial u_y}=\frac{y_b-u_y}{\sqrt{{(x_b-u_x)}^2+{(y_b-u_y)}^2}}-\frac{y_a-u_y}{\sqrt{{(x_a-u_x)}^2+{(y_a-u_y)}^2}}$$

(44)

$$\frac{\partial k_{A,C,B,D}}{\partial u_x}=\frac{x_c-u_x}{\sqrt{{(x_c-u_x)}^2+{(y_c-u_y)}^2}}-\frac{x_a-u_x}{\sqrt{{(x_a-u_x)}^2+{(y_a-u_y)}^2}}$$

(45)

$$\frac{\partial k_{A,C,B,D}}{\partial u_y}=\frac{y_c-u_y}{\sqrt{{(x_c-u_x)}^2+{(y_c-u_y)}^2}}-\frac{y_a-u_y}{\sqrt{{(x_a-u_x)}^2+{(y_a-u_y)}^2}}$$

(46)

The FIM with respect to the local parameter $\mathit{u}$, i.e., Equation (15) becomes

$$\mathit{G}\left(\mathit{u}\right)=\nabla {\mathit{\theta }}^T\left(\mathit{u}\right)\mathit{G}\left(\mathit{\theta }\right)\nabla \mathit{\theta }\left(\mathit{u}\right)={\nabla }_{\mathit{u}}^T\mathit{k}\left(\mathit{u}\right){\Sigma }^{-1}{\nabla }_{\mathit{u}}\mathit{k}\left(\mathit{u}\right)$$

(47)

Therefore, the iterative MLE estimator for estimating the location of a free mote using RIPS measurements is implemented as

$$\begin{array}{cc}\hfill {\mathit{u}}^{(k+1)}& ={\mathit{u}}^{\left(k\right)}+\lambda {\left[{\nabla }_{\mathit{u}}\mathit{k}({\mathit{u}}^{\left(k\right)})\right]}^{-1}[\mathit{x}-\mathit{k}({\mathit{u}}^{\left(k\right)})]\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill \mathit{G}({\mathit{u}}^{(k+1)})& ={\left[{\nabla }_{\mathit{u}}\mathit{k}({\mathit{u}}^{(k+1)})\right]}^T{\Sigma }^{-1}{\nabla }_{\mathit{u}}\mathit{k}({\mathit{u}}^{(k+1)})\hfill \end{array}$$

(49)

The covariance of the estimator ${\mathit{u}}^{(k+1)}$ is the inverse of the Fisher information matrix given in Equation (49).

As with other gradient optimization algorithms, a reasonable guess of the initial state value ${\mathit{u}}^{\left(0\right)}$ is required to facilitate the optimization converges to the correct local minimum. In this application, the initial state ${\mathit{u}}^{\left(0\right)}$ may be obtained from the RIPS measurements via the RIPS trilateration algorithm described in [29], which is then used to calculate $\mathit{G}\left({\mathit{u}}^{\left(0\right)}\right)$.

The performance of the proposed localization algorithm is illustrated by analyzing a scenario illustrated in Figure 6. In this example, three RIPS nodes are located at $(40,50)$, $(70,50)$, and $(60,70)$ m. The noise of a RIPS measurement is assumed to be zero-mean Gaussian with a standard deviation $\sigma =1$ m.

Figure 6a shows 5 cases of localization results of the algorithm. The initial values are randomly generated in the simulations. We use the label CRLB to signify the ellipse which corresponds to the inverse of the Fisher information matrix of the network $\mathit{G}\left(\mathit{u}\right)$, centered at the true location $\mathit{u}$. Figure 6b shows the covariances of the iterative MLE estimator at different iterations, where the kth error ellipse is calculated using the inverse of $\mathit{G}\left({\mathit{u}}^{\left(k\right)}\right)$ in Equation (49) and is centered at ${\mathit{u}}^{\left(k\right)}$. Figure 7a,b demonstrate the localization results of different initial state values with the same measurements and results based on a set of measurements with the same initial state values, respectively. The estimator performance under this scenario is summarized in

Table 1. Estimator performance summary for the mote localization scenario shown in Figure 7b.

Measure	Results
Number of Monte Carlo runs	$N=100$
Standard deviation of sensor noise	$\sigma =2$ m
Location of the free mote	$\mathit{u}={[40,65]}^T$
CRLB for estimating the state $\mathit{u}$	${\mathit{G}}^{-1}\left(\mathit{u}\right)=\left[\begin{array}{cc}24.4279& -15.2115\\ -15.2115& 11.4530\end{array}\right]$
Sample mean of the estimator	$E\left\{\widehat{\mathit{u}}\right\}=\frac{1}{N}{\sum }_{n=1}^N{\widehat{\mathit{u}}}_n={[39.9776,64.9758]}^T$
Sample covariance of the estimator	Cov$\left(\widehat{\mathit{u}}\right)=\frac{1}{N-1}{\sum }_{n=1}^N\left({\widehat{\mathit{u}}}_n-E\left\{\widehat{\mathit{u}}\right\}\right){\left({\widehat{\mathit{u}}}_n-E\left\{\widehat{\mathit{u}}\right\}\right)}^T=\left[\begin{array}{cc}2.9817& -1.7683\\ -1.7683& 1.2706\end{array}\right]$
Average RMS location error	$\frac{1}{N}{\sum }_{n=1}^N\left|\right|{\widehat{\mathit{u}}}_n-\mathit{u}\left|\right|=1.5928$ m
Number of iterations M	$M=44$, when $\lambda =0.1,\epsilon =0.01$ m
($\epsilon$—iteration stopping threshold)	$M=25$, when $\lambda =0.2,\epsilon =0.01$ m
($\lambda$—learning rate)	$M=35$, when $\lambda =0.2,\epsilon =0.001$ m

.

5. Conclusions

In this paper, an iterative maximum likelihood estimator based on the natural gradient method is described to address a class of nonlinear estimation problems for distributions of curved exponential families. We show that the underlying nonlinear stochastic filtering problem is solved by a natural gradient optimization technique which operates over statistical manifolds under dual affine connections. In this way, information geometry offers an interesting insight into the natural gradient algorithm and connects the stochastic estimation problem to a deterministic optimization problem. In this respect, the underlying philosophy is far more significant than the algorithm itself. Furthermore, based on an information geometric analysis it is promising that better algorithms for solving non-linear estimation problems can be derived. For instance, a “whitened gradient” which whitens the tangent space of a manifold has been presented in [30]. The whitened gradient replaces the Riemannian metric $\mathit{G}\left(\mathit{u}\right)$ in the natural gradient updates by its square root ${\mathit{G}}^{-\frac{1}{2}}\left(\mathit{u}\right)$ and results in a faster and more robust convergence.

The work in this paper indicates that the methods of differential/information geometry provide useful tools for systematically solving certain non-linear problems commonly encountered in signal processing. Future work involves extrapolation of these techniques to handle the filtering problem for nonlinear stochastic dynamics.