^{1}

^{2}

^{*}

^{3}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

Distributed estimation of Gaussian mixtures has many applications in wireless sensor network (WSN), and its energy-efficient solution is still challenging. This paper presents a novel diffusion-based EM algorithm for this problem. A diffusion strategy is introduced for acquiring the global statistics in EM algorithm in which each sensor node only needs to communicate its local statistics to its neighboring nodes at each iteration. This improves the existing consensus-based distributed EM algorithm which may need much more communication overhead for consensus, especially in large scale networks. The robustness and scalability of the proposed approach can be achieved by distributed processing in the networks. In addition, we show that the proposed approach can be considered as a stochastic approximation method to find the maximum likelihood estimation for Gaussian mixtures. Simulation results show the efficiency of this approach.

Recently, with the advances in micro-electronics and wireless communications, the large scale wireless sensor networks (WSNs) have found applications in many domains, such as environment monitoring, vehicle tracking, healthcare,

Mixture density is a powerful family of probability density to represent some physical quantities. The most wildly used mixture is Gaussian mixture model (GMM), which has wide applications in WSNs such as energy based multi-source localization and maneuvering target tracking [

The EM algorithm is used for finding maximum likelihood estimates of parameters in probabilistic models, where the model depends on unobserved latent variables. The EM algorithm was explained and given its name in a classic 1977 paper by Arthur Dempster, Nan Laird, and Donald Rubin [

The EM algorithm has been widely used in many areas, such as econometric, clinical, engineering and sociological studies [

The distributed processing framework in WSNs depends on the strategies of cooperation that are allowed among the nodes. Two of the strategies used in the literature for distributed processing are the incremental strategy and the consensus strategy. In the incremental strategy, information flows in a sequential manner from one node to the other. This mode of operation requires a cyclic pattern of collaboration among the nodes [

In the consensus strategy, distributed estimation algorithms are proposed in [

For the incremental strategy-based methods, finding the communication path through all nodes is the well-known Hamiltonian circuit problem, which is proved to be NP-complete [

In this paper, we propose a diffusion scheme of EM (DEM) algorithm for Gaussian mixtures in WSNs. At each iteration, the time-varying communication network is modeled as a random graph. A diffusion-step (D-step) is implemented between the E-step and the M-step. In the E-step, sensor nodes compute the local statistics by using local observation data and parameters estimated at the last iteration. In the D-step, each node exchanges local information only with its current neighbors and updates the local statistics with exchanged information. In the M-step, the sensor nodes compute the estimation of parameter using the updated local statistics by the D-step at this iteration. Compared with the existing distributed EM algorithms, the proposed approach can extensively save communication for each sensor node while maintain the estimation performance. Different from the linear estimation methods such as the least-squares and the least-mean squares estimation algorithms, each iteration of EM algorithm is a nonlinear transform of measurements. The steady-state performance of the proposed DEM algorithm can not be analyzed by linear way. Instead, we show that the DEM algorithm can be considered as a stochastic approximation method to find the maximum likelihood estimation for Gaussian Mixtures.

The rest of this paper is organized as follows. In Section 2, the centralized EM algorithm for Gaussian mixtures in WSNs is introduced, and two distributed strategies for EM algorithms are reviewed. In Section 3, a randomly time-varying network is introduced and the DEM algorithm in WSNs is proposed. Section 4 discusses the asymptotical property of the proposed DEM algorithm, and shows that the proposed algorithm can be viewed as a stochastic approximation to the maximum likelihood estimator according to the

We consider a network of _{m}_{mn}_{m}_{1}, ···, _{K}

Our task is using the observations to estimate the unknown parameters

The EM algorithm seeks to find the maximum likelihood estimation of the marginal likelihood by iteratively applying the E-step and M-step. In the E-step, we calculate the expectation of the log likelihood function, with respect to the conditional distribution of the unobserved data ^{t}^{t}

In the M-step, we want to maximize the expected log-likelihood ^{t}^{t+1},

By solving this maximization problem, we can get the estimation of

The centralized EM algorithm enables the calculation of the global solution using all the observation data from each sensor in the network. We can rewrite

The membership function is the probability of which the observation data _{mn}

The estimation of parameters at the

We denote
^{t}_{mn}

Incremental Updating Strategy

In this strategy, a cyclic sequential communication path between sensor nodes is pre-determined. In the forward path, the global statistics is updated by the pre-selected path for all sensor nodes according to
^{t}

Consensus Strategy

The estimation of parameter set ^{t}

Therefore, the idea of the average consensus filter can be used after the computing local statistics at the E-step. Each sensor node can get the global statistics by exchanging the local statistics with its neighbors until reaching consensus all over the network. The global solution for each sensor node at the M-step can be obtained by

For both strategies, individual sensor node first computes its own local statistics using the local observation data and estimation of the last step, and then gets the global statistics or global estimation for the parameters by changing information with its neighbors in some specific way. Therefore, each node can obtain the estimation of the parameters equivalent to the estimation in centralized way at every step, and get the same convergence point as the standard EM.

Both of the strategies described in Section 2 can be adopted to implement the EM algorithm in distributed way. However, as noted earlier in [

For the

Consider a WSN in which each sensor node has local computation functions and can only communicate with sensors in its neighborhood. The network can be modeled as a graph _{ij}_{i}

Power consumption of the sensor nodes in WSN can be divided into three domains: sensing, communication, and data processing [_{ij}_{ij}

By the definition of communication cost, the incremental strategy tends to require the least amount of communications. However, it inherently requires a Hamiltonian cycle through which signal estimates are sequentially circulated from sensor to sensor. In addition, the eventuality of a sensor failure also challenge the applicability of incremental schemes in large scale WSNs. The major disadvantage of the consensus-based algorithm is its high communication overhead requirements of each node. Motivated by these results and by diffusion strategies of [

E-step

At time t, the nodes compute their local statistics with local observation data and the estimated parameter at the last step as follows,

D-step

Each node communicates its local statistics with its neighbors as follows,

M-step

Each node computes the pre-estimation and communicates it to its neighbors to perform the average according to
_{i}

In our proposed method, each sensor node needs to transmit the local statistics and preestimates to neighboring nodes at each iteration while the distributed EM with consensus strategy need much more amount of communication to achieve consensus, especially in large scale WSNs. Without loss of generality, we consider the one-dimensional Gaussian mixtures with _{i}^{2}), since the local statistics need to be exchanged between all of the neighbors until they achieve consensus [_{i}_{i}

An important question is how well does the proposed DEM algorithm perform the estimation task. That is, how close does the local estimate get to the desired parameter as time evolves. Analyzing the performance of such diffusion scheme is challenging since each node is influenced by the local statistics from the neighboring nodes. In this section, we analyze the performance of the proposed DEM algorithm and show that it can be considered as a stochastic approximation method to find the maximum likelihood estimator. That is, if an infinite number of data, which are drawn independently according to the probability distribution

First, we introduce a symbol _{i}

If an infinite number of data are drawn independently according to the unknown probability distribution density _{ρ}

At an equilibrium point of this EM algorithm, the estimated parameter at the last step and current step becomes the same: ^{t}^{t+1}. The equilibrium condition of this EM algorithm is equivalent to the maximum likelihood condition,

Since we consider the asymptotical property of the DEM algorithm, one more data can be taken at each iteration. Therefore, the local standard EM algorithm for each sensor node is a modified on-line version of the EM algorithm [

The modified weighted mean _{i}_{i}_{t}^{t−1}), and ^{t}

After computing the modified weighted means according to ^{n}

In the basic on-line EM algorithm, for the t-th observation _{t}

For the WSN case, the weighted means are actually the local statistics, which are denoted as

A distributed dynamic system consists of _{m}

The D-step of DEM algorithm, which is proposed in the previous section, can be formulated by the following Laplacian dynamics in the discrete form

Therefore, the abstract form of standard on-line EM algorithm proposed in

From

For the convergence proof of the stochastic approximation in _{ρ}

Both terms on the right side of

The normalized coefficient

Recalling the definition of the coefficients

Therefore, we define the

In this section, we will present simulation results to illustrate the effectiveness of our proposed algorithm. We randomly generate

First, we consider the 1-dimensional Gaussian mixtures with two components. Each of the Gaussian components is with mean 6 and 15, variance 1 and 9, respectively. In the first 50 nodes (node 1 to node 50), 80% observations come from the first Gaussian component, and the rest 20% are from the second component. In the last 50 (node 51 to node 100), 40% observations come from the first component while the rest 60% are from the second one. The communication range is set to be 1.5 in this simulation.

The performance of estimation for each sensor node is shown in

The estimation performance for local, diffusion and consensus EM algorithms is demonstrated in _{i}

The scalability of the proposed diffusion EM algorithm is also investigated, with

In this subsection, we consider the 2D Gaussian density with

The first estimated mean values during the iteration process for ten random selected sensor nodes are demonstrated in

We also investigated the impact of estimation performance and communication overhead with different communication ranges. Since local processing/computing is much less expensive than communication, we emphasize on the tradeoff between estimation performance and communication overhead. From

Different communication ranges for each node are also considered as well as the different exchange steps at each iteration for each node. In

In addition, we consider the fixed communication range of each node with different steps of information exchange in the D-step of each iteration. From

This paper has presented a diffusion scheme of EM algorithm for distributed estimation of Gaussian mixtures in WSNs. In the E-step of this method, sensor nodes compute the local statistics by using local observation data and parameters estimated at the last iteration. A diffusion step is implemented over the communication networks after the E-step. In this step, the communication network is modeled as a connected graph, and each node exchanges local information only with its neighbors. In the M-step, the sensor nodes compute the estimation of parameter using the updated local statistics by the D-step at this iteration. Compared with the existing distributed EM algorithms, our proposed method can extensively reduce communication overhead for each sensor node while maintaining the estimation performance. In addition, we have shown that the proposed DEM algorithm can be considered as a stochastic approximation method to find the maximum likelihood estimation for Gaussian mixtures. Simulation results show good performance of our method.

This work was partly supported by the National Natural Science Foundation of China-Key Program (No. 61032001) and the National Natural Science Foundation of China (No. 60828006).

The connected network of 100 randomly distributed sensors with different radio ranges.

Estimated mean and covariance with and without diffusion.

Performance comparison for different EM algorithms.

Communication overhead for diffusion EM and consensus EM.

Likelihood function for different EM algorithms.

Estimated mean and covariance with and without diffusion.

Performance comparison for different EM algorithms with 1,000 sensor nodes.

Data distribution with 3-components Gaussian mixtures.

Estimated mean values by EM with and without diffusion.

Estimation performance and communication overhead versus radio range.

Estimated mean values versus radio range.

Estimated mean values versus steps of information exchange.