Distributed Density Estimation Based on a Mixture of Factor Analyzers in a Sensor Network

Abstract

Distributed density estimation in sensor networks has received much attention due to its broad applicability. When encountering high-dimensional observations, a mixture of factor analyzers (MFA) is taken to replace mixture of Gaussians for describing the distributions of observations. In this paper, we study distributed density estimation based on a mixture of factor analyzers. Existing estimation algorithms of the MFA are for the centralized case, which are not suitable for distributed processing in sensor networks. We present distributed density estimation algorithms for the MFA and its extension, the mixture of Student’s t-factor analyzers (MtFA). We first define an objective function as the linear combination of local log-likelihoods. Then, we give the derivation process of the distributed estimation algorithms for the MFA and MtFA in details, respectively. In these algorithms, the local sufficient statistics (LSS) are calculated at first and diffused. Then, each node performs a linear combination of the received LSS from nodes in its neighborhood to obtain the combined sufficient statistics (CSS). Parameters of the MFA and the MtFA can be obtained by using the CSS. Finally, we evaluate the performance of these algorithms by numerical simulations and application example. Experimental results validate the promising performance of the proposed algorithms.

1. Introduction

Sensor networks are composed of tiny, intelligent sensor nodes that are deployed over a geographic region. This type of network has a broad range of applications, such as environmental monitoring, precision agriculture and military surveillance [1,2,3]. Distributed estimation over sensor networks is to estimate some parameters of interest through local computation and information exchange among neighbor nodes. Compared to centralized estimation, it does not need to send observations collected by all of the sensors to a powerful central node, so the complexity and resource consumption can be reduced. Furthermore, distributed estimation is more flexible and robust to node and/or link failure [2,4]. Recently, many distributed estimation algorithms have been proposed, such as distributed LMS [5], distributed recursive least squares (RLS) [6], distributed source location [7], distributed power allocation [8], distributed sparse estimation [9,10], distributed information theoretic learning [11] and distributed Gaussian process regression [12].

Mixture of Gaussians (GMM) is a flexible and powerful probabilistic modeling tool for density estimation. It has been used in several areas, such as pattern recognition, computer vision, signal and image analysis and machine learning. When estimating parameters in the GMM by the maximum likelihood criterion, the expectation maximization (EM) algorithm [13,14] is usually adopted. It iteratively performs the expectation step (E-step) to calculate the conditional expectations of unobserved/hidden variables and runs the maximization step (M-step) to estimate parameters of data distributions based on the result of the E-step. However, when the dimension of observations is high, the fitting performance of the GMM deteriorates or even the associated EM algorithm cannot work [15]. The main reason is that the GMM cannot realize dimensionality reduction, which is to compress highly-correlated components of observations. In this case, a mixture of factor analyzers (MFA) [16,17] can be considered. The MFA combines local factor analysis in a form of a finite mixture. As factor analysis can describe variability among high-dimensional observations in terms of potentially low-dimensional latent factors, the MFA can carry out dimensionality reduction simultaneously when finishing specific tasks. Moreover, in order to process the non-normality of data or outliers, normal distributions in the MFA can be replaced by Student’s t-distributions, obtaining the mixture of Student’s t-factor analyzers (MtFA) [18,19]. Therefore, the MFA and its extension MtFA are effective tools for processing high-dimensional observations [20]. They have been successfully applied in the domains of signal processing [21,22], bioinformatics [23,24] and other applied fields.

In sensor networks, GMM has been introduced for density estimation of observations [25,26,27,28,29,30,31,32]. The estimation process for the GMM needs to be realized by distributed EM algorithms. According to the way by which nodes communicate with each other, distributed EM algorithms can be classified into the incremental type [25,26], the consensus type [27] and the diffusion type [28,29,30,31,32]. In the incremental scheme [25,26], a long way from the first node to last node of the pre-selected path is needed. When any node along the path fails, reliability problem may happen. In the consensus-based distributed EM algorithm for the GMM [27], a consensus filter, by which global statistics for each node is achieved, is carried out between the E-step and the M-step at each iteration. The objective is to obtain the same estimations for all nodes at each iteration. In the diffusion type of distributed estimation for the GMM, each node exchanges information only with its neighbors through a diffusion cooperative protocol. Good performance is obtained while communication overhead is kept low [2,4]. In this paper, we focus on a diffusion type of distributed estimation. Among previous studies, a distributed model order and parameter estimation algorithm for the GMM was proposed in [28]. Moreover, algorithm performance was analyzed. In [29], a diffusion-based EM algorithm was presented for distributed estimation in unreliable sensor networks. In this scenario, some nodes may be subject to data failures and report only noise. The aim of the algorithm was to achieve the optimal performance within the whole range of SNRs. In [30], information diffusion and averaging were considered and performed simultaneously. In [31], an adaptive diffusion scheme was proposed. In [32], the performance of the diffusion-based EM algorithm was analyzed. It could be considered as a stochastic approximation method [33] to find the maximum likelihood estimation of the GMM.

As the MFA can handle high-dimensional observations, which are also usually encountered in sensor networks, in this paper, we propose distributed density estimation algorithms for the MFA and its extension MtFA. We represent these two algorithms as D-MFA and D-MtFA, respectively. Specially, for each node in the sensor network, we define an objective function as the linear combination of local log-likelihoods, whose combination weights are determined by the number of observations in the corresponding neighbor nodes. After local sufficient statistics are computed, the current node calculates its combined sufficient statistics by a linear weighed combination of these local sufficient statistics from nodes in its neighborhood set. Finally, parameters of the MFA and MtFA are updated using the combined sufficient statistics. Apart from the distributed processing of the MFA and the MtFA in this paper, there are two other differences from the existing algorithms. First, in the relevant algorithms [25,27,28], mixing proportions in the GMM are different at each node, whereas means and covariances are the same throughout the network. On the contrary, in this paper, all of the parameters in the MFA or the MtFA are the same throughout the network. Using this design, distributed clustering and classification can be done in arbitrary nodes after the estimation process finishes. Second, for each node, the objective function is directly defined. The combination of weights in the objective function is effectively designed.

The rest of this paper is organized as follows. In Section 2, a brief overview of the MFA and the MtFA are provided. In Section 3, the D-MFA algorithm and D-MtFA algorithm are formulated. In Section 4, numerical simulations for the synthetic observations are performed to illustrate the effectiveness and advantages of the proposed algorithms. Moreover, the application of these algorithms to distributed clustering is also presented. Finally, conclusions are drawn in Section 5.

The acronyms mentioned in this paper are listed in the following.

Acronym list:
GMM	Gaussian mixture model
EM	expectation maximization
E-step	expectation step
M-step	maximization step
MFA	mixture of factor analyzers
MtFA	mixture of Student’s t-factor analyzers
D-MFA	distributed density estimation algorithm for the MFA
D-MtFA	distributed density estimation algorithm for the MtFA
CSS	combined sufficient statistics
LSS	local sufficient statistics
S-MFA	standard EM algorithm for the MFA
S-MtFA	standard EM algorithm for the MtFA
NC-MFA	non-cooperation MFA
NC-MtFA	non-cooperation MtFA
D-GMM	distributed density estimation algorithm for the GMM
D-tMM	distributed density estimation algorithm for the Student’st-mixture model

2. Preliminaries: MFA and MtFA

2.1. Mixture of Factor Analyzers

Let the observed dataset be ${Y}=\{{\mathrm{y}}_1,...,{\mathrm{y}}_N\}$. In the MFA, it assumes that each p-dimensional data vector ${\mathrm{y}}_n$ is generated as:

$${\mathrm{y}}_n={{\mu }}_i+{{A}}_i{{u}}_n+{\mathrm{e}}_{ni}\text{ with prob. }{\pi }_i(i=1,...,I)$$

(1)

where I is the number of mixing components. The corresponding q-dimensional ($q<p$) factor ${{u}}_n\sim \mathrm{𝒩}\left({{u}}_n\right|\mathbf{0},{{I}}_q)$ is independent of the ${\mathrm{e}}_{ni}\sim \mathrm{𝒩}\left({\mathrm{e}}_{ni}\right|\mathbf{0},{{D}}_i)$, where ${{D}}_i$ is a $p\times p$ diagonal matrix. The parameter ${{\mu }}_i$ is the mean of the i-th analyzer, and ${{A}}_i$ ($p\times q$) is the linear transformation known as the factor loading matrix. The so-called mixing proportions ${\pi }_i(i=1,...,I)$ are nonnegative and sum to one. The standard EM algorithm for the MFA is given in [15,16].

2.2. Mixture of Student’s t-Factor Analyzers

Since the MFA adopts the normal family for the distributions of the errors and the latent factors, it is sensitive to outliers. An obvious way to improve the robustness of this model for observations having longer tails than normal is using the t-family of elliptically-symmetric distributions. Therefore, the MtFA has been proposed in [18]. In the MtFA, it assumes that p-dimensional data vector ${{y}}_n$ is generated in the same way as that in the MFA, as shown in Equation (1). However, the distributions of q-dimensional $(q<p)$ factor ${{u}}_n$ and noise ${\mathrm{e}}_{ni}$ are $t\left({{u}}_n\right|\mathbf{0},{{I}}_q,{\nu }_i)$ and $t\left({\mathrm{e}}_{ni}\right|\mathbf{0},{{D}}_i,{\nu }_i)$, respectively. In the above Student’s t distributions, ${\nu }_i$ is called the degree of freedom that controls the length of the tails of the distributions. With this modification, the MtFA is more robust to outliers and can process the non-normality of observations in a better way [23]. In essence, $t\left({{u}}_n\right|\mathbf{0},{{I}}_q,{\nu }_i)$ and $t\left(e_{ni}\right|\mathbf{0},{{D}}_i,{\nu }_i)$ can be respectively regarded as average Gaussian scale distributions $\mathrm{𝒩}\left({{u}}_n\right|\mathbf{0},{{I}}_q/w_{ni})$ and $\mathrm{𝒩}\left(e_{ni}\right|\mathbf{0},{{D}}_i/w_{ni})$ with the Gamma distributed precision scalar $w_{ni}$, that is:

$$\begin{array}{ccc}\hfill t\left({{u}}_n\right|\mathbf{0},{{I}}_q,{\nu }_i)& =& \int \mathrm{d}{w}_{ni}\mathrm{𝒩}\left({{u}}_n\right|\mathbf{0},{{I}}_q/w_{ni})\mathrm{𝒢}\left(w_{ni}\right|{\nu }_i/2,{\nu }_i/2)\hfill \\ \hfill t\left({\mathrm{e}}_{ni}\right|\mathbf{0},{{D}}_i,{\nu }_i)& =& \int \mathrm{d}{w}_{ni}\mathrm{𝒩}\left({\mathrm{e}}_{ni}\right|\mathbf{0},{{D}}_i/w_{ni})\mathrm{𝒢}\left(w_{ni}\right|{\nu }_i/2,{\nu }_i/2)\hfill \end{array}$$

where $\mathrm{𝒢}(·)$ denotes the Gamma distribution. The standard EM algorithm for the MtFA is given in [14,18].

3. Distributed Estimation Algorithms for the MFA and MtFA

3.1. Network Model and Objective Function

Consider a sensor network with M nodes. The m-th node has $N_m$ data observations ${{Y}}_m={\left\{{\mathrm{y}}_{m,n}\right\}}_{n=1,...,N_m}(m=1,...,M)$, and ${\mathrm{y}}_{m,n}$ denotes the n-th observation in node m. The distribution of each p-dimensional observation ${\mathrm{y}}_{m,n}$ is modeled by the MFA, defined in Equation (1). It is noted that the factor associated with ${\mathrm{y}}_{m,n}$ here is represented as ${{u}}_{m,n}$. The parameter set of the MFA is ${\Theta }={\{{\pi }_i,{{\mu }}_i,{{A}}_i,{{D}}_i\}}_{i=1,...,I}$, which is to be estimated.

The network topology is described by a graph. Let W denote the distance that a node can communicate via wireless radio links. Nodes m and l are connected if the Euclidean distance $d_{m,l}$ between m and l is less than or equal to W. Moreover, a graph is connected if for any pair of nodes $(m,n)$, there exists a path from m to n. The neighborhood set of node m, denoted by $R_m$, is defined as the one-hop neighbors of node m (including m itself). For example, in Figure 1, the dashed circle represents the neighborhood set of node m, containing Node 1, Node 2, node l and node m itself.

Figure 1. A sensor network consists of a collection of cooperating nodes. Node m only exchanges information (e.g., local sufficient statistics (LSS) in the proposed D-MFA and D-MtFA algorithms) with nodes in $R_m$.

Figure 1. A sensor network consists of a collection of cooperating nodes. Node m only exchanges information (e.g., local sufficient statistics (LSS) in the proposed D-MFA and D-MtFA algorithms) with nodes in $R_m$.

In order to design the D-MFA and the D-MtFA algorithms, the objective functions should be carefully specified at first. Here, we take node m in the sensor network for example. We define the objective function ${\mathrm{ℱ}}_m\left({\Theta }\right)$ of the MFA at node m as a linear combination of local log-likelihoods $logp\left({{Y}}_l\right|{\Theta })$ associated with nodes l in its neighborhood $R_m$.

$${\mathrm{ℱ}}_m\left({\Theta }\right)=\sum _{l\in R_m}{c}_{lm}·logp\left({{Y}}_l\right|{\Theta })=\sum _{l\in R_m}{c}_{lm}\sum _{n=1}^{N_l}log\left(\sum _{i=1}^I{\pi }_{i}N\left({\mathrm{y}}_{l,n}\right|{{\mu }}_i,{{A}}_i,{{D}}_i)\right)$$

(2)

where ${\left\{c_{lm}\right\}}_{l\in R_m}$ are some non-negative combination coefficients satisfying the condition ${\sum }_{l\in R_m}{c}_{lm}=1$, $c_{lm}=0$ if node $l\notin R_m$.

It should be emphasized that when defining ${\mathrm{ℱ}}_m\left({\Theta }\right)$, we consider two important factors. First, as node m can only communicate with its neighbors, it is reasonable to define ${\mathrm{ℱ}}_m\left({\Theta }\right)$ as a combination of local log-likelihoods $logp\left({{Y}}_l\right|{\Theta }),(l\in R_m)$. When estimating Θ, node m can make use of the information from nodes in $R_m$. Due to the effect of information diffusion, each node can obtain all information directly or indirectly from other nodes. Second, the contributions of different local log-likelihoods $logp\left({{Y}}_l\right|{\Theta }),(l\in R_m)$ for the estimation of Θ at node m may also be different. The combination coefficient $c_{lm}$ weights the importance of information flow from the node $l(l\in R_m)$. Therefore, how to choose $c_{lm}$ is important. Here, we adopt a simple, but effective mechanism that $c_{lm}$ is determined by:

$$c_{lm}=\frac{N_l}{{\sum }_{l^{\prime }\in R_m}{N}_{l^{\prime }}}$$

(3)

If node l has a larger number of observations $N_l$, the information from this node makes a larger contribution to obtain more accurate parameter estimations. Therefore, a larger combination coefficient $c_{lm}$ in Equation (3) can further make this contribution prominent. In the future, a more effective implementation, such as an adaptive strategy [4], can be considered to determine these combination coefficients better.

3.2. Distributed Density Estimation Algorithm for the MFA

After the objective functions ${\mathrm{ℱ}}_m\left({\Theta }\right)(m=1,...,M)$ have been determined, the next task is to estimate parameters Θ in the MFA by maximizing ${\mathrm{ℱ}}_m\left({\Theta }\right)$. For node l, an I-dimensional binary latent variable $z_{l,n}$, which is associated with ${\mathrm{y}}_{l,n}$, is introduced. As the MFA is a mixture model, $z_{l,n,i}(=1)$ denotes that ${\mathrm{y}}_{l,n}$ belongs to the i-th component of the MFA. The latent variables for node l in the neighborhood $R_m$ are ${{U}}_l={\left\{{{u}}_{l,n}\right\}}_{n=1,...,N_l}$, ${{Z}}_l={\left\{{{z}}_{l,n}\right\}}_{n=1,...,N_l}$, $(l\in R_m)$. Now, ${\mathrm{ℱ}}_m\left({\Theta }\right)(m=1,...,M)$ in Equation (2) can be expressed as:

$${\mathrm{ℱ}}_m\left({\Theta }\right)=\sum _{l\in R_m}{c}_{lm}·\left\{log\sum _{Z_l}\int \mathrm{d}{{U}}_{l}p({{Y}}_l,{{Z}}_l,{{U}}_l|{\Theta })\right\}$$

where:

$$p({{Y}}_l,{{Z}}_l,{{U}}_l|{\Theta })=p\left({{Z}}_l\right|{\Theta })p\left({{U}}_l\right|{{Z}}_l,{\Theta })p\left({{Y}}_l\right|{{Z}}_l,{{U}}_l{\Theta })$$

(4)

The three conditional probabilities in Equation (4) are:

$$\begin{array}{ccc}\hfill p\left({{Z}}_l\right|{\Theta })& =& \prod _{n=1}^{N_l}\prod _{i=1}^I{\pi }_i^{z_{l,n,i}}\hfill \\ \hfill p\left({{U}}_l\right|{{Z}}_l,{\Theta })& =& \prod _{n=1}^{N_l}\prod _{i=1}^I\mathrm{𝒩}{\left({{u}}_{l,n}\right|\mathbf{0},{{I}}_q)}^{z_{l,n,i}}\hfill \\ \hfill p\left({{Y}}_l\right|{{Z}}_l,{{U}}_l,{\Theta })& =& \prod _{n=1}^{N_l}\prod _{i=1}^I\mathrm{𝒩}{\left({\mathrm{y}}_{l,n}\right|{{\mu }}_i+{{A}}_i{{u}}_{l,n},{{D}}_i)}^{z_{l,n,i}}\hfill \end{array}$$

Here, we derive the distributed estimation algorithm with the aid of the standard EM algorithm [13]. First, we introduce two distributions $q\left({{Z}}_l\right)$ and $q\left({{U}}_l\right|{{Z}}_l)$ defined over the latent variables. For any choice of $q\left({{Z}}_l\right)$ and $q\left({{U}}_l\right|{{Z}}_l)$, the following decomposition holds:

$${\mathrm{ℱ}}_m\left({\Theta }\right)=\sum _{l\in R_m}{c}_{lm}·\mathrm{ℒ}(q_l,{\Theta })+\sum _{l\in R_m}{c}_{lm}·\mathrm{KL}(q_l\left|\right|p_l)$$

(5)

where:

$$\begin{array}{ccc}\hfill \mathrm{ℒ}(q_l,{\Theta })& =& \sum _{{{Z}}_l}q\left({{Z}}_l\right)\int \mathrm{d}{{U}}_{l}q\left({{U}}_l\right|{{Z}}_l)log\left\{\frac{p({{Y}}_l,{{Z}}_l,{{U}}_l|{\Theta })}{q\left({{Z}}_l\right)q\left({{U}}_l\right|{{Z}}_l)}\right\}\hfill \\ \hfill \mathrm{KL}\left(q_l\right|\left|p_l\right)& =& -\sum _{{{Z}}_l}q\left({{Z}}_l\right)\int \mathrm{d}{{U}}_{l}q\left({{U}}_l\right|{{Z}}_l)log\left\{\frac{p\left({{Z}}_l\right|{{Y}}_l,{\Theta })p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{\Theta })}{q\left({{Z}}_l\right)q\left({{U}}_l\right|{{Z}}_l)}\right\}\hfill \end{array}$$

The verification of log-likelihood decomposition can be found in [34]. As ${\mathrm{ℱ}}_m\left({\Theta }\right)$ as a combination of local log-likelihoods, the whole decomposition can also be expressed by a combination of local log-likelihood decompositions, as shown in Equation (5).

Moreover, $\mathrm{KL}\left(q_l\right|\left|p_l\right)$ in Equation (5) is the Kullback–Leibler divergence between $q\left({{Z}}_l\right)q\left({{U}}_l\right|{{Z}}_l)$ and $p\left({{Z}}_l\right|{{Y}}_l,{\Theta })p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{\Theta })$, which satisfies $\mathrm{KL}\left(q_l\right|\left|p_l\right)\ge 0$. Therefore, it can be seen from Equation (5) that ${\sum }_{l\in R_m}{c}_{lm}·\mathrm{ℒ}(q_l,{\Theta })\le {\mathrm{ℱ}}_m\left({\Theta }\right)$. In other words, ${\sum }_{l\in R_m}{c}_{lm}·\mathrm{ℒ}(q_l,{\Theta })$ is a lower bound on ${\mathrm{ℱ}}_m\left({\Theta }\right)$. As direct maximization of the ${\mathrm{ℱ}}_m\left({\Theta }\right)$ is difficult, it can be solved by the maximization of this lower bound instead.

Suppose that the parameters estimated in the last iteration are ${{\Theta }}^{\mathrm{old}}={\{{\pi }_i^{\mathrm{old}},{{\mu }}_i^{\mathrm{old}},{{A}}_i^{\mathrm{old}},{{D}}_i^{\mathrm{old}}\}}_{i=1,...,I}$. In the first stage, the lower bound ${\sum }_{l\in R_m}{c}_{lm}·\mathrm{ℒ}(q_l,{{\Theta }}^{\mathrm{old}})$ is maximized with respect to $q\left({{Z}}_l\right)q\left({{U}}_l\right|{{Z}}_l)$ while holding ${{\Theta }}^{\mathrm{old}}$ fixed. From Equation (5), this maximum can be achieved when $\mathrm{KL}\left(q_l\right|\left|p_l\right)=0$. In other words, $q\left({{Z}}_l\right)q\left({{U}}_l\right|{{Z}}_l)=p\left({{Z}}_l\right|{{Y}}_l,{{\Theta }}^{\mathrm{old}})p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{{\Theta }}^{\mathrm{old}})$. Therefore, two conditional distributions, $p\left({{Z}}_l\right|{{Y}}_l,{{\Theta }}^{\mathrm{old}})$ and $p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{{\Theta }}^{\mathrm{old}})$, should be computed.

Concretely, for node l ($l\in R_m$), $p\left({{Z}}_l\right|{{Y}}_l,{{\Theta }}^{\mathrm{old}})$ can be calculated by:

$$p\left({{Z}}_l\right|{{Y}}_l,{{\Theta }}^{\mathrm{old}})=\prod _{n=1}^{N_l}\prod _{i=1}^{I}p\left(z_{l,n,i}\right|{\mathrm{y}}_{l,n})$$

where:

$$p\left(z_{l,n,i}\right|{\mathrm{y}}_{l,n})=\frac{{\pi }_i^{\mathrm{old}}N\left({\mathrm{y}}_{l,n}\right|{{\mu }}_i^{\mathrm{old}},{{A}}_i^{\mathrm{old}}{\left({{A}}_i^{\mathrm{old}}\right)}^{\mathrm{T}}+{{D}}_i^{\mathrm{old}})}{{\sum }_{i^{\prime }=1}^I{\pi }_{i^{\prime }}^{\mathrm{old}}N\left({\mathrm{y}}_{l,n}\right|{{\mu }}_{i^{\prime }}^{\mathrm{old}},{{A}}_{i^{\prime }}^{\mathrm{old}}{\left({{A}}_{i^{\prime }}^{\mathrm{old}}\right)}^{\mathrm{T}}+{{D}}_{i^{\prime }}^{\mathrm{old}})}$$

(6)

Moreover, $p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{{\Theta }}^{\mathrm{old}})$ should also be obtained by:

$$\begin{array}{c}\hfill p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{{\Theta }}^{\mathrm{old}})=\prod _{n=1}^{N_l}\prod _{i=1}^{I}p\left({{u}}_{l,n}\right|{\mathrm{y}}_{l,n},z_{l,n,i})=\prod _{n=1}^{N_l}\prod _{i=1}^{I}N\left({{u}}_{l,n}\right|{\overline{{u}}}_{l,n,i},{{\Omega }}_i)\end{array}$$

(7)

The mean ${\overline{{u}}}_{m,n,i}$ and covariance ${{\Omega }}_i$ are:

$$\begin{array}{ccc}\hfill {\overline{{u}}}_{l,n,i}& =& {\mathrm{g}}_i^{\mathrm{T}}({\mathrm{y}}_{l,n}-{{\mu }}_i^{\mathrm{old}})\hfill \\ \hfill {{\Omega }}_i& =& {{I}}_q-{\mathrm{g}}_i^{\mathrm{T}}{{A}}_i^{\mathrm{old}}\hfill \end{array}$$

(8)

where:

$${\mathrm{g}}_i={\left[{{A}}_i^{\mathrm{old}}{\left({{A}}_i^{\mathrm{old}}\right)}^{\mathrm{T}}+{{D}}_i^{\mathrm{old}}\right]}^{-1}·{{A}}_i^{\mathrm{old}}$$

(9)

is an intermediate variable introduced to simplify expressions in the following steps.

When the above two conditional distributions have been obtained, $q\left({{Z}}_l\right)q\left({{U}}_l\right|{{Z}}_l)$ is determined and held fixed, and the lower bound ${\sum }_{l\in R_m}{c}_{lm}·\mathrm{ℒ}(q_l,{\Theta })$ is maximized with respect to Θ to get new estimated ${{\Theta }}^{\mathrm{new}}$. This will cause the lower bound to increase, which will necessarily cause the corresponding ${\mathrm{ℱ}}_m\left({\Theta }\right)$ to increase.

Concretely, the current lower bound is expressed as:

$$\begin{array}{ccc}\hfill \sum _{l\in R_m}{c}_{lm}\mathrm{ℒ}(q_l,{\Theta })& =& \sum _{l\in R_m}{c}_{lm}\sum _{{{Z}}_l}p\left({{Z}}_l\right|{{Y}}_l,{{\Theta }}^{\mathrm{old}})\int \mathrm{d}{{U}}_{l}p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{{\Theta }}^{\mathrm{old}})\hfill \\ & & \times \left\{logp({{Y}}_l,{{Z}}_l,{{U}}_l|{\Theta })-log\left[p\left({{Z}}_l\right|{{Y}}_l,{{\Theta }}^{\mathrm{old}})p\left({{U}}_l\right|{{Z}}_l,{{Y}}_l,{{\Theta }}^{\mathrm{old}})\right]\right\}\hfill \end{array}$$

(10)

Discard the second logarithmic term unrelated to Θ in Equation (10); the objective function represented by $Q_m\left({\Theta }\right)$ at node m is:

$$\begin{array}{ccc}\hfill Q_m\left({\Theta }\right)& =& \sum _{l\in R_m}{c}_{lm}\sum _{n=1}^{N_l}\sum _{i=1}^{I}p\left(z_{l,n,i}\right|{\mathrm{y}}_{l,n})p\left({{u}}_{l,n}\right|{\mathrm{y}}_{l,n},z_{l,n,i})\hfill \\ & & \times \left\{z_{l,n,i}log\left[{\pi }_{i}N\left({{u}}_{l,n}\right|\mathbf{0},{{I}}_q)N\left({\mathrm{y}}_{l,n}\right|{{\mu }}_i+{{A}}_i{{u}}_{l,n},{{D}}_i)\right]\right\}\hfill \end{array}$$

Now, parameters in Θ can be obtained by taking derivation of $Q_m\left({\Theta }\right)$ with respect to Θ.

First, ${\pi }_i$ and ${{\mu }}_i$ are updated to:

$${\pi }_i=\frac{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle }{{\sum }_{i^{\prime }=1}^I{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i^{\prime }}\rangle }$$

(11)

and:

$${{\mu }}_i=\frac{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle {\mathrm{y}}_{l,n}}{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle }$$

(12)

respectively. Subsequently, by performing derivation of $Q_m\left({\Theta }\right)$ with respect to ${{A}}_i$, we have:

$${{A}}_i=\frac{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle ({\mathrm{y}}_{l,n}-{{\mu }}_i){\langle {{u}}_{l,n}\rangle }^{\mathrm{T}}}{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle ·\langle {{u}}_{l,n}{{u}}_{l,n}^{\mathrm{T}}\rangle }$$

(13)

Finally, the expression of ${{D}}_i$ can be obtained in the same way,

$${{D}}_i=\mathrm{diag}\left\{\frac{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle \left[({\mathrm{y}}_{l,n}-{{\mu }}_i){({\mathrm{y}}_{l,n}-{{\mu }}_i)}^{\mathrm{T}}-{{A}}_i\langle {{u}}_{l,n}{{u}}_{l,n}^{\mathrm{T}}\rangle {{A}}_i^{\mathrm{T}}\right]}{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle }\right\}$$

(14)

where $\mathrm{diag}\{·\}$ denotes the operator setting off-diagonal terms to zero. In Equations (11,12,13,14), $\langle z_{l,n,i}\rangle$ is the expectation of $z_{l,n,i}$ given by Equation (6). $\langle {{u}}_{l,n}\rangle$ and $\langle {{u}}_{l,n}{{u}}_{l,n}^{\mathrm{T}}\rangle$ can be obtained from Equation (7), which are:

$$\langle {{u}}_{l,n}\rangle ={\overline{{u}}}_{l,n,i}\mathrm{and}\langle {{u}}_{l,n}{{u}}_{l,n}^{\mathrm{T}}\rangle ={{\Omega }}_i+{\overline{{u}}}_{l,n,i}{\overline{{u}}}_{l,n,i}^{\mathrm{T}}$$

(15)

respectively. Substituting Equation (15) into Equations (13) and (14), we have:

$${{A}}_i={{V}}_i{{g}}_i{({{g}}_i^{\mathrm{T}}{{V}}_i{{g}}_i+{{\Omega }}_i)}^{-1}$$

(16)

and:

$${{D}}_i=\mathrm{diag}\left\{{{V}}_i-{{A}}_i({{g}}_i^{\mathrm{T}}{{V}}_i{{g}}_i+{{\Omega }}_i){{A}}_i^{\mathrm{T}}\right\}$$

(17)

where:

$${{V}}_i=\frac{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle ({\mathrm{y}}_{l,n}-{{\mu }}_i){({\mathrm{y}}_{l,n}-{{\mu }}_i)}^{\mathrm{T}}}{{\sum }_{l\in R_m}{c}_{lm}{\sum }_{n=1}^{N_l}\langle z_{l,n,i}\rangle }$$

(18)

From Equations (11), (12) and (16)–(18), we can see, when estimating parameters Θ at node m, that three combined sufficient statistics (CSS) must be obtained, represented as:

$$\begin{array}{ccc}\hfill {\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]& =& \sum _{l\in R_m}{c}_{lm}·{\mathrm{LSS}}_l^{\left(1\right)}\left[i\right]\hfill \\ \hfill {\mathrm{CSS}}_m^{\left(2\right)}\left[i\right]& =& \sum _{l\in R_m}{c}_{lm}·{\mathrm{LSS}}_l^{\left(2\right)}\left[i\right]\hfill \\ \hfill {\mathrm{CSS}}_m^{\left(3\right)}\left[i\right]& =& \sum _{l\in R_m}{c}_{lm}·{\mathrm{LSS}}_l^{\left(3\right)}\left[i\right]\hfill \end{array}$$

(19)

where:

$$\begin{array}{ccc}\hfill {\mathrm{LSS}}_l^{\left(1\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle \hfill \\ \hfill {\mathrm{LSS}}_l^{\left(2\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle ·{\mathrm{y}}_{l,n}\hfill \\ \hfill {\mathrm{LSS}}_l^{\left(3\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle ·{\mathrm{y}}_{l,n}·{\mathrm{y}}_{l,n}^{\mathrm{T}}\hfill \end{array}$$

(20)

${\mathrm{LSS}}_l={\{{\mathrm{LSS}}_l^{\left(1\right)}\left[i\right],{\mathrm{LSS}}_l^{\left(2\right)}\left[i\right],{\mathrm{LSS}}_l^{\left(3\right)}\left[i\right]\}}_{i=1,...,I}$ are local sufficient statistics (LSS) of node l. Therefore, CSS in node m is a linear combination of the LSS of nodes in $R_m$. If node l has a large number of observations, the accuracy of calculated ${\mathrm{LSS}}_l$ should be high and should make an important contribution to the CSS of node m. A relatively large $c_{lm}$ in Equation (3) can make this contribution prominent, obtaining accurate estimation of Θ.

In the following, we summarize the realization process of the D-MFA algorithm.

Step 1 (Initialization): This initializes the parameters ${\{{\pi }_i,{{\mu }}_i,{{A}}_i,{{D}}_i\}}_{i=1,...,I}$. Each node l broadcasts the number of its observations $N_l$ to its neighbors. When receiving this information, each node calculates the combination coefficient by Equation (3).

Step 2 (Computation): Each node l in the sensor network computes $\langle z_{l,n,i}\rangle$, ${{\Omega }}_i$ and ${\mathrm{g}}_i$ by Equations (6), (8) and (9), respectively. Then, it computes three local sufficient statistics ${\mathrm{LSS}}_l={\{{\mathrm{LSS}}_l^{\left(1\right)}\left[i\right],{\mathrm{LSS}}_l^{\left(2\right)}\left[i\right],{\mathrm{LSS}}_l^{\left(3\right)}\left[i\right]\}}_{i=1,...,I}$ according to its own observations ${{Y}}_l$, by Equation (20).

Step 3 (Diffusion): Each node l in sensor networks diffuses its local sufficient statistics ${\mathrm{LSS}}_l$, as shown in Figure 1.

Step 4 (Combination): When node m ($m=1,...,M$) receives the local sufficient statistics from all of its one-hop neighbor nodes l ($l\in R_m$), it computes the combined sufficient statistics $\{{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]$, ${\mathrm{CSS}}_m^{\left(2\right)}\left[i\right]$, ${\mathrm{CSS}}_m^{\left(3\right)}{\left[i\right]\}}_{i=1,...,I}$ by Equation (19).

Step 5 (Estimation): Node m ($m=1,...,M$) estimates ${\pi }_i$, ${{\mu }}_i$, ${{A}}_i$ and ${{D}}_i$ according to Equations (11), (12), (16) and (17), respectively. Here, we substitute Equation (19) into Equations (11), (12) and (18), reformulating the estimation step as follows:

$$\begin{array}{ccc}\hfill {\pi }_i& =& \frac{{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]}{{\sum }_{i^{\prime }=1}^I{\mathrm{CSS}}_m^{\left(1\right)}\left[i^{\prime }\right]}\hfill \\ \hfill {{\mu }}_i& =& \frac{{\mathrm{CSS}}_m^{\left(2\right)}\left[i\right]}{{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]}\hfill \\ \hfill {{A}}_i& =& {{V}}_i{\mathrm{g}}_i{({\mathrm{g}}_i^{\mathrm{T}}{{V}}_i{\mathrm{g}}_i+{{\Omega }}_i)}^{-1}\hfill \\ \hfill {{D}}_i& =& \mathrm{diag}\left\{{{V}}_i-{{A}}_i({\mathrm{g}}_i^{\mathrm{T}}{{V}}_i{\mathrm{g}}_i+{{\Omega }}_i){{A}}_i^{\mathrm{T}}\right\}\hfill \end{array}$$

where:

$${{V}}_i=\frac{{\mathrm{CSS}}_m^{\left(3\right)}\left[i\right]-2{\mathrm{CSS}}_m^{\left(2\right)}\left[i\right]·{{\mu }}_i+{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]·{{\mu }}_i{{\mu }}_i^{\mathrm{T}}}{{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]}$$

Step 6 (Termination): Node m ($m=1,...,M$) calculates its current local log-likelihood as:

$$logp\left({{Y}}_m\right|{{\Theta }}^{\mathrm{new}})=\sum _{n=1}^{N_m}log\left(\sum _{i=1}^I{\pi }_{i}N\left({\mathrm{y}}_{m,n}\right|{{\mu }}_i,{{A}}_i,{{D}}_i)\right)$$

where superscript “new” denotes the newly estimated parameters at the current iteration. If $logp\left({{Y}}_m\right|{{\Theta }}^{\mathrm{new}})-logp\left({{Y}}_m\right|{{\Theta }}^{\mathrm{old}})<ϵ$, node m enters the terminated state; else, go to Step 2 and start the next iteration. It is noted that the terminated nodes do no computation or communication in the following iterations. If one node cannot receive information from a neighbor node in the next iteration, the node will use the received and saved LSS information from that neighbor node at the last iteration when updating CSS. When there is no message communication or information exchange in the network, implying all nodes reach the terminated state, the algorithm ends.

3.3. Distributed Density Estimation Algorithm for the MtFA

Compared to the MFA, the main difference of the MtFA is that it has an additional degree of freedom parameter ${\nu }_i(i=1,...,I)$. Therefore, the parameter set of the MtFA to be estimated is ${\Theta }={\{{\pi }_i,{{\mu }}_i,{{A}}_i,{{D}}_i,{\nu }_i\}}_{i=1,...,I}$. Moreover, apart from ${{Z}}_l$ and ${{U}}_l$, the latent variable ${{W}}_l={\left\{w_{l,n,i}\right\}}_{i=1,...,I}^{n=1,...,N_l}$ should be introduced, explained in Section 2.2. Similarly, for node m in a sensor network, a linear combination of local log-likelihoods associated with nodes in $R_m$ is defined as:

$$\begin{array}{c}\hfill {\mathrm{ℱ}}_m\left({\Theta }\right)=\sum _{l\in R_m}{c}_{lm}·logp\left({{Y}}_l\right|{\Theta })=\sum _{l\in R_m}{c}_{lm}\sum _{n=1}^{N_l}log\left(\sum _{i=1}^I{\pi }_i·t\left({{y}}_{l,n}\right|{{\mu }}_i,{{A}}_i{{A}}_i^{\mathrm{T}}+{{D}}_i,{\nu }_i)\right)\end{array}$$

(21)

The derivation process of the D-MtFA algorithm is similar to that of the D-MFA, except that in Step 2, the posterior distribution of $p\left(w_{m,n,i}\right|{{y}}_{m,n},z_{m,n,i})$ and $p\left({{u}}_{m,n}\right|{{y}}_{m,n},z_{m,n,i},w_{m,n,i})$ should be computed, and in Step 5, ${\nu }_i$ needs to be estimated. We put this derivation in the Appendix in detail and directly describe the D-MtFA algorithm here.

Step 1 (Initialization): This initializes the values of the parameters ${\{{\pi }_i,{{\mu }}_i,{{A}}_i,{{D}}_i,{\nu }_i\}}_{i=1,...,I}$. Each node l broadcasts the number of its observations $N_l$ to its neighbors. When receiving this information, each node calculates the combination coefficient by Equation (3).

Step 2 (Computation): Each node l in the sensor network computes five local sufficient statistics ${\mathrm{LSS}}_l=\{{\mathrm{LSS}}_l^{\left(1\right)}\left[i\right],{\mathrm{LSS}}_l^{\left(2\right)}\left[i\right],{\mathrm{LSS}}_l^{\left(3\right)}\left[i\right]$, ${\mathrm{LSS}}_l^{\left(4\right)}\left[i\right],{\mathrm{LSS}}_l^{\left(5\right)}{\left[i\right]\}}_{i=1,...,I}$ according to its observations ${{Y}}_l$, given as:

$$\begin{array}{ccc}\hfill {\mathrm{LSS}}_l^{\left(1\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle \hfill \\ \hfill {\mathrm{LSS}}_l^{\left(2\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle ·\langle w_{l,n,i}\rangle \hfill \\ \hfill {\mathrm{LSS}}_l^{\left(3\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle ·\langle w_{l,n,i}\rangle ·{{y}}_{l,n}\hfill \\ \hfill {\mathrm{LSS}}_l^{\left(4\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle ·\langle w_{l,n,i}\rangle ·{{y}}_{l,n}{{y}}_{l,n}^{\mathrm{T}}\hfill \end{array}$$

$$\begin{array}{ccc}\hfill {\mathrm{LSS}}_l^{\left(5\right)}\left[i\right]& =& \sum _{n=1}^{N_l}\langle z_{l,n,i}\rangle ·log\langle w_{l,n,i}\rangle \hfill \end{array}$$

(22)

The expressions of expectations $\langle z_{l,n,i}\rangle$ and $\langle w_{l,n,i}\rangle$ in Equation (22) are given in the Appendix. Moreover, the intermediate variables ${{\Omega }}_i$ and ${{g}}_i$ should also be prepared for simplifying expressions in Step 5, shown in the Appendix.

Step 3 (Diffusion): Each node l in the sensor network diffuses its local sufficient statistics ${\mathrm{LSS}}_m$, as shown in Figure 1.

Step 4 (Combination): When node m ($m=1,...,M$) receives the local sufficient statistics from all of its one-hop neighbor nodes l ($l\in R_m$), it calculates the combined sufficient statistics, shown as:

$${\mathrm{CSS}}_m^{\left(H\right)}\left[i\right]=\sum _{l\in R_m}{c}_{lm}·{\mathrm{LSS}}_l^{\left(H\right)}\left[i\right]H=1,2,3,4,5$$

(23)

Step 5 (Estimation): Node m $(m=1,...,M)$ estimates the parameters of the MtFA:

$$\begin{array}{ccc}\hfill {\pi }_i& =& \frac{{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]}{{\sum }_{i^{\prime }=1}^I{\mathrm{CSS}}_m^{\left(1\right)}\left[i^{\prime }\right]}\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill {{\mu }}_i& =& \frac{{\mathrm{CSS}}_m^{\left(3\right)}\left[i\right]}{{\mathrm{CSS}}_m^{\left(2\right)}\left[i\right]}\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill {{A}}_i& =& {{V}}_i{{g}}_i{({{g}}_i^{\mathrm{T}}{{V}}_i{{g}}_i+{{\Omega }}_i)}^{-1}\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill {{D}}_i& =& \mathrm{diag}\left\{{{V}}_i-{{A}}_i({{g}}_i^{\mathrm{T}}{{V}}_i{{g}}_i+{{\Omega }}_i){{A}}_i^{\mathrm{T}}\right\}\hfill \end{array}$$

(27)

where:

$${{V}}_i=\frac{{\mathrm{CSS}}_m^{\left(4\right)}\left[i\right]-2{\mathrm{CSS}}_m^{\left(3\right)}\left[i\right]·{{\mu }}_i^{\mathrm{T}}+{\mathrm{CSS}}_m^{\left(2\right)}\left[i\right]·{{\mu }}_i{{\mu }}_i^{\mathrm{T}}}{{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]}$$

(28)

In addition, ${\nu }_i$ is updated by solving the following equation:

$$\begin{array}{c}\hfill log\left(\frac{{\nu }_i}{2}\right)-\psi \left(\frac{{\nu }_i}{2}\right)+1-\frac{{\mathrm{CSS}}_m^{\left(5\right)}\left[i\right]-{\mathrm{CSS}}_m^{\left(2\right)}\left[i\right]}{{\mathrm{CSS}}_m^{\left(1\right)}\left[i\right]}-log\left(\frac{{\nu }_i^{\mathrm{old}}+p}{2}\right)+\psi \left(\frac{{\nu }_i^{\mathrm{old}}+p}{2}\right)=0\end{array}$$

(29)

where $\psi (·)$ is the digamma function and ${\nu }_i^{\mathrm{old}}$ is the value of ${\nu }_i$ on the last iteration of this algorithm. Equation (29) can be solved by some numerical methods, i.e., the Newton method.

Step 6 (Termination): Node m ($m=1,...,M$) calculates its current local log-likelihood $logp\left({{Y}}_l\right|{{\Theta }}^{\mathrm{new}})$, expressed in Equation (21). The superscript “new” denotes the newly-estimated parameters at the current iteration. The termination condition of the algorithm is the same as that in the D-MFA algorithm.

4. Experimental Results

4.1. Synthetic Data

In this subsection, we test the performances of the proposed algorithms on synthetic data. Here, we consider a sensor network composed of 100 nodes to evaluate the estimation performance of the proposed algorithms. Nodes are randomly placed in a square of $5\times 5$. The communication distance is taken as 0.8. In this setting, the connected graph reflecting network topology is shown in Figure 2.

Figure 2. Network connection.

Figure 2. Network connection.

In the first 30 nodes (Node 1–Node 30), each node has 80 observations. In the next 40 nodes (Node 31–Node 70), each node contains 100 observations. In the last 30 nodes (Node 71–Node 100), each node has 120 observations. All of the 10-dimensional observations in the 100 nodes are assumed to be generated from three-component Gaussian mixtures. The parameters are as follows:

$$\begin{array}{c}({\pi }_1,{\pi }_2,{\pi }_3)=(0.3,0.5,0.2)\hfill \\ {{\mu }}_1=\left(3333300000\right), {{\mu }}_2=\left(0000000000\right),\hfill \\ {{\mu }}_3=(-3-3-3-3-300000);\hfill \\ {{\Sigma }}_1=\mathrm{diag}(111110.10.10.10.10.1),\hfill \\ {{\Sigma }}_2={{\Sigma }}_1, {{\Sigma }}_3={{\Sigma }}_1.\hfill \end{array}$$

We adopt several models to represent the distributions of these observations, and the task is to estimate parameters in the models. Here, we compare the performance of four schemes. In the first scheme, the standard EM algorithm for the MFA (S-MFA) is implemented in a centralized unit using all observations from 100 nodes. In the second scheme, the D-MFA algorithm proposed in Section 3.2 performs simultaneously in all nodes. In the third scheme, the EM algorithm for the MFA runs in each node using only local observations of that node. In other words, there is no information exchange among nodes. We abbreviate it as non-cooperation MFA (NC-MFA) for description convenience. In the last scheme, the distributed EM algorithm for the GMM (D-GMM) is implemented. In the D-GMM, the objective function is similar to that of the proposed D-MFA, except that MFA is replaced by GMM. It should be emphasized that the centralized unit is assumed to be always reliable in the S-MFA under ideal conditions. However, this condition is not always fulfilled when the centralized unit fails. Therefore, the S-MFA is seldom adopted in sensor networks. The aim is to test whether the estimation performance of the D-MFA can approach that of the S-MFA.

In the initialization of these MFA schemes, the dimension of factors are set to five. $({\pi }_1^0,{\pi }_2^0,{\pi }_3^0)=(1/3,1/3,1/3)$, $\{{{\mu }}_1^0,{{\mu }}_2^0,{{\mu }}_3^0\}$ are set as randomly-selected observations in the those nodes. The initial elements in $\{{{D}}_1^0,{{D}}_2^0,{{D}}_3^0\}$ and $\{{{A}}_1^0,{{A}}_2^0,{{A}}_3^0\}$ are generated by standardized normal distributions. In order to make the estimation results visible, the principal component analysis is performed for observations, obtaining the two largest eigenvalues and the associated eigenvectors. Then, the observations, the estimated means ${{\mu }}_i(i=1,2,3)$ and the covariances ${{\Sigma }}_i({{\Sigma }}_i={{A}}_i{{A}}_i^{\mathrm{T}}+{{D}}_i)$ after the termination of the algorithms can be projected into 2D principal subspace [34]. Figure 3 illustrates the results of the estimated parameters at the 2D principal subspace in these four schemes. In this figure, the estimated mean ${{\mu }}_i$ of each component is denoted by “+”, and the estimated covariance ${{\Sigma }}_i$ is represented by shaded ellipse. Concretely, in Figure 3a, parameters can be correctly estimated by the S-MFA, as the centralized unit can use all of the observations directly. In Figure 3b–d, the results of a randomly-selected node are given. For the NC-MFA, the appropriate parameters are incorrectly estimated, as it can only use its own observations, which also happens in other nodes. For the D-GMM, as it is based on GMM, it cannot describe and process high-dimensional observations well. Finally, in the D-MFA, each node can receive the calculated LSS from nodes in its neighborhood set and combine them for parameter estimation. Compared to GMM, MFA can reflect the properties of these high-dimensional observations more accurately. Therefore, the estimated means and covariances are correct in the D-MFA, as shown in Figure 3b. The other nodes have the same results as this selected node, which are not given here due to space limitation. Moreover, as the same observations and models are used in three MFA schemes, the changes of the average log-likelihood over all nodes in the S-MFA, log-likelihood of the D-MFA and the NC-MFA are shown in different lines in Figure 4. We can see that as the iteration increases, the D-MFA is convergent. Its convergence performance approaches that of the S-MFA.

In order to further show the estimation accuracy of the D-MFA at all of the nodes in a sensor network, we select two kinds of parameters $({\pi }_1,{\pi }_2,{\pi }_3)$ and ${{\mu }}_1$, giving the estimation results of these parameters. In Figure 5, the estimated $({\pi }_1,{\pi }_2,{\pi }_3)$ in the D-MFA and the NC-MFA at all 100 nodes are provided. In the NC-MFA, each node cannot correctly estimate parameters due to limited observations and no information exchange with other nodes, shown by dashed lines. On the contrary, after the D-MFA converges, the estimated values $({\pi }_1,{\pi }_2,{\pi }_3)$ in all 100 nodes approach their true values $(0.3,0.5,0.2)$. In Figure 6, we compare all of the vector components in ${{\mu }}_1$ estimated by these three MFA schemes. It is noted that for the D-MFA and the NC-MFA, we give the mean and standard deviation of each vector component over 100 nodes to reflect the whole performance of network. We can clearly see that the S-MFA can correctly estimate ${{\mu }}_1$, as it can use all of the observations. For the D-MFA, the mean of estimated ${{\mu }}_1$ in each vector component approaches the corresponding true value $\left(3333300000\right)$, while the mean of ${{\mu }}_1$ obtained by the NC-MFA is not consistent with true value. Moreover, the standard deviation of D-MFA is smaller than that of the NC-MFA. Since the other parameters lead to similar results, we omit them here.

Figure 3. Scatter plot of observations with the estimated parameters at 2D principal subspace using different schemes: (a) S-MFA; (b) D-MFA; (c) NC-MFA; (d) D-GMM.

Figure 3. Scatter plot of observations with the estimated parameters at 2D principal subspace using different schemes: (a) S-MFA; (b) D-MFA; (c) NC-MFA; (d) D-GMM.

Figure 4. Log-likelihood changes of three MFA schemes during 30 iterations.

Figure 4. Log-likelihood changes of three MFA schemes during 30 iterations.

Figure 5. The estimated mixing proportions $({\pi }_1,{\pi }_2,{\pi }_3)$ in the D-MFA and the NC-MFA at 100 nodes.

Figure 5. The estimated mixing proportions $({\pi }_1,{\pi }_2,{\pi }_3)$ in the D-MFA and the NC-MFA at 100 nodes.

Figure 6. The mean and standard deviation of all of the vector components in estimated ${{\mu }}_1$ over 100 nodes.

Figure 6. The mean and standard deviation of all of the vector components in estimated ${{\mu }}_1$ over 100 nodes.

For the D-MtFA algorithm, we test its performance and compare it to the S-MtFA, the NC-MtFA and the D-tMM. It is noted that the S-MtFA, the NC-MtFA and the D-tMM can be realized by replacing Gaussian distributions in the S-MFA, the NC-MFA and the D-GMM with Student’s t-distributions, respectively. Here, observations are generated by mixtures of Student’s t-distributions. The parameters ${\{{\pi }_i,{{\mu }}_i,{{\Sigma }}_i\}}_{i=1,2,3}$ are unchanged while ${\nu }_1={\nu }_2={\nu }_3=5$. In Figure 7, the scatter plot of observations with the estimated parameters at the 2D principal subspace is shown. It is noted that for the D-MtFA, the NC-MtFA and the D-tMM, the results of a randomly-selected node are given. From this figure, we can see several observations located out of ordinary regions, which can be taken as outliers. The S-MtFA in the centralized unit, shown in Figure 7a, can grasp the distributions, as it can make use of all observations. On the contrary, the performance of the NC-MtFA and the D-tMM are bad. The reasons are similar to those of NC-MFA and D-GMM, which have been explained. When implementing the D-MtFA, parameters can be accurately estimated, while the property of robustness to outliers is still maintained, as shown in Figure 7b. In summary, the proposed D-MFA and the D-MtFA can accurately estimate parameters in a distributed way when each node in sensor network has part of the high-dimensional observations.

Figure 7. Scatter plot of observations with the estimated parameters at the 2D principal subspace using different schemes: (a) S-MtFA; (b) D-MtFA; (c) NC-MtFA; (d) D-tMM.

Figure 7. Scatter plot of observations with the estimated parameters at the 2D principal subspace using different schemes: (a) S-MtFA; (b) D-MtFA; (c) NC-MtFA; (d) D-tMM.

4.2. Real Data

In several countries, there are monitoring sites located in different regions, whose tasks are to detect nutritional ingredients in the wine samples. These sites form a sensor network, in which each site only communicates with its neighbors and can implement local computations. The wine samples sent to these monitoring sites may belong to different cultivars. Therefore, in each monitoring site, these samples need to be classified, which is good for analyzing in-depth the relationship of nutritional ingredients in the wine and their cultivars. It is certain that the more the references are available for each site, the better the results will be. Therefore, the network and cooperation between sites are required.

In this subsection, we consider the wine cultivar clustering problem as a simulation of the above scenario. The database of this problem is the wine dataset, which is one of the most popular datasets in the UCImachine learning repository [35]. In this wine dataset, 178 samples are collected from a chemical analysis of wines grown in three different cultivars in Italy (the No. 1∼No. 59 samples belong to the first class; the No. 60∼No.130 samples belong to the second class; the No. 131∼No. 178 samples belong to the third class). Each sample has 13 attributes, so the dimension of observations is 13. The sensor network is composed of eight nodes, represented as eight monitoring sites. The average number of nodes in the neighborhood set is two, and the graph is guaranteed to be connected. The average number of samples in each site is 22.

Clustering belongs to the unsupervised learning paradigm in machine learning. When the D-MFA (or the D-MtFA) is adopted for clustering, initial values of parameters in the D-MFA are set, which are the same as those in Section 4.1. Then, the corresponding algorithm derived in Section 3.2 (or Section 3.3) is performed. After algorithm converges, an additional computation step based on the estimated parameters Θ at node m is carried out to obtain $\langle z_{m,n,i}\rangle$ by Equation (6). Finally, the cluster decision for each observation ${{y}}_{m,n}$ is:

$${𝒞}_{m,n}={\mathrm{argmax}}_{i=1}^I\langle z_{m,n,i}\rangle .m=1,...M,n=1,...,N_m$$

The clustering results of the D-MFA at Node 1∼Node 8 are shown in Figure 8. In this figure, blue “∘” represents correctly-clustered observations, while red “×” denotes wrongly-clustered ones. From these figures, we can see that the correct ratio in eight nodes are $100\%,100\%,95.2\%,95.5\%,100\%,95.5\%,100\%,92.9\%$. There are five wrongly-clustered observations in all. The correct ratio in the entire network is $97.2\%$. In order to compare the performance of the D-MFA with that of the S-MFA, we perform the D-MFA and the S-MFA algorithms 20 times. The average correct ratio of these 20 runs by the D-MFA is $96.9\%$, approaching that by the S-MFA, which is $98.2\%$. The reason for the small performance gap between the S-MFA and the D-MFA may be that the number of observations for each node is small and the dimension is relatively high. The accuracy of the calculated LSS in Step 2 of the D-MFA are a little worse than those global sufficient statistics obtained by all of the observations in the E-step of the S-MFA. For the NC-MFA, as the number of observations in this example at each node is small, the clustering cannot be implemented. For the D-GMM, as the dimension of observations is high, it also cannot finish the task of this example effectively. Moreover, as there are no outliers in this dataset, the clustering result of the D-MtFA is the same as that of the D-MFA, which is not shown here. In summary, we can use the proposed schemes to realize distributed clustering.

Figure 8. Clustering results of the wine dataset at (a–h) Node 1∼Node 8.

Figure 8. Clustering results of the wine dataset at (a–h) Node 1∼Node 8.

5. Conclusions

In this paper, we propose a distributed density estimation method base on a mixture of factor analyzers in sensor networks. First, a linear combination of local log-likelihoods associated with nodes in its neighborhood (including itself) is defined as the objective function. In this objective function, the combination coefficients are determined by the number of observations in corresponding nodes. Then, the D-MFA and the D-MtFA algorithms are derived. In these algorithms, the combined sufficient statistics of each node are used to estimate parameters, which can be obtained by performing a linear combination of local sufficient statistics from nodes in its neighborhood. Finally, we evaluate the performances of the proposed algorithms and apply them to the tasks of clustering and classification. Experimental results show that they are promising and effective statistical tools for processing the high-dimensional datasets in a distributed way in sensor networks.

In our future work, we will investigate distributed algorithms that can automatically determine the structure of MFA, e.g., the number of components. We also intend to design adaptive strategies to adjust combination coefficients more flexibly. Moreover, the coverage problem [36] in a sensor network is important when implementing distributed algorithms. We will consider this issue in the future.