A Nonparametric Model for Multi-Manifold Clustering with Mixture of Gaussians and Graph Consistency

Abstract

Multi-manifold clustering is among the most fundamental tasks in signal processing and machine learning. Although the existing multi-manifold clustering methods are quite powerful, learning the cluster number automatically from data is still a challenge. In this paper, a novel unsupervised generative clustering approach within the Bayesian nonparametric framework has been proposed. Specifically, our manifold method automatically selects the cluster number with a Dirichlet Process (DP) prior. Then, a DP-based mixture model with constrained Mixture of Gaussians (MoG) is constructed to handle the manifold data. Finally, we integrate our model with the k-nearest neighbor graph to capture the manifold geometric information. An efficient optimization algorithm has also been derived to do the model inference and optimization. Experimental results on synthetic datasets and real-world benchmark datasets exhibit the effectiveness of this new DP-based manifold method.

1. Introduction

Over the past decades, clustering has been the most fundamental task in many computer vision and data mining applications [1,2], e.g., image/motion segmentation [3,4], community detection [5], feature selection [6] and biological/network information analysis [7,8]. However, most of the conventional clustering methods assume that data samples are scattered in the feature space, which ignores the intrinsic underlying data structure that many real datasets have [3,9]. To overcome this problem, various manifold-based clustering (multi-manifold clustering) methods have been proposed and developed. Compared to the conventional clustering method, which regards the cluster as the data points with small distances between cluster members or dense areas of the feature space, the multi-manifold approach aims to gather the given data points into disparate groups, which come from different underlying submanifolds [10].

Unlike the conventional clustering methods [11,12], multi-manifold clustering can be classified into two different categories, the linear method and the nonlinear method [13]. In the first category, linear methods (also known as subspace clustering) construct the multi-manifold clustering by assuming that the underlying cluster can be well approximated by a union of low dimensional linear manifolds [14]. For example, Gholami [14] and Vidal [15] used a linear function to fit the underlying submanifold and cluster the clusters with the mixture model. Sparse Subspace Clustering (SSC)- [16], Low-Rank Representation (LRR)- [17] and Least Squares Regression (LSR)-based [18] methods approach the linear manifold clustering problem by finding a sparse representation of each point in terms of other data points. After forming a similarity graph with the learned sparse representation, spectral clustering methods are used to cluster data into distinctive clusters. As an expanding framework of the linear multi-manifold clustering methods, non-linear algorithms can be naturally applied to linear and/or nonlinear manifolds. For example, the K-manifold clusters the nonlinear subspace dataset by expanding the conventional K-means with geodesic distance [19]. Spectral Multi-Manifold Clustering (SMMC) integrates the local geometric information within the subspace clustering framework to handle the manifold structure [13], Multi-Manifold Matrix Decomposition for Co-clustering (M3DC) handles the manifold dataset by considering the geometric structures of both the sample manifold and the feature manifold simultaneously [20]. Recently, the state-of-the-art method may be deep subspace clustering, which assembles the deep framework and the conventional subspace clustering method [21,22].

However, a drawback of most conventional manifold clustering methods is that the clustering accuracy depends on the cluster number, which is always unavailable in advance [23]. To overcome this model selection problem, one category of the most widely-studied methods is that equipping the conventional methods with a Dirichlet process prior, e.g., Dirichlet Process Mixture (DPM) models [24,25], Multilevel Clustering with Context (${\mathrm{MC}}^2$) [26] and Dirichlet Process Variable Clustering (DPVC) [27]. Since the distributions adopted in these nonparametric models are defined in the Euclidean space, those conventional Dirichlet process clustering methods suffer difficulty when dealing with the manifold data. To overcome this problem, many manifold DP clustering models have been proposed. Wang [28] and Gholami [14] assumed that the submanifold is lying on the linear manifold and can be fitted with the hyperplane. Straub et al. [29,30] defined the Gaussian distribution on the sphere surface and introduced an auxiliary indicator vector zwith a DP prior. More than the sphere manifold, Simo et al. [31] expanded the distribution to the manifold space with the logarithmic and exponential mapping. Although these models are quite powerful and have been widely studied in many applications, they have their drawbacks when the manifold structure is not prespecified [31]. For example, the DP-space and temporal subspace clustering model is an expanding method of the linear manifold clustering method. It lacks the capability to handle a non-linear manifold dataset. In the geodesic mixture model, the logarithmic and exponential mapping algorithms [32,33] used in this model depend mainly on the pre-defined geometric structure, which is always unavailable. For the sphere mixture model, the sphere manifold has not been extended to arbitrary manifolds [31].

In this paper, we investigate the manifold clustering method with no prespecified manifold structure and cluster number in the DPM framework. In order to model the complicated manifold cluster distributions, we integrate the original DPM with the conventional Mixture of Gaussians (MoG) [34,35] to handle the manifold distribution (Figure 1a). Furthermore, we also notice that an unconstrained MoG distribution fails to capture the manifold geometrical information (Figure 1b). Inspired by the previous study [23,36], we regularize our model with a k-nearest neighbor graph. To form a meaningful cluster, in which samples from the same cluster are closed and related, we constrain the MoG mean with a Mahalanobis distance.

The main contributions are as follows:

• A constrained MoG distribution has been applied to model the non-Gaussian manifold distribution.
• We integrate the graph theory with DPM to capture the manifold geometrical information.
• The variational inference-based optimization framework is proposed to carry out the model inference and learning.

The organization of our paper proceeds as follows. In Section 2, we review the background knowledge of the Dirichlet process mixture model. Simultaneously, we present the generation procedure of the proposed manifold Dirichlet process mixture model and give the variational expectation maximization inference algorithm. Experimental comparisons will be presented in Section 3. In Section 4, we give the detailed discussions and present the limitations and advantages. Section 5 concludes the paper.

2. Materials and Methods

In this section, we firstly review the basic concept of the Dirichlet Process Mixture (DPM) model. Then, we propose the multi-manifold clustering method by equipping DPM with MoG and the k-nearest neighbor graph. In our method, the Dirichlet process is used to generate the suitable cluster number. MoG and the k-nearest neighbor graph are applied to model the non-Gaussian manifold distribution and capture the manifold geometric information. Finally, variational inference is derived to do the model inference and learning.

2.1. Dirichlet Process Mixture Model

The Dirichlet Process Mixture (DPM) model is an approach that extends the mixture model by introducing a Dirichlet process prior within the Bayesian framework. In DPM, we firstly sample a prior distribution G from the Dirichlet process and then sample the likelihood parameters ${\left\{{\mathit{\theta }}_n\right\}}_{n=1}^N$ from G. With the sampled likelihood parameters, observation data $x_n$ can be generated from the likelihood distribution $F(x|{\mathit{\theta }}_n)$. This procedure can be concluded as follows:

$$\begin{array}{c}G|G_0(\mathit{\lambda })\sim DP(G_0(\mathit{\lambda }),\alpha )\hfill \\ {\mathit{\theta }}_n|G\sim Gn=1,2,3,\dots ,N\hfill \\ {\mathit{x}}_n\sim F(\mathit{x}|{\mathit{\theta }}_n)n=1,2,3,\dots ,N,\hfill \end{array}$$

(1)

where $F(\mathit{x}|{\mathit{\theta }}_n)$ is a likelihood distribution and $G_0$ is a base distribution. $x_n$ is the observation sample.

By integrating out G, the joint distribution of the likelihood parameters ${\left\{{\mathit{\theta }}_n\right\}}_{n=1}^N$ exhibits a clustering effect. Suppose that we have $N-1$ parameters ${\left\{{\mathit{\theta }}_n\right\}}_{n=1}^{N-1}$ sampled from our Dirichlet process. We then have the following probability for the N-th value of $\mathit{\theta }$.

$$\begin{array}{c}\hfill p({\mathit{\theta }}_N|{\left\{{\mathit{\theta }}_n\right\}}_{n=1}^{N-1})=\frac{\alpha G_0}{\alpha +N-1}+{\sum }_{i=1}^I\frac{n_i\mathit{\delta }(i)}{\alpha +N-1},\end{array}$$

(2)

where $n_i$ denotes the $\mathit{\theta }$ frequency of occurrence in ${\left\{{\mathit{\theta }}_n\right\}}_{n=1}^{N-1}$ and $\mathit{\delta }(j)$ represents the delta function. I denotes the number of unique values in ${\left\{{\mathit{\theta }}_n\right\}}_{n=1}^N$. (2) reveals the fact that a new sample ${\mathit{\theta }}_n$ is either generated from a new cluster with probability $G_0$ or extracted from the existing clusters ${\left\{{\mathit{\theta }}_n\right\}}_{n=1}^{N-1}$ with probability $n_i/(\alpha +N-1)$.

2.2. Our Proposed Method

In this section, we expand the original DPM model to a multi-manifold clustering framework named the Similarity Dirichlet Process Mixture (SimDPM) model. The main notations and descriptions used in our method are summarized in

Table 1. The main notations and descriptions.

Notations	Descriptions
$\mathit{\lambda }$	Hyper parameter ${\mathit{u}}_0,c_0,{\mathit{W}}_0,{\nu }_0$ of normal-Wishart
M	The mixture number
N	Number of the observation samples
${\mathit{\theta }}_k^{*}$	Gaussian parameter $\mathit{u},\mathit{\delta }$
${\tilde{\mathit{\theta }}}_k$	MoG parameter ${\tilde{\mathit{u}}}_k,{\tilde{\mathit{\delta }}}_k,{\tilde{\mathit{\pi }}}_k$
s	Tradeoff parameter
K	The maximum cluster number
$\alpha$	Parameter of the Beta distribution
$z_n^{(1)}$	Class indicator of a Gaussian distribution
$z_n^{(2)}$	Class indicator of the MoG distribution
X	Unlabeled dataset
${\mathit{\gamma }}_k$	Variational parameter ${\mathit{\gamma }}_{k,1},{\mathit{\gamma }}_{k,2}$
${\mathit{\tau }}_k$	Variational parameter of normal-Wishart
${\mathsf{\Phi }}_n$	Variational parameter of categorical distribution
A	An auxiliary parameter that equals $\mathsf{\Phi }$
${\mathit{\lambda }}_A$	Penalty parameter used in the graph Laplacian
L	Graph Laplacian
$L^k$	k-nearest neighbor graph
$D^k$	Diagonal matrix whose entries are column sums of $L^k$
R	Posterior penalty term with graph Laplacian
r	Neighbor number used in the graph Laplacian

.

As we have debated, DPM is unable to model the manifold dataset since the conventional likelihood distribution $F(\mathit{x}|{\mathit{\theta }}_n)$ is defined in the Euclidean space or prespecified manifold. To overcome this problem, we approximate the manifold distribution with MoG (Figure 1a). Then, we construct the sample generation process with two phases, a single Gaussian distribution and a mixture of Gaussians distribution. The reason we generate the data with both the single Gaussian distribution and the MoG distribution is that some simple submanifolds and non-manifold clusters can be modeled by the single Gaussian distribution.

Suppose that we are given N observation samples $\mathit{X}={\left\{{\mathit{x}}_n\right\}}_{n=1}^N$ where $x_n\in {\mathbb{R}}^D$. Given the additional parameters of the MoG distribution, we assume the following generative process for each observation data ${\mathit{x}}_{\mathit{i}}$:

• For $i=1,2,3,\dots ;$ draw $v_i|\alpha \sim Beta(1,\alpha )$
• For $i=1,2,3,\dots ;$ draw ${\mathit{\theta }}_i^{*}|G_0\sim G_0$
• For every data point i:

  (a)

  Choose $z_i^{(1)}|\mathit{v}\sim mult(s\mathit{\pi }(\mathit{v}))$

  (b)

  Choose $z_i^{(2)}|\mathit{v}\sim mult((1-s)\mathit{\pi }(\mathit{v}))$

  (c)

  Draw ${\mathit{x}}_i|z_i^{(1)}\sim N(\mathit{x}|{\mathit{\theta }}_{z_i^{(1)}}^{*})$

  (d)

  Draw ${\mathit{x}}_i|z_i^{(2)}\sim MoG(\mathit{x}|{\tilde{\mathit{\theta }}}_{z_i^{(2)}})$

where $Beta(1,\alpha )$ is a beta distribution with parameter one and $\alpha$, $mult(\mathit{\pi }(\mathit{v}))$ is a categorical distribution parameterized by $\mathit{\pi }(\mathit{v})$, $\mathit{v}$ and $\mathit{\pi }(\mathit{v})$ are vectors with $\mathit{v}={\left\{v_k\right\}}_{k=1}^{\infty }$ and $\mathit{\pi }(v_i)=v_i{\prod }_{j=1}^{i-1}(1-v_j)$ and $G_0$ is a normal-Wishart distribution with parameter $\mathit{\lambda }=({\mathit{u}}_0,c_0,{\mathit{W}}_0,v_0)$ where ${\mathit{u}}_0\in {\mathbb{R}}^D$ ${\mathit{W}}_0\in {\mathbb{R}}^{D\times D}$, ${\mathit{\theta }}_{z_i^{(1)}}^{*}$ is a Gaussian distribution parameterized by ${\mathit{u}}_{z_i^{(1)}}\in {\mathbb{R}}^D,{\mathit{\delta }}_{z_i^{(1)}}\in {\mathbb{R}}^{D\times D}$. ${\tilde{\mathit{\theta }}}_{z_i^{(2)}}$ is the MoG distribution with parameter ${\tilde{\mathit{u}}}_{z_i^{(2)}}\in {\mathbb{R}}^D,{\tilde{\mathit{\delta }}}_{z_i^{(2)}}\in {\mathbb{R}}^{D\times D},{\tilde{\mathit{\pi }}}_{z_i^{(2)}}\in {\mathbb{R}}^M$ where M denotes the mixture number in every ${\tilde{\mathit{\theta }}}_{z_i^{(2)}}$. Tradeoff parameter s denotes how likely the observation sample $x_n$ is sampled from a single Gaussian distribution. The corresponding probability graph model representation of manifold DPM can be described as Figure 2.

To form a meaningful cluster (samples from the same cluster are closely related) and respect the manifold geometrical information, we constrain the MoG mean with:

$$\frac{1}{2}{({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})}^T{\tilde{\mathit{\delta }}}_{k,m}^{-1}({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})ϵ,m1,$$

and use a k-nearest neighbor graph to regularize the posterior probability inspired by [23], in which the graph Laplacian is used to capture the geometric information that has been missed by the MoG distribution.

$$\begin{array}{c}\hfill R=\sum _{k=1}^{\infty }p{(k|\mathit{X})}^{T}Lp(k|\mathit{X}),\end{array}$$

(3)

where $p(k|\mathit{X})={\left\{p(k|{\mathit{x}}_{\mathit{n}})\right\}}_{n=1}^N$ is the posterior probability. L is the graph Laplacian constructed by the k-nearest neighbor graph [13]. Note that the constraint of ${\tilde{\mathit{u}}}_{k,m}$ depends only on the previous ${\tilde{\mathit{u}}}_{k,m-1}$, but not on ${\tilde{\mathit{u}}}_{k,m+1}$. Below, we characterize the k-nearest neighbor graph $L^k$. Given the unlabeled data X, for any point $x_i$, we sort the rest of the data samples and select the top-k nearest neighbors. If node $x_j$ is in the top-k nearest points of node $x_i$, we set:

$$L_{i,j}^k=e^{\frac{-\Vert x_i-x_j{\Vert }^2}{Te}}.$$

Here, we define the L as the equation $L=D^k-L^k$. $D^k$ is a diagonal matrix whose entries are column (or row, since Sis symmetric) sums of $L^k$. For convenience, the neighbor number used in our graph is denoted as r.

2.3. Variational Expectation Maximization Inference

Our scheme for estimating the data cluster depends mainly on our capability to infer the posterior distribution. We solve this using variational expectation maximization inference.

Unlike the conventional expectation maximization algorithm, the posterior probability in our model will be estimated via the variational inference, and then, we optimize the MoG parameter by maximizing the lower bound with the fixed variational parameter. Following the general variational inference framework, we firstly give the Evidence Lower BOund (ELBO) for the SimDPM with the truncated stick-breaking process (when applying this process, the maximum cluster number ∞ is truncated to K) [37].

$$\begin{array}{cc}\hfill \log (p(\mathit{X}|\alpha ,\mathit{\lambda },\tilde{\mathsf{\Theta }})-{\mathit{\lambda }}_{R}R\ge & E_q\left[\log p({\mathsf{\Theta }}^{*}|\mathit{\lambda })\right]+\sum _{n=1}^N{E}_q\left[\log p(z_n^{(1)},z_n^{(2)}|\mathit{V})\right]+\sum _{n=1}^N{E}_q\left[\log p(x_n|{\mathsf{\Theta }}^{*},z_n^{(1)})\right]\hfill \\ & -E_q\left[\log q(\mathit{v},{\mathsf{\Theta }}^{*},{\mathit{z}}^{(\mathit{1})},{\mathit{z}}^{(\mathit{2})})\right]+\sum _{n=1}^N{E}_q\left[\log p({\mathit{x}}_n|\tilde{\mathsf{\Theta }},z_n^{(2)})\right]\hfill \\ & +E_q\left[\log p(\mathit{v}|\alpha )\right]-{\mathit{\lambda }}_{R}R,\hfill \end{array}$$

(4)

where $\mathit{X}={\left\{{\mathit{x}}_n\right\}}_{n=1}^N$ is the observation sample, $p({\mathsf{\Theta }}^{*}|\mathit{\lambda })$ is a normal-Wishart distribution with hyper parameters $\mathit{\lambda }=({\mathit{u}}_0,c_0,{\mathit{W}}_0,v_0)$ and $p({\mathit{x}}_n|\tilde{\mathsf{\Theta }})$ is the constrained MoG distribution parameterized by $\tilde{\mathsf{\Theta }}={\left\{{\tilde{\mathit{\theta }}}_k\right\}}_{k=1}^K$ where ${\tilde{\mathit{\theta }}}_k=\{{\tilde{\mathit{u}}}_k,{\tilde{\mathit{\delta }}}_k,{\tilde{\mathit{\pi }}}_k\}$, $p({\mathit{x}}_n|{\mathsf{\Theta }}^{*})$ is a single Gaussian distribution. ${\mathit{z}}^{(\mathit{1})}={\left\{z_n^{(1)}\right\}}_{n=1}^N$ and ${\mathit{z}}^{(\mathit{2})}={\left\{z_n^{(2)}\right\}}_{n=1}^N$ are the indicator variables sampled from the categorical distribution $p(z_n^{(1)},z_n^{(2)}|\mathit{v})$. Following the factorized family variational inference [37], which can make the posterior distribution computable, q can be expressed as:

$$\begin{array}{cc}\hfill q(\mathit{v},{\mathsf{\Theta }}^{*},{\mathit{z}}^{(\mathit{1})},& {\mathit{z}}^{(\mathit{2})})=\prod _{k=1}^{K-1}{q}_{{\mathit{\gamma }}_k}(v_k)\prod _{k=1}^K{q}_{{\mathit{\tau }}_k}({\mathit{\theta }}_k^{*})\prod _{n=1}^N{q}_{{s\mathit{\varphi }}_n}(z_n^{(1)})\prod _{n=1}^N{q}_{{(1-s)\mathit{\varphi }}_n}(z_n^{(2)}),\hfill \end{array}$$

(5)

where $q_{{\mathit{\gamma }}_k}(v_k)$ is the Beta distribution with ${\mathit{\gamma }}_k=\{{\mathit{\gamma }}_{k,1},{\mathit{\gamma }}_{k,2}\}$ and $q_{{\mathit{\tau }}_t}({\mathit{\theta }}_t^{*})$ is a normal-Wishart distribution with the parameter ${\mathit{\tau }}_k=\{{\mathit{u}}_k,c_k,{\mathit{W}}_k,v_k\}$. For $q_{s{\mathit{\varphi }}_n}(z_n^{(1)})$ and $q_{(1-s){\mathit{\varphi }}_n}(z_n^{(2)})$, we denote it as two categorical distributions with parameter ${\mathit{\varphi }}_n={\left\{{\varphi }_{n,k}\right\}}_{k=1}^K$ ($\mathsf{\Phi }={\left\{{\mathit{\varphi }}_n\right\}}_{n=1}^N\in {\mathbb{R}}^{N\times K}$). s is the tradeoff parameter.

For derivation convenience, we denote ELBO as $L(\mathit{\gamma },\mathit{\tau },\mathsf{\Phi },\tilde{\mathsf{\Theta }})$. By using this inequality relaxation, we note that learning the model and estimating the model parameters are altered to maximize the following equation.

$$\begin{array}{c}\underset{\{{\mathit{\gamma }}_k,{\mathit{u}}_k,c_k,{\mathit{W}}_k,v_k,{\tilde{\mathit{u}}}_k,{\tilde{\mathit{\delta }}}_k,{\tilde{\mathit{\pi }}}_k\}}{\arg \max }L(\mathit{\gamma },\mathit{\tau },\mathsf{\Phi },\tilde{\mathsf{\Theta }})-{\mathit{\lambda }}_{R}R\hfill \\ s.t.\frac{1}{2}{({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})}^T{\tilde{\mathit{\delta }}}_{k,m}^{-1}({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})ϵ,m1,\hfill \\ R=\sum _{k=1}^{\infty }p{(k|\mathit{X})}^{T}Lp(k|\mathit{X}).\hfill \end{array}$$

We also notice that, since we have truncated the maximum cluster number to K, the penalty term R is altered to be $R={\sum }_{k=1}^{K}p{(k|\mathit{X})}^{T}Lp(k|\mathit{X})$.

Variational E-step: In the variational inference framework, the variational parameter can be estimated by maximizing the lower bound of likelihood function $\log p(\mathit{X}|\alpha ,\mathit{\lambda })$ with the coordinate ascent algorithm.

For ${\mathit{\varphi }}_{n,k}$ in ${\left\{{\mathit{\varphi }}_{n,k}\right\}}_{k=1}^K$, note that this is a constrained maximization since ${\sum }_{k=1}^K{\mathit{\varphi }}_{n,k}=1$, and the probability $p(k|\mathit{X})$ can be approximated by the variational parameter ${\mathsf{\Phi }}_{.,k}$. To solve this problem, we use an auxiliary variable ${\mathit{A}}_k={\mathsf{\Phi }}_{.,k}$ where $A_k\in {\mathbb{R}}^N$ and form the Lagrangian by isolating the terms in ELBO, which contain ${\varphi }_{n,k}$ as:

$$\begin{array}{cc}\hfill \underset{{\mathit{\varphi }}_n}{\arg \max }& L(\mathit{\gamma },\mathit{\tau },\mathsf{\Phi },\tilde{\mathsf{\Theta }})+{\mathit{\lambda }}_L(\sum _{k=1}^K{\mathit{\varphi }}_{n,k}-1)-{\mathit{\lambda }}_R\sum _{k=1}^K{\mathit{A}}_k^{T}L{\mathit{A}}_k,\hfill \\ & s.t.{\mathit{A}}_k={\mathsf{\Phi }}_{.,k},\hfill \end{array}$$

(6)

where ${\mathit{\lambda }}_L$ is a Lagrangian multiplier and ${\mathit{\lambda }}_R$ is a penalty parameter.

Fix ${\mathit{A}}_k$ to update ${\mathsf{\Phi }}_{.,k}$. The updating rule for ${\varphi }_{n,k}$ can be achieved by taking the derivation.

$$\begin{array}{cc}\hfill \log {\mathit{\varphi }}_{n,k}\propto & s{E}_q\left[\log p({\mathit{x}}_n|{\mathit{\theta }}_k^{*})\right]+(1-s)\log \mathrm{p}({\mathit{x}}_n|{\tilde{\mathit{\theta }}}_k)\hfill \\ & +\sum _{jk,n}^{}(\mathsf{\Psi }({\mathit{\gamma }}_{j,2})-\mathsf{\Psi }({\mathit{\gamma }}_{j,1}+{\mathit{\gamma }}_{j,2}))\hfill \\ & +\mathsf{\Psi }({\mathit{\gamma }}_{k,1})-\mathsf{\Psi }({\mathit{\gamma }}_{k,1}+{\mathit{\gamma }}_{k,2}).\hfill \end{array}$$

(7)

We now fix $\mathsf{\Phi }$ to update ${\mathit{H}}_k$.

$$\begin{array}{c}\underset{{\mathit{A}}_k}{\arg \min }{\mathit{A}}_k^{T}L{\mathit{A}}_k+{\mathit{\lambda }}_A\Vert {\mathit{A}}_k-{\mathsf{\Phi }}_{.,k}^{old}{\Vert }_2^2\hfill \\ ⟹{\mathit{A}}_k={\mathsf{\Phi }}_{.,k}={\mathit{\lambda }}_H{({\mathit{\lambda }}_{A}I+L)}^{-1}{\mathsf{\Phi }}_{.,k}^{old},\hfill \end{array}$$

(8)

where ${\mathit{\lambda }}_A$ is the penalty parameter. For the other variational parameter, we can attain the following closed-form solutions when taking the derivation of the previous proposed ELBO function and setting it to zero:

$$\begin{array}{c}{\mathit{\gamma }}_{k,1}=1+\sum _{i=1}^N{\mathit{\varphi }}_{i,k},\hfill \\ {\mathit{\gamma }}_{k,2}=\alpha +\sum _{i=1}^N{\sum }_{jk}{\mathit{\varphi }}_{i,j},\hfill \end{array}$$

(9)

$$\begin{array}{c}{c}_k=c_0+N_k,\hfill \\ v_k=v_0+N_k,\hfill \end{array}$$

(10)

$$\begin{array}{c}{\mathit{u}}_k=\frac{1}{c_k}(c_0{\mathit{u}}_0+N_k{\overline{\mathit{x}}}_k),\hfill \\ {\mathit{W}}_k^{-1}={\mathit{W}}_0^{-1}+N_k{\mathit{S}}_k+\frac{c_0{N}_k}{c_0+N_k}({\overline{\mathit{x}}}_k-{\mathit{u}}_0){({\overline{\mathit{x}}}_k-{\mathit{u}}_0)}^T\hfill \end{array}$$

(11)

where $N_k$, $\mathit{S}$${}_k$ and $\stackrel{-}{\mathit{x}}$${}_k$ can be estimated as follows:

$$\begin{array}{c}{N}_k=\sum _{n=1}^{N}s{\mathit{\varphi }}_{n,k},\hfill \\ {\overline{\mathit{x}}}_k=\frac{s}{N_k}\sum _{n=1}^N{\mathit{\varphi }}_{n,k}{\mathit{x}}_n,\hfill \\ {\mathit{S}}_k=\frac{s}{N_k}\sum _{n=1}^N{\mathit{\varphi }}_{n,k}({\mathit{x}}_n-{\overline{\mathit{x}}}_k){({\mathit{x}}_n-{\overline{\mathit{x}}}_k)}^T.\hfill \end{array}$$

(12)

For the prior parameters ${\mathit{u}}_0,c_0,{\mathit{W}}_0,v_0$, we use them in a non-informative manner to make them influence as little as possible the inference of the variational posterior distributions. For the other variational parameters, we initialize them in a random way.

Variational M-step: To optimize the lower bound parameter $\tilde{\mathit{\theta }}$, we apply the EM framework again, in which we introduce an auxiliary posterior variable $q(k,m|{\mathit{x}}_n)$ and Jensen’s inequality [37].

$$\begin{array}{c}L(\mathit{\gamma },\mathit{\tau },\mathsf{\Phi },\tilde{\mathsf{\Theta }})+H\sum _{m=2}^M\left\{{({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})}^T{\tilde{\mathit{\delta }}}_{k,m}^{-1}({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})\right\}\hfill \\ \ge C+\sum _{n=1}^N\sum _{k,m=1}^{K,M}\left\{(1-s){\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)\log \frac{{\tilde{\mathit{\pi }}}_{k,m}N({\mathit{x}}_n|{\tilde{\mathit{u}}}_{k,m},{\tilde{\mathit{\delta }}}_{k,m})}{q(k,m|{\mathit{x}}_n)}\right\}\hfill \\ +H\sum _{m=2}^M\left\{{({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})}^T{\tilde{\mathit{\delta }}}_{k,m}^{-1}({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})\right\},\hfill \end{array}$$

(13)

where C is a constant value with no respect to $\tilde{\mathsf{\Theta }}$ in $L(\mathit{\gamma },\mathit{\tau },\mathsf{\Phi },\tilde{\mathsf{\Theta }})$. By using the inequality relaxation, the variational M-step can be reformulated as the optimization problem:

$$\begin{array}{cc}\hfill \max & \{\sum _{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|x_n)\log \frac{{\tilde{\mathit{\pi }}}_{k,m}N(x_n|{\tilde{\mathit{u}}}_{k,m},{\tilde{\mathit{\delta }}}_{k,m})}{q(k,m|x_n)}\hfill \\ & +H{({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})}^T{\tilde{\mathit{\delta }}}_{k,m}^{-1}({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})\hfill \\ & +{\mathit{\lambda }}_L({\sum }_{m=1}^{M}q(k,m|{\mathit{x}}_n)-1)\},\hfill \end{array}$$

(14)

where ${\mathit{\lambda }}_L$ and H are the Lagrangian multipliers. We therefore achieve the following closed-form solution by taking the derivative and setting the lower bound of (14) to zero:

$$\begin{array}{c}\hfill {\tilde{\mathit{u}}}_{k,1}=\frac{{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,1|{\mathit{x}}_n){\mathit{x}}_n}{{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,1|{\mathit{x}}_n)};\end{array}$$

(15)

when m is greater than one, we have:

$$\begin{array}{c}\hfill {\tilde{\mathit{u}}}_{k,m}=\frac{{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n){\tilde{\mathit{\delta }}}_{k,m-1}{\mathit{x}}_n-H{\tilde{\mathit{u}}}_{k,m-1}}{{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)-H}.\end{array}$$

(16)

Similar to the mean parameter ${\tilde{\mathit{u}}}_{k,m}$, for ${\tilde{\mathit{\delta }}}_{k,m}$, we have:

$$\begin{array}{c}{\tilde{\mathit{\delta }}}_{k,m}=\frac{T_{k,m}}{{\sum }_{n-1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)},m1,\hfill \end{array}$$

(17)

where:

$$\begin{array}{cc}\hfill T_{k,m}& =-H({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1}){({\tilde{\mathit{u}}}_{k,m}-{\tilde{\mathit{u}}}_{k,m-1})}^T\hfill \\ & +\sum _{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)({\mathit{x}}_n-{\tilde{\mathit{u}}}_{k,m}){({\mathit{x}}_n-{\tilde{\mathit{u}}}_{k,m})}^T,\hfill \end{array}$$

since the constraint does not exist in the components where $m2$, the updating rule for $m=1$ is a little different.

$$\begin{array}{c}{\tilde{\mathit{\delta }}}_{k,m}=\frac{{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)({\mathit{x}}_n-{\tilde{\mathit{u}}}_{k,m}){({\mathit{x}}_n-{\tilde{\mathit{u}}}_{k,m})}^T}{{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)}.\hfill \end{array}$$

(18)

For the computation of ${\mathit{\pi }}_{k,m}$, we have:

$$\begin{array}{c}{\tilde{\mathit{\pi }}}_{k,m}=\frac{{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)}{{\sum }_m^M{\sum }_{n=1}^N{\mathit{\varphi }}_{n,k}q(k,m|{\mathit{x}}_n)}.\hfill \end{array}$$

(19)

The computation of $q(k,m|{\mathit{x}}_n)$ will be identical to the standard mixture of Gaussians model learning algorithm [37].

2.4. Agorithm

The full learning and inference algorithm is summarized in Algorithm 1. The flowchart of our proposed framework is demonstrated in Figure 3. Below, we analyze the computational complexity.

Algorithm 1 Semi-supervised DPM clustering algorithm.

Require:   unlabeled dataset ${\mathit{X}}_u$.

Ensure:   variational parameters, ${\{{\mathit{\gamma }}_{\mathit{k}},{\mathit{\tau }}_{\mathit{k}},{\mathit{\varphi }}_{\mathit{k}}\}}_{k=1}^K$, and model parameter, ${\left\{\widetilde{{\mathit{\theta }}_{\mathit{k}}}\right\}}_{k=1}^K.$

1: Construct the k-nearest neighbor graph $L^k$ and L. Initialize the variational parameter randomly. 2: while not convergent do 3:  Expectation step: 4:  while not convergent do 5:   for all $n,k$ do 6:    Update the variational parameters $\left\{{\mathit{\varphi }}_{n,k}^{old}\right\}$ using (7). 7:    Update the variational parameters $\{{\nu }_k,{\mathit{\varphi }}_k,c_k,{\mathit{u}}_k,{\mathit{W}}_k^{-1},v_k\}$ using (9), (10) and (11). 8:   end for 9:   Update the variational parameters $\left\{{\mathit{\varphi }}_{n,k}\right\}$ using (8). 10:  end while 11:  Maximization step: 12:  for $k=1;k\le K;k=k+1$ do 13:   Update ${\tilde{\mathit{u}}}_{k,1}$ and ${\tilde{\mathit{\delta }}}_{k,1}$ with (15) and (18). 14:   for $m=2;m\le M;m=m+1$ do 15:    Update ${\tilde{\mathit{u}}}_{k,m}$, ${\tilde{\mathit{\delta }}}_{k,m}$ and ${\tilde{\mathit{\pi }}}_{k,m}$ using (16), (17) and (19). 16:   end for 17:  end for 18:  $q(k,m|x_n)\leftarrow \frac{{\tilde{\mathit{\pi }}}_{k,m}N(x_n|{\tilde{\mathit{u}}}_{k,m},{\tilde{\mathit{\delta }}}_{k,m})}{{\sum }_{m=1}^M{\tilde{\mathit{\pi }}}_{k,m}N(x_n|{\tilde{\mathit{u}}}_{k,m},{\tilde{\mathit{\delta }}}_{k,m})}$ 19: end while

Algorithm complexity: Suppose that we have N samples, each sample has D dimensions. The maximum cluster number in our experiment is K. Expectation step converges after running T_e times. The whole algorithm converges after T times. From the derivation, we know that the main computation lies on the Equations (7), (8) and (11), in which we need to calculate the inverse and determinant of the matrix. For Equations (7) and (11), we need O(K · D³). For Equation (8), we need O(N³). Another major computation is the Equations (16) and (17), which takes the computational complexity of O((M − 1) · K · N · D²). According to the debates, we know that the whole algorithm computational complexity is O(T · (T_e · (N³ + K · D³) + (M − 1) · K · N · D²)).

For the space complexity, the main cost is the variational parameters which takes O(K · D² + N · K). Another cost is the MoG parameters which needs O(M · K · D²). Then, the total space complexity is O(K · D² + N · K + M · K · D²).

3. Results

To demonstrate the usefulness of the proposed manifold model, we tested our method on both synthetic and real-world datasets and compared it with the following methods:

• Original Dirichlet Process Mixture (DPM) model [38].
• Affinity Propagation (AP) clustering [39].
• A Dirichlet process-based linear manifold clustering method, DP-space [28].
• Density-based Clustering algorithm by Fast Search and Find of Density Peaks (CFSFDP) [40].
• Another category is the clustering method, which needs to specify the class number, K-means, LRR [17] and LatLRR [41].

Clustering accuracy in our experiment was measured through Normalized Mutual Information (NMI) [32]. Suppose $U=\{U_1,U_2,U_3,\dots ,U_{\left|U\right|}\}$ denotes the real cluster labels obtained from the ground truth and $V=\{V_1,V_2,V_3,\dots ,V_{\left|V\right|}\}$ obtained from a clustering algorithm. $\left|U\right|$ and $\left|V\right|$ denote the cluster number. Then, a mutual information metric between U and V can be defined as:

$$\begin{array}{c}\hfill MI(U,V)=\sum _{i=1}^{\left|U\right|}\sum _{j=1}^{\left|V\right|}P(i,j)log(\frac{P(i,j)}{P(i)P(j)})\end{array}$$

(20)

where $P(i)$ and $P(j)$ are the probability that a sample picked at random falls into class $U_i$ or $V_j$ and $P(i,j)$ denotes the probability that a sample falls into both classes $U_i$ and $V_j$. The Normalized Mutual Information (NMI) then can be defined as:

$$\begin{array}{c}\hfill NMI(U,V)=\frac{MI(U,V)}{\sqrt{(H(U)H(V))}}\end{array}$$

(21)

where $H(U)$ and $H(V)$ denote the entropy.

Experimental setup: In our experiment, we ran every algorithm 10 times and report the average accuracy. The parameters of the SimDPM algorithm were selected using the ground-truth labels of less than 40% according to the clustering accuracy. The default value for $\alpha$ and the maximum cluster number K were set at 20 and 30. The other variational parameters were initialized randomly except ${\mathit{u}}_k$ and ${\mathit{W}}_k$, for which we used the mean and covariance of the observation data to initialize. All our algorithms were implemented in MATLAB R2016a on a DELL Precision Workstation with 8.00 G RAM and a Xeon(R) E3 CPU.

For the original DPM, we used the $\alpha =20$ and set the other variational parameters randomly. When operating the DP-space, we used $\mathit{\lambda }$ and s from the values, as this was suggested in the original codes, and we selected this by using 30% ground-truth labels. The parameters used in LatLRR and LRR were that $\alpha =1$, $\beta =1.4$ and $\mathit{\lambda }=4$. For CFSFDP, we chose the determination points that were significantly different from the other points in the decision graph. In the setting of AP, we used the preference value as a scalar one. Both CFSFDP and AP used the K-nearest neighbor graph as the similarity matrix.

3.1. Synthetic Dataset

In this section, we evaluate our SimDPM model on a synthetic dataset. We show the results in Figure 4. Clearly, there are two patterns.

Visual comparison on synthetic hybrid data shows that SimDPM performed better than the traditional DPM model. In our result, Figure 4b shows the result using the original variational DPM. As can be seen, the original DPM tended to partition the synthetic dataset in a hard manner. Our manifold method yielded an ideal clustering result. The reason is that our model handled the dataset with a Gaussian expanding distribution and reserved the local geometrical structure of the data space by applying a k-nearest neighbor graph.

3.2. Real Dataset

(a) Motion segmentation: Motion segmentation usually refers to the task of separating the movements of multiple rigid-body objects from video sequences. Linear manifold clustering methods are popular in this task [42]. In our experiment, we used the Hopkin155 dataset [43] and cast it into a general multi-manifold clustering task. We show some samples in Figure 5. According to the dataset itself, we divided the universal set into checkerboard and others [43], in which each contained 26 and nine subsets. For the checkerboard dataset, we separated it into the Linear manifold dataset (L) and Non-Linear manifold dataset (Non-L) according to the 3D projection of PCA. When applying our algorithm, we projected point trajectories into 10D features. The clustering result and the estimated cluster number are presented in

Table 2. Clustering accuracy on the Hopkin155 dataset with 3 motions. AP, Affinity Propagation; CFSFDP, Density-based Clustering algorithm by Fast Search and Find of Density Peaks; LRR, Low-Rank Representation. Bolded numbers denote the highest clustering accuracy.

Method	Checkerboard			Others
	L	Non-L	Average
SimDPM	0.80	0.73	0.79	0.83
DPM [38]	0.42	0.37	0.41	0.45
DP-space [28]	0.84	0.48	0.78	0.94
AP [39]	0.29	0.32	0.29	0.31
CFSFDP [40]	0.40	0.19	0.36	0.47
K-means	0.48	0.48	0.49	0.47
LRR [17]	0.51	0.33	0.48	0.33
LatLRR [41]	0.52	0.31	0.47	0.34

and

Table 3. The estimated cluster number on the Hopkin155 dataset with 3 motions. L, Linear.

Method	Checkerboard			Others
	L	Non-L	Average
Ground truth	3.00	3.00	3.00	3.00
The estimated cluster number	3.33	3.60	3.10	3.09

. As can be seen, our proposed method performed the best on the Non-L dataset. On the others and L dataset, DP-space was the first best, and our method was the second best. For the estimated cluster number, we can observe that our model could produce the suitable cluster size compared with the ground truth.

(b) Coil20 image dataset: The coil20 [44] image database is a popular manifold database containing 20 objects from the Columbia university image library. Some image samples are demonstrated in Figure 6. Each image is taken from five degrees apart as the object is rotated on a turntable. Thus, each object in coil20 has 72 images. The size of the object is 128 × 128, with 256 grey levels per pixel. In our experiment, each image was firstly represented by a 128 × 128 dimensional vector, and then, we projected it into a 10D feature using the PCA method. To test the general clustering performance, we used five coil20 subsets. For the overall testing, we also gave the universal dataset (Dataset 20). The clustering result and the estimated cluster number are demonstrated in

Table 4. Clustering accuracy on the coil20 dataset. Bolded numbers denote the highest clustering accuracy.

Method	Subdataset					20
	2	4	6	8	10
SimDPM	0.30	0.55	0.56	0.60	0.69	0.72
DPM [38]	0.29	0.42	0.50	0.53	0.57	0.69
DP-space citewang2015dp	0.01	0.34	0.10	0.19	0.10	0.26
AP [39]	0.12	0.22	0.18	0.11	0.25	0.36
CFSFDP [40]	0	0.57	0.57	0.53	0.46	0.42
K-means	0	0.52	0.46	0.57	0.59	0.73
LRR [17]	0.11	0.62	0.56	0.47	0.52	0.70
LatLRR [41]	0	0.57	0.57	0.48	0.50	0.58

and

Table 5. The estimated cluster number on the coil20 dataset.

Method	Subdataset					20
	2	4	6	8	10
Ground truth	2.0	4.0	6.0	8.0	10.0	20.0
The estimated cluster number	3.1	5.0	6.3	6.7	11.3	21.7

. From the result, we know that our method consistently outperformed the DP-based algorithms such as DP-space and DPM. When comparing with the other methods, our method was the first or the second best, especially compared with the approaches that do not need to specify the cluster number.

(c) Swedish leaf image dataset: The Swedish dataset introduced in [45] consists of 1125 leaves of 15 species with 75 images per species. In this dataset, we firstly extracted the outer contour and then achieved the contour features by applying the Fourier transform [46]. Every leaf in our experiments was represented as a 10-dimensional feature. Some samples are shown in Figure 7.

Similar to the coil20 dataset, we demonstrated the efficiency on five subsets and the universal dataset. We ran every algorithm in our experiment 10 times, and took the accuracy by averaging the 10 results. The experimental results are demonstrated in

Table 6. Clustering accuracy on the leaf dataset. Bolded numbers denote the highest clustering accuracy.

Method	Subdataset					15
	2	4	6	8	10
SimDPM	0.29	0.62	0.46	0.50	0.54	0.38
DPM [38]	0.26	0.43	0.45	0.49	0.51	0.34
DP-space [28]	0.49	0.33	0	0	0	0.03
AP [39]	0	0.03	0.11	0	0.02	0
CFSFDP [40]	0.17	0.56	0.24	0.28	0.42	0
K-means	0.45	0.39	0.61	0.50	0.44	0.22
LRR [17]	0.76	0.45	0.48	0.33	0.40	0.22
LatLRR [41]	0.65	0.44	0.32	0.20	0.41	0.23

and

Table 7. The estimated cluster number on the leaf dataset.

Method	Subdataset					15
	2	4	6	8	10
Ground truth	2.0	4.0	6.0	8.0	10.0	15.0
The estimated class number	4.7	6.3	10.2	14.5	16.2	20.2

, which present some observations: (1) compared with the original DPM, the improvement of the clustering accuracy (average 0.07) was lower than the improvement in the coil20 dataset (average 0.05); (2) the cluster number was consistently increasing as the ground truth cluster number was increasing.

From the experimental results in Table 2, Table 3, Table 4, Table 5, Table 6 and Table 7, we can draw some points as follows.

• The proposed method obtained the highest clustering accuracy especially on the Non-L and coil20 dataset compared with the non-prespecified cluster number methods, which validates the effectiveness of our non-linear assumption.
• DP-space performed better than our method on the L and others dataset, the reason being that DP-space has a prior structure assumption, which introduces additional manifold geometric information.
• LRR, LatLRR and K-means outperformed our algorithm on some coil20 and leaf subdatasets, the reason being that our method needed to estimate the cluster number along with clustering. This made our algorithm hard to optimize.
• Compared to the coil20 and leaf dataset, our method achieved an all-around performance boosting on the motion segmentation dataset; this is because the simple clustering task (the linear manifold has only three classes) was easy for our algorithm to optimize and model.
• Compared to the leaf dataset, our method achieved a better clustering performance boosting on the coil20 dataset. The reason is that coil20 is a well-defined manifold dataset, in which the structure among samples is easy to capture by the graph Laplacian.
• Our manifold model consistently produced the suitable cluster number with the increasing of the data cluster size, which indicates that our model could provide a flexible model size when fitting different datasets.

3.3. The Effect of the Algorithm Parameters

In this section, we firstly investigate the effects of the parameters ${\mathit{\lambda }}_A$ and s on the Non-L dataset. More specifically, in the experiment, when one parameter is being tuned, the value of the other parameter is fixed. The parameters ${\mathit{\lambda }}_A$ and s were sampled from $\{1000,100,80,60,40,20\}$ and $\{1,0.9,0.8,0.7,0.6,0.5\}$. We show the clustering accuracy in Figure 8.

As we can see in Figure 8, experimental results indicate that the proposed model was sensitive to ${\mathit{\lambda }}_A$. Empirically, the best clustering accuracy was achieved when ${\mathit{\lambda }}_A=100$. We also observed that our method achieved the best clustering accuracy when $s=0.7,0.8$. This reveals that the MoG had improved the clustering accuracy. Besides, we measured the clustering accuracy with the different M and the neighbor number r using the k-nearest neighbor graph. The clustering accuracy is demonstrated in Figure 9 and Figure 10. As can be seen, the clustering accuracy achieved the best performance on the leaf subdataset and the coil20 dataset when the neighbor number $r=\{5,6,7,8,9,10\}$. Clustering accuracy increased along with the increasing of the M in the subdataset of the coil20 dataset and leaf dataset. Unlike the subdataset, parameter M and the neighbor number r had little effect on the full dataset of coil20 and leaf. The reason is that our model is a non-convex model, and the complicated dataset led to a much more complicated optimization.

4. Discussion

Compared to the previous linear manifold and geodesic mixture models, our theoretical analysis has shown that our method is a prespecified manifold and cluster number-free model. This is because we use a DP prior to generate the cluster indicator with the suitable cluster number and use the MoG and K-nearest neighbor graph to capture the submanifold rather than using a predefined manifold. Additionally, compared with different multi-manifold clustering methods with prespecified manifolds and cluster numbers like DP-space, LRR and LatLRR, our method has shown superior performance on the general manifold clustering task (coil20 and leaf dataset). This indicates our method can fill the research gap we have mentioned in the Introduction.

Although our method can handle the problems we have mentioned (estimating the cluster number and handling the general manifold clustering task), limitations still exist. That is, our method is not a full Bayesian framework. Thereby, some parameters should be tuned manually. This may be unacceptable in some real applications. Moreover, we note that MoG and the Gaussian distribution are sensitive to the dimension of the data. In future work, we will explore a full generative model, in which the parameters can be generated by using some Bayesian priors. Since our approach is sensitive to the dimension, we will also explore certain methods to integrate the dimension reduction method and the manifold clustering.

5. Conclusions

In this paper, we have proposed a nonparametric generative model to handle the manifold dataset with no prespecified cluster number and manifold distribution. In the course of the theoretical and experimental analysis, we have demonstrated that MoG can extend the application scope of the original DPM and can significantly improve the clustering accuracy compared to the previous proposed method. However, to be frank, the proposed method can only partially handle the problem we state in the Introduction due to the facts that: (1) MoG, the mean constraint and K-nearest neighbor graph are hard to optimize when we incorporate them into the DP framework; this can be observed when we use it in the coil20 and full leaf dataset; (2) the DP prior has a limitation when generating the suitable cluster number.