Probabilistic Non-Negative Matrix Factorization with Binary Components

Abstract

Non-negative matrix factorization is used to find a basic matrix and a weight matrix to approximate the non-negative matrix. It has proven to be a powerful low-rank decomposition technique for non-negative multivariate data. However, its performance largely depends on the assumption of a fixed number of features. This work proposes a new probabilistic non-negative matrix factorization which factorizes a non-negative matrix into a low-rank factor matrix with $\left\{0,1\right\}$ constraints and a non-negative weight matrix. In order to automatically learn the potential binary features and feature number, a deterministic Indian buffet process variational inference is introduced to obtain the binary factor matrix. Further, the weight matrix is set to satisfy the exponential prior. To obtain the real posterior distribution of the two factor matrices, a variational Bayesian exponential Gaussian inference model is established. The comparative experiments on the synthetic and real-world datasets show the efficacy of the proposed method.

1. Introduction

Non-negative matrix factorization (NMF) is an important nonlinear technique to find purely additive, parts-based representations of non-negative multivariate data [1]. It aims to reveal the latent structure or pattern in the data. Compared with the classical low-rank matrix factorization methods, such as the singular value decomposition and principal component analysis, NMF has turned out to be a powerful tool in data-mining analysis because of its stronger explanatory properties (parts constitute the whole) and rationality (decomposing factors must be non-negative in some real scenarios). For example, image information can be compressed and summarized into a combination of some features [2], and some feature words can be learned from the document to represent the document for clustering tasks [3]. This original information is generally stored in the computer as a data matrix, which is decomposed by NMF into the product of a basis factor matrix and a weight.

Most research on the NMF algorithm is based on the specific expected characteristics to impose other restrictions except negativity on the factorization factor matrix [4,5,6]. Although these works have made some achievements, some significant problems remain to be solved, such as the way to construct efficient algorithm to avoid complex iterative calculation, and the initialization problem which greatly weaken the flexibility of the model due to its unknowability. Wild et al. [7] proposed that the use of a structured initial value can improve the efficiency of the NMF algorithm at the cost of converging to a poor local solution.

The probability non-negative matrix factorization from statistical view may be enlightening. Bayesian NMF is the NMF model with KL penalty that was extended to the Bayesian setting [8]. Using the Bayesian nonparametric model of infinite dimensional parameter space, the number of features is determined as a random quantity in the process of posterior inference. Indian buffet process (IBP) is a classical latent feature model. This method generates a prior of a binary matrix, which has finite dimensional rows (number of objects) and infinite dimensional columns (number of features) [9]. The real latent finite element information can be obtained completely by the posterior inference with IBP. Many studies have proposed the application of IBP in NMF. For example, Yang et al. projected a non-negative binary matrix tri-factorization model that better reveals the underlying structure between samples and features, including two binary matrices and a weight matrix [10].

In this work, we come up with a new special probabilistic non-negative matrix factorization which requires the basis matrix $Z$ atisfies $\left\{0,1\right\}$ constraints and the weighted matrix $A$ is non-negative. This is clearer and more intuitive than the definition of tri-factorization proposed in [10]. For the binary matrix $Z$, the IBP is introduced to solve, and we assume that the weight matrix $A$ obeys the exponential distribution under the non-negative constraint. In addition, based on synthetic dataset and real-world datasets, we compare our approach with benchmark methods to verify its flexibility and effectiveness. Main contributions are following three aspects:

(1)

We present a pNMF with single binary component. Compared with other methods, the binary matrix can be regarded as the mapping from real object to binary codes, which is more intuitive.

(2)

IBP is applied to the pNMF and we explain its use as a prior in the variational Bayesian exponential Gaussian model. The real latent information is completely obtained by inference and the sensitivity of the model to initialization parameter setting is greatly reduced.

(3)

The experiments on the synthesized dataset and real-world datasets show the validity of the proposed method.

The rest of this paper is organized as follows: the related work is outlined in the next section. Section 3 explains the proposed method and the derivation process. Section 4 gives the empirical research results on four different types of dataset. The last section concludes the work.

2. Related Work

Recent research has inspired us to consider pNMF with single components on the basis of IBP. This section lists the literature review of NMF and IBP.

2.1. Non-Negative Matrix Factorization

NMF was firstly posed by Paatero and tapper [1], but enjoyed wide popularity due to the research results of Lee and Seung [2]. According to its definition, the factor elements after factorization must be non-negative, and simultaneously achieves nonlinear dimension reduction. Mathematically, given the non-negative matrix $V$, the problem is to find two factor matrices $W\ge 0$ and $H\ge 0$ to approximate $V$, so that $V\approx WH$. It is also usually interpreted as $V=WH+\epsilon$ by researchers with blind source signal separation background [11], where $\epsilon$ is a noise matrix. Lee and Seung [2] also introduced two simpler algorithms whose objective functions are based on Euclid distance and Kullback–Leibler divergence (KL), respectively, and they proved the convergence of iterative rules. As an unsupervised learning method, NMF is widely used in many fields.

Many improved NMF algorithms have been advocated in recent years. Hoyer et al. [12] combined sparse penalty term with NMF to construct the sparse NMF model, in which parameters were solved by the gradient projection method and the EM algorithm. Wang et al. [13] combined Fisher’s discriminant criterion with the objective function of non-negative matrix factorization to extract local features. Lin et al. [14] brought forth an optimization algorithm of NMF based on projection gradient method, which refined the convergence rate of factorization. Liu et al. [6] found a restrictive NMF algorithm under semi-supervised learning that effectively combined the category information. Guillamet et al. [15] pointed out a weighted NMF algorithm to solve the image classification problem by better representation ability of local features.

For many NMF methods, the rank needs to be determined in advance, and its value was adjusted and searched through trial and error. Hinrich et al. [16] mentioned a probabilistic sparse NMF model that extends the variational Bayesian NMF and explicitly considers sparsity. Mohammadiha et al. [17] developed a hidden Markov model whose output density function is a gamma distribution. This model was reformulated as a probabilistic NMF, and it can capture the time dependence internally. To derive the maximum a posterior (MAP) of the non-negative factors, Shterenberg et al. [18] extended the NMF deterministic framework to the probabilistic such that the original data are no longer deterministic, which is considered to be a sample drawn from a multinomial distribution.

2.2. Indian Buffet Process

Griffiths et al. [9] proposed the IBP in 2005. They defined the probability distribution over equivalence classes of a binary matrix with a finite number of rows and an unbounded number of columns, and illustrated the use of it as a prior in an infinite binary linear–Gaussian model. At the same time, they have proved that this distribution as a prior is suitable for probabilistic models that use potentially infinite feature arrays to represent objects [19]. Many studies have been extended on the basis of IBP to generate more general distribution categories.

Ghahramani et al. [20] introduced a two-parameter generalization of the IBP that focused on separating the coupling relationship between the total number of features and the distribution on the feature number of each object. The et al. [21] analyzed the stick-breaking construction of IBP and developed a slice sampler with higher efficiency and easier application. Thibaux and Jordan [22] demonstrated the exchangeable distribution produced by IBP corresponds to the use of a latent measure based on the beta process. Gael et al. [23] rendered a strategy for modifying the IBP to capture the dependency which can arise as the observations being generated in a specific sequence. Miller et al. [24] used the phylogenetic IBP to seek a dependency that is taken as the result of known degrees of relatedness among observations.

The significance of IBP is to produce a distribution of infinite sparse binary matrix, which can be used to define many new non-parametric Bayesian models with different likelihood functions. Miller et al. [25] defined a class of non-parametric latent feature models that can be used for link prediction. Meeds et al. [26] presented a model of dyadic data based on the two-parameter IBP and developed some novel Metropolis-Hasting proposals for inference. Wood et al. [27] adopted a non-parametric Bayesian approach based on the IBP to model the structure graphs with many hidden causes. Navarro et al. [28] proposed a non-parametric Bayesian model for inferring features from similarity judgements.

3. The Proposed Methods

3.1. Model Framework

Given a non-negative matrix $X\in {ℝ}^{N\times D}$, the product of two non-negative factor matrices $Z$ and $A$ is expected to be as close to $X$ as possible. Our data model is shown in Figure 1.

Where $Z$ is a binary matrix of $N\times K$ dimension, and $A$ is a non-negative weight matrix of $K\times D$ dimension. In this model, $X$ follows the Gaussian conditional distribution:

$$X\approx ZA \mathrm{s}.\mathrm{t}. Z\in {\{0,1\}}^{N\times K}and A\in {ℝ}_{+}^{K\times D}.$$

(1)

$$X_n\sim Normal\left(Z_{n}A,{\sigma }_n^2\right), for n\in \left\{1,\dots ,N\right\}.$$

(2)

According to Bayesian rule:

$$p(Z,A|X)\propto \underset{model \mathrm{s}pecific}{\underbrace{p(X|Z,A)p\left(A\right)}}\hspace{1em}\times \underset{prior on binary matrix}{\underbrace{p\left(Z\right)}}.$$

(3)

It can be seen that each column in $Z$ corresponds to the existence of a latent feature, and all elements of $Z$ are non-negative due to the $\left\{0,1\right\}$ constraints. $p\left(Z\right)$ is just the prior probability function defined by IBP on the binary matrix $Z$ in (1). For $A$, we need a prior to express rules about its value, so we assumed that $A$ obeys the exponential distribution to meet the non-negative limitation. Thus, each element of $A$ obeys the exponential distribution [8].

Therefore, the defined model can be considered as an exponential Gaussian model, and the prior representation of these two matrices is:

$$\begin{gathered} A_{jk}\sim \epsilon \left(A_{jk}|{\lambda }_k\right), {\lambda }_k\sim \mathcal{G}\left({\lambda }_k|{\alpha }_0,{\beta }_0\right), z_{nk}\sim Bernoulli\left({\pi }_k\right), \\ s.t. k\in \left\{1,\dots ,\infty \right\},j\in \left\{1,\dots ,D\right\}, n\in \left\{1,\dots ,N\right\}. \end{gathered}$$

(4)

In accordance with the automatic association determination algorithm proposed by Thomas et al. [29], the parameter $\lambda$ is set using gamma prior, which helps the automatic selection of the model. The whole factor $k$ is either “activated” or “off”, and only a few effective factors are set to 0 based on the magnitude of $\lambda$. Figure 2 indicates the graph structure of the exponential Gaussian model.

According to (3), the goal of our model is to obtain the true posterior distribution $ln(p(Z,A|X,{\sigma }_n^2,\alpha ,\lambda ))$ of factor matrices. It is very difficult to solve this problem with conventional methods, but the variational approximation algorithm is likely to be an effective method. Many studies have revealed that this algorithm simplifies inference process and guarantees accuracy to a certain extent.

Like Gibbs sampling, the mean field variational inference is a common approximate Bayesian inference method, which is to simplify the calculation when the introduced variational distribution $q\left(\theta \right)$ is as close to the real distribution $p(\theta |X)$ as possible. The theory assumes that the variational distribution $q$ can be completely decomposed as $q\left(\theta \right)={{\displaystyle \prod }}_{i}q\left({\theta }_i\right)$, and all variables are calculated independently in the process of posteriori inference. It has been proved in [30] that the optimal distribution $q^{*}\left({\theta }_i\right)$ of the ith parameter can be expressed as follows:

$$log q^{*}\left({\theta }_i\right)={\mathbb{E}}_{q\left(-{\theta }_i\right)}\left[log p\left(\theta ,X\right)\right]+C.$$

(5)

where $C$ is a constant. The objective function is usually to maximize the lower bound of evidence (ELBO) or minimize the KL divergence as much as possible, and it is essential to choose a suitable approximate distribution $q$ for solving the variational problems.

On the basis of the model framework, our work is mainly to find the best approximate distribution of two factor matrices that are consistent with the constraints.

3.2. The Solution of Weighted Matrix A

For the weight matrix $A$, the exponential distribution is used as a prior in the model to limit its non-negativity. According to Bayes’ rule, the posterior of $A$ is approximately a truncated normal distribution if $Z$ is fixed. The formula is as follows:

$$\begin{array}{l}{q}^{*}\left(A_{jk}\right)\propto exp\left\{{\mathbb{E}}_{q\left({\theta }_{-A_{jk}}\right)}\left[log p\left(X|\theta \right)\right]+log p\left(\theta \right)\right\}\\ \propto exp\left\{{\mathbb{E}}_{q\left({\theta }_{-A_{jk}}\right)}\left[{\displaystyle {\displaystyle \sum }_{i\in {\Omega }_j}}log\left[\sqrt{\frac{\tau }{2\pi }}exp\left\{-\frac{\tau }{2}{(X_{ij}-Z_i{A}_j)}^2\right\}\right]+log\left[{\lambda }_k exp\{-{\lambda }_k{A}_{jk}\}\right]\right]\right\}\times u\left(x\right)\\ \propto exp\left\{{\mathbb{E}}_{q\left({\theta }_{-A_{jk}}\right)}\left[{\displaystyle {\displaystyle \sum }_{i\in {\Omega }_j}}-\frac{\tau }{2}{(X_{ij}-Z_i{A}_j)}^2-{\lambda }_k{A}_{jk}\right]\right\}\times u\left(x\right)\\ \propto exp\left\{{\mathbb{E}}_{q\left({\theta }_{-A_{jk}}\right)}\left[-\frac{\tau }{2}{\displaystyle {\displaystyle \sum }_{i\in {\Omega }_j}}\left[Z_{ik}^2{A}_{jk}^2-2Z_{ik}{A}_{jk}\left(X_{ij}-{\displaystyle {\displaystyle \sum }_{k^{\prime }\ne k}}{Z}_{i{k}^{\prime }}{A}_{j{k}^{\prime }}\right)\right]\right]-A_{jk}\widetilde{{\lambda }_k}\right\}\times u\left(x\right)\\ \propto exp\left\{A_{jk }\left[-\widetilde{{\lambda }_k}+\tilde{\tau }{\displaystyle {\displaystyle \sum }_{i\in {\Omega }_j}}\left(X_{ij}-{\displaystyle {\displaystyle \sum }_{k^{\prime }\ne k}}\widetilde{Z_{i{k}^{\prime }}}\widetilde{A_{j{k}^{\prime }}}\right)\widetilde{Z_{ik}}\right]-\frac{A_{jk}^2}{2}\left[\tilde{\tau }{\displaystyle {\displaystyle \sum }_{i\in {\Omega }_j}}\widetilde{Z_{ik}^2}\right]\times u\left(x\right)\right\}\\ \propto exp\left\{-\frac{{\tau }_{jk}}{2}{\left(A_{jk}-{\mu }_{jk}\right)}^2\right\}\times u\left(x\right)\\ \propto \mathcal{T}\mathcal{N}(A_{jk}|{\mu }_{jk},{\tau }_{jk}).\end{array}$$

(6)

Here,

$$\mathcal{T}\mathcal{N}\left(x|\mu ,\tau \right)=\left\{\begin{array}{c}\frac{\sqrt{\frac{\tau }{2\pi }}exp\{-\frac{\tau }{2}{(x-\mu )}^2\}}{1-\Phi \left(-\mu \sqrt{\tau }\right)} if x\ge 0 \\ 0 if x<0\end{array}\right.,$$

(7)

where $A_{-jk}$ represents all elements in the matrix $A_{jk}$ except the $k-th$ column, and it can be seen that the posterior probabilities of non-positive elements are all set to 0. This process is not a strict optimization of $A$, which may cause some errors in the final reconstruction results.

According to the derivation process of (6), the mean and variance of the true posterior $q^{*}\left(A_{jk}\right)$ can be obtained as follows:

$$\begin{gathered} {\tau }_{jk}=\tilde{\tau }{\displaystyle \sum }_{i\in {\Omega }_j}\widetilde{Z_{ik}^2}, \\ {\mu }_{jk}=\frac{1}{{\tau }_{jk}}\left(-\widetilde{{\lambda }_k}+\tilde{\tau }{\displaystyle \sum }_{i\in {\Omega }_j}\left(X_{ij}-{\displaystyle \sum }_{k^{\prime }\ne k}\widetilde{Z_{i{k}^{\prime }}}\widetilde{A_{j{k}^{\prime }}}\right)\widetilde{Z_{ik}}\right). \end{gathered}$$

(8)

It can be found that the smaller the mean (negative), the larger the variance. The truncated distribution is closest to the exponential distribution when the expected scale parameter is $\left|\mu \ast \tau \right|$. In order to prevent the calculation of mean and variance effectively from being invalidated by numerical errors, it is necessary to restrict $\left|\mu \right|$. The setting of parameter $\lambda$ is also an important factor affecting the lower bound of $X$. We assume that it satisfies the following distribution and update the relevant parameters:

$$\begin{gathered} q\left(A_{jk}\right)=\mathcal{T}\mathcal{N}\left(A_{jk};{\mu }_{jk},{\tau }_{jk}\right), q\left({\lambda }_k\right)=\mathcal{G}\left({\lambda }_k;{\alpha }_k^{*},{\beta }_k^{*}\right), \\ {\alpha }_k^{*}={\alpha }_0+D, {\beta }_k^{*}={\beta }_0+{{\displaystyle \sum }}_{j=1}^D\widetilde{A_{jk}}. \end{gathered}$$

(9)

3.3. The Solution of Binary Matrix Z

The generation process of IBP is similar to many Indian cafeterias in London. Endless plates line up, and customers enter one by one, choosing food from left to right. If selected by a customer, this dish is marked as “1”, otherwise “0”. The number of dishes selected by each customer satisfies the Poisson distribution [9]. The process is shown in Figure 3.

As shown in Figure 3, the binary sequence representing the choice of the third customer can be expressed as $\left[1, 0, 1, 0, 0,\Vert \right]$. By connecting $N$ pieces of such sequence vertically, IBP generates a priori of a binary matrix with finite number of rows (number of data) and infinite number of columns (number of latent features) to define many new non-parametric Bayesian models with different likelihood functions. Combined with the analysis of the model framework, IBP is used to approximate the binary basis matrix $Z$. In order to facilitate the process of variational derivation, the stick breaking construction based on the research of The et al. [16] is introduced as follows:

$$v_j\sim Beta\left(\alpha ,1\right), {\pi }_k={{\displaystyle \prod }}_{j=1}^k{v}_j, z_{nk}\sim Bernoulli\left({\pi }_k\right).$$

(10)

where $v_{j }$ is the variable, $\alpha$ is a parameter of the beta distribution and ${\pi }_k$ is the truncated weight of column $k$. The basis matrix $Z$ in our model adopts the approximate form similar to the infinite variational approach of linear Gaussian model proposed by Doshi et al. [12]:

$$q_{v_{nk}}\left(z_{nk}\right)=Bernoulli\left(z_{nk};v_{nk}\right), q_{{\tau }_k}\left(v_k\right)=Beta\left(v_k;{\tau }_{k1},{\tau }_{k2}\right).$$

(11)

To facilitate calculation, the $\pi$ is replaced by $v$. The derivation of $v$ and its parameters are given in [12]. We will present this information in detail in the following derivation.

3.4. Variational Inference Process

Based on the above analysis, the variation distribution of our model is shown in Figure 4.

Since it is difficult to take the real expectation of the log joint likelihood $log p(X|\theta )$, this paper gives its lower bound through reasoning. The variational objective function of our model is as follows:

$$\begin{array}{c}log p(X|\theta )\ge {\displaystyle {\displaystyle \sum }_{k=1}^K}{\mathbb{E}}_v[log p(v_k|\alpha )]+{\displaystyle {\displaystyle \sum }_{k=1}^K}{\displaystyle {\displaystyle \sum }_{n=1}^N}{\mathbb{E}}_{v,Z}[log p(Z_{nk}|v)]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\displaystyle {\displaystyle \sum }_{k=1}^K}{\mathbb{E}}_{\lambda }[log p({\lambda }_k|{\alpha }_0,{\beta }_0)]+{\displaystyle {\displaystyle \sum }_{k=1}^K}{\displaystyle {\displaystyle \sum }_{j=1}^D}{\mathbb{E}}_{\lambda ,A}[log p(A_{jk}|{\lambda }_{jk})]\\ \hspace{1em}+{\displaystyle {\displaystyle \sum }_{n=1}^N}{\mathbb{E}}_{Z,A}[log p(X_n|Z,A,{\sigma }_n^2)]+H\left[q\right],\end{array}$$

(12)

where $\theta =\left\{\alpha ,{\alpha }_0,{\beta }_0,{\sigma }_n^2\right\}$ is the parameter. From (12), It can be found from (12) that except the second term, all items are calculated by exponential distribution family, and their expected value derivation can refer to the study of Doshi et al. [12]. The ${\mathbb{E}}_{\lambda }\left[\ast \right]$ is:

$${\displaystyle \prod }_{k=1}^K{\mathbb{E}}_{\lambda }\left[log p\left({\lambda }_k|{\alpha }_0,{\beta }_0\right)\right]={\alpha }_0 log {\beta }_0-ln \Gamma \left({\alpha }_0\right)+\left({\alpha }_0-1\right){\displaystyle \sum }_{k=1}^K\psi \left({\alpha }_k\right)+{\beta }_0{\displaystyle \sum }_{k=1}^K\frac{{\alpha }_k}{{\beta }_k}.$$

(13)

According to the research results of Chopin and Mazet [31,32], we can draw the truncated distribution formula and simulate the Gaussian distribution defined on the finite interval $\left[a,b\right]$. The semi finite interval can still be considered by setting $b=+\infty$. The principle of this method is to divide the interval into the same area and use a properly distributed accept–reject algorithm. Since the $A_{jk}$ is a truncated normal distribution, the ${\mathbb{E}}_{\lambda ,A}\left[\ast \right]$ is:

$$\begin{gathered} {\mathbb{E}}_{\lambda ,A}\left[log p\left(A_k|{\lambda }_k\right)\right]={\mathbb{E}}_{\lambda ,A}\left[log({\lambda }_k·exp(-{\lambda }_k{A}_k\right))\left]={\mathbb{E}}_{\lambda ,A}\right[log {\lambda }_k-{\lambda }_k{A}_k] \\ =log {\lambda }_k-{\lambda }_k\mathbb{E}\left[A_k\right] \\ =log\frac{{\alpha }_k}{{\beta }_k}-\frac{{\alpha }_k}{{\beta }_k}\left({\mu }_k+{\tau }_k·\frac{1}{\sqrt{2\pi \cdot {\tau }_k}}·\right.\left.exp\left(-\frac{u_k^2}{2{\tau }_k}\right)/\frac{1}{2}erfc\left(-\sqrt{\frac{{\tau }_k}{2}}·{\mu }_k\right)\right). \end{gathered}$$

(14)

For the lower truncated normal distribution, the expectation is as follows:

$$\mathbb{E}\left(x|x<c\right)=\mu +{\sigma }^2\frac{f\left(c\right)}{S\left(c\right)}.$$

(15)

Here $f\left(x\right)$ is the probability density function and $S\left(x\right)$ is the complementary cumulative distribution function, and their formulas are as follows:

$$\begin{gathered} f\left(x\right)=\frac{1}{\sqrt{2\pi }\sigma }exp\left(-\frac{{\left(x-\mu \right)}^2}{2{\sigma }^2}\right), \\ S\left(x\right)=1-F\left(x\right)=1-cdf\left(x\right)=0.5\ast \mathrm{erfc}\left(\frac{x}{\sqrt{2}}\right) . \end{gathered}$$

(16)

The entropy $H\left[q\right]$ in (11) is:

$$\begin{array}{c}H\left[q\right]=-{\mathbb{E}}_{q}log\left[{\displaystyle {\displaystyle \prod }_{k=1}^K}{q}_{{\tau }_k}\left({\pi }_k\right){\displaystyle {\displaystyle \prod }_{k=1}^K}q\left({\lambda }_k\right){\displaystyle {\displaystyle \prod }_{k=1}^K}{q}_{{\mu }_k}\left(A_k\right){\displaystyle {\displaystyle \prod }_{k=1}^K}{\displaystyle {\displaystyle \prod }_{n=1}^N}{q}_{v_{nk}}\left(z_{nk}\right)\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\displaystyle {\displaystyle \sum }_{k=1}^K}{\mathbb{E}}_{\pi }\left(-log q_{{\tau }_k}\left({\pi }_k\right)\right)+{\displaystyle {\displaystyle \sum }_{k=1}^K}{\mathbb{E}}_{\lambda }\left(-log q\left({\lambda }_k\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\displaystyle {\displaystyle \sum }_{k=1}^K}{\mathbb{E}}_A\left(-log q_{{\mu }_k}\left(A_k\right)\right)+{\displaystyle {\displaystyle \sum }_{k=1}^K}{\displaystyle {\displaystyle \sum }_{n=1}^N}{\mathbb{E}}_Z\left(-log q_{v_{nk}}\left(z_{nk}\right)\right)\end{array}$$

(17)

where,

$$\begin{array}{l}{\mathbb{E}}_{\pi }\left(-log q_{{\tau }_k}\left({\pi }_k\right)\right)=log\left(\frac{\Gamma \left({\tau }_{k1}\right)\Gamma \left({\tau }_{k2}\right)}{\Gamma \left({\tau }_{k1}+{\tau }_{k2}\right)}\right)-\left({\tau }_{k1}-1\right)\psi \left({\tau }_{k1}\right)-\left({\tau }_{k2}-1\right)\psi \left({\tau }_{k2}\right)\\ +\left({\tau }_{k1}+{\tau }_{k2}-2\right)\psi \left({\tau }_{k1}+{\tau }_{k2}\right),\\ {\mathbb{E}}_{\lambda }\left(-log q\left({\lambda }_k\right)\right)=ln \Gamma \left({\alpha }_k^{*}\right)-\left({\alpha }_k^{*}-1\right)\psi \left({\alpha }_k^{*}\right)-ln {\beta }_k^{*}+{\alpha }_{k }^{*},\\ {\mathbb{E}}_A\left(-log q_{{\mu }_k}\left(A_k\right)\right)=0.5\ast log(\left(2\pi e{)}^D\left|{\tau }_k\right|\right) ,\\ {\mathbb{E}}_Z\left(-log q_{v_{nk}}\left(z_{nk}\right)\right)=-\left(1-v_{nk}\right)log(1-v_{nk})-v_{nk}log v_{nk} .\end{array}$$

(18)

The $\psi \left(·\right)$ is a digamma function. According to the analysis, the derivation formula of the final objective function of (12) is as follows:

$$\begin{array}{l}log p\left(X|\theta \right)\ge {\displaystyle {\displaystyle \sum }_{k=1}^K}\left[log \alpha +\left(\alpha -1\right)\left(\psi \left({\tau }_{k1}\right)-\psi \left({\tau }_{k1}+{\tau }_{k2}\right)\right)\right]\\ +{\displaystyle {\displaystyle \sum }_{k=1}^K}{\displaystyle {\displaystyle \sum }_{n=1}^N}\left[v_{nk}\left({\displaystyle {\displaystyle \sum }_{m=1}^k}\psi \left({\tau }_{k2}\right)-\psi \left({\tau }_{k1}+{\tau }_{k2}\right)\right)+\left(1-v_{nk}\right){\mathbb{E}}_{\upsilon }\left[log\left(1-{\displaystyle {\displaystyle \prod }_{m=1}^k}{\upsilon }_m\right)\right]\right]\\ +\begin{array}{c}{\alpha }_{0}log {\beta }_0-ln \Gamma \left({\alpha }_0\right)+\left({\alpha }_0-1\right){\displaystyle {\displaystyle \sum }_{k=1}^K}\psi \left({\alpha }_k\right)+{\beta }_0{\displaystyle {\displaystyle \sum }_{k=1}^K}\frac{{\alpha }_k}{{\beta }_k}\end{array}\\ +log\frac{{\alpha }_k}{{\beta }_k}-\frac{{\alpha }_k}{{\beta }_k}\left({\mu }_k+{\tau }_k·\frac{1}{\sqrt{2\pi \cdot {\tau }_k}}· \left.exp\left(-\frac{u_k^2}{2{\tau }_k}\right)/\frac{1}{2}erfc\left(-\sqrt{\frac{{\tau }_k}{2}}\cdot {\mu }_k\right)\right)\right.\\ +{\displaystyle {\displaystyle \sum }_{n=1}^N}\left[-\frac{D}{2}log\left(2\pi {\sigma }_n^2\right)\right]\\ +{\displaystyle {\displaystyle \sum }_{n=1}^N}\left[-\frac{1}{2{\sigma }_n^2}\left(X_n{X}_n^T-2{\displaystyle {\displaystyle \sum }_{k=1}^K}{v}_{nk}{\mu }_k{X}_n^T+2{\displaystyle {\displaystyle \sum }_{k<k^{\prime }}}{v}_{nk}{v}_{n{k}^{\prime }}{\mu }_k{\mu }_k^T+{\displaystyle {\displaystyle \sum }_{k=1}^K}{v}_{nk}\left(tr\left({\tau }_k\right)+{\mu }_k{\mu }_k^T\right)\right)\right]\\ +{\displaystyle {\displaystyle \sum }_{k=1}^K}\left[log\left(\frac{\Gamma \left({\tau }_{k1}\right)\Gamma \left({\tau }_{k2}\right)}{\Gamma \left({\tau }_{k1}+{\tau }_{k2}\right)}\right)-\left({\tau }_{k1}-1\right)\psi \left({\tau }_{k1}\right)-\left({\tau }_{k2}-1\right)\psi \left({\tau }_{k2}\right)+\left({\tau }_{k1}+{\tau }_{k2}-2\right)\psi \left({\tau }_{k1}+{\tau }_{k2}\right)\right]\\ +{\displaystyle {\displaystyle \sum }_{k=1}^K}\frac{1}{2}log(\left(2\pi e{)}^D\left|{\tau }_k\right|\right)+{\displaystyle {\displaystyle \sum }_{k=1}^K}ln \Gamma \left({\alpha }_k^{*}\right)-\left({\alpha }_k^{*}-1\right)\psi \left({\alpha }_k^{*}\right)-ln{\beta }_k^{*}+{\alpha }_k^{*}\\ +{\displaystyle {\displaystyle \sum }_{k=1}^K}{\displaystyle {\displaystyle \sum }_{n=1}^N}\left[-v_{nk}log v_{nk}-\left(1-v_{nk}\right)log(1-v_{nk})\right].\end{array}$$

(19)

4. Experimental Result

The ability of the proposed method to perform matrix factorization and learn the number of latent features through the synthetic dataset is first demonstrated, and then the performance of the proposed method is verified by comparing with the benchmark methods on real datasets, including Swimmer dataset, Cora document dataset, and CBCL face dataset. While showing the effect of the model, we list the initialization setting of the parameters in each task, which have a great impact on the quality of convergence.

The main parameters involved in the experiment include IBP parameter $\alpha$, lambda parameter ${\alpha }_0$,${\beta }_0$, Gaussian parameter ${\sigma }_n$ and truncated feature number $K$. The optimal effect parameter will be given for reference at the end of each experiment. The algorithm experiment is carried out in the environment of Matlab2015b.

4.1. Sythetic Dataset

We apply the model to a simple composite dataset generated from known features. The dataset is simple enough, and the accuracy of graph reconstruction can reach almost 100%, and there are enough differences to resolve the qualitative features of latent feature inference. We have generated a specific $20\times 16$-dimensional non-negative matrix, some of which is shown in Figure 5.

Figure 5a shows the basis matrix $Z$ with $20\times 4$ dimension. Figure 5b shows the extracted latent features, there are four in total. The original image is sufficient to justify it. For example, the first image is made up of three feature elements in Figure 5b except for the item in the upper right corner, while the third image contains that single item.

Figure 5d shows the reconstructed image using the extracted latent features, and the comparison shows that the accuracy can reach almost 100%.

The above analysis fully proves the effectiveness of our algorithm for automatic learning of latent features. Then, the parameters are initialized as $\alpha =1$,${\alpha }_0=0.1$, ${\beta }_0=10$, $K=6$.

4.2. Swimmer Dataset

The swimmer dataset contains all possible combinations of swimmer’s body postures (including limbs and trunk), with a total of 256 grayscale images. Figure 6 shows some samples of the dataset, depicting four simplified human figures with limbs and showing the different joints of the limbs. In this paper, the dataset is represented as a $256\times 1024$-dimensional matrix $X$, and each column represents a $32\times 32$-dimensional image.

The experiment is designed to observe 16 variable limb features and 1 fixed trunk feature through model learning. The trunk and limbs should be separate parts. According to our model, 17 features are accurately extracted, as shown in Figure 7.

On the contrary, most previous studies cannot successfully separate the backbone part. Since our algorithm is based on the NMF framework, the result of the basic NMF is first shown in Figure 8a. This result is derived from the research results of Gao Liang et al. in [33]. It can be seen that the eighth base image in the first line is almost identical to the sixth in the second line. The body part exists in each extracted feature, so the basic NMF method cannot independently extract the inherent or variable features of the dataset, and there is obvious overlap between the base feature values.

Figure 8b shows the result of feature extraction in [34]. It can be seen that the four basic images that have been specially labeled are all a combination of limbs and the trunk. This is contrary to common sense.

Referring to two comparative experimental methods, a series of actual factors have been “polluted” to a certain extent. The results show that the proposed method has good data representation and feature recognition ability.

In addition, Figure 9 shows the basis matrix $Z$. The parameters are initialized as $\alpha =1, {\sigma }_n=0.1, {\alpha }_0=1, {\beta }_0=1, K=20$.

4.3. Document Clustering

In this subsection, we apply the proposed model to a practical task: document clustering. The Cora dataset is used for experimental data that contains 2708 papers, including 1433 unique words after removing the stop words and the words that appear less than 10 times in the literature. The labels of these documents are their research field, about 7 categories. In order to better measure the effect of clustering task, we use three evaluation indicators: Jaccard coefficient (JC), Fowlkes and Mallows (FM), and F-measurement (F1). The larger their value, the better the clustering effect is, and their formulas are as follows:

$$\begin{gathered} JC=a/(a+b+c), \\ FM=\sqrt{a/\left(a+b\right)}\cdot \sqrt{a/\left(a+c\right)} , \\ F1=2a^2/\left(2a^2+ac+ab\right). \end{gathered}$$

(20)

where $a$ is the number of point pairs belonging to the same cluster in the benchmark results and test results, $b$ is the number of point pairs in the same benchmark but different cluster, and $c$ is the number of point pairs in the same cluster but different benchmark cluster.

The results of our method are compared with some parametric models, such as spectral clustering (SP), sparse NMF (SNMF), NMF, and also some Bayesian non-parametric models, which will be described in

Table 1. Clustering result of the Cora dataset.

Index		JC	FM	F1
Parametric Model or Algorithms	SNMF	0.1678	0.2882	0.2870
	NMF	0.1524	0.2729	0.2688
	SP	0.1691	0.3123	0.2737
Bayesian Non-parametric NMF Models	IBP	0.1650	0.2770	0.2890
	Bivariate-Beta doubly IBP	0.1720	0.2790	0.2930
	Gaussian process doubly IBP	0.1650	0.2630	0.2610
	Ours	0.1749	0.3173	0.2977

.

The benchmark comparison data are obtained from [35]. The Bivariate-Beta distribution (BB-dIBP-NMF) and Gaussian process (GP-dIBP-NMF) are proposed based on doubly IBP. Finally, in our model, 14 latent features are obtained from the dataset, others are 17 (from BB-dIBP-NMF), 22 (from GP-dIBP-NMF). It can be seen that, compared with various methods mentioned in other papers, our algorithm obtains the best clustering results with the least features. Figure 10 shows this result more visually.

The parameters are initialized as $\alpha =20, {\sigma }_n=0.1, \lambda =100.$ And the square difference $\sigma$ of spectral clustering is set to 0.2.

4.4. Face Feature Extraction

Not only can NMF decompose quickly and accurately, but the decomposition result has definite physical meaning. Lee and Seung [2] took face recognition as an example to decompose the face into separate parts such as nose, eyes, mouth and eyebrows. This is consistent with the intuitive visual concept of “the parts constitute the whole” in human thinking. And they compared the decomposition results of the NMF algorithm with those of common matrix factorization methods such as vector quantization (VQ) and PCA. The difference is that NMF is based on partial facial representations, while VQ and PCA are global representations. Therefore, we aim to apply the proposed model to the same face dataset (CBCL Face Database) [36] and observe whether our model can accurately extract similar local features and completely reconstruct the entire face, which is conducive to verifying the effectiveness of our model.

The CBCL Face Database contains many face and non-face images. It is widely used in the Biology and Computing Learning Center of the MIT. We use 2429 face images in the training set. Figure 11 shows some face images in this dataset.

According to the algorithm model, we extracte 51 “local” facial features from 2429 images, while 49 features are extracted by Lee and Seung [2]. Figure 12 shows facial feature extracted by our model, where white pixels represent features and the black is set as background.

Figure 13 shows the comparison between the reconstructed specific face image and the original image. By calculation, the mean square error rate of the reconstruction of the whole dataset is only 8%.

The parameters are initialized as $\alpha =12, {\sigma }_n=1, {\alpha }_0=3, {\beta }_0=1$.

5. Conclusions

As we know, traditional NMF models need to assume the number of basis to be fixed which makes them less flexible for many applications. IBP provides a priori of an infinite binary matrix, which is usually used to infer latent features and their quantities from a set of observation data. In this paper, we propose an improved IBP based probabilistic NMF framework, which uses exponential distribution and $\left\{0,1\right\}$ onstraints to limit the non-negativity of the weight matrix and the basis matrix. This method allows us to flexibly choose appropriate parameters as the basis to avoid the problems caused by the initial decomposition rank setting, and to automatically learn the latent features. At the same time, an exponential Gaussian model is constructed based on this framework, and the true posterior of two factor matrices is derived with the variational Bayes method. Compared with the benchmark methods, the proposed method has achieved better results on both synthetic and real datasets, and it shows higher decomposition efficiency. However, the proposed model has great computational complexity. The efficiency of execution and the reduction of model complexity may be one of the future research contents.