Latent Multi-View Semi-Nonnegative Matrix Factorization with Block Diagonal Constraint

Abstract

Multi-view clustering algorithms based on matrix factorization have gained enormous development in recent years. Although these algorithms have gained impressive results, they typically neglect the spatial structures that the latent data representation should have, for example, the ideal data representation owns a block structure just like the indicator matrix has. To address this issue, a new algorithm named latent multi-view semi-nonnegative matrix factorization with block diagonal constraint (LMSNB) is proposed. First, latent representation learning and Semi-NMF are combined to get a lower-dimensional representation with consistent information from different views. Second, the block diagonal constraint is able to capture the global structure of original data. In addition, the graph regularization is considered in our model to preserve the local structure. LMSNB can deal with negative data matrix and be applied to more fields. Although the low dimensional representation from semi-nonnegative matrix factorization loses some valuable information, it still has same structure as original data with the help of block diagonal constraint and graph regularization. Finally, an iterative optimization algorithm is proposed for our objective problem. Experiments on several multi-view benchmark datasets demonstrate the effectiveness of our approach against other state-of-the-art methods.

1. Introduction

Clustering, as a fundamental topic in machine learning, is able to divide the data into several disjoint sets according to the similarity between samples. Many clustering algorithms [1,2,3,4], based on different model assumptions [5,6], are proposed to deal with different data types [7] and can be applied into different scientific areas [8,9,10].

Multi-view data, as a new data type, has emerged in many fields with the advent of the information era. For example, a video can be described by subtitles, audio and images, respectively. Text news can be translated into different languages. A picture can be described by LBP, HOG and GIST, etc. Generally, an object can be depicted by several ways and thus multi-view data is common in our world. In the past two decades, the clustering analysis for multi-view data has become a hot research topic and attracted much attention in machine learning [11,12] and computer vision communities [13].

For multi-view clustering (MVC), the primary idea tends to concatenate the features of different views directly to make a unified data [14]. However, the unified data usually suffers from redundant information and over-high dimensional features, and thus causes a poor clustering performance. Therefore, researchers have tried to extract the complementary information from different views to enhance clustering performance, and numerous MVC methods have been proposed in recent years [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32]. We divide them into four categories roughly: co-training algorithm [20,21,22], multi-kernel learning [23,24], graph-based method [25,26,27], subspace-learning-based model [28,29,30,31,32]. In particular, subspace learning for multi-view data have gained significant development due to the outstanding performance.

The “intact space” [33], which generated from subspace learning, is a good paradigm for fusing the multi-view information. The main assumption of “intact space” is that multi-view data can be viewed as the subspaces of a latent intact representation. Driven by this idea, many new multi-view clustering methods have been proposed [34,35,36,37,38,39,40]. To pursue a more robust and effective subspace representation, Zhang et al.  [34] proposed the Latent Multi-view Subspace Clustering (LMSC) method where the latent representation learning and subspace learning are performed at the same time. Chen et al.  [38] proposed Multi-view Clustering in Latent Embedding Space (MCLES) algorithm, which learnt the similarity and the indicator matrix simultaneously. Noticed that the multi-view data usually contains too much noise, Xie et al.  [39] proposed Adaptive Latent Similarity Learning for multi-view clustering (ALSL) algorithm, which integrated the similarity structure of each view deploying the idea of latent space.

The latent space learning mentioned above is a matrix factorization technique intrinsically. In recent years, many other algorithms based on matrix factorization, such as nonnegative matrix factorization (NMF) [41] and its extensions [42,43], were introduced into multi-view clustering [44,45,46,47]. NMF algorithm is a significant method to get a desirable low-dimensional data representation for the data matrices whose elements are nonnegative, e.g., face and text. Based on NMF, Cai et al.  [42] considered the graph regularization, and proposed Graph Regularized Non-negative Matrix Factorization (GNMF). The new data representation in GNMF captures the intrinsic geometric structure of the original data in most cases. In addition, considering the data that we confront are not always nonnegative, an extension named Semi-NMF was proposed [43], which only imposed the nonnegative constraint on encoding matrix but free the basis matrix. In 2013, Liu et al.  [44] first introduced NMF into multi-view clustering, the proposed method supplied a compatible encoding matrix from multiple views. The literatures [45,46] deployed GNMF to deal with multi-view data, both of them got good clustering results. Zhao et al.  [47] proposed a deep matrix factorization for multi-view clustering, where Semi-NMF was deployed to learn a common nonnegative representation from multiple views.

GNMF for multi-view clustering is able to capture the local structure by graph regularized term. Moreover, deep matrix factorization for multi-view clustering can reduce the dimension of original data step by step and thus gets a layer structure. This is the reason that the existing approaches based on matrix factorization achieved impressive results on multi-view clustering. However, they usually ignore the block diagonal structure of the new data representation, which may cause degraded results. To address this issue, a k-block diagonal constraint is deployed to capture the block diagonal structure and thus we propose an algorithm named latent multi-view semi-nonnegative matrix factorization with block diagonal constraint (LMSNB). Along with Semi-NMF, the latent space learning can get a robust fused representation. In addition, graph regularization is adopted to capture the local geometry. Furthermore, the k-block diagonal constraint in our model not only deploys the prior information sufficiently, but preserves the global structure of the original data. By applying the Augmented Lagrangian Multiplier with Alternating Direction Method of Multipliers (ALM-ADMM) algorithm, our objective model can be optimized effectively. Extensive experiments are carried on several benchmark datasets to assess the performance of our proposed method. Figure 1 illustrates the general framework of our method.

In summary, the main contributions of this paper are:

• Benefited from the latent representation and Semi-NMF, our algorithm can get a robust low-dimensional representation that fused the consistent information of the multiple views.
• Deploying the graph regularization, our model is able to keep the local geometry consistency between the new low-dimensional representation and original multi-view data.
• By adding the k-block diagonal constraint, not only our model sufficiently utilizes the prior information, but the new low-dimensional representation captures the global structure.

The remainder of this paper is organized as follows. In Section 2, we give a brief reviewing of the related works, such as Semi-NMF and block diagonal constraint. After that, we present our proposed multi-view clustering method in Section 3. The optimization of our unified objective problem is displayed in Section 4. The performance of our algorithm and other comparative methods are summurized in Section 5. Finally, Section 6 concludes our work.

2. Notations and Related Works

In this section, we briefly review some notions closely related to our proposed method. Firstly, we give some symbols this paper used.

2.1. Notations

In this paper, matrices are represented with capital symbols, and vectors with lower-case symbols. For an arbitrary matrix $A\in {\mathbb{R}}^{m\times n}$, $A_{i,:}$ and $A_{:,j}$ denote the i-th row and j-th column of A respectively. $A\ge 0$ means all elements of A are nonnegative. $\langle A,B\rangle$ represents the inner product of A and B. ${∥A∥}_F$ is the Frobenius norm which defined as ${∥A∥}_F=\sqrt{{\sum }_{i=1}^m{\sum }_{j=1}^n{A}_{ij}^2}$. ${∥A∥}_{2,1}$ is the $l_{2,1}$ norm which defined as ${∥A∥}_{2,1}={\sum }_{j=1}^n\sqrt{{\sum }_{i=1}^m{A}_{ij}^2}$. If A is positive semi-definite, we write as $A⪰0$. For matrices $A,B\in {\mathbb{R}}^{n\times n}$, we denote $A⪯B$ or $B⪰A$ if $B-A⪰0$. $\mathrm{Tr}\left(A\right)$ is the trace of A. Infinity norm ${∥·∥}_{\infty }$ is the largest value in absolute elements. For vector a, ${∥a∥}_2$ represents the $l_2$ norm, and $\mathrm{Diag}\left(a\right)$ stands for a diagonal matrix with the elements of vector a. $\mathrm{diag}\left(A\right)$ is a column vector composed by the diagonal elements of matrix A. In addition, vector $\mathbf{1}$ denotes a column vector where all elements are ones. I refers to an identity matrix with proper dimension.

2.2. NMF and Semi-NMF

The nonnegative matrix factorization (NMF) [41] is a significant method to obtain a desirable low-dimensional data representation. Conceretly, let $X=\left[x_1,x_2,\dots ,x_n\right]\in {\mathbb{R}}_{+}^{m\times n}$ be the data matrix that contains n data points with m features, here, the symbol ${\mathbb{R}}_{+}$ represents nonnegative real number. The goal of NMF is to approximate the original data matrix as much as possible by using the product of two matrices in the following,

$$X\approx U{V}^{\mathrm{T}},$$

(1)

where $U\in {\mathbb{R}}_{+}^{m\times k}$, $V\in {\mathbb{R}}_{+}^{n\times k}$, and k is the desired lower dimension ($k<m$). Matrices U and V are also called as basis matrix and encoding matrix, respectively. The i-th row of V is regarded as the low dimensional representation of the original data point $x_i$. Matrices U and V are mainly founded by Frobenius norm, Kullback-Leibler divergence [48], or other divergences [49,50].

Although NMF has achieved great success in clustering analysis, it’s hard to process the data with negative elements. Thus, Ref. [43] proposed semi-nonnegative matrix factorization (Semi-NMF) algorithm to deal with the arbitrary data matrices. In detail, Semi-NMF restricts V in Equation (1) to be nonnegative while places no constraint on U. Based on Frobenius norm, the objective function of Semi-NMF can be written as follows,

$$\underset{U,V}{min}{∥X-U{V}^{\mathrm{T}}∥}_F^{2}s.t.V\ge 0.$$

(2)

2.3. Block Diagonal Constraint

The block diagonal constraint [51] was first introduced to pursue the block diagonal structure of affinity matrix in subspace segmentation. The block diagonal constraint can be guaranteed by the following theorem.

Theorem 1 

([2]). For any $M\ge 0$, $M=M^{\mathrm{T}}$, the Laplacian matrix $L_M$ is defined as $L_M=\mathrm{Diag}\left(M\mathbf{1}\right)-M$. Then, the multiplicity k of the eignvalue 0 of $L_M$ equals the number of connected components (blocks) in M.

By Theorem 1, both the block diagonal SSC and LRR algorithms in [51] obtain affinity matrices Z with better block diagonal structure under the constraint

$$\mathcal{K}=\left\{Z|\mathrm{rank}\left(L_M\right)=n-k,M=(\left|Z\right|+\left|Z^{\mathrm{T}}\right|)/2\right\}.$$

Then, the clustering results are superior to the original SSC and LRR methods.

Moreover, Lu et al. [52] proposed the k-block diagonal regularizer ${∥M∥}_k$ ($M\ge 0$, $M=M^{\mathrm{T}}$), which is the sum of the k smallest eignvalues of $L_M$. The k-block diagonal regularizer has the property that ${∥M∥}_k=0$ iff the affinity matrix M owns the k-block diagonal structure. Namely, ${∥M∥}_k$ will pursue the block diagonal structure when the subspaces are independent.

Naturally, we consider the Semi-NMF with block diagonal constraint. For the given data matrix $X=[X_1,X_2,\dots ,X_k]$, which is composed by k clusters and each $X_i$ represents a group of data samples, $V{V}^{\mathrm{T}}$ will be a k-block diagonal matrix in the ideal case, where V is the encoding matrix in Semi-NMF just as problem (2) shows. Inspired by [52], it’s reasonable to impose the k-block diagonal constraint on $V{V}^{\mathrm{T}}$, i.e., ${∥V{V}^{\mathrm{T}}∥}_k$.

3. Proposed Method

In this section, we will systematically introduce the LMSNB method.

3.1. Latent Representation Learning

Given a multi-view data matrix $X=[X^{\left(1\right)};X^{\left(2\right)};\dots ;X^{\left(v\right)}]\in {\mathbb{R}}^{M\times N}$ composed by v different views, $X^{\left(i\right)}\in {\mathbb{R}}^{m_i\times N}$ is the data of the i-th view, where $m_i$ and N are the number of features and samples in the i-th view, respectively. The purpose of latent representation is to learn a shared representation from different views. Following the idea in [34], the latent representation model can be written as follows,

$$X^{\left(i\right)}=P^{\left(i\right)}H+E^{\left(i\right)},i=1,\dots ,v,$$

(3)

where $P^{\left(i\right)}\in {\mathbb{R}}^{m_i\times K}$ and $E^{\left(i\right)}\in {\mathbb{R}}^{m_i\times N}$ are the projection matrix and the error matrix corresponding to i-th view, respectively. $H\in {\mathbb{R}}^{K\times N}$ is the shared latent representation, K is the dimension of the latent space. Since $X^{\left(i\right)}-P^{\left(i\right)}H=X^{\left(i\right)}-\left(a{P}^{\left(i\right)}\right)(H/a)$ (a is a positive constant), we have $a\to \infty$, $H/a\to 0$. Thus, we impose extra constraint $P^{\left(i\right)}{P}^{\left(i\right)\mathrm{T}}=I$ to prevent the obtained H arbitrarily close to zero. For v views, we can rewrite Equation (3) as Equation (4).

$$X=PH+E,$$

(4)

where $X\in {\mathbb{R}}^{M\times N}$ is the multi-view data matrix mentioned above, $P=[P^{\left(1\right)};P^{\left(2\right)};\dots ;P^{\left(v\right)}]\in {\mathbb{R}}^{M\times K}$ and $E=[E^{\left(1\right)};E^{\left(2\right)};\dots ;E^{\left(v\right)}]\in {\mathbb{R}}^{M\times N}$.

In the practical applications, various loss measures are used on E, such as, $l_1$ loss for random corruptions, $l_2$ loss for white noise, and $l_{2,1}$ loss for sample-specific outliers. In this paper, we concentrate on the robustness of our model for outliers, the objective function of latent representation learning can be formulated as follows,

$$\begin{array}{cc}\hfill & \underset{E\in {\mathbb{R}}^{M\times N}}{min}{∥E∥}_{2,1}\hfill \\ \hfill & s.t.X=PH+E,P^{\left(i\right)}{P}^{\left(i\right)\mathrm{T}}=I,i=1,\dots ,v.\hfill \end{array}$$

(5)

3.2. Lower Dimensional Representation Learning

Noticed that the learned latent representation H may be high dimensional features, we tend to reduce the number of features by using dimension reduction algorithm. Semi-NMF [43] is a canonical algorithm for dimension reduction. Thus, we introduce the Semi-NMF to achieve the dimension reduction of latent representation H. The objective function for Semi-NMF of H can be written as follows,

$$\begin{array}{cc}\hfill & \underset{U\in {\mathbb{R}}^{K\times k},V\in {\mathbb{R}}^{N\times k}}{min}{∥H-U{V}^{\mathrm{T}}∥}_F^2,\hfill \\ \hfill & s.t.V\ge 0.\hfill \end{array}$$

(6)

3.3. Local Geometry Preserving

Generally speaking, if the nearby data points in the original space will also be close in the new space, we say that the model can preserve original local geometry. For this purpose, the graph regularization is applied to capture the local structure in our model.

Considering a graph with n vertices, each vertex corresponds to a data point, an edge is established between $x_i$ and $x_j$ if $i\in {\mathcal{N}}_k\left(j\right)\mathrm{or}j\in {\mathcal{N}}_k\left(i\right)$, where ${\mathcal{N}}_k\left(i\right)$ denotes the k nearest neighbors (Euclidean distance) of data point i. $S_{ij}$ is used to measure the similarity between the i-th and j-th point. There are several methods to define the weight matrix S, i.e., 0–1 Weighting, Heat Kernel Weighting [53] and Dot-Product Weighting [42]. In this paper, we choose the Heat Kernel Weighting, which is defined as

$$S_{ij}=\left\{\begin{array}{cccc}\hfill e^{-\frac{{∥X_{:,i}-X_{:,j}∥}_2^2}{\sigma }}& ,\hfill & \hfill & \mathrm{if}i\in {\mathcal{N}}_k\left(j\right)\mathrm{or}j\in {\mathcal{N}}_k\left(i\right),\hfill \\ \hfill 0& ,\hfill & \hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(7)

where $\sigma$ is the bandwidth parameter, $X\in {\mathbb{R}}^{M\times N}$ is the multi-view data matrix mentioned in Section 3.1. It’s obvious that S is a symmetric matrix. Based on the weight matrix S, the graph regularization term can be formulated as

$$\frac{1}{2}\sum _{i,j=1}^N{∥V_{i,:}-V_{j,:}∥}_2^2{S}_{ij}=\mathrm{Tr}\left(V^{\mathrm{T}}LV\right),$$

(8)

where V is the encoding matrix in problem (6), $L=D-S$ is the Laplacian matrix and D is a diagonal matrix with the i-th diagonal element $D_{ii}={\sum }_{j=1}^N{S}_{ij}$. By minimizing Equation (8), the nearby data points in the original space are likely to be close in the low-dimensional data space V. So, the objective function of graph regularization can be written as

$$\underset{V\in {\mathbb{R}}^{N\times k}}{min}\mathrm{Tr}\left(V^{\mathrm{T}}LV\right).$$

(9)

3.4. Our Proposed Model

By combining the aforementioned ideas, our model can be expressed as (10).

$$\begin{array}{cc}\hfill & \underset{E,P,H,U,V}{min}{∥E∥}_{2,1}+\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2+\alpha \mathrm{Tr}\left(V^{\mathrm{T}}LV\right)+\beta {∥V{V}^{\mathrm{T}}∥}_k\hfill \\ \hfill & s.t.X=PH+E,V\ge 0,P^{\left(i\right)}{P}^{\left(i\right)\mathrm{T}}=I^{\left(i\right)},i=1,\dots ,v,\hfill \end{array}$$

(10)

where $\lambda ,\alpha ,\beta >0$ are the trade-off parameters to balance different terms, $I^{\left(i\right)}$ is an identity matrix with proper dimension. Dealing with ${∥V{V}^{\mathrm{T}}∥}_k$, we need the following theorem.

Theorem 2 

([54], p. 545). Let $L_G\in {\mathbb{R}}^{n\times n}$ and $L_G⪰0$. Then,

$$\sum _{i=n-k+1}^n{\lambda }_i\left(L_G\right)=\underset{W}{min}\langle L_G,W\rangle s.t.0⪯W⪯I,\mathrm{Tr}\left(W\right)=k.$$

In our model, $L_G$ in Theorem 2 is just as $L_{V{V}^{\mathrm{T}}}=\mathrm{Diag}\left(V{V}^{\mathrm{T}}\mathbf{1}\right)-V{V}^{\mathrm{T}}$. It’s easy to prove that $L_{V{V}^{\mathrm{T}}}$ is a positive semi-definite matrix. Then, by Theorem 2, problem (10) is further equivalent to

$$\begin{array}{cc}\hfill & \underset{E,P,H,U,V,W}{min}{∥E∥}_{2,1}+\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2+\alpha \mathrm{Tr}\left(V^{\mathrm{T}}LV\right)+\beta \langle L_{V{V}^{\mathrm{T}}},W\rangle \hfill \\ \hfill & s.t.X=PH+E,V\ge 0,0⪯W⪯I,\mathrm{Tr}\left(W\right)=k,P^{\left(i\right)}{P}^{\left(i\right)\mathrm{T}}=I^{\left(i\right)},i=1,\dots ,v.\hfill \end{array}$$

(11)

4. Optimization

In this section, we will give a detailed optimization to (11) by the Augmented Lagrangian Multiplier with Alternating Direction Method of Multipliers (ALM-ADMM). Our objective problem (11) can be solved by minimizing the following ALM problem

$$\begin{array}{cc}\hfill \mathcal{L}(E,P,H,U,V,W)=& {∥E∥}_{2,1}+\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2+\alpha \mathrm{Tr}\left(V^{\mathrm{T}}LV\right)\hfill \\ \hfill & +\beta \langle L_{V{V}^{\mathrm{T}}},W\rangle +\Phi (Y,X-PH-E)\hfill \\ \hfill & +\langle \Psi ,V\rangle \hfill \\ \hfill s.t.0⪯W⪯I,\mathrm{Tr}\left(W\right)& =k,P^{\left(i\right)}{P}^{\left(i\right)\mathrm{T}}=I^{\left(i\right)},i=1,\dots ,v,\hfill \end{array}$$

(12)

where $\Phi (Y,A)=\frac{\mu }{2}{∥A∥}_F^2+\langle Y,A\rangle$, $\mu >0$ is a penalty scalar. For optimization, We separate our problem into the following seven subproblems.

1. $P^{\left(i\right)}$-subproblem

For solving $P^{\left(i\right)}$, we fix other variables and introduce Theorem 3, which could output an approximate solution for the following problem (13). Notice that ${\left\{P^{\left(i\right)}\right\}}_{i=1}^v$ are independent from each other, we can solve each $P^{\left(i\right)}$ individually. Moreover, we use $Y^{\left(i\right)}$ to denote the Lagrange multiplier corresponding to $X^{\left(i\right)}-P^{\left(i\right)}H-E^{\left(i\right)}$ of i-th views.

$$\begin{array}{cc}\hfill P^{\left(i\right)*}& =\underset{P^{\left(i\right)}}{argmin}\Phi (Y^{\left(i\right)},X^{\left(i\right)}-P^{\left(i\right)}H-E^{\left(i\right)})\hfill \\ \hfill & =\underset{P^{\left(i\right)}}{argmin}\frac{\mu }{2}{∥{(X^{\left(i\right)}+\frac{Y^{\left(i\right)}}{\mu }-E^{\left(i\right)})}^{\mathrm{T}}-H^{\mathrm{T}}{P}^{\left(i\right)\mathrm{T}}∥}_F^2\hfill \\ \hfill s.t.P& {}^{\left(i\right)}{P}^{\left(i\right)\mathrm{T}}=I^{\left(i\right)},i=1,\dots ,v.\hfill \end{array}$$

(13)

Theorem 3 

([55]). Given the objective function ${min}_R{∥Q-GR∥}_F^2$, s.t. $R^{\mathrm{T}}R=R{R}^{\mathrm{T}}=I$, the optimal solution is $R=U{V}^{\mathrm{T}}$, where U and V are left and right singular vectors of SVD of $G^{\mathrm{T}}Q$, respectively.

By Theorem 3, we have the optimal solution $P^{\left(i\right)*}=V{U}^{\mathrm{T}}$, where U and V are the left and right singular vectors of SVD of $H{(X^{\left(i\right)}+Y^{\left(i\right)}/\mu -E^{\left(i\right)})}^{\mathrm{T}}$.

Actually, $P^{\left(i\right)}$ may not be an orthogonal matrix as the Theorem 3 needs. But, according to the suggestion in [34], we relax the matrix R in Theorem 3 to be row orthogonal and obtain the promising results in our experiments.

2. H-subproblem

Fixing other variables, the terms with H are as (14).

$$\begin{array}{cc}\hfill H^{*}& =\underset{H}{argmin}\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2+\Phi (Y,X-PH-E)\hfill \\ \hfill & =\underset{H}{argmin}\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2+\frac{\mu }{2}{∥X-PH-E∥}_F^2+\mathrm{Tr}\left(Y^{\mathrm{T}}(X-PH-E)\right).\hfill \end{array}$$

(14)

Taking the partial derivative of Equation (14) with respect to H and setting the formula equals to zero, we can get the closed-form solution of H as follows,

$$H^{*}={(2\lambda I+\mu P^{T}P)}^{-1}(2\lambda U{V}^T+\mu P^{T}X-\mu P^{T}E+P^{T}Y).$$

(15)

3. U-subproblem

Similar to H-subproblem, we obtain the optimal solution of U as Equation (16).

$$U^{*}=\underset{U}{argmin}\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2.$$

$$U^{*}=HV{\left(V^{\mathrm{T}}V\right)}^{-1}.$$

(16)

4. V-subproblem

The objective problem about V can be written as follows,

$$V^{*}=\underset{V\ge 0}{argmin}\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2+\alpha \mathrm{Tr}\left(V^{\mathrm{T}}LV\right)+\beta \langle L_{V{V}^{\mathrm{T}}},W\rangle +\langle \Psi ,V\rangle .$$

(17)

Since

$$\begin{array}{cc}\hfill \langle \mathrm{Diag}\left(V{V}^{\mathrm{T}}\mathbf{1}\right),W\rangle & =\langle \mathrm{Diag}\left(V{V}^{\mathrm{T}}\mathbf{1}\right),\mathrm{Diag}\left(w\right)\rangle \hfill \\ \hfill & =\langle V{V}^{\mathrm{T}}\mathbf{1},w\rangle \hfill \\ \hfill & =\langle V{V}^{\mathrm{T}},w{\mathbf{1}}^{\mathrm{T}}\rangle ,\hfill \end{array}$$

where $w=\mathrm{diag}\left(W\right)$. Then, for the the third term in (17), we have

$$\langle L_{V{V}^{\mathrm{T}}},W\rangle =\langle \mathrm{Diag}\left(V{V}^{\mathrm{T}}\mathbf{1}\right)-V{V}^{\mathrm{T}},W\rangle =\langle V{V}^{\mathrm{T}},w{\mathbf{1}}^{\mathrm{T}}-W\rangle .$$

Finally, the problem (17) is equivalent to

$$V^{*}=\underset{V\ge 0}{argmin}\lambda {∥H-U{V}^{\mathrm{T}}∥}_F^2+\alpha \mathrm{Tr}\left(V^{\mathrm{T}}LV\right)+\beta \langle V{V}^{\mathrm{T}},w{\mathbf{1}}^{\mathrm{T}}-W\rangle +\langle \Psi ,V\rangle .$$

(18)

Taking the partial derivative of Equation (18) with respect to V and setting it to zero. Using the KKT condition ${\Psi }_{ij}{V}_{ij}=0$, we have

$${\left(-2\lambda H^{\mathrm{T}}U+2\lambda V{U}^{\mathrm{T}}U+\beta \left(w{\mathbf{1}}^{\mathrm{T}}-W+{(w{\mathbf{1}}^{\mathrm{T}}-W)}^{\mathrm{T}}\right)V\right)}_{ij}{V}_{ij}=0.$$

As we all know, $Z=Z^{+}-Z^{-}$, where $Z_{ij}^{+}=\left(\right|Z_{ij}|+Z_{ij})/2$, $Z_{ij}^{-}=\left(\right|Z_{ij}|-Z_{ij})/2$. Finally the updating equation of V can be written as the entry-wise in the following,

$$V_{ij}=V_{ij}\frac{{(2\lambda {\left(H^{\mathrm{T}}U\right)}^{+}+2\lambda V{\left(U^{\mathrm{T}}U\right)}^{-}+2\alpha SV+2\beta W^{+}V)}_{ij}}{{(2\lambda {\left(H^{\mathrm{T}}U\right)}^{-}+2\lambda V{\left(U^{\mathrm{T}}U\right)}^{+}+2\alpha DV+\beta (w{\mathbf{1}}^{\mathrm{T}}+\mathbf{1}{w}^{\mathrm{T}})V+2\beta W^{-}V)}_{ij}}.$$

(19)

Regarding the updating rule Equation (19), we have the following theorem.

Theorem 4. 

The objective function (11) is nonincreasing under the updating rule (19).

Due to the space, please see the Appendix A for a detailed proof about the above theorem.

5. W-subproblem

$$\begin{array}{cc}\hfill W^{*}& =\underset{W}{argmin}\beta \langle L_{V{V}^{\mathrm{T}}},W\rangle \hfill \\ \hfill & =\underset{W}{argmin}\langle \mathrm{Diag}\left(V{V}^{\mathrm{T}}\mathbf{1}\right)-V{V}^{\mathrm{T}},W\rangle ,\hfill \\ \hfill s.t.& 0⪯W⪯I,\mathrm{Tr}\left(W\right)=k.\hfill \end{array}$$

(20)

It follows from [54] that $W=F{F}^{\mathrm{T}}$ is the optimal solution of (20), where $F\in {\mathbb{R}}^{N\times k}$ consists of k eigenvectors associated with the k smallest eigenvalues of $\mathrm{Diag}\left(V{V}^{\mathrm{T}}\mathbf{1}\right)-V{V}^{\mathrm{T}}$.

6. E-subproblem

$$\begin{array}{cc}\hfill E^{*}& =\underset{E}{argmin}{∥E∥}_{2,1}+\Phi (Y,X-PH-E)\hfill \\ \hfill & =\underset{E}{argmin}{∥E∥}_{2,1}+\frac{\mu }{2}{∥X-PH-E+\frac{Y}{\mu }∥}_F^2\hfill \\ \hfill & =\underset{E}{argmin}\frac{1}{\mu }{∥E∥}_{2,1}+\frac{1}{2}{∥E-(X-PH+\frac{Y}{\mu })∥}_F^2\hfill \\ \hfill & =\underset{E}{argmin}\frac{1}{\mu }{∥E∥}_{2,1}+\frac{1}{2}{∥E-G∥}_F^2,\hfill \end{array}$$

(21)

where $G=X-PH-Y/\mu$. This subproblem can be efficiently solved by Lemma 3.3 in [56].

7. Updating Multiplier

The update for the Lagrange multiplier Y is a simple dual ascent step.

$$Y=Y+\mu (X-PH-E).$$

(22)

The optimization steps are summarized in Algorithm 1. Just as the analysis in Section 2.3, $V{V}^{\mathrm{T}}$ will be a block diagonal matrix in the idea case, and each block corresponds to one cluster. Then we perform the standard spectral clustering on $V{V}^{\mathrm{T}}$ to get the final results.

Algorithm 1:LMSNB

Input: Multi-view data X, $\lambda$, $\alpha$, $\beta$, the dimension K of latent representation H. Initialize:$P=\mathbf{0}$, $E=\mathbf{0}$, $Y=\mathbf{0}$, $\mu =0.2$, $\rho =1.3$, ${\mu }_{max}={10}^5$, $t=0$, $maxiter=50$, $\epsilon ={10}^{-5}$. Initialize H, U, V with random values. Generating a weight matrix S of Multi-view data X by Equation (7). whilenot convergeddo end$\phantom{(}$ Output:P, H, E, U, V.

Now, we analyze the computational complexity of our proposed algorithm. Since the weight matrix S has been finished before the iteration, we omit the complexity about it. As mentioned in the above section, our method contains seven subproblems. Updating P costs $O({\sum }_{i=1}^v{m}_i^3+MNK+M{K}^2)$, where $N,M$ and K are the number of data samples, the number of features from all views and the dimension of latent space, respectively. The complexity of updating H is $O(K^3+N{K}^2+MNK+NKk+M{K}^2)$, where k is the number of clusters. For U-subproblem, the matrix inverse operation ${\left(V^{\mathrm{T}}V\right)}^{-1}$ costs $O(N{k}^2+k^3)$, and the total complexity for updating U is $O(k^3+N{k}^2+NKk+K{k}^2)$. For updating V, the complexity is $O(MNk+NKk+N^{2}k)$. The main complexity of updating W is from eigen-decomposition, which is $O\left(N^{2}k\right)$. Updating E and multiplier Y both need the complexity for $O\left(MNK\right)$. Overall, the computational complexity of our algorithm is $O(N^{2}k+{\sum }_{i=1}^v{m}_i^3+K^3)$ (since $k\ll K$) for each iteration.

5. Experiment

In this section, we give a detailed evaluation about the effectiveness of our proposed algorithm LMSNB. All the experiments are conducted on a laptop with a 3.20 GHz Intel Core i5 CPU, 16 GB RAM and MATLAB 2018b (64 bit).

5.1. Compared Algorithms

We compare our method with the state-of-art multi-view clustering algorithms, they are:

LMSC [34] seeks the shared latent representation for multi-view data and learns the self-expression matrix for the latent representation. This model is able to fuse the complementary information from several views, and thus produces a more accurate subspace representation.

LCRSR [37] recovers the latent complete row space and the sparse errors of the multi-view data under the assumption of latent representation. The dimension of learned row space is low enough and thus causes a low time consumption.

ECMSC [32] introduces a position-aware exclusivity constraint to obtain affinity matrices in different views, which have complementary information of different views. Thus ECMSC can obtain an indicator matrix with diverse information. Hence it has a good clustering performance.

MSC_IAS [40] constructs an intactness-aware similarity matrix deploying the intact space, which is learnt by fusing the multi-view information.

5.2. Evaluation Metrics

Following most of clustering researches, we adopt four commonly used metrics, including Accuracy (ACC), Normalized Mutual Information (NMI), F-score and Rand Index (RI), to measure the performance of the proposed method. For the above metrics, a higher value means a better clustering result.

5.3. Datasets

Six benchmark datasets are adopted in our experiments.

3Source (http://mlg.ucd.ie/datasets/3sources.html (accessed on 15 April 2022)): This dataset is collected from three online news sources: BBC, Reuters and the Guardian. In totally, there are 948 news articles covering 416 distinct news stories. We select the 169 stories among them which are reported in all three sources.

MSRCv1 (https://www.microsoft.com/en-us/research/project/image-understanding (accessed on 15 April 2022)): This dataset consists of 240 images and 9 objects classes. We select 7 classes containing 210 images in our experiments, such as tree, building, airplane, cow, face, car and bicycle. And 5 types of features are used by us: Centrist features, Color moment, GIST, Histogram of Oriented Gradient (HOG) and Local Binary Pattern (LBP).

Handwritten (HW) (https://archive.ics.uci.edu/ml/datasets/Multiple+Features (accessed on 15 April 2022)): This dataset contains 2000 data points with ten digit classes from 0 to 9, and there are 200 data points in each class. It consists of six views: Profile correlation, Fourier coefficient, Karhunen-loeve coefficient, Morphological features, Pixel average in $2\times 3$ windows and Zernike moments.

NUS-WIDE [57]: It is a real world web image dataset for object recognition problem. It contains six views: Color moments, Color histogram, Color correlogram, Wavelet texture, Edge distribution and Visual words.

MNIST: This dataset contains 2000 images of handwritten numerals from 0 to 9. Each image is representated by three types, which are IsoProjection (ISO), Linear Discriminant Analysis (LDA) and Neighborhood Preserving Embedding (NPE).

Scene15 [58]: This dataset contains 15 scene categories with indoor and outdoor environments, such as kitchen, store, highway and mountain, etc. And there are 4485 images in total. Three views [59] are used in our experiments: GIST, HOG and LBP.

Table 1. Statistics of the six datasets.

Datasets	Samples	Views	Clusters
3Source	169	3	6
MSRCv1	210	5	7
HW	2000	6	10
NUS-WIDE	1600	6	8
MNIST	2000	3	10
Scene15	4485	3	15

summarizes the basic information of the six datasets.

5.4. Clustering Results

In our experiments, the codes for the compared algorithms are given by the authors, and the parameters are tuned to the optimal performance. For our algorithm, the input data samples are normalized to unit $l_2$ norm. Similar to [34], we set the dimension K of the latent representation H as 100 for all datasets. As for similarity matrix S, we set the number of the neighborhood as 6 for all datasets, and the bandwidth parameter $\sigma$ equals to the mean of the Euclidean distances between every two data points. Now, we only need to tune the parameter $\lambda$, $\alpha$ and $\beta$, which are all searched from the candidate set $\{2^{-2},2^0,2^2,2^4,2^6,2^8,2^{10}\}$. We select the parameters corresponding to the optimal performances for each dataset (see

Table 2. The optimal values of the three parameters.

Parameter	3Source	MSRCv1	HW	NUS-WIDE	MNIST	Scene15
$\lambda$	$2^4$	$2^2$	$2^{-2}$	$2^2$	$2^0$	$2^{-2}$
$\alpha$	$2^4$	$2^4$	$2^{10}$	$2^0$	$2^4$	$2^2$
$\beta$	$2^0$	$2^0$	$2^0$	$2^0$	$2^0$	$2^0$

for a detail). For the optimal parameters, we repeat each algorithm 30 times and report the means and standard deviations in

Table 3. Clustering performance on six datasets (mean ± standard deviation %).

Datasets	Methods	ACC	NMI	F-Score	RI
3Sourse	LMSC	$\underline{64.97\pm 3.00}$	$53.07\pm 1.94$	$\underline{62.09\pm 3.59}$	$\underline{82.47\pm 2.11}$
	LCRSR	$55.56\pm 0.18$	$\underline{53.94\pm 0.93}$	$49.18\pm 0.92$	$78.91\pm 0.42$
	ECMSC	$33.61\pm 2.88$	$7.21\pm 1.22$	$33.31\pm 3.35$	$37.68\pm 10.96$
	MSC_IAS	$58.95\pm 6.92$	$50.20\pm 5.43$	$52.42\pm 6.00$	$79.96\pm 2.53$
	LMSNB	$\mathbf{75}.\mathbf{48}\pm \mathbf{3}.\mathbf{89}$	$\mathbf{66}.\mathbf{71}\pm \mathbf{3}.\mathbf{17}$	$\mathbf{72}.\mathbf{81}\pm \mathbf{3}.\mathbf{39}$	$\mathbf{88}.\mathbf{26}\pm \mathbf{1}.\mathbf{34}$
MSRCv1	LMSC	$70.49\pm 5.86$	$61.69\pm 6.04$	$58.59\pm 6.58$	$88.28\pm 1.82$
	LCRSR	$73.16\pm 0.26$	$63.41\pm 0.64$	$60.78\pm 0.43$	$88.80\pm 0.12$
	ECMSC	$\underline{81.90\pm 0.00}$	$74.62\pm 0.00$	$\underline{73.47\pm 0.00}$	$\underline{92.52\pm 0.00}$
	MSC_IAS	$80.24\pm 2.04$	$\underline{75.92\pm 1.33}$	$70.97\pm 1.49$	$91.53\pm 0.44$
	LMSNB	$\mathbf{90}.\mathbf{30}\pm \mathbf{0}.\mathbf{61}$	$\mathbf{83}.\mathbf{04}\pm \mathbf{0}.\mathbf{71}$	$\mathbf{81}.\mathbf{76}\pm \mathbf{0}.\mathbf{99}$	$\mathbf{94}.\mathbf{86}\pm \mathbf{0}.\mathbf{29}$
HW	LMSC	$82.53\pm 5.48$	$81.27\pm 1.84$	$77.86\pm 3.36$	$95.40\pm 0.78$
	LCRSR	$81.52\pm 4.29$	$74.24\pm 1.76$	$70.75\pm 3.12$	$94.12\pm 0.66$
	ECMSC	$81.60\pm 0.00$	$85.55\pm 0.00$	$81.39\pm 0.00$	$96.09\pm 0.00$
	MSC_IAS	$\mathbf{96}.\mathbf{96}\pm \mathbf{0}.\mathbf{10}$	$\mathbf{93}.\mathbf{41}\pm \mathbf{0}.\mathbf{17}$	$\mathbf{94}.\mathbf{05}\pm \mathbf{0}.\mathbf{19}$	$\mathbf{98}.\mathbf{82}\pm \mathbf{0}.\mathbf{04}$
	LMSNB	$\underline{96.48\pm 0.02}$	$\underline{92.59\pm 0.05}$	$\underline{93.16\pm 0.05}$	$\underline{98.64\pm 0.01}$
NUS-WIDE	LMSC	$31.66\pm 1.79$	$15.92\pm 1.14$	$21.87\pm 1.01$	$79.83\pm 0.29$
	LCRSR	$\underline{32.78\pm 0.52}$	$16.66\pm 0.40$	$23.39\pm 0.30$	$\underline{80.51\pm 0.15}$
	ECMSC	$30.31\pm 0.00$	$\underline{19.13\pm 0.00}$	$\mathbf{24}.\mathbf{95}\pm \mathbf{0}.\mathbf{00}$	$77.57\pm 0.00$
	MSC_IAS	$29.33\pm 0.75$	$18.65\pm 0.56$	$22.59\pm 0.39$	$77.61\pm 0.35$
	LMSNB	$\mathbf{34}.\mathbf{53}\pm \mathbf{1}.\mathbf{63}$	$\mathbf{20}.\mathbf{15}\pm \mathbf{0}.\mathbf{91}$	$\underline{24.73\pm 0.39}$	$\mathbf{80}.\mathbf{70}\pm \mathbf{0}.\mathbf{37}$
MNIST	LMSC	$57.68\pm 4.63$	$52.64\pm 7.18$	$53.24\pm 4.36$	$84.92\pm 5.37$
	LCRSR	$\underline{86.94\pm 0.03}$	$74.87\pm 0.07$	$\underline{78.35\pm 0.06}$	$\underline{95.17\pm 0.01}$
	ECMSC	$79.50\pm 0.00$	$74.15\pm 0.00$	$73.97\pm 0.00$	$94.51\pm 0.00$
	MSC_IAS	$78.27\pm 3.35$	$\underline{76.99\pm 1.62}$	$74.63\pm 2.80$	$94.51\pm 0.72$
	LMSNB	$\mathbf{90}.\mathbf{28}\pm \mathbf{0}.\mathbf{12}$	$\mathbf{80}.\mathbf{10}\pm \mathbf{0}.\mathbf{19}$	$\mathbf{83}.\mathbf{82}\pm \mathbf{0}.\mathbf{21}$	$\mathbf{96}.\mathbf{39}\pm \mathbf{0}.\mathbf{05}$
Scene15	LMSC	$\underline{52.54\pm 2.32}$	$54.30\pm 2.19$	$\underline{42.50\pm 2.12}$	$\underline{91.28\pm 0.42}$
	LCRSR	$44.94\pm 1.58$	$45.89\pm 0.63$	$33.02\pm 0.96$	$90.59\pm 0.14$
	ECMSC	$49.39\pm 2.35$	$46.84\pm 0.90$	$37.75\pm 1.14$	$90.63\pm 0.40$
	MSC_IAS	$50.16\pm 1.39$	$\underline{54.71\pm 0.64}$	$38.41\pm 0.83$	$90.49\pm 0.39$
	LMSNB	$\mathbf{56}.\mathbf{99}\pm \mathbf{0}.\mathbf{95}$	$\mathbf{58}.\mathbf{36}\pm \mathbf{0}.\mathbf{18}$	$\mathbf{46}.\mathbf{06}\pm \mathbf{0}.\mathbf{60}$	$\mathbf{91}.\mathbf{84}\pm \mathbf{0}.\mathbf{15}$

. Moreover, we also plot the histograms of clustering performance on six datasets for a better visualization, see Figure 2.

In Table 3, the best results are marked in bold, while the second best results are represented by the underline values. We can observe that LMSNB achieves better performance compared with LMSC, LCRSR, ECMSC and MSC_IAS in most cases.

To be specific, for the results on textual dataset 3Source, our proposed approach achieves $10.51\%$, $13.64\%$, $10.72\%$ and $5.79\%$ average improvements over the most competitive algorithm LMSC in terms of ACC, NMI, F-score and RI, respectively. For most of the image datasets, our LMSNB achieves high clustering accuracy as well as small standard deviations. Taking MSRCv1 for an example, the average ACC and standard deviation are $90.30\%$ and $0.61\%$, respectively. Our LMSNB also gets the similar clustering results on MNIST. For the results on Scene15, LMSNB achieves $4.45\%$ improvement compared with LMSC in terms of ACC. The clustering results on NUS-WIDE are generally poor for all the compared algorithms, however, LMSNB achieves improvements around $3.74\%$ and $5.32\%$ in terms of ACC and NMI compared with the second best method LCRSR. On HW dataset, both MSC_IAS and LMSNB produce encouraging results. Our LMSNB performs only $0.48\%$, $0.82\%$, $0.89\%$ and $0.18\%$ lower than MSC_IAS in terms of ACC, NMI, F-score and RI, respectively, which are negligible differences.

On the whole, our LMSNB is outperformed with the compared algorithms for most of the datasets we used. The compared algorithm MSC_IAS and LMSC are proposed based on the latent intact space, however, MSC_IAS gains better results than LMSC in most cases. The reason may be that the former considers the local geometry, i.e., the intactness-aware similarity in MSC_IAS is learned by the graph regularization. Further, the superior of our algorithm is that LMSNB not only preseves the local geometry deploying the graph regularization, but captures the global information by adding the block diagonal constraint.

5.5. Parameter Sensitivity Analysis

In this part, we show the sentivity analysis of different parameters. There are three tuning parameters $\lambda$, $\alpha$ and $\beta$ in our final model (11). So, we tend to fix one and tune the others. Firstly, we tune $\lambda \in \{2^{-2},2^0,2^2,2^4,2^6,2^8,2^{10}\}$ and $\alpha \in \{2^{-2},2^0,2^2,2^4,2^6,2^8,2^{10}\}$ when $\beta$ is fixed to 1. Figure 3 shows that how the ACC changes with varying $\lambda$ and $\alpha$ when $\beta =1$. Next, we fix $\lambda =1$, and tune the $\alpha \in \{2^{-2},2^0,2^2,2^4,2^6,2^8,2^{10}\}$ and $\beta \in \{2^{-2},2^0,2^2,2^4,2^6,2^8,2^{10}\}$. Figure 4 shows that how the ACC changes with varying $\alpha$ and $\beta$ when $\lambda =1$.

From Figure 3, we can see that our method maintains relatively stable clustering performance over a wide range under most of the datasets we used. Specifically, for NUS-WIDE, MNIST and Scene15, our algorithm supplies a good performance on all the parameters we used. For 3Source and MSRCv1, the ACC performance would be better when the parameter $\alpha$ takes the middle value of its range. As for HW, a larger $\alpha$ will lead to the better results for our algorithm.

From Figure 4, our algorithm performs the relatively stable clustering results on NUS-WIDE and MNIST dataset. For 3Source, HW and Scene15, we can observe that a larger $\alpha$ may produce the better clustering results, and a smaller $\beta$ also performs the better clustering results.

5.6. Ablation Study

In this section, we will investigate the effectiveness of the block diagonal constraint ${∥V{V}^{\mathrm{T}}∥}_k$. We remove the k-block diagonal constraint term in model (10) as comparative experiment (LMSN) and show the results in

Table 4. The ACC performance of ablation study (mean ± standard deviation %).

Datasets	LMSNB	LMSN	t-Test
3Sourse	$\mathbf{75}.\mathbf{48}\pm \mathbf{3}.\mathbf{89}$	$73.22\pm 3.26$	Yes
MSRCv1	$\mathbf{90}.\mathbf{30}\pm \mathbf{0}.\mathbf{61}$	$87.49\pm 1.35$	Yes
HW	$96.48\pm 0.02$	$\mathbf{96}.\mathbf{50}\pm \mathbf{0}.\mathbf{01}$	Yes
NUS-WIDE	$34.53\pm 1.63$	$\mathbf{36}.\mathbf{88}\pm \mathbf{0}.\mathbf{98}$	Yes
MNIST	$\mathbf{90}.\mathbf{28}\pm \mathbf{0}.\mathbf{12}$	$81.78\pm 1.78$	Yes
Scene15	$\mathbf{56}.\mathbf{99}\pm \mathbf{0}.\mathbf{95}$	$54.35\pm 1.79$	Yes

The best results are marked in bold.

. We run each experiment 30 times and then use t-test at 0.05 significance level to illustrate the statistical significance of the performance produced by LMSNB and LMSN.

From Table 4, we can observe that LMSNB gains better clustering results than LMSN in most of the datasets we used. In particular, our algorithm LMSNB achieves nearly $9\%$ promotion on MNIST dataset due to the block-diagonal constraint. In a word, the k-block diagonal constraint ${∥V{V}^{\mathrm{T}}∥}_k$ can be an effecitve constraint and lead to better results in most cases.

5.7. Visualization of $V{V}^{\mathrm{T}}$

Now, we show the visualization of $V{V}^{\mathrm{T}}$ on HW and MNIST (see Figure 5a,c), where the point tends to be yellow corresponding to a larger value while point rich in blue means a smaller value. It should be noted that the rows of V are permuted according to the true label of the samples. Since the existence of noise or samples located in the boundary of classes, the block diagonal structures are not obvious. Thus, for a better visualization, we also plot the binarized version of $V{V}^{\mathrm{T}}$ on the same datasets (see Figure 5b,d). The binarization of matrix A is defined as follows,

$$\mathrm{Binary}{\left(A\right)}_{ij}=\left\{\begin{array}{cccc}\hfill 1& ,\hfill & \hfill & \mathrm{if}{A}_{ij}⩾\delta ,\hfill \\ \hfill 0& ,\hfill & \hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(23)

where $\delta =A\left({\sum }_{g=1}^k{\left|C_g\right|}^2\right)$, $A\left(t\right)$ represents the t-th maximum element in matrix A. $\left|C_g\right|$ denotes the number of samples in the g-th cluster. We give a short explanation about the value of $\delta$. Actually, we hope that the number of nonzero elements in $\mathrm{Binary}\left(V{V}^{\mathrm{T}}\right)$ for g-th cluster equals to the square of the size of samples in the same cluster, i.e., ${\left|C_g\right|}^2$. Then, threshold parameter $\delta$ can be defined as the above. From Figure 5b,d, we can observe that $V{V}^{\mathrm{T}}$ indeed owns the block diagonal structure on the two given datasets. Based on this fact, it’s reasonable to impose the k-block diagonal constraint on $V{V}^{\mathrm{T}}$.

5.8. Time Analysis

In this section, we show the time consumptions of each algorithm, see

Table 5. The time consuming of different algorithms.

Methods	3Source	MSRCv1	HW	NUS-WIDE	MNIST	Scene15
LMSC	9.41	3.00	552.98	278.37	620.81	6253.14
LCRSR	2.93	1.30	13.79	14.56	4.98	465.73
ECMSC	183.04	2.21	158.14	97.06	53.43	1476.50
MSC_IAS	11.02	1.02	14.93	14.72	16.32	80.34
LMSNB	69.40	1.12	27.15	17.58	30.60	306.88

. From Table 5, we can see that the speed of our LMSNB is faster than LMSC and ECMSC on most of datasets, but usually slower than LCRSR and MSC_IAS. It should be noted that the time complexity of LMSNB is $O(N^{2}k+{\sum }_{i=1}^v{m}_i^3+K^3)$, which is cubic to $m_i$ (the number of features in i-th views). So when M is large, our LMSNB would cost too much time. Take 3Source for an example, although it contains only 169 samples, the numbers of features for three views are 3560, 3631 and 3068, respectively. Thus our algorithm has a longer time consuming.

6. Conclusions

In this paper, we propose an algorithm named latent multi-view semi-nonnegative matrix factorization with block diagonal constraint (LMSNB). Based on Section 5, k-block diagonal structure is usually found in most of datasets. Thus, the assumption of k-block diagonal constraint ${∥V{V}^{\mathrm{T}}∥}_k$ is reasonable. It can capture the global structure of the original data and lead to improved clustering performance. Moreover, multi-view latent representation learning, Semi-NMF and graph regularization are utilized to get a shared low-dimensional representation which captures the local geometry. Thus, our model LMSNB enable to produce a new representation which captures the global and local geometry. Experiment results on six multi-view datasets demonstrate the effectiveness of our algorithm.

There are two limitations for our LMSNB. First, for some datasets with high dimension, the time consuming of our algorithm is large. Second, there are three trade-off parameters need to be tuned in our model, which is a barrier for the real applications.

There are also several directions for our future work. First, however it’s unreasonable to set the dimension K as 100 for some datasets (since the constraint $P^{\left(i\right)}{P^{\left(i\right)}}^{\mathrm{T}}=I^{\left(i\right)}$), the promising results are still obtained, such as MSRCv1. We will try to explain it in our future work. Second, noticed that the importance of different views are ignored in our model, we will design a framework that considers the weight of different views. Finally, we are also interested in extending our model to semi-supervised tasks.