Consistent Regularized Non-Negative Tucker Decomposition for Three-Dimensional Tensor Data Representation

Abstract

Non-negative Tucker decomposition (NTD) is one of the general and prominent decomposition tools designed for high-order tensor data, with its advantages reflected in feature extraction and low-dimensional representation of data. Most NTD-based methods only apply intrinsic and different constraints to the last factor matrix that is a low-dimensional representation of the original tensor information. This processing procedure may result in the loss of the relationship between the factor matrices in all dimensions. To enhance the representation ability of NTD, we propose a consistent regularized non-negative Tucker decomposition for three-dimensional tensor data representation. Consistent regularization is symmetrically presented and mathematically expressed by intrinsic cues in multiple dimensions, that is, manifold structure and orthogonality information. The paired constraint constructed by the double parameter operator is utilized to unlock hidden semantics and maintain the consistent geometric structure of the three-dimensional tensor. Moreover, we develop the iterative updating method based on the multiplicative update rule to solve the proposed model and provide its convergence and computational complexity. The extensive numerical results of unsupervised image clustering experiments on eight real-world datasets demonstrated the feasibility and efficiency of the new method.

1. Introduction

In the realm of image restoration, image clustering, and image learning tasks, the curse of dimension poses a significant challenge. It becomes crucial to delve into the underlying data information and effectively condense it. Images traditionally are first transformed into one-dimensional long feature vectors representing data points during the data processing process. Subsequently, various valid methods are employed for dimension reduction and feature extraction. These include principal component analysis (PCA) [1], group component analysis (GCA) [2] and non-negative matrix factorization (NMF) [3,4]. When data exhibit symmetry, symmetric non-negative matrix factorization (symNMF) [5] is supposed to a more appropriate way to replace NMF work. In addition, inspired by matrix factorization and the idea of deep learning, deep matrix factorization (DMF) [6] have emerged. Actually, the one-dimensional vector representation of a gray scale image generally ignores the two-dimensional spatial relationships within the image, and formally overlooks the fundamental nature that an image should naturally be presented in the form of two-dimensional matrix. To explore low-dimensional representations in higher-order data, various tensor decomposition methods [7] have been developed, including non-negative Tucker decomposition [8], Bayesian tensor factorisation [9], tensor ring decomposition [10], and  CANDECOMP/PARAFAC (CP) decomposition [11]. Non-negative Tucker decomposition (NTD) gained popularity due to preserve the multi-way structure inherent in tensor data.

Numerous researchers have incorporated diverse graph Laplacian regularization terms or orthogonal constraints into non-negative matrix factorization and non-negative Tucker decomposition models, developing various model variants. Stemmed from matrix factorization, Cai et al. [12] stated the graph regularized NMF (GNMF) model, which constructs a nearest neighborhood graph to consider the immanent manifold structure of low-dimensional representation matrices. Shang et al. [13] raised a graph dual regularization model that makes use of both the geometric information of data matrix and the feature matrix based on the NMF concept. Huang et al. [14] utilized the global geometrical structure and formed a robust structure regularized NMF. Ding et al. [15,16] designed orthogonal NMF (ONMF) that is theoretically described to have a deep connection to K-means clustering. Li et al. [17] proposed a new model that can adjust the degree and strength of orthogonality, called approximately orthogonal NMF (AONMF), and corresponds to the fuzzy means. The above methods are aimed at generating vector objects, where each row or column of two-dimensional data represents a data point vector. Stemmed from Tucker decomposition, Qiu et al. [18] developed a graph Laplacian regularized NTD (GNTD) model that has good performance in image clustering problems. Subsequently, Qiu et al. [19] used an alternating proximate gradient descent method to solve the proposed GNTD model (UGNTD). Liu et al. [20] proposed a new effective model that mixes the graph regularization and $L_p$ smooth constraint into the NTD framework. Chen et al. [21] adaptively obtained the optimal graph and affirmed an adaptive graph regularized NTD (AGRNTD) method to preserve the geometric structure. Li et al. [22] asserted a manifold regularization NTD (MR-NTD), which captured the potential structure of the core tensor generated by NTD. The orthogonal NTD (ONTD) model [23] presents nice properties, in  which it acquires more clustering information from the orthogonal factor matrix. Qiu et al. [24] presented a flexible and elastic method without relying on external clustering tools, which is referred to as approximately orthogonal NTD (AONTD). Both single-constraint and multiple-constraint NTD frameworks demonstrate their excellent performance in image processing tasks.

Most NTD-based methods nowadays apply single or multiple constraints to the low-dimensional representation of NTD to enhance performance. Actually, the factor matrices obtained from NTD in each direction can be regarded as low-rank representation, and the factor matrices in each direction have duality and mutual correlation. Only delving deeply into the intrinsic structure of the last-dimensional direction factor matrix is incomplete consideration. This inspection and judgment loses the sophisticated semantics of other dimensions and, to some extent, disrupts the organic nexus between different dimensions. Motivated by the ideas, in this paper we construct a multi-linear NTD framework with consistent regularization to obtain more global and local information for image representation. The  graph regularization for each factor matrix catches the latent manifold implication and multi-dimensional linear structure of data. By regulating the quality of approximation for each factor matrix, we can detect more complex information among data. Furthermore, we propose multi-directional paired constrained NTD (MPCNTD) model, by jointly blending the geometrical structure and resilient orthogonality. The calculation rules and the intelligible proof of convergence about the new method are offered. Eight widely used datasets about clustering task are adopted to justify the effectiveness and robustness of the proposed MPCNTD method.

The remainder of this article is organized as follows. In Section 2, we provide the related models. In Section 3, we introduce the new MPCNTD model in detail. An efficient MU method is developed to solve the corresponding optimization problem. Section 4 presents the results of experiments on eight real-world datasets, and  conclusions and outlook are finally drawn in Section 5.

2. Preliminaries

Let $\mathcal{X}$ be a third-order tensor of size $I_1\times I_2\times I_3$, its frontal slice is $X^i$, $i=1,2,\cdots ,n$. In the clustering problem of image data, each frontal slice $X^i$ can represent a gray image. GNTD [18] consider the geometrical structure and decomposes the original tensor $\mathcal{X}$ into the multi-linear product of core tensor $\mathcal{G}$ and factor matrices $\mathbf{U}$, $\mathbf{V}$, $\mathbf{W}$. It is expressed as follows:

$$\begin{array}{cc}\hfill \underset{\mathcal{G},\mathbf{U},\mathbf{V},\mathbf{W}}{min}& {\mathcal{F}}_{GNTD}={\displaystyle \frac{1}{2}}{∥\begin{array}{c}\mathcal{X}-\mathcal{G}{\times }_1\mathbf{U}{\times }_2\mathbf{V}{\times }_3\mathbf{W}\end{array}∥}_F^2+\lambda \mathrm{Tr}\left({\mathbf{W}}^{\mathrm{T}}\mathbf{L}\mathbf{W}\right)\hfill \\ & s.t.\mathcal{G}⩾0,\mathbf{U},\mathbf{V},\mathbf{W}⩾0,\hfill \end{array}$$

(1)

where the operator ${\times }_r$ means r-mode product, and ${\left|\right|\mathcal{A}\left|\right|}_F^2={\sum }_i^{I_1}{\sum }_j^{I_2}{\sum }_k^{I_3}{\mathcal{A}}_{ijk}^2$ denotes Frobenius norm of tensor $\mathcal{A}$. The r-mode product of a tensor $\mathcal{Y}\in {\mathbb{R}}^{J_1\times J_2\times \cdots \times J_N}$ and a matrix $\mathbf{H}\in {\mathbb{R}}^{I_r\times J_r}$, denoted by $\mathcal{Y}{\times }_r\mathbf{H}$, is ${\left(\mathcal{Y}{\times }_r\mathbf{H}\right)}_{j_1\dots j_{r-1}{i}_r{j}_{r+1}\dots j_N}={\sum }_{j_r=1}^{J_r}{y}_{j_1\dots j_{r-1}{j}_r{j}_{r+1}\dots j_N}{h}_{i_r{j}_r}$. The graph-based tensor decomposition method, especially Tucker-based decomposition, effectively reveals the underlying geometric information of the obtained tensor. The theoretical knowledge of spectral graph and manifold learning indicate that the local geometric structures are naturally established by the nearest neighborhood graph of target data. The raw data points ${\mathbf{x}}_i,{\mathbf{x}}_j$ is converted into ${\mathbf{w}}_i,{\mathbf{w}}_j$ in the low-dimensional space, so $\mathbf{d}=\left|\right|{\mathbf{w}}_i-{\mathbf{w}}_j{\left|\right|}^2$ is used to measure the dissimilarity of ${\mathbf{w}}_i$ and ${\mathbf{w}}_j$. The  paradigm of graph Laplacian term is as follows:

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{2}}\sum _{i,j=1}^n\left|\right|{\mathbf{w}}_i-{\mathbf{w}}_j{\left|\right|}^2{\mathbf{S}}_{ij}& =\sum _{i=1}^n{\mathbf{w}}_i^{\top }{\mathbf{w}}_i\sum _{j=1}^n{\mathbf{S}}_{ij}-\sum _{i,j=1}^n{\mathbf{w}}_i^{\top }{\mathbf{w}}_j{\mathbf{S}}_{ij}=\sum _{i=1}^n{\mathbf{w}}_i^{\top }{\mathbf{w}}_i{\mathbf{D}}_{ii}-\sum _{i,j=1}^n{\mathbf{w}}_i^{\top }{\mathbf{w}}_j{\mathbf{S}}_{ij}\hfill \\ & =\mathrm{Tr}\left({\mathbf{W}}^{\mathrm{T}}\mathbf{D}\mathbf{W}\right)-\mathrm{Tr}\left({\mathbf{W}}^{\mathrm{T}}\mathbf{S}\mathbf{W}\right)=\mathrm{Tr}\left({\mathbf{W}}^{\mathrm{T}}\mathbf{L}\mathbf{W}\right)\hfill \end{array}$$

where $\mathbf{L}=\mathbf{D}-\mathbf{S}$ is the graph Laplacian, and $\mathbf{S}$ is a weight matrix that needs to be reasonably constructed. Minimizing the manifold regularization $\mathrm{Tr}\left({\mathbf{W}}^{\mathrm{T}}\mathbf{L}\mathbf{W}\right)$ means that if the data points ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ are closer in the high-dimensional space, then the data points ${\mathbf{w}}_i$ and ${\mathbf{w}}_j$ are also closer in the low-dimensional space, that is, the trend of change is accordant.

3. Model and Algorithm

In this section, the multi-directional paired constrained non-negative Tucker decomposition model is developed at length, the optimization formula is proposed, the convergence of the algorithm is demonstrated, and their computational complexity is provided.

3.1. Motivations and Model

It is theoretically rational to extend the orthogonality of two matrices in non-negative matrix factorization to the orthogonality of the factor matrices in non-negative Tucker decomposition. However, in view of fuzzy concepts, samples may have a certain degree of membership in each cluster. The orthogonality of factor matrix reduces the accuracy of the model solution, so we focus on its approximate orthogonal form. Taking the factor matrix in the last direction, $\mathbf{W}$, as an example here. The soft orthogonality of conception is executed through the following two steps:

• Step one: The unit-norm constraints ${\mathbf{w}}_r^T{\mathbf{w}}_r=1$ on the column of $\mathbf{W}$ are not mandatory, and direct removal is also feasible. For this item, we choose not to ignore it but to handle it to achieve a more comprehensive consideration. It can be implemented through ${\parallel \mathbf{W}\parallel }_2=1$ to attain unit-norm constraint, where r is number of columns in matrix $\mathbf{W}$.

• Step two: The orthogonality constraint is ${\mathbf{w}}_r^T{\mathbf{w}}_j=0, r\ne j,$ where ${\mathbf{w}}_i$ denotes the i-th column of the matrix $\mathbf{W}$. The orthogonality constraint of independent relationships between columns is of utmost importance. Considering the non-negativity constraints, we set ${\mathbf{w}}_r^T{\mathbf{w}}_j⩾0$ for any r and j. From the previous text, we have learned that ${\sum }_{r=1}^R{\sum }_{j=1,j\ne r}^R{\mathbf{w}}_r^{\top }{\mathbf{w}}_j=\mathrm{Tr}\left({\mathbf{W}}^{\top }\mathbf{Q}\mathbf{W}\right)$ is held, where $\mathbf{Q}=\mathbf{P}-\mathbf{I}=\mathbf{1}{\mathbf{1}}^{\top }-\mathbf{I}$, $\mathbf{1}$ be an all-one vector, $\mathbf{I}$ be an identity matrix.

The formula $\mathrm{Tr}\left({\mathbf{W}}^{\mathrm{T}}\mathbf{Q}\mathbf{W}\right)$ and ${\parallel \mathbf{W}\parallel }_2=1$ together form the concept of soft or approximate orthogonality of the three-dimensional representation matrix. The closer the value of $\mathrm{Tr}\left({\mathbf{W}}^{\mathrm{T}}\mathbf{Q}\mathbf{W}\right)$ is to 0, the higher the orthogonality of data. In this paper, we not only impose approximate orthogonal constraints on the three-dimensional representation matrix but also perform such operations on factor matrices in other directions.

For three-dimensional tensor data, we can obtain factor matrices in three directions using the NTD basic framework. Merely mining deep information from the matrix in the last direction, which will inevitably lack data information in the other two directions. However, these three factor matrices have equal status in terms of decomposition. For practical clustering tasks and convenience in writing, we define pairwise constraint operators as

$$\mathfrak{E}\left(\mathbf{A}\right)={\mu }_A\mathrm{Tr}\left({\mathbf{A}}^T\mathbf{Q}\mathbf{A}\right)+{\lambda }_A\mathrm{Tr}\left({\mathbf{A}}^T\mathbf{L}\mathbf{A}\right),$$

where the forms of $\mathbf{Q}$ and $\mathbf{L}$ are the same as the approximate orthogonal term and the graph regularization term mentioned earlier, respectively. The consistent regularization that the geometrical structure and softly orthogonality apply for three-factor matrices is used to better preserve multidimensional results and obtain more stable solutions. This operator linearly adds two mathematical expressions of two constraints. Moreover, $\mu$ and $\lambda$ are positive integers and are used to adjust constraint terms balancing the error in different problems.

We apply paired constraint operators to the three-factor matrix obtained through the NTD decomposition. The object function of the MPCNTD method is as follows:

$$\begin{array}{cc}\hfill \underset{\mathcal{G},\mathbf{U},\mathbf{V},\mathbf{W}}{min}& {\mathcal{F}}_{MPCNTD}={\displaystyle \frac{1}{2}}{∥\begin{array}{c}\mathcal{X}-\mathcal{G}{\times }_1\mathbf{U}{\times }_2\mathbf{V}{\times }_3\mathbf{W}\end{array}∥}_F^2+\mathfrak{E}\left(\mathbf{U}\right)+\mathfrak{E}\left(\mathbf{V}\right)+\mathfrak{E}\left(\mathbf{W}\right)\hfill \\ & s.t.\mathcal{G}⩾0,\mathbf{U},\mathbf{V},\mathbf{W}⩾0.\hfill \end{array}$$

(2)

In this model, ${\parallel \mathcal{X}-\mathcal{G}{\times }_1\mathbf{U}{\times }_2\mathbf{V}{\times }_3\mathbf{W}\parallel }_F^2$ is the main part. The variation of adjacent pixels in each factor matrix should be minimal to preserve more multidimensional linear information of the tensor $\mathcal{X}$. Applying $\mathfrak{E}(·)$ to matrices in three directions, it is consistent in form and symmetric in space. The $\mathfrak{E}(·)$ not only ensure that the factor matrices are approximate orthogonality, but also save internal information by graph optimization to conserve the smooth structure. It should be noted that the dimensions of $\mathbf{L}$ and $\mathbf{Q}$ will vary with the dimensions of the corresponding factor matrix. To reduce the definition, we will no longer emphasize it. The proposed MPCNTD model can plumb comprehensive and localized information, which is visually showed the flow of MPCNTD in Figure 1.

3.2. Computational Algorithm

The objective function (2) of MPCNTD is non-convex, and its global optimal solution is not easy to solve. The classic technique to overcome this challenge is the block coordinate descent method. Solving the sub-problems of MPCNTD, we use the multiplication updating rules to optimize. By using the Lagrange multiplier method, the converted objective function is as follows:

$$\begin{array}{cc}\hfill \mathcal{L}=& {\displaystyle \frac{1}{2}}{∥\begin{array}{c}\mathcal{X}-\mathcal{G}{\times }_1\mathbf{U}{\times }_2\mathbf{V}{\times }_3\mathbf{W}\end{array}∥}_F^2+\mathfrak{E}\left(\mathbf{U}\right)+\mathfrak{E}\left(\mathbf{V}\right)+\mathfrak{E}\left(\mathbf{W}\right)+\mathrm{vec}{\left(\Phi \right)}^T\mathrm{vec}\left(\mathcal{G}\right)\hfill \\ & +\mathrm{Tr}\left({\Psi }_1{\mathbf{U}}^T\right)+\mathrm{Tr}\left({\Psi }_2{\mathbf{V}}^T\right)+\mathrm{Tr}\left({\Psi }_3{\mathbf{W}}^T\right),\hfill \end{array}$$

(3)

where $\mathrm{vec}\left(\Phi \right)⩾0$, ${\Psi }_1⩾0$, ${\Psi }_2⩾0$ and ${\Psi }_3⩾0$ are the Lagrange multipliers matrices of core tensor $\mathcal{G}$, factor matrices $\mathbf{U}$, $\mathbf{V}$ and $\mathbf{W}$, respectively. The whole optimization process are the solutions of factor matrices $\mathbf{U}$, $\mathbf{V}$, $\mathbf{W}$ and core tensor $\mathcal{G}$.

The updating rule for 3-th factor matrix $\mathbf{W}$ is given first. We apply the mode-3 unfolding form, the Function (3) is the following format:

$$\begin{array}{cc}\hfill \mathcal{L}=& {\displaystyle \frac{1}{2}}\mathrm{Tr}\left({\mathbf{X}}_{\left(3\right)}{\mathbf{X}}_{\left(3\right)}^{\top }\right)-\mathrm{Tr}\left({\mathbf{X}}_{\left(3\right)}(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }{\mathbf{W}}^{\top }\right)+{\displaystyle \frac{1}{2}}\mathrm{Tr}\left(\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^T{\mathbf{W}}^{\top }\right)\hfill \\ & +{\mu }_U\mathrm{Tr}\left({\mathbf{U}}^T\mathbf{Q}\mathbf{U}\right)+{\lambda }_U\mathrm{Tr}\left({\mathbf{U}}^T\mathbf{L}\mathbf{U}\right)+{\mu }_V\mathrm{Tr}\left({\mathbf{V}}^T\mathbf{Q}\mathbf{V}\right)+{\lambda }_V\mathrm{Tr}\left({\mathbf{V}}^T\mathbf{L}\mathbf{V}\right)+{\mu }_W\mathrm{Tr}\left({\mathbf{W}}^T\mathbf{Q}\mathbf{W}\right)\hfill \\ & +{\lambda }_W\mathrm{Tr}\left({\mathbf{W}}^T\mathbf{L}\mathbf{W}\right)+\mathrm{vec}{\left(\Phi \right)}^{\mathrm{T}}\mathrm{vec}\left(\mathcal{G}\right)+\mathrm{Tr}\left({\Psi }_1{\mathbf{U}}^{\mathrm{T}}\right)+\mathrm{Tr}\left({\Psi }_2{\mathbf{V}}^{\mathrm{T}}\right)+\mathrm{Tr}\left({\Psi }_3{\mathbf{W}}^{\mathrm{T}}\right).\hfill \end{array}$$

(4)

When $\mathbf{W}$ is updated, other variables are constants. The partial derivative of $\mathcal{L}$ in (4) with respect to $\mathbf{W}$ is as follows: $\partial \mathcal{L}/\partial \mathbf{W}=-{\mathbf{X}}_{\left(3\right)}(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\lambda }_W\mathbf{L}\mathbf{W}+{\mu }_W\mathbf{Q}\mathbf{W}+{\Psi }_3.$ According to the Karush-Kuhn-Tucker (KKT) condition, $\partial \mathcal{L}/\partial \mathbf{W}=0$ and the equation $\mathbf{W}\odot {\Psi }_3=0$, where and in the following, ⊙ denotes the Hadamard product. We obtain the following: ${\Psi }_3={\mathbf{X}}_{\left(3\right)}(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }-\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }-{\lambda }_W\mathbf{L}\mathbf{W}-{\mu }_W\mathbf{Q}\mathbf{W}.$ By calculating $\mathbf{W}\odot {\Psi }_3=\mathbf{W}\odot \left({\mathbf{X}}_{\left(3\right)}(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }\right)-\mathbf{W}\odot (\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\lambda }_W\mathbf{L}\mathbf{W}+{\mu }_W\mathbf{Q}\mathbf{W}),$ the updating rule for $\mathbf{W}$ is as follows:

$${\mathbf{W}}_{ij}\leftarrow {\mathbf{W}}_{ij}{\displaystyle \frac{{\mathcal{P}}_{+}\left[{[{\mathbf{X}}_{\left(3\right)}(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\mu }_W\mathbf{I}\mathbf{W}+{\lambda }_W\mathbf{S}\mathbf{W}]}_{ij}\right]}{{[\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\mu }_W\mathbf{P}\mathbf{W}+{\lambda }_W\mathbf{D}\mathbf{W}]}_{ij}}},$$

(5)

where and in the following, ${\mathcal{P}}_{+}\left[\eta \right]=max(0,\eta )$.

Using the Lagrangian function and Karush-Kuhn-Tucker condition, we can similarly obtain the updating formulas for $\mathbf{U}$ and $\mathbf{V}$. Unfolding $\mathcal{X}$ along the mode-1 and mode-2 directions, the following updating rule   

$${\mathbf{V}}_{ij}\leftarrow {\mathbf{V}}_{ij}{\displaystyle \frac{{\mathcal{P}}_{+}\left[{[{\mathbf{X}}_{\left(2\right)}(\mathbf{W}\otimes \mathbf{U}){\mathbf{G}}_{\left(2\right)}^{\top }+{\mu }_V\mathbf{I}\mathbf{V}+{\lambda }_V\mathbf{S}\mathbf{V}]}_{ij}\right]}{{[\mathbf{V}{\mathbf{G}}_{\left(2\right)}{(\mathbf{W}\otimes \mathbf{U})}^{\top }(\mathbf{W}\otimes \mathbf{U}){\mathbf{G}}_{\left(2\right)}^{\top }+{\mu }_V\mathbf{P}\mathbf{V}+{\lambda }_V\mathbf{D}\mathbf{V}]}_{ij}}},$$

(6)

and

$${\mathbf{U}}_{ij}\leftarrow {\mathbf{U}}_{ij}{\displaystyle \frac{{\mathcal{P}}_{+}\left[{[{\mathbf{X}}_{\left(1\right)}(\mathbf{W}\otimes \mathbf{V}){\mathbf{G}}_{\left(1\right)}^{\top }+{\mu }_U\mathbf{I}\mathbf{U}+{\lambda }_U\mathbf{S}\mathbf{U}]}_{ij}\right]}{{[\mathbf{U}{\mathbf{G}}_{\left(1\right)}{(\mathbf{W}\otimes \mathbf{V})}^{\top }(\mathbf{W}\otimes \mathbf{V}){\mathbf{G}}_{\left(1\right)}^{\top }+{\mu }_U\mathbf{P}\mathbf{U}+{\lambda }_U\mathbf{D}\mathbf{U}]}_{ij}}}.$$

(7)

are gained.

To update $\mathcal{G}$, we fix three factor matrices. Considering the vectorization form of $\mathcal{X}$, the Function (3) about $\mathcal{G}$ is as follows:

$$\mathcal{L}={\displaystyle \frac{1}{2}}\left|\right|\mathrm{vec}\left(\mathcal{X}\right)-{\mathbf{F}\mathrm{vec}\left(\mathcal{G}\right)\left|\right|}_2^2+\mathrm{vec}{\left(\mathcal{G}\right)}^{\mathrm{T}}\mathrm{vec}\left(\Phi \right),$$

(8)

where $\mathbf{F}=\mathbf{W}\otimes \mathbf{V}\otimes \mathbf{U}\in {\mathbb{R}}^{I_1{I}_2{I}_3\times J_1{J}_2{J}_3}$. The meaning of $\mathrm{vec}\left(\mathcal{X}\right)$ is that expanding the tensor $\mathcal{X}$ into a 1-Dimension vector. And $\mathrm{vec}\left(\Phi \right)$ stands for the Lagrange multiplier of $\mathrm{vec}\left(\mathcal{G}\right)$. The gradient of $\mathcal{L}$ in (8) with respect to $\mathrm{vec}\left(\mathcal{G}\right)$ is $\partial \mathcal{L}/\partial \mathrm{vec}\left(\mathcal{G}\right)={\mathbf{F}}^{\top }\mathbf{F}\mathrm{vec}\left(\mathcal{G}\right)-{\mathbf{F}}^{\top }\mathrm{vec}\left(\mathcal{X}\right)+\mathrm{vec}\left(\Phi \right).$ The KKT conditions are $\partial \mathcal{L}/\partial \mathrm{vec}\left(\mathcal{G}\right)=0$ and ${\left(\mathrm{vec}\left(\mathcal{G}\right)\right)}_{\mathrm{i}}{\left(\mathrm{vec}\left(\Phi \right)\right)}_{\mathrm{i}}=0$, so we gain the following formula:

$${\left(\mathrm{vec}\left(\mathcal{G}\right)\right)}_{\mathrm{i}}\leftarrow {\left(\mathrm{vec}\left(\mathcal{G}\right)\right)}_{\mathrm{i}}{\displaystyle \frac{{\mathrm{P}}_{+}\left[{\left({\mathbf{F}}^{\top }\mathrm{vec}\left(\mathcal{X}\right)\right)}_{\mathrm{i}}\right]}{{\left({\mathbf{F}}^{\top }\mathbf{F}\mathrm{vec}\left(\mathcal{G}\right)\right)}_{\mathrm{i}}}}.$$

(9)

The optimization rules has been finished, which is summarized in Algorithm 1. Initialize $\mathcal{G}$, $\mathbf{U}, \mathbf{V}, \mathbf{W}$ randomly by rand without other additional processing. The initialization of other methods is the same operation.

Algorithm 1 The MPCNTD method

Input:  Data tensor, cluster number, parameter, the stopping criterion. Output:  Core tensor $\mathcal{G}$, nonnegative matrices $\mathbf{U}$, $\mathbf{V}$, $\mathbf{W}$.   1: Initialize $\mathbf{U}$, $\mathbf{V}$, $\mathbf{W}$ as random matrices and $\mathcal{G}$ as an arbitrary positive tensor.   2: Obtain the weight matrix $\mathbf{S}$.   3: repeat   4:     Update $\mathbf{U}$ by (7) and adjust the columns of $\mathbf{U}$ so that its norm value is equal to 1.   5:     Update $\mathbf{V}$ by (6) and adjust the columns of $\mathbf{V}$ so that its norm value is 1.   6:     Update $\mathbf{W}$ by (5) and adjust the columns of $\mathbf{W}$ so that its norm value is 1.   7:     Update $\mathcal{G}$ by (9).   8: until the convergence condition of method (set maximum iteration step) is satisfied.

3.3. Discussion on Convergence

In this part, we discussed the convergence of the algorithm by proving the following theorem:

Theorem 1.

The objective function in (3) is non-increasing under the updating rules of (5)–(9). The objective function is invariant under these updates if and only if $\mathbf{U}$, $\mathbf{V}$, $\mathbf{W}$ and $\mathcal{G}$ are at a stationary point.

Proof. 

Following the proof paradigm of the multiplicative updating, we will illustrate the convergence of MPCNTD. The ingenious design of auxiliary functions is crucial and significant in the process of proving the theorem [4]. We assume $G(u,u^{\prime })$ as an auxiliary function for $F\left(u\right)$. Then $F\left(u\right)$ is non-increasing under the equation

$$u^{t+1}=\underset{u}{\mathrm{argmin}}G(u,u^t).$$

(10)

$F\left(u^{t+1}\right)=F\left(u^t\right)$ establishes only if $u^t$ is a local minimum of $G(u,u^t)$. Only when each sub-problem about $\mathbf{U}$, $\mathbf{V}$, $\mathbf{W}$ and $\mathcal{G}$ has a unique solution, convergence can be implemented. We inspect that the objective function of MPCNTD is non-increasing under the updating rule (5) of 3th dimensional representation $\mathbf{W}$. For any element ${\mathbf{W}}_{ij}$ in $\mathbf{W}$, let $F_{ij}\left({\mathbf{W}}_{ij}\right)$ stand for the part of the objective function in (2) related to ${\mathbf{W}}_{ij}$. The first-order derivative of the $F_{ij}\left({\mathbf{W}}_{ij}\right)$ with respect to ${\mathbf{W}}_{ij}$ is $F_{ij}^{\prime }\left({\mathbf{W}}_{ij}\right)={[-{\mathbf{X}}_{\left(3\right)}(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\lambda }_W\mathbf{L}\mathbf{W}+{\mu }_W\mathbf{Q}\mathbf{W}]}_{ij}$ and $F_{ij}^{″}\left({\mathbf{W}}_{ij}\right)={\left[{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }\right]}_{jj}+{[{\lambda }_W\mathbf{L}+{\mu }_W\mathbf{Q}]}_{ii}$ is the second order derivative.

Conditions (a): The function

$$\begin{array}{cc}\hfill G({\mathbf{W}}_{ij},{\mathbf{W}}_{ij}^t)=& F_{ij}\left({\mathbf{W}}_{ij}^t\right)+F_{ij}^{\prime }\left({\mathbf{W}}_{ij}^t\right)({\mathbf{W}}_{ij}-{\mathbf{W}}_{ij}^t)\hfill \\ & +{\displaystyle \frac{1}{2}}{\displaystyle \frac{{[\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\lambda }_W\mathbf{D}\mathbf{W}+{\mu }_W\mathbf{P}\mathbf{W}]}_{ij}}{{\mathbf{W}}_{ij}^t}}{({\mathbf{W}}_{ij}-{\mathbf{W}}_{ij}^t)}^2\hfill \end{array}$$

(11)

is an auxiliary function for $F_{ij}\left({\mathbf{W}}_{ij}\right)$, which is only relevant to ${\mathbf{W}}_{ij}$.

Replacing $G(u,u^{\prime })$ in (10) by (11), we obtain $\partial G({\mathbf{W}}_{ij},{\mathbf{W}}_{ij}^t)/\partial {\mathbf{W}}_{ij}=F_{ij}^{\prime }\left({\mathbf{W}}_{ij}^t\right)+[{[\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\lambda }_W\mathbf{D}\mathbf{W}+{\mu }_W\mathbf{P}\mathbf{W}]}_{ij}/{\mathbf{W}}_{ij}^t]({\mathbf{W}}_{ij}-{\mathbf{W}}_{ij}^t)=0$ which yields ${\mathbf{W}}_{ij}\leftarrow {\mathbf{W}}_{ij}{[{\mathbf{X}}_{\left(3\right)}(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\mu }_W\mathbf{I}\mathbf{W}+{\lambda }_W\mathbf{S}\mathbf{W}]}_{ij}/{[\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\mu }_W\mathbf{P}\mathbf{W}+{\lambda }_W\mathbf{D}\mathbf{W}]}_{ij}.$ It complies with (5). So it is established that the objective function of MPCNTD is non-increasing under the updating rule of factor matrix $\mathbf{W}$. From the iteration format, it can be seen that the iteration rules of the three factor matrices have symmetry and similarity. The setting of auxiliary function about other variables and the entire proof process are easy to analogize. Below $\mathcal{G}$ is being considered. Let $g_i$ is the element of $\mathrm{vec}\left(\mathcal{G}\right)$ and $F_i\left(g_i\right)$ is the part of the objective function in (3) relevant to $g_i$.

Conditions (b): The function

$$G(g_i,g_i^t)=F_i\left(g_i^t\right)+F_i^{\prime }\left(g_i^t\right)(g_i-g_i^t)+{\displaystyle \frac{{\left({\mathbf{F}}^{\top }\mathbf{F}\mathrm{vec}\left({\mathcal{G}}^{\mathrm{t}}\right)\right)}_{\mathrm{i}}}{g_i^t}}{(g_i-g_i^t)}^2$$

(12)

is an auxiliary function for $F_i\left(g_i\right)$.

Putting $G(g_i,g_i^t)$ of (12) into (10), we have $g_i^{t+1}=g_i^t{\left({\mathbf{F}}^{\top }\mathrm{vec}\left(\mathcal{X}\right)\right)}_{\mathrm{i}}/{\left({\mathbf{F}}^{\top }\mathrm{Fvec}\left({\mathcal{G}}^{\mathrm{t}}\right)\right)}_{\mathrm{i}}.$ $F_i\left(g_i\right)$ is non-increasing under the updating rule (9). As of now, the theoretical analysis of the theorem has been completed. □

To prove the theorem, we need to validate that the following conditions (a) and (b) hold. Below we elaborate on the proof of auxiliary functions about ${\mathbf{W}}_{ij}$ for the sake of simplicity.

Proof for Condition (a).

The auxiliary function must satisfy the two terms: $G({\mathbf{W}}_{ij},{\mathbf{W}}_{ij}^t)\ge F\left({\mathbf{W}}_{ij}\right)$ and $G({\mathbf{W}}_{ij},{\mathbf{W}}_{ij})=F\left({\mathbf{W}}_{ij}\right)$. The second point is obviously valid, so what we need to prove is the first inequality $G({\mathbf{W}}_{ij},{\mathbf{W}}_{ij}^t)\ge F\left({\mathbf{W}}_{ij}\right)$. The Taylor expansion of $F_{ij}\left({\mathbf{W}}_{ij}\right)$ at ${\mathbf{W}}_{ij}^t$ is as follows:

$$\begin{array}{c}\hfill F_{ij}\left({\mathbf{W}}_{ij}\right)=F_{ij}\left({\mathbf{W}}_{ij}^t\right)+F_{ij}^{\prime }\left({\mathbf{W}}_{ij}^t\right)({\mathbf{W}}_{ij}-{\mathbf{W}}_{ij}^t)+{\displaystyle \frac{1}{2}}{F}_{ij}^{″}\left({\mathbf{W}}_{ij}^t\right){({\mathbf{W}}_{ij}-{\mathbf{W}}_{ij}^t)}^2.\end{array}$$

(13)

Comparing (11) with (13), we can see that $G({\mathbf{W}}_{ij},{\mathbf{W}}_{ij}^t)\ge F\left({\mathbf{W}}_{ij}\right)$ is equivalent to the following:

$$\begin{array}{cc}\hfill {\displaystyle \frac{{[\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\lambda }_W\mathbf{D}\mathbf{W}+{\mu }_W\mathbf{P}\mathbf{W}]}_{ij}}{{\mathbf{W}}_{ij}^t}}& \ge {\left[{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }\right]}_{jj}\hfill \\ & +{[{\lambda }_W\mathbf{L}+{\mu }_W\mathbf{Q}]}_{ii}.\hfill \end{array}$$

(14)

After calculation and deformation, we obtain the following: ${[\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }+{\lambda }_W\mathbf{D}\mathbf{W}+{\mu }_W\mathbf{P}\mathbf{W}]}_{ij}\ge {\mathbf{W}}_{ij}^t{\left[{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }\right]}_{jj}+{[{\lambda }_W\mathbf{L}+{\mu }_W\mathbf{Q}]}_{ii}{\mathbf{W}}_{ij}^t$. Furthermore, we can convert ${\left[\mathbf{W}{\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }\right]}_{ij}$ to ${\sum }_{k=1,k\ne j}^{J_3}{\mathbf{W}}_{ik}{\left({\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }\right)}_{kj}+{\mathbf{W}}_{ij}^t{\left({\mathbf{G}}_{\left(3\right)}{(\mathbf{V}\otimes \mathbf{U})}^{\top }(\mathbf{V}\otimes \mathbf{U}){\mathbf{G}}_{\left(3\right)}^{\top }\right)}_{jj}.$ We have ${\left[\mathbf{D}\mathbf{W}\right]}_{ij}={\sum }_{k=1}^{I_3}{\mathbf{D}}_{ik}{\mathbf{W}}_{kj}\ge {\mathbf{D}}_{ii}{\mathbf{W}}_{ij}\ge {\mathbf{D}}_{ii}{\mathbf{W}}_{ij}^t=({\mathbf{L}}_{ii}+{\mathbf{S}}_{ii}){\mathbf{W}}_{ij}^t\ge {\mathbf{L}}_{ii}{\mathbf{W}}_{ij}^t$, ${\left[\mathbf{P}\mathbf{W}\right]}_{ij}={\sum }_{k=1}^{I_3}{\mathbf{P}}_{ik}{\mathbf{W}}_{kj}\ge {\mathbf{P}}_{ii}{\mathbf{W}}_{ij}\ge {\mathbf{P}}_{ii}{\mathbf{W}}_{ij}^t=({\mathbf{Q}}_{ii}+{\mathbf{I}}_{ii}){\mathbf{W}}_{ij}^t\ge {\mathbf{Q}}_{ii}{\mathbf{W}}_{ij}^t,$ which means (14) establishes. Thus, the point $G({\mathbf{W}}_{ij},{\mathbf{W}}_{ij}^t)\ge F\left({\mathbf{W}}_{ij}\right)$ holds. □

3.4. Computational Complexity

In this subsection, we estimate the computational complexity of the proposed MPCNTD. We heavily replace the Kronecker product operations that is time-consuming with tensor operation in our holistic updating rules. The fladd (a floating-point addition), flmlt (a floating-point multiplication) and fldiv (a floating-point division) is used as three indicators for subdividing computational complexity. Divide the specific calculation into three parts to make the sorting process clear and concise. The specific calculations of fladd, flmlt and fldiv for $\mathbf{W}$ respectively are $I_1{I}_2{I}_3{J}_1+I_2{I}_3{J}_1{J}_2+J_1^2{J}_2{J}_3+J_1{J}_2^2{J}_3+I_1{J}_1^2+I_2{J}_2^2+3I_3{J}_1{J}_2{J}_3+4I_3{J}_3(I_3+2),$$I_1{I}_2{I}_3{J}_1+I_2{I}_3{J}_1{J}_2+J_1^2{J}_2{J}_3+J_1{J}_2^2{J}_3+I_1{J}_1^2+I_2{J}_2^2+3I_3{J}_1{J}_2{J}_3+4I_3{J}_3(I_1+1)+I_3{J}_3$ and $I_3{J}_3$. Due to its symmetry, the computational complexity of other variables ($\mathbf{U}$,$\mathbf{V}$) can be easily analogized and obtained. Additionally, the specific calculations of fladd, flmlt and fldiv for $\mathcal{G}$ respectively are $I_1{I}_2{I}_3{J}_1+I_2{I}_3{J}_1{J}_2+J_1^2{J}_2{J}_3+J_1{J}_2^2{J}_3+J_1{J}_2{J}_3^2+I_1{J}_1^2+I_2{J}_2^2+I_3{J}_3^2+I_3{J}_1{J}_2{J}_3$, $I_1{I}_2{I}_3{J}_1+I_2{I}_3{J}_1{J}_2+J_1^2{J}_2{J}_3+J_1{J}_2^2{J}_3+I_1{J}_1^2+I_2{J}_2^2+3I_3{J}_1{J}_2{J}_3+J_1{J}_2{J}_3$ and $J_1{J}_2{J}_3$. After adding the above computational complexity and referencing the proportion $I_i\gg J_i,i=1,2,3$, the complexity of the proposed MPCNTD method is $\mathcal{O}\left(I_1{I}_2{I}_3(J_1+J_2+J_3)\right)$. If our proposed method converges after T iterations, then the total computational complexity is $\mathcal{O}\left(\left(I_1{I}_2{I}_3(J_1+J_2+J_3)\right)T\right)$.

4. Experiments

In this section, we compare MPCNTD with classic clustering methods (K-means), NMF [3], and NMF-based methods (ONMF [15,16], GNMF [12] and GDNMF [13]). In addition, we also compare MPCNTD with some tensor decomposition methods including NTD [8] and GNTD [18].

4.1. Datasets Description

COIL20 dataset contains 1440 images of 20 objects. The subjects were placed on an electric turntable that rotated 360 degrees to change the orientation of the object relative to the fixed camera.

COIL100 dataset collects 7200 images of 100 subjects. The objects have a wide variety of complex geometric and reflectance characteristics.

PIE dataset contains faces of 68 people, 13 different poses, 43 different lighting conditions, and with 4 different expressions.

USPS dataset is a handwritten digit database. It contains 9298 handwritten digit gray images in total. The size of each digit image is $16\times 16$.

YALE dataset collects 165 gray-scale images in GIF format of 15 people. Each object has 11 images, with one image for each different facial expression or configuration.

ORL dataset contains 400 gray-scale faces images which consist of 40 different subjects with 10 distinct images. For some subjects, the images were taken under different lighting, times, facial expressions. The size of each face image is 112 × 92.

JAFFE dataset includes 213 images of 7 different emotional facial expressions.

UMIST dataset includes 564 images of 20 people. Each covers a series of postures from the side to the front. Subjects covered a range of races, genders, and appearances. Each theme exists in its own directory and is labelled, with images numbered in order during filming. The detailed information of the dataset is shown in

Table 1. Benchmark dataset.

	Datasets	Samples	Dimensions	Classes	Type	Tensor Size
1	COIL20	1440	1024	20	Object	$32\times 32\times 1440$
2	COIL100	7200	1024	100	Object	$32\times 32\times 7200$
3	PIE	2856	1024	68	Face	$32\times 32\times 2856$
4	USPS	9298	256	10	Handwritten	$16\times 16\times 9298$
5	YALE	165	1024	15	Face	$32\times 32\times 165$
6	ORL	400	1024	40	Face	$32\times 32\times 400$
7	JAFFE	213	4096	10	Face	$64\times 64\times 213$
8	UMIST	575	1024	20	Face	$32\times 32\times 575$

.

4.2. Evaluation Measures

Three indicators, accuracy (ACC), normalized mutual information (NMI) and cluster purity (PUR), are calculated to assess the clustering performance of all methods. The ACC is denoted as $\delta (k_i,map\left(c_i\right))/n$, where $c_i$ is the label arising from clustering method, $k_i$ is real label of $x_i$, $\delta (x,y)$ is the indicator function, n is the overall number of objects, and $map\left(c_i\right)$ is the mapping function. Accuracy measures the degree to which each cluster contains data points from the same category. The NMI is denoted as ${\sum }_{u=1}^c{\sum }_{v=1}^k\log ({\displaystyle \frac{n{n}_{uv}}{n_u{n}_v^{\prime }}})/\sqrt{({\sum }_{u=1}^c{n}_u\log ({\displaystyle \frac{n_u}{n}}))({\sum }_{v=1}^k{n}_v^{\prime }\log ({\displaystyle \frac{n_v^{\prime }}{n}}))}$, where $n_{uv}$ is the number of cluster, $n_i$ and $n_i^{\prime }$ are the data points in the real label cluster and the clustered label cluster, respectively. NMI measures the quality of the clusters. The PUR is denoted as ${\sum }_{i=1}^k{m}_i/m$, where k is the number of classes in the sample, m is the total sample size. Purity measures the extent to which each cluster contains samples from primarily the same class. The larger values of three indicators, the better quality of clustering.

The number of grayscale images is set to the three-dimensional mode of tensors, forming a third-order tensor applied for clustering experiments. After obtaining the low dimensional representation of the tensor data using different methods, K-means clustering is used to analyze its low dimensional representation and generate results. Other settings about our experiments are shown. We uniformly construct the weighting matrix $\mathbf{S}$ by heat-kernel weight. The heat kernel is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathbf{S}}_{ij}=\left\{\begin{array}{cc}{e}^{-{\displaystyle \frac{{∥\begin{array}{c}{\mathbf{x}}_i-{\mathbf{x}}_j\end{array}∥}^2}{{\sigma }^2}}},\hfill & {\mathbf{x}}_j\in \mathcal{N}\left({\mathbf{x}}_i\right)or{\mathbf{x}}_i\in \mathcal{N}\left({\mathbf{x}}_j\right)\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}\end{array}$$

where $\sigma =1$ and $\mathcal{N}\left({\mathbf{x}}_i\right)$ is the p-nearest neighbourhood of data ${\mathbf{x}}_i$. The maximum number of iteration is 100, and all methods are run independently 10 times. The average of total measurement indicators is the final experimental record, and the optimal result is displayed in bold. To demonstrate the fairness, the whole experiments are run with MATLAB R2023a on a PC equipped with 8 GB of memory.

4.3. Parameter Sensitivity

MPCNTD is susceptible to the input of parameters, that is to say, the reasonable selection of parameter determines the performance of the model. Below, we have discussed concretely the impact of three influencing factors on outcome.

• Simulation of incomplete data situation
  As an unsupervised algorithm, MPCNTD is directly affected by the proportion of extracted data. In this part, we consider the performance of the model under incomplete data conditions, which is crucial for practical applications. It’s not that the more samples extracted, the better the algorithm’s behavior. Multiple sampling results to simulate incomplete data can help explore the overall performance of the algorithm. The sampling ratio for each dataset is 10%, 20%, 30%, 40%, 50%, 60%, and the results are recorded as the average of 10 replicates each time. Figure 2 shows the clustering performance of the algorithm and the curve of the percentage of sampled data. ACC, NMI, and PUR are represented by solid blue, red, and green lines, respectively. The corresponding dashed line displays the average level of the datasets.

• Graph construction details
  Parameter p in figure construction is discussed. For the graph in MPCNTD, the nearest neighbours of KNN are set to $\{1,3,5\}$. The average classification ACC, NMI and PUR results are reported in Figure 3. Parameter $p=5$, $p=3$ and $p=1$ are presented in blue, red, and orange columns, respectively. The optimal result can be seen at its columnar top. We found that smaller values are not necessarily better. Due to the strong correlation between points (i.e., data and data) during the graphic construction process, redundant graphic information is generated. Select appropriate p values in practical problems.

• Hyper-parameter protocol about $\lambda$ and $\mu$
  There are multiple adjustable regularized parameters ${\lambda }_r,r=U,V,W$ and ${\mu }_r,r=U,V,W$ in our model to balance the fundamental decomposition and consistent regularization terms. To diminish confusion in parameter selection, we set ${\lambda }_r=\lambda , r=U,V,W$ and ${\mu }_r=\mu , r=U,V,W$ and deeply explore the clustering ability set in this way. The grid control method is considered to acquire the relatively optimal parameters. The numerical selection of two parameters $\lambda ,\mu$ is $[0.001,0.01,0.1,1,10,100,1000]$ by using prior-free criteria. We select $7\times 7$ situations to achieve the best clustering effect and provide robustness plots. To weigh the three measurement criteria, the numerical effect of ACC is the essential decisive factor, assisted with NMI and PUR. Specific experimental results are presented in Figure 4.

4.4. Experiments for Effectiveness and Analysis

In this subsection, we apply the maximum number of categories as the number of clusters for the eight datasets, and this is identical treatment for all comparative experimental methods.

From

Table 2. Clustering performance based on eight datasets.

	K-Means	NMF	ONMF	GNMF	GDNMF	NTD	GNTD	MPCNTD
	ACC (%)
COIL20	53.47 ± 4.05	55.37 ± 3.90	49.18 ± 1.91	69.11 ± 6.82	70.05 ± 4.75	44.24 ± 2.19	69.17 ± 6.60	73.16 ± 4.76
COIL100	43.91 ± 2.08	45.19 ± 1.60	37.12 ± 0.93	55.69 ± 2.66	55.75 ± 2.81	21.71 ± 0.31	55.98 ± 1.88	58.97 ± 1.85
PIE	23.84 ± 1.18	41.11 ± 2.68	21.69 ± 0.48	68.17 ± 3.58	68.43 ± 4.09	53.73 ± 1.42	70.31 ± 2.35	71.32 ± 2.84
USPS	64.46 ± 3.38	38.55 ± 1.80	36.26 ± 3.53	68.77 ± 10.74	69.22 ± 10.35	27.88 ± 1.40	72.34 ± 7.21	77.82 ± 12.98
Yale	35.58 ± 2.89	37.21 ± 1.62	35.88 ± 1.80	38.48 ± 2.08	38.48 ± 2.08	36.55 ± 2.08	41.03 ± 2.32	47.21 ± 2.53
ORL	49.87 ± 2.71	48.62 ± 2.7	37.00 ± 0.89	51.08 ± 2.08	52.17 ± 1.86	19.62 ± 0.84	51.05 ± 1.53	55.88 ± 2.02
JAFFE	48.92 ± 4.31	45.68 ± 2.83	37.79 ± 2.74	50.23 ± 4.64	58.08 ± 4.65	22.58 ± 1.20	59.15 ± 3.60	61.55 ± 4.37
UMIST	40.28 ± 2.18	38.12 ± 3.00	34.37 ± 1.48	60.00 ± 5.54	60.45 ± 4.82	27.57 ± 1.17	60.14 ± 4.93	62.47 ± 4.35
	NMI (%)
COIL20	69.61 ± 1.56	69.65 ± 2.33	61.87 ± 2.01	84.82 ± 3.50	84.92 ± 2.22	55.13 ± 0.91	85.58 ± 2.81	86.95 ± 1.68
COIL100	72.33 ± 0.81	72.41 ± 0.34	62.48 ± 0.51	79.09 ± 1.47	79.31 ± 1.33	47.04 ± 0.11	80.30 ± 0.73	80.31 ± 0.65
PIE	53.57 ± 0.69	70.30 ± 1.49	46.13 ± 0.39	85.91 ± 1.33	85.88 ± 1.34	79.45 ± 0.52	85.39 ± 1.10	86.60 ± 0.87
USPS	60.35 ± 1.48	27.09 ± 1.72	23.64 ± 3.86	78.69 ± 4.33	78.61 ± 4.35	16.31 ± 0.69	78.82 ± 3.07	80.83 ± 4.93
Yale	40.67 ± 3.95	42.06 ± 2.24	39.64 ± 1.30	44.15 ± 1.79	44.13 ± 1.72	43.82 ± 2.07	47.26 ± 1.61	53.51 ± 1.50
ORL	69.37 ± 1.40	69.11 ± 2.04	58.36 ± 0.78	70.26 ± 0.92	70.17 ± 1.11	41.66 ± 0.92	70.25 ± 0.68	72.98 ± 0.83
JAFFE	58.52 ± 2.28	56.47 ± 2.46	48.44 ± 1.38	76.53 ± 2.53	76.70 ± 2.29	38.94 ± 0.93	76.62 ± 2.02	77.22 ± 1.81
UMIST	52.41 ± 3.76	45.40 ± 2.09	35.60 ± 2.02	57.18 ± 3.47	62.30 ± 2.37	13.25 ± 1.41	61.08 ± 1.93	64.79 ± 2.39
	PUR (%)
COIL20	66.42 ± 2.14	65.56 ± 2.40	60.18 ± 2.36	85.09 ± 3.70	85.66 ± 2.83	50.85 ± 1.06	88.24 ± 2.69	87.10 ± 1.61
COIL100	57.87 ± 1.14	56.84 ± 1.12	45.74 ± 0.50	72.70 ± 1.29	72.48 ± 1.65	25.12 ± 0.38	70.72 ± 1.35	70.02 ± 1.38
PIE	29.61 ± 0.89	49.52 ± 1.39	29.56 ± 0.69	84.70 ± 1.29	84.85 ± 1.25	62.70 ± 1.19	83.18 ± 1.36	84.68 ± 1.26
USPS	70.79 ± 1.71	53.18 ± 2.39	56.12 ± 5.24	84.94 ± 4.16	85.30 ± 3.88	31.23 ± 1.11	84.84 ± 2.68	88.52 ± 4.17
Yale	48.42 ± 3.66	44.61 ± 2.44	44.67 ± 1.83	45.94 ± 0.94	46.00 ± 0.92	42.79 ± 2.25	48.48 ± 2.41	53.33 ± 1.59
ORL	59.25 ± 2.69	56.63 ± 2.80	44.65 ± 1.12	58.30 ± 0.79	59.25 ± 0.62	24.38 ± 1.60	58.20 ± 0.91	62.60 ± 0.66
JAFFE	60.00 ± 4.43	52.35 ± 4.04	41.31 ± 2.25	64.55 ± 3.19	68.03 ± 1.87	27.42 ± 2.72	68.69 ± 1.98	70.56 ± 2.53
UMIST	45.51 ± 2.29	43.29 ± 2.69	40.12 ± 1.98	69.13 ± 3.20	69.23 ± 3.07	32.49 ± 1.37	69.86 ± 3.19	72.35 ± 2.01

, our algorithm presents better performance on the eight evaluated datasets. After analyzing the content of Table 2, the following phenomena can be summarized:

(1)

Based on eight image datasets, MPCNTD attains the competent clustering performance in most situations, which demonstrates that MPCNTD can reveal additional discriminative information for tensor data. For example, the average clustering ACC of MPCNTD on the datasets COIL20, COIL100, PIE, USPS, YALE, ORL, JAFFE, and UMIST is higher than the second best method by 3.11%, 2.99%, 1.01%, 5.48%, 6.18%, 3.71%, 2.40% and 2.02%, respectively.

(2)

The matrix dimension reduction approach NMF, GNMF, GDNMF outperform the k-means method with regard to existing outcome. This is because matrix dimension reduction retain more structures of the original data while discovering more accurate data representations. Moreover, the numerical result of NTD-based model is better than that of the NMF-based methods, which illustrates that tensor-based method have advantages over matrix-based methods in data representation and reduction in dimension.

(3)

We found that the NTD method has weak performance in clustering problems because our iteration steps are uniformly set to 100 and we choose a unified initial state. When the iteration process is relaxed and appropriate initial values are selected, this method is still feasible. The result of ONMF losing to NMF is the same. This situation indirectly illustrates that considering consistent regularization not only accelerates the iteration process, but also makes data representation more accurate.

(4)

MPCNTD obtains contestable clustering performance in all experiments. This situation is due to the truth that MPCNTD captures multiple pieces of information from various directions, and this consistent regularization contains graph info and a softly orthogonal structure. Graph learning can maintain the consistency of geometric information of matrices in different spaces, and the balance between multi-graph regularization maximizes the exploration of geometric information of tensor data. The normalization and orthogonality rules in approximate orthogonality work together to maintain data independence as much as possible, resulting in better clustering performance.

4.5. Convergence Study and Running Times

In this part, we pursue the convergence performance of MPCNTD experimentally on all eight evaluated datasets. From Figure 5, we can directly discover that the objective function values are monotonically decreasing and generally reach stability within 50 iterations, which proves the effectiveness of our model and the feasibility of the MU-based iteration method. In addition, we tested the computation time of each method using the maximum number of clusters, and the specific details are described in

Table 3. Running time (seconds).

Datasets	K-Means	NMF	ONMF	GNMF	GDNMF	NTD	GNTD	MPCNTD
COIL20	0.9575	4.8707	17.8615	7.7992	6.7093	8.5009	7.6108	11.6861
COIL100	12.7224	55.5005	389.9492	59.3759	60.6811	106.4293	44.9294	87.1066
PIE	4.3523	18.2649	77.1014	19.7781	19.7723	84.8791	34.1724	36.5172
USPS	3.4253	2.5031	179.4944	10.4931	11.8063	9.3728	34.7790	44.8650
YALE	0.0628	1.6680	0.5067	1.2088	0.6632	1.5704	1.3235	1.4317
ORL	0.2675	1.5499	3.2105	2.0715	2.3844	2.1336	3.4087	2.3716
JAFFE	0.2869	2.5672	4.2324	3.0784	2.6312	3.6669	3.3649	3.8293
UMIST	0.3352	1.5462	5.4409	1.9811	2.6882	3.3493	3.5304	3.6853

. It is not difficult to identify that our method has a longer running time due to the addition of consistent regularization from the comparison of time. But given the superior results, the time consumption can also be tolerated.

5. Conclusions and Expectation

In this article, we propose an NTD-based framework that includes the concepts of graph Laplace and approximate orthogonality, taking into account pairwise constraint in multiple directions. This framework not only utilizes partial representations in tensor models, but also utilizes the structural information of consistent regularization following the decomposition direction. Our proposed model ensures that the minimum point of the objective function converges to the specified local region. Multitudinous experiments were operated on eight real image datasets in unsupervised clustering tasks to make verification and validation of the MPCNTD framework. In the future, we would like to keep on our exploration in the following two aspects. Firstly, it is worth studying whether there are more reasonable constraints to obtain a better solution. More constraint information, such as label information and smoothness, can be considered. Secondly, it is crucial for developing more efficient parallel algorithms about factor matrices. Due to the similarity in the iterative process of factor matrices based on the tensor framework.