Hypergraph-Regularized L_p Smooth Nonnegative Matrix Factorization for Data Representation

Abstract

Nonnegative matrix factorization (NMF) has been shown to be a strong data representation technique, with applications in text mining, pattern recognition, image processing, clustering and other fields. In this paper, we propose a hypergraph-regularized $L_p$ smooth nonnegative matrix factorization (HGSNMF) by incorporating the hypergraph regularization term and the $L_p$ smoothing constraint term into the standard NMF model. The hypergraph regularization term can capture the intrinsic geometry structure of high dimension space data more comprehensively than simple graphs, and the $L_p$ smoothing constraint term may yield a smooth and more accurate solution to the optimization problem. The updating rules are given using multiplicative update techniques, and the convergence of the proposed method is theoretically investigated. The experimental results on five different data sets show that the proposed method has a better clustering effect than the related state-of-the-art methods in the vast majority of cases.

1. Introduction

Data representation plays an important role in information retrieval [1], computer vision [2], pattern recognition [3], and other applied fields [4,5]. There are many approaches to deal with high-dimensional data, such as data dimensionality reduction, random forests (RF) [6], multilayer perceptrons (MPL) [7], graph neural networks (GNN) [8], and hypergraph neural networks (HGNN) [9]. The dimensions of data matrices are extremely high in these practical applications. The high-dimensional data can not only cause storage difficulties, but also possible dimensional curses. Therefore, it is necessary to find an effective and low-dimensional representation of the original high-dimensional data matrix. It is also important to preserve the multidimensional structure of the image data and to preserve the multidimensional structure of the image data. Matrix factorization is one of the important data representation techniques, and typical matrix decomposition methods mainly include the following: principal component analysis (PCA) [10], linear discriminant analysis (LDA) [11], singular value decomposition (SVD) [12], nonnegative matrix factorization (NMF) [13,14], and so on. The tensor of a multidimensional array is well suited to representing such an image space, and, in order to extract valuable information from a given large tensor, a low-rank Tucker decomposition (TD) [15] is usually considered. In the real world, many data images, video volumes, and test data are non-negative, and, for these types of data, nonnegative Tucker decomposition (NTD) has recently received attention [16,17,18,19].

NMF has been gaining popularity through the works of Lee and Seung that were published in Nature [13] and NIPS [14]. It has been widely applied in clustering [20,21,22], face recognition [23,24], text mining [25,26], image processing [27,28,29], hyperspectral unmixing (HU) [30,31], and other fields [32,33,34,35]. Several NMF versions have been presented in order to improve data representation capabilities by introducing different regularization terms or constraint terms into the basis NMF model. For example, by considering the orthogonality of factor matrices, Ding et al. [21] presented a orthogonal nonnegative matrix tri-factorization (ONMF) approach. By incorporating the graph regularization term into the standard NMF model, Cai et al. [2] presented a graph-regularized nonnegative matrix factorization (GNMF) method, where a simple nearest neighborhood graph is constructed by considering the pairwise geometric relationships between two sample points. However, the model did not take into account the high-order relationships among multiple sample points. Shang et al. [36] presented a graph dual regularization nonnegative matrix factorization (DNMF) approach, which simultaneously considers the intrinsic geometry matrix structures of both the data manifold and the feature manifold. However, their DNMF approach neglects the high-order relationships among multiple sample points or multiple features. To solve the above problem, Zeng et al. [37] presented the hypergraph-regularized nonnegative matrix factorization (HNMF) method, which incorporates the hypergraph regularization term into NMF and constructs a hypergraph by considering high-order relationships among multiple sample points. However, HNMF is unable to produce a smooth and precise solution, because it does not take into account the smoothness of the basis matrix. Recently, Leng et al. [38] proposed the graph-regularized $L_p$ smooth nonnegative matrix factorization (GSNMF) method by incorporating the graph regularization term and the $L_p$ smoothing term into the basis NMF model, which considers the intrinsic geometric structures of the sample data and may produce a smooth and more accurate solution to the optimization problem with the addition of the graph regularization term and the $L_p$ smoothing constraint. However, in GSNMF, only the pairwise relationships between two sample points are considered, and the high-order relationships among multiple sample points are ignored.

Base on NTD, Qiu et al. [39] proposed a graph-regularized non-negative Tucker decomposition (GNTD) method, which is able to extract a representation based on low-dimensional parts from high-dimensional tensor data and retain geometric information. Immediately after, Qiu et al. [40] gave an alternating approximate gradient descent method to solve the proposed GNTD framework (UGNTD).

1.1. Problem Statement

In this paper, by incorporating hypergraph regularization and $L_p$ smoothing constraint terms into the standard NMF model, we propose a hypergraph-regularized $L_p$ smooth nonnegative matrix factorization (HGSNMF) method. The hypergraph regularization term considers the high-order relationships among multiple samples. The $L_p$ smoothing constraint term takes into account the smoothness of the basis matrix, which has been proven to be significant in data representation [41,42,43]. To solve the optimization problem of the HGSNMF model, we offer an effective optimization algorithm using the multiplicative update technique and theoretically prove the convergence of the HGSNMF algorithm. Finally, we conducted comprehensive experiments on five data sets to demonstrate the effectiveness of the proposed method.

1.2. Research Contribution

The main contributions of this work can be summarized as follows:

(1) Considering the complex relationships as pairwise relationships in a simple graph will inevitably lead to loss of the important information. Therefore, we construct the hypergraph regularization term to better discover the hidden semantics and simultaneously capture the underlying intrinsic geometric structure of high-dimensional spatial data samples. When constructing a hypergraph, each vertex of the hypergraph represents a data sample, and each vertex forms a hyperedge with its k nearest neighboring samples. Each hyperedge represents the similarity relationship between a group of samples with higher similarity.

(2) We consider the $L_p$ smoothing constraint of the basis matrix, which not only removes the noise of the basis matrix to make it smooth, but also obtains a smooth and more accurate solution to the optimization problem by combining the advantages of isotropic and diffusion anisotropic smoothing.

(3) We solve the optimization problem using an efficient iterative technique and conducted comprehensive experiments to empirically analyze our approach on five data sets; the experimental results validate the effectiveness of our proposed method.

The rest of the paper is organized as follows. In Section 2, we introduce some related works, including NMF, GSNMF, and hypergraph learning. In Section 3, we propose the novel HGSNMF model in detail, as well as its updating rules, and prove the convergence of the HGSNMF method. We also analyse the complexity of the proposed method. In Section 4, we provide the results extensive experiments that were conducted to validate the proposed method. Finally, we conclude this paper in Section 5.

2. Related Work

2.1. Nonnegative Matrix Factorization

Given a nonnegative matrix $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_n]\in {\mathbb{R}}_{+}^{m\times n}$, each column ${\mathbf{x}}_i\in {\mathbb{R}}_{+}^m(i=1,2,\dots ,n)$ of $\mathbf{X}$ represents a data point. The purpose of NMF is to decompose a nonnegative matrix into two low-rank nonnegative factor matrices $\mathbf{B}\in {\mathbb{R}}_{+}^{m\times r}$ and $\mathbf{C}\in {\mathbb{R}}_{+}^{r\times n}$, whose product is an approximation of the original matrix. In particular, the objective function of NMF is

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\min \parallel \mathbf{X}-\mathbf{BC}\parallel }_F^2,\\ \hfill s.t.\mathbf{B}\ge 0,\mathbf{C}\ge 0,\end{array}\end{array}$$

(1)

where ${\parallel ·\parallel }_F$ is the Frobenius norm of a matrix, $\mathbf{B}$ is the basis matrix, and $\mathbf{C}$ is the coefficient matrix (also called the encoding matrix). Obviously, the objective function is convex for $\mathbf{B}$ or $\mathbf{C}$, but nonconvex for both $\mathbf{B}$ and $\mathbf{C}$. Lee et al. [14] proposed the iterative multiplicative updating technique to solve the problem (1) as follows:

$$\begin{array}{c}\hfill {\mathbf{B}}_{ik}\leftarrow {\mathbf{B}}_{ik}\frac{{\left({\mathbf{XC}}^{\top }\right)}_{ik}}{{\left({\mathbf{BCC}}^{\top }\right)}_{ik}},\\ \hfill {\mathbf{C}}_{kj}\leftarrow {\mathbf{C}}_{kj}\frac{{\left({\mathbf{B}}^{\top }\mathbf{X}\right)}_{kj}}{{\left({\mathbf{B}}^{\top }\mathbf{BC}\right)}_{kj}},\end{array}$$

where ${\mathbf{B}}^{\top }$ is the transpose of $\mathbf{B}$.

2.2. Graph Regularization $L_p$ Smooth Nonnegative Matrix Factorization

GSNMF takes full account of the similarity between pairs of data points and the smoothness of the basis matrix, so it adds the graph regularity term and the $L_p$ smoothness term into the basis NMF model. Specifically, the objective function of GSNMF is

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & {\min \parallel \mathbf{X}-\mathbf{BC}\parallel }_F^2+\alpha \mathrm{Tr}\left(\mathbf{C}\mathbf{L}{\mathbf{C}}^{\top }\right)+{2\mu \left|\right|\mathbf{B}\left|\right|}_p^p,\hfill \\ \hfill & s.t.\mathbf{B}\ge 0,\mathbf{C}\ge 0,\hfill \end{array}\end{array}$$

(2)

where $\mathrm{Tr}(·)$ denotes the trace of a matrix, $\alpha >0,\mu >0$, $1<p\le 2$, $\mathbf{L}=\mathbf{D}-\mathbf{W}$ is called as the graph Laplacian, and $\mathbf{D}$ is a diagonal matrix with ${\mathbf{D}}_{ii}={\Sigma }_j{\mathbf{W}}_{ij}$. $\mathbf{W}$ is the weight matrix whose value is given by [2]

$$\begin{array}{c}\hfill {\mathbf{W}}_{ij}=\left\{\begin{array}{ccc}& \sum _{x_j\in e_i}exp(-\frac{\left|\right|{\mathbf{x}}_i-{\mathbf{x}}_j{\left|\right|}^2}{{\delta }^2}),\hfill & \hfill {\mathbf{x}}_i\in {\mathcal{N}}_k\left(x_j\right)or{\mathbf{x}}_j\in {\mathcal{N}}_k\left(x_i\right)\\ & 0,\hfill & \hfill otherwise\end{array}\right.\end{array}$$

Leng et al. [38] used the iterative multiplicative updating technique to solve problem (2) as follows:

$$\begin{array}{c}\hfill {\mathbf{B}}_{ik}\leftarrow {\mathbf{B}}_{ik}\frac{{\left({\mathbf{XC}}^{\top }\right)}_{ik}}{{\left({\mathbf{BCC}}^{\top }\right)}_{ik}+\mu p{\mathbf{B}}_{ik}^{p-1}},\\ \hfill {\mathbf{C}}_{kj}\leftarrow {\mathbf{C}}_{kj}\frac{{\left({\mathbf{B}}^{\top }\mathbf{X}\right)}_{kj}+\alpha {\left(\mathbf{CW}\right)}_{kj}}{{\left({\mathbf{B}}^{\top }\mathbf{BC}\right)}_{kj}+\alpha {\left(\mathbf{CD}\right)}_{kj}}.\end{array}$$

2.3. Hypergraph Learning

The simple graph only considers the pairwise geometric relationship between data samples; the complex internal structure of the data samples could not be used efficiently. To remedy this defect, the hypergraph takes into account the high-order geometric relationships among multiple samples, which can better capture the potential geometric information of the data [30]. Thus, hypergraph learning [44,45,46,47,48,49] is an extension of simple graph learning theory.

The hypergraph $G=(V,E)$ takes into account the high-order relationships among multiple vertices and consists of a non-empty vertex set $V=\{v_1,v_2,\dots ,v_n\}$ and a non-empty hypergraph set $E=\{e_1,e_2,\dots ,e_m\}$. Each element $v_i\in V$ is called a vertex, and each element $e_j\in E$ is a subset of V, which is known as a hyperedge of G. G is a hypergraph defined on V if $e_j\ne ⌀$ for $j=1,2,\dots ,m$ and $e_1\cup e_2\cup \cdots \cup e_m=V$.

When constructing a hypergraph, we generate the hyperedge by calculating the Euclidean distance between the k nearest neighbors (kNN) of each vertex. The parameter k is manually set. The kNN method allows the following steps to select k sample data points to construct the hypergraph. First, it calculates the distance between the sample data and the individual sample data. Then, it sorts according to the increasing relationship of the distances. Finally, it selects the k sample data points with the smallest distance. Obviously, kNN has the advantages of being simple, easy to understand, and easy to implement. Thus, we use kNN to construct the hypergraph. An incidence matrix $\mathbf{H}\in {\mathbb{R}}_{+}^{\left|V\right|\times \left|E\right|}$ is used to describe the incidence relationship between a vertex and a hyperedge, which is formalized as $\mathbf{H}(v,e)=1$ if $v\in e$ and as $\mathbf{H}(v,e)=0$ otherwise.

Figure 1 gives an illustration of the simple graph, the hypergraph $\overline{G}=(\overline{V},\overline{E})$, and an incidence matrix. If a vertex is in the k nearest neighbors of another vertex in the undirected simple graph, the two vertices are connected by an edge. The hypergraph $\overline{G}=(\overline{V},\overline{E})$ considers the high-order relationships among multiple vertices and is made up of a non-empty vertex set $\overline{V}=\{v_1,v_2,v_3,v_4,v_5,v_6,v_7,v_8\}$ and a non-empty hypergraph set $\overline{E}=\{e_1=\{v_1,v_2,v_4\},e_2=\{v_3,v_4,v_5,v_6\},e_3=\{v_6,v_7,v_8\}\}$. In Figure 1, the solid nodes stand for the vertices, while the node sets denoted by the solid line segment and the ellipses represent the hyperedges. Furthermore, each vertex in the hypergraph is connected to at least one hypheredge, which is associated with a weight, and each hyperedge can have multiple vertices.

Figure 1. An illustration of the simple graph, the hypergraph $\overline{G}$, and an indication matrix.

Each hyperedge e can be assigned with a positive integer $\omega \left(e\right)$ that represents the weight of the hyperedge. The degree of a vertex v and the degree of a hyperedge can be expressed as $d\left(v\right)={\displaystyle \sum _{e\in E}}\omega \left(e\right)\mathbf{H}(v,e)$ and $\delta \left(e\right)={\displaystyle \sum _{v\in V}}\mathbf{H}(v,e)$, respectively. According to [44], the unnormalized hypergraph Laplacian matrix can be expressed as

$$\begin{array}{c}\hfill {\mathbf{L}}_{Hyper}={\mathbf{D}}_v-\mathbf{S},\end{array}$$

(3)

where $\mathbf{S}={\mathbf{HWD}}_e^{-1}{\mathbf{H}}^{\top }$, $\mathbf{H}$ is an incidence matrix, $\mathbf{W}$ is a diagonal weight matrix composed of $\omega \left(e\right)$, and ${\mathbf{D}}_v$ and ${\mathbf{D}}_e$ denote the diagonal matrices composed of $d\left(v\right)$ and $\delta \left(e\right)$, respectively.

There is a wide range for hypergraphs in computer vision, including classification and retrieval tasks. Feng et al. [9] put up the idea of a hypergraph neural network framework (HGNN) for learning data representation. Links are used in social networks. Chen et al. [50] made a methodical and thorough forecast regarding links. Yin et al. [51] presentend a hypergraph-regularized nonnegative tensor factorization for dimensionality reduction. In addition, Wang et al. [30] presented a hypergraph-regularized sparse NMF (HGLNMF) for hyperspectral unmixing, which incorporates the sparse term into HNMF. HGLNMF takes the sparsity of the coefficient matrix into account, and the hypergraph can simulate the higher-order relationship between multiple pixels by using multiple vertices in its hyperedges. Wu et al. [52] presented nonnegative matrix factorization with mixed hypergraph regularization (MHGNMF), by taking into account the higher-order information between the vertices. Some scholars apply nonnegative matrix factorization to multiple perspectives, Zhang et al. [53] presented semi-supervised multi-view clustering with dual-hypergraph-regularized partially shared nonnegative matrix factorization (DHPS-NMF). Huang et al. [54] presented diverse deep matrix factorization with hypergraph regularization for multiview data representation. To some extent, these approaches focus more on the multiple vertices hypergraph, which reflects the higher-order relationships between the multiple vertices, but they ignore the base matrix or its smoothness. To overcome this deficiency, we suggest the following hypergraph-regularized $L_p$ smooth nonnegative matrix factorization, which takes into account the higher-order relationships of numerous vertices, as well as the smoothness of the basis matrix.

3. Hypergraph-Regularized ${\mathit{L}}_{\mathit{p}}$ Smooth Nonnegative Matrix Factorization

In this section, we will describe the proposed HGSNMF approach in detail, as well as the iterative updating rules of two factor matrices. Then, the convergence of the proposed iterative updating rules is proven. Finally, the cost of calculating this approach is shown.

First, we give the construction of the hypergraph regularization term. Given a nonnegative data matrix $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_n]\in {\mathbb{R}}_{+}^{m\times n}$, we expect that, if two data samples ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ are close, the corresponding encoding vectors ${\mathbf{c}}_i$ and ${\mathbf{c}}_j$ in the low-dimensional space are also close to each other. We encode geometrical information in the coefficient matrix hypergraph by linking each data sample with its k nearest neighbors and denoting their hypergraph connections with heat kernel weight:

$$\begin{array}{c}\hfill \omega \left(e_i\right)=\sum _{x_j\in e_i}exp(-\frac{\left|\right|{\mathbf{x}}_i-{\mathbf{x}}_j{\left|\right|}^2}{{\delta }^2}),\end{array}$$

(4)

where $\delta =\frac{1}{kn}{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{x_j\in e_i}}\left|\right|{\mathbf{x}}_i-{\mathbf{x}}_j\left|\right|$ denotes the average distance among all the vertices.

With the weight $\omega \left(e\right)$ matrix defined above, the hypergraph regularization of the matrix $\mathbf{C}$ can be caculated by the following optimization problem:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathfrak{R}& =\frac{1}{2}\sum _{e\in E}\sum _{i,j\in e}\frac{\omega \left(e\right)}{\delta \left(e\right)}\left|\right|{\mathbf{c}}_i-{\mathbf{c}}_j{\left|\right|}^2\hfill \\ & =\mathrm{Tr}\left(\mathbf{C}({\mathbf{D}}_v-\mathbf{S}){\mathbf{C}}^{\top }\right)\hfill \\ & =\mathrm{Tr}\left({\mathbf{CL}}_{Hyper}{\mathbf{C}}^{\top }\right),\hfill \end{array}\end{array}$$

where ${\mathbf{L}}_{Hyper}$ is the hypergraph Laplacian matrix of the hyergraph G and is defined by (3).

3.1. The Objective Function

To discover the intrinsic geometric structure information of a data set and produce a smooth and more accurate solution, we propose the HGSNMF method by incorporating hypergraph regularization and the $L_p$ smoothing constraint into NMF. The objective function of our HGSNMF is defined as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}& min\mathcal{O}=\underset{\mathbf{B},\mathbf{C}}{min}\left|\right|\mathbf{X}-{\mathbf{BC}\left|\right|}_F^2+\alpha \mathrm{Tr}\left(\mathbf{C}{\mathbf{L}}_{Hyper}{\mathbf{C}}^{\top }\right)+{2\mu \left|\right|\mathbf{B}\left|\right|}_p^p,\hfill \\ & s.t.\mathbf{B}\ge 0,\mathbf{C}\ge 0,\hfill \end{array}\end{array}$$

(5)

where $\mathbf{B}$ is the basis matrix, ${\left|\right|\mathbf{B}\left|\right|}_p={\left({\displaystyle \sum _{i=1,j=1}^{m,r}}{\left({\mathbf{B}}_{ij}\right)}^p\right)}^{\frac{1}{p}},$$0<p\le 2$ and $p\ne 1$, ${\mathbf{B}}_{ij}$ is the ith row and jth column entry of matrix $\mathbf{B}$, $\mathbf{C}$ is the coefficient matrix, and $\alpha$ and $\mu$ are the positive regularization parameters for balancing the reconstruction error in (5). The hypergraph regularization term and $L_p$ smoothing regularization term are presented in the second and the third terms, respectively. The hypergraph regularization term can more effectively discover the hidden semantics and simultaneously capture the underlying intrinsic geometric structure of the high-dimensional space data. The $L_p$ smoothing constraint of the basis matrix not only smooths the basis matrix by removing noise, but it also produces a smooth and more accurate solution to the optimization problem by combining the advantages of isotropic and diffusion anisotropic smoothing. Then, we give the detailed derivation of the updating rules, the theoretical proof of convergence, and an analysis of the computing complexity of the HGSNMF approach, as well as further comparative experiments.

3.2. Optimization Method

The objective function $\mathcal{O}$ in (5) is not convex in both $\mathbf{B}$ and $\mathbf{C}$, so it is unrealistic to find the global optimal solution. Thus, we can only obtain the local optimal solution by using the iterative method. There are the multiplicative update, projective gradient, alternating direction multiplier, and dictionary learning algorithm for solving optimization problems. Because the multiplicative update algorithm has the advantages of convergence, simple operation, and small computation, we use the multiplicative update algorithm to solve optimization problems. We can turn the objective function into the following unconstrained objective function by using the Lagrange multiplier:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L=& \left|\right|\mathbf{X}-{\mathbf{BC}\left|\right|}_F^2+\alpha \mathrm{Tr}\left(\mathbf{C}{\mathbf{L}}_{Hyper}{\mathbf{C}}^{\top }\right)+{2\mu \left|\right|\mathbf{B}\left|\right|}^p-\mathrm{Tr}\left(\mathsf{{\rm Y}}{\mathbf{B}}^{\top }\right)-\mathrm{Tr}\left(\mathsf{\Lambda }{\mathbf{C}}^{\top }\right),\hfill \end{array}\end{array}$$

where $\mathsf{{\rm Y}}=\left[{\mathsf{{\rm Y}}}_{ik}\right]$, $\mathsf{\Lambda }=\left[{\mathsf{\Lambda }}_{kj}\right]$, and ${\mathsf{{\rm Y}}}_{ik}$ and ${\mathsf{\Lambda }}_{kj}$ are the Lagrange multipliers for the constrains ${\mathbf{B}}_{ik}\ge 0$ and ${\mathbf{C}}_{kj}\ge 0$, respectively.

By taking the partial derivatives of L with respect to $\mathbf{B}$ and $\mathbf{C}$, respectively, we have that

$$\begin{array}{c}\hfill \frac{\partial L}{\partial \mathbf{B}}=2{\mathbf{BCC}}^{\top }-2{\mathbf{XC}}^{\top }+2\mu p{\mathbf{B}}^{p-1}-\mathsf{{\rm Y}},\end{array}$$

$$\begin{array}{c}\hfill \frac{\partial L}{\partial \mathbf{C}}=2{\mathbf{B}}^{\top }\mathbf{BC}-2{\mathbf{B}}^{\top }\mathbf{X}+2\alpha \mathbf{C}{\mathbf{L}}_{Hyper}-\mathsf{\Lambda }.\end{array}$$

By using the Karush–Kuhn–Tucker (KKT) conditions $\frac{\partial L}{\partial \mathbf{B}}=0$, $\frac{\partial L}{\partial \mathbf{C}}=0$, ${\mathsf{{\rm Y}}}_{ik}·{\mathbf{B}}_{ik}=0$, and ${\mathsf{\Lambda }}_{kj}·{\mathbf{C}}_{kj}=0$, we obtain that

$$\begin{array}{c}\hfill {\left({\mathbf{BCC}}^{\top }\right)}_{ik}{\mathbf{B}}_{ik}+\mu p{\mathbf{B}}_{ik}^{p-1}{\mathbf{B}}_{ik}-{\left({\mathbf{XC}}^{\top }\right)}_{ik}{\mathbf{B}}_{ik}=0,\end{array}$$

(6)

$$\begin{array}{c}\hfill \begin{array}{cc}& {\left({\mathbf{B}}^{\top }\mathbf{BC}\right)}_{kj}{\mathbf{C}}_{kj}+\alpha {\left(\mathbf{C}{\mathbf{L}}_{Hyper}\right)}_{kj}{\mathbf{C}}_{kj}-{\left({\mathbf{B}}^{\top }\mathbf{X}\right)}_{kj}{\mathbf{C}}_{kj}=0.\hfill \end{array}\end{array}$$

(7)

According to (6) and (7), we can obtain the following updating rules for $\mathbf{B}$ and $\mathbf{C}$:

$$\begin{array}{c}\hfill {\mathbf{B}}_{ik}\leftarrow {\mathbf{B}}_{ik}\frac{{\left({\mathbf{XC}}^{\top }\right)}_{ik}}{{\left({\mathbf{BCC}}^{\top }\right)}_{ik}+\mu p{\mathbf{B}}_{ik}^{p-1}}\end{array}$$

(8)

and

$$\begin{array}{c}\hfill {\mathbf{C}}_{kj}\leftarrow {\mathbf{C}}_{kj}\frac{{\left({\mathbf{B}}^{\top }\mathbf{X}\right)}_{kj}+\alpha {\left(\mathbf{CS}\right)}_{kj}}{{\left({\mathbf{B}}^{\top }\mathbf{BC}\right)}_{kj}+\alpha {\left(\mathbf{C}{\mathbf{D}}_v\right)}_{kj}},\end{array}$$

(9)

respectively.

3.3. Convergence Analysis

In this part, we demonstrate the convergence of our proposed HGSNMF in (5) by utilizing the updating rules (8) and (9). First of all, we introduce some related definitions and lemmas.

Definition 1 

([14]). $G(x,x^{\prime })$ is an auxiliary function of $F\left(x\right)$ if $G(x,x^{\prime })$ satisfies the condition

$$G(x,x^{\prime })\ge F\left(x\right),G(x,x)=F\left(x\right).$$

The auxiliary function plays an important role due to the following lemma.

Lemma 1 

([14]). If G is an auxiliary function of F, then F is nonincreasing under the updating rule

$$\begin{array}{c}\hfill x^{t+1}=arg\underset{x}{min}G(x,x^t).\end{array}$$

(10)

To prove the convergence of HGSNMF under the updating step for $\mathbf{B}$ in (8), we fix the matrix $\mathbf{C}$. For any element ${\mathbf{B}}_{ik}$ in $\mathbf{B}$, we use ${\tilde{F}}_{ik}$ to denote the part of the objective function $\mathcal{O}$ that is relevant only to ${\mathbf{B}}_{ik}$. The first and second derivatives of $\tilde{F}\left({\mathbf{B}}_{ik}\right)$ are given as follows:

$$\begin{array}{c}\hfill {\tilde{F}}_{ik}^{\prime }={\left(\frac{\partial \mathcal{O}}{\partial \mathbf{B}}\right)}_{ik}=2{({\mathbf{BCC}}^{\top }-{\mathbf{XC}}^{\top })}_{ik}+2\mu p{\left({\mathbf{B}}_{ik}\right)}^{p-1}\end{array}$$

and

$$\begin{array}{c}\hfill {\tilde{F}}_{ik}^{″}={\left(\frac{{\partial }^2\mathcal{O}}{\partial {\mathbf{B}}^2}\right)}_{ik}=2{\left({\mathbf{CC}}^{\top }\right)}_{kk}+2\mu p(p-1){\left({\mathbf{B}}_{ik}\right)}^{p-2},\end{array}$$

respectively.

Lemma 2.

The function

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \tilde{G}(x,{\mathbf{B}}_{ik}^t)=\tilde{F}\left({\mathbf{B}}_{ik}^t\right)+{\tilde{F}}^{\prime }\left({\mathbf{B}}_{ik}^t\right)(x-{\mathbf{B}}_{ik}^t)+\frac{{\left({\mathbf{BCC}}^{\top }\right)}_{ik}+\mu p{\left({\mathbf{B}}_{ik}^t\right)}^{p-1}}{{\mathbf{B}}_{ik}^t}{(x-{\mathbf{B}}_{ik}^t)}^2\end{array}\end{array}$$

(11)

is an auxiliary function of ${\tilde{F}}_{ik}\left(x\right)$, which is only relevant to ${\mathbf{B}}_{ik}$.

Proof. 

From Taylor series expansion, we have that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\tilde{F}}_{ik}\left(x\right)=& \tilde{F}\left({\mathbf{B}}_{ik}^t\right)+{\tilde{F}}^{\prime }\left({\mathbf{B}}_{ik}^t\right)(x-{\mathbf{B}}_{ik}^t)+{\left({\mathbf{CC}}^{\top }\right)}_{kk}{(x-{\mathbf{B}}_{ik}^t)}^2\hfill \\ & +\mu p(p-1){\left({\mathbf{B}}_{ik}^t\right)}^{p-2}){(x-{\mathbf{B}}_{ik}^t)}^2.\hfill \end{array}\end{array}$$

(12)

Clearly, $\tilde{G}(x,x)={\tilde{F}}_{ik}\left(x\right)$. By Definition 1, we just need to prove that $\tilde{G}(x,{\mathbf{B}}_{ik}^t)\ge {\tilde{F}}_{ik}\left(x\right).$ By comparing (11) with (12), we can find that $\tilde{G}(x,{\mathbf{B}}_{ik}^t)\ge {\tilde{F}}_{ik}\left(x\right)$ is equivalent to

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \frac{{\left(\mathbf{B}{\mathbf{CC}}^{\top }\right)}_{ik}+\mu p{\left({\mathbf{B}}_{ik}^t\right)}^{p-1}}{{\mathbf{B}}_{ik}^t}\ge {\left({\mathbf{CC}}^{\top }\right)}_{kk}+\mu p(p-1){\left({\mathbf{B}}_{ik}^t\right)}^{p-2}.\end{array}\end{array}$$

(13)

$\mathbf{B}\ge 0,\mathbf{C}\ge 0,0<p\le 2$, and $p\ne 1$, we have

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\left(\mathbf{B}{\mathbf{CC}}^{\top }\right)}_{ik}=\sum _{l=1,l\ne k}^r{\mathbf{B}}_{il}^t{\left({\mathbf{CC}}^{\top }\right)}_{lk}+{\mathbf{B}}_{ik}^t{\left({\mathbf{CC}}^{\top }\right)}_{kk}\ge {\mathbf{B}}_{ik}^t{\left({\mathbf{CC}}^{\top }\right)}_{kk}\end{array}\end{array}$$

and

$$p{\left({\mathbf{B}}_{ik}^t\right)}^{p-1}\ge p(p-1){\left({\mathbf{B}}_{ik}^t\right)}^{p-1}.$$

Thus, (13) holds, which implies that $\tilde{G}(x,{\mathbf{B}}_{ik}^t)\ge {\tilde{F}}_{ik}\left(x\right)$. □

Theorem 1.

The objective function $\mathcal{O}$ in (5) is nonincreasing under the updating rule of (8).

Proof. 

By replacing $G(x,x^t)$ in (10) with (11), we can obtain the updating rule

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbf{B}}_{ik}^{t+1}& ={\mathbf{B}}_{ik}^t-{\mathbf{B}}_{ik}^t\frac{{\tilde{F}}^{\prime }\left({\mathbf{B}}_{ik}^t\right)}{2{\left(\mathbf{B}{\mathbf{CC}}^{\top }\right)}_{ik}+2\mu p{\left({\mathbf{B}}_{ik}^t\right)}^{p-1}}\hfill \\ & ={\mathbf{B}}_{ik}^t\frac{{\left({\mathbf{XC}}^{\top }\right)}_{ik}}{{\left(\mathbf{B}{\mathbf{CC}}^{\top }\right)}_{ik}+\mu p{\left({\mathbf{B}}_{ik}^t\right)}^{p-1}}.\hfill \end{array}\end{array}$$

(14)

Since (11) is an auxiliary function of ${\tilde{F}}_{ik}$, ${\tilde{F}}_{ik}$ is nonincreasing under the updating rule of (8). □

Next, we fix the matrix $\mathbf{B}$. For any element ${\mathbf{C}}_{kj}$ in $\mathbf{C}$, we use ${\overline{F}}_{kj}$ to denote the part of the objective function $\mathcal{O}$ that is relevant only to ${\mathbf{C}}_{kj}$. By calculation,

$$\begin{array}{c}\hfill {\overline{F}}_{kj}^{\prime }={\left(\frac{\partial \mathcal{O}}{\partial \mathbf{C}}\right)}_{kj}=2{({\mathbf{B}}^{\top }\mathbf{BC}-{\mathbf{B}}^{\top }\mathbf{X}+\alpha {\mathbf{CL}}_{Hyper})}_{kj}\end{array}$$

and

$$\begin{array}{c}\hfill {\overline{F}}_{kj}^{″}={\left(\frac{{\partial }^2\mathcal{O}}{\partial {\mathbf{C}}^2}\right)}_{kj}=2{\left({\mathbf{B}}^{\top }\mathbf{B}\right)}_{kk}+2\alpha {\left({\mathbf{L}}_{Hyper}\right)}_{jj}.\end{array}$$

Lemma 3.

The function

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \overline{G}(x,{\mathbf{C}}_{kj}^t)=\overline{F}\left({\mathbf{C}}_{kj}^t\right)+{\overline{F}}^{\prime }\left({\mathbf{C}}_{kj}^t\right)(x-{\mathbf{C}}_{kj}^t)+\frac{{({\mathbf{B}}^T\mathbf{BC}+\alpha \mathbf{C}{\mathbf{D}}_v)}_{kj}}{{\mathbf{C}}_{kj}^t}{(x-{\mathbf{C}}_{kj}^t)}^2\end{array}\end{array}$$

(15)

is an auxiliary function of ${\overline{F}}_{kj}\left(x\right)$, which is only relevant to ${\mathbf{C}}_{kj}$.

Proof. 

From Taylor series expansion, we have that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\overline{F}}_{kj}\left(x\right)=\overline{F}\left({\mathbf{C}}_{kj}^t\right)+{\overline{F}}^{\prime }\left({\mathbf{C}}_{kj}^t\right)(x-{\mathbf{C}}_{kj}^t)+\left({\left({\mathbf{B}}^{\top }\mathbf{B}\right)}_{kk}+\alpha {\left({\mathbf{L}}_{Hyper}\right)}_{jj})\right(x-{\mathbf{C}}_{kj}^t{)}^2.\end{array}\end{array}$$

(16)

Clearly, $\overline{G}(x,x)={\overline{F}}_{kj}\left(x\right)$. According to Definition 1, we only need to prove that $\overline{G}(x,{\mathbf{C}}_{kj}^t)\ge {\overline{F}}_{kj}\left(x\right).$ By comparing (15) with (16), we see that $\overline{G}(x,{\mathbf{C}}_{kj}^t)\ge {\overline{F}}_{kj}\left(x\right)$ is equivalent to

$$\begin{array}{c}\hfill \frac{{({\mathbf{B}}^{\top }\mathbf{BC}+\alpha \mathbf{C}{\mathbf{D}}_v)}_{kj}}{{\mathbf{C}}_{kj}^t}\ge {\left({\mathbf{B}}^{\top }\mathbf{B}\right)}_{kk}+\alpha {\left({\mathbf{L}}_{Hyper}\right)}_{jj}.\end{array}$$

(17)

Since $\mathbf{B}\ge 0$ and $\mathbf{C}\ge 0$, we have

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\left({\mathbf{B}}^{\top }\mathbf{BC}\right)}_{kj}=\sum _{l=1,l\ne j}^n{\mathbf{C}}_{kl}^t{\left({\mathbf{B}}^{\top }\mathbf{B}\right)}_{kk}+{\mathbf{C}}_{kj}^t{\left({\mathbf{B}}^{\top }\mathbf{B}\right)}_{kk}\ge {\mathbf{C}}_{kj}^t{\left({\mathbf{B}}^{\top }\mathbf{B}\right)}_{kk}\end{array}\end{array}$$

and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\left(\mathbf{C}{\mathbf{D}}_v\right)}_{kj}=\sum _{l=1,l\ne j}^n{\mathbf{C}}_{kl}^t{\left({\mathbf{D}}_v\right)}_{lj}+{\mathbf{C}}_{kj}^t{\left({\mathbf{D}}_v\right)}_{jj}\ge {\mathbf{C}}_{kj}^t{({\mathbf{D}}_v-\mathbf{S})}_{jj}={\mathbf{C}}_{kj}^t{\left({\mathbf{L}}_{Hyper}\right)}_{jj}.\end{array}\end{array}$$

Thus, (17) holds, which implies that $\overline{G}(x,{\mathbf{C}}_{kj}^t)\ge {\overline{F}}_{kj}\left(x\right)$. □

Theorem 2.

The objective function $\mathcal{O}$ in (5) is nonincreasing under the updating rule of (9).

Proof. 

By replacing $G(x,x^t)$ in (10) with (15), we can obtain the updating rule:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbf{C}}_{kj}^{t+1}& ={\mathbf{C}}_{kj}^t-{\mathbf{C}}_{kj}^t\frac{{\overline{F}}^{\prime }\left({\mathbf{C}}_{kj}^t\right)}{2{({\mathbf{B}}^{\top }\mathbf{BC}+\alpha \mathbf{C}{\mathbf{D}}_v)}_{kj}}\hfill \\ & ={\mathbf{C}}_{kj}^t\frac{{({\mathbf{B}}^{\top }\mathbf{X}+\alpha \mathbf{C}\mathbf{B})}_{kj}}{{({\mathbf{B}}^{\top }\mathbf{BC}+\alpha \mathbf{C}{\mathbf{D}}_v)}_{kj}}.\hfill \end{array}\end{array}$$

(18)

Since (15) is an auxiliary function of ${\overline{F}}_{kj}$, ${\overline{F}}_{kj}$ is nonincreasing under the updating rule of (9). □

Similar to NMF, it is known from Theorems 1 and 2 that the convergence of the model (5) can be guaranteed under the updating rules of (8) and (9).

The specific procedure for finding the local optimal $\mathbf{B}$ and $\mathbf{C}$ of HGSNMF is summarized in Algorithm 1.

For the specific implementation of Algorithm 1, we first repeat HGSNMF 10 times on the original data, and then this low-dimensional reduced data repeats K-means clustering 10 times.

Algorithm 1 HGSNMF algorithm.

Input: Data matrix $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n]\in {\mathbb{R}}_{+}^{m\times n}$. The number of neighbors k. The algorithm

parameters r, p and regularization parameters $\alpha$, $\mu$. The stopping criterion $ϵ$, and the maximum

number of iterations maxiter. Let ${\Delta }_0=1e-5$.

Output: Factors $\mathbf{B}$ and $\mathbf{C}$;

1: Initialize $\mathbf{B}$ and $\mathbf{C}$;

2: Construct the weight matrix $\mathbf{W}$ using (4), and calculate the

matrix ${\mathbf{D}}_v$, $\mathbf{S}$;

3: for $t=1,2,\cdots ,$ maxiter do

4:    Update ${\mathbf{B}}^t$ and Update ${\mathbf{C}}^t$ according to (8), (9), respectively.

5:    Compute the objective function value $\mathcal{O}$ of (5) to denote ${\Delta }_t$.

6:    if $\frac{|{\Delta }_t-{\Delta }_{t-1}|}{{\Delta }_{t-1}}<ϵ,$

Break and return $\mathbf{B},\mathbf{C}$.

7:    end if

8: end for

3.4. Computational Complexity Analysis

In this section, we analyze the computational complexity of the proposed HGSNMF method compared to other nonnegative matrix methods. The fladd, flmlt, and fldiv denote floating point addition, multiplication, and division, respectively. The $“O^{”}$ notation denotes the computational cost. The parameters n, m, r, and k denote the number of sample points, features, factors, and nearest neighbors to construct an edge or hyperedge, respectively.

According to the updating rules, we calculate the arithmetic operations of each iteration in HGSNMF and summarise the results of the proposed HGSNMF. From Table 1, we can see that the total cost of our proposed HGSNMF is $O\left(mnr\right)$.

Table 1. Computational operation counts for each iteration in NMF, GNMF, HNMF, GSNMF, HGLNMF, and HGSNMF.

	fladd	flmlt	fldiv	Overall
NMF	$2nmr+2(m+n)r^2$	2$nmr$+2$(m+n)r^2+mr+nr$	$mr+nr$	$O\left(mnr\right)$
GNMF	2$nmr$+2$(m+n)r^2+nr+3n$	2$nmr$+2$(m+n)r^2+(m+2n+nk)r$	$mr+nr$	$O\left(mnr\right)$
HNMF	2$nmr$+2$(m+n)r^2+nr+3n$	2$nmr$+2$(m+n)r^2+(m+2n+nk)r$	$mr+nr$	$O\left(mnr\right)$
GSNMF	2$nmr$+2$(m+n)r^2+nk+3n+mr$	2$nmr$+2$(m+n)r^2+(m+2n+nk)r$	$mr+nr$	$O\left(mnr\right)$
HGLNMF	2$nmr$+2$(m+n)r^2+nk+3n+mr$	2$nmr$+2$(m+n)r^2+(m+2n+nk)r$	$mr+nr$	$O\left(mnr\right)$
HGSNMF	2$nmr$+2$(m+n)r^2+nk+3n+mr$	2$nmr$+2$(m+n)r^2+(m+2n+nk)r$	$mr+nr$	$O\left(mnr\right)$

4. Numerical Experimentation

In this section, we compare the results of data clustering on five popular data sets to estimate the performance of the proposed HGSNMF method with the related state-of-the-art methods, such as K-means, NMF [13], GNMF [2], HNMF [37], GSNMF [38], HGLNMF [30], GNTD [39], and UGNTD [40]. All tests were performed on a computer with a 2-Ghz Intel(R) Core(TM) i5-2500U CPU and 8-GB memory 64-bit using MATLAB R2015a in Windows 10. The stopping criterion was $ϵ={10}^{-5}$, and the maximum number of iterations was set to ${10}^4$.

4.1. Data Sets

The clustering performance was evaluated on five widely used data sets, including COIL20 (https://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php (accessed on 26 October 2021)), YALE, Mnist, ORL (https://www.cad.zju.edu.cn/home/dengcai.Data/data.html, accessed on: 26 October 2021), and Georgia ((https://www.anefian.com/research/face-reco.htm (accessed on 26 October 2021)). The important statistical information of the five data sets is listed in Table 2, with more details given as follows.

Table 2. Statistics of the five data sets.

Data Sets	Samples	Features	Classes
COIL20	1440	1024	20
YALE	165	1024	15
ORL	400	1024	40
Georgia	750	1024	50
Mnist	500	784	10

(1) COIL20: The data set contains 72 grey-scale images for each of 20 objects viewed at varying angles.They were resized to $32\times 32$.

(2) ORL: The data set contains 10 different images of each of 40 human subjects. For some subjects, the images were taken at different times and different light conditions. They capture different facial expressions and different facial details. We resized them to $32\times 32$.

(3) YALE: The data set contains 11 grey-scale images for each of 15 individuals viewed at different facial expressions or configurations. These images were resized to $32\times 32$.

(4) Georgia: The data set contains 15 color JPEG grey face images for each of 50 people, with cluttered backgrounds for each object. We also resized them to $32\times 32$.

(5) Mnist: The data set contains 700 grey images of handwritten digits from zero to nine. During the experiment, we randomly selected 50 digit images from each category. Each image was resized to $28\times 28$.

4.2. Evaluation Metrics

Two popular evaluation metrics were used: the clustering accuracy (ACC) and the normalized mutual information (NMI) [55], which evaluate the clustering performance by comparing the obtained cluster label of each sample with the label provided by the data set. ACC is defined as follows

$$\begin{array}{c}\hfill ACC=\frac{{\displaystyle \sum _{i=1}^n}\delta (r_i,map\left(q_i\right))}{n},\end{array}$$

where $r_i$ is the correct label provided by the real data set, $q_i$ is the clustering label obtained by the clustering result, n is the total number of documents, $\delta (x,y)$ is the delta function that equals one if $x=y$ and equals to zero otherwise, and $map(·)$ is a mapping function that maps each cluster label $q_i$ to a given equivalent label from the data set. By using the Kuhn–Munkers algorithm [56], one can find the best mapping.

Given two clusters C and $C^{\prime }$, the mutual information metric $MI(C,C^{\prime })$ can be defined as follows

$$\begin{array}{c}\hfill MI(C,C^{\prime })=\sum _{c_i\in C,c_j^{\prime }\in C{}^{\prime }}p(c_i,c_j^{\prime })log\frac{p(c_i,c_j^{\prime })}{p\left(c_i\right)·p\left(c_j^{\prime }\right)},\end{array}$$

where $p\left(c_i\right)$ and $p\left(c_j^{\prime }\right)$ denote the probabilities that an arbitrarily chosen sample from the data set belongs to the clusters $c_i$ and $c_j^{\prime }$, respectively, and $p(c_i,c_j^{\prime })$ denotes the joint possibility that this arbitrarily selected image belongs to the cluster $c_i$ and the cluster $c_j^{\prime }$ at the same time. The normalized mutual information (NMI) is defined as follows

$$\begin{array}{c}\hfill NMI(C,C^{\prime })=\frac{MI(C,C^{\prime })}{max(H\left(C\right),H\left(C^{\prime }\right))},\end{array}$$

where C is a set of the true labels, and $C^{\prime }$ is a set of clusters obtained from the clustering algorithms. $H\left(C\right)$ and $H\left(C^{\prime }\right)$ are the entropies of C and $C^{\prime }$, respectively.

4.3. Performance Evaluations and Comparisons

To evaluate the performance of our proposed method, we chose K-means, NMF, GNMF, HNMF, and GSNMF as the comparison clustering algorithms:

(1) K-means: The K-means performs clustering on the original data; we employed it to uncove whether low-dimension data can improve clustering performance on high-dimension data.

(2) NMF [13]: The original NMF represents the data by imposing nonnegative constraints on the factor matrices.

(3) GNMF [2]: Based on NMF, it constructs a local geometric structure of the original data space as a regularization term.

(4) HNMF [37]: It incorporates the hypergraph regularization term into NMF.

(5) GSNMF [38]: It incorporates both graph regularization and the $L_p$ smoothing constraint into NMF.

(6) HGLNMF [30]: It incorporates both hypergraph regularization and the $L_{\frac{1}{2}}$ sparse constraint into NMF.

(7) GNTD [39]: It is a graph-regularized nonnegative Tucker decomposition method that incorporates the graph regularization term into the NTD, which can preserve the geometrical information for high-dimensional tensor data.

(8) UGNTD [40]: It is a graph-regularized nonnegative Tucker decomposition method that incorporates the graph regularization term into the NTD, and the alternating proximate gradient descent method is used to optimize the GNTD model.

(9) HGSNMF: We proposed the hypergraph-regularized $L_p$ smooth nonnegative matrix factorization by incorporating hypergraph regularization term and $L_p$ smoothing constraint into NMF.

The general clustering number k was fixed, and we describe the experiment as follows.

For the NMF, GNMF, HNMF, GSNMF, HGLNMF, and HGSNMF, we initialized two low-rank nonnegative matrix factors using a random strategy in the experiments. Next, we set the dimensionality of the low-dimensional space to the number of clusters, and we used the classical K-means method to cluster the samples in the new data representation space.

For the NMF, GNMF, HNMF, GSNMF, HGLNMF, and HGSNMF, we used the Frobenius norm as the reconstruction error of the objective function. For the GNMF, GSNMF, and GNTD, the heat kernel weighting scheme was used to generate the five-nearest-neighbors graph for constructing the weight matrix. For the HNMF, HGLNMF, and HGSNMF, the heat kernel weighting scheme was used to generate the five-nearest-neighbors graph for constructing the weight matrix in the hypergraph. For the GNMF, GSNMF, and GNTD, the graph regularization parameter was set to 100 for each of them. For the UGNTD, the graph regularization parameter was set to 1000. For the HNMF and HGLNMF, the hypergraph regularization parameter was set to 100 for each of them, and for the GSNMF, the parameter was set as $p=1.7$. For the HGLNMF, the parameter µ was tuned from $\{0.1,10,100\}$, and the best results are reported. For the GNTD, for the Mnist data set, the first two directional sizes of the kernel tensor were chosen from $\{10,12,14\}$, and the third direction was taken as the class number k. For the UGNTD, the size of the core tensor was set to 30 × 3 × 10. For the HGSNMF, the parameters $\alpha$ and $\beta$ were tuned from $\{100,500,1000\}$ for COIL20, $\{0.1,10,500,1000\}$ for YALE, $\{1,10,100\}$ for ORL, $\{500,1000\}$ for Georgia, and $\{100,500\}$ for Mnist; the parameter p was tuned from $\{0.3,0.5,1.1\}$ for COIL20, $\{0.1,0.5,1.1,1.3\}$ for YALE, $\{0.1,1.2,1.5\}$ for ORL, $\{0.5,0.6\}$ for Georgia, and $\{1.4,1.7\}$ for Mnist. The best results are reported.

For K-means, we repeated K-means clustering 10 times on the original data. For the NMF, GNMF, HNMF, GSNMF, HGLNMF, and HGSNMF, we first repeated NMF, GNMF, HNMF, GSNMF, HGLNMF, GNTD, UGNTD, and HGSNMF 10 times on the original data and then repeated K-means clustering 10 times on this low-dimensional reduced data, respectively. We report the average clustering performance and standard deviation in Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, Table 10, Table 11 and Table 12, where the best results are in bold.

Table 3. Normalized mutual information (NMI) on COIL20 data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
4	65.13 ± 16.60	69.63 ± 15.76	72.86 ± 14.86	79.53 ± 13.65	73.79 ± 13.51	79.52 ± 13.64	$\mathbf{84}.\mathbf{45}\pm \mathbf{15}.\mathbf{24}$
6	67.70 ± 9.79	69.65 ± 11.27	72.79 ± 10.70	80.72 ± 10.58	68.63 ± 10.00	80.76 ± 10.54	$\mathbf{83}.\mathbf{84}\pm \mathbf{8}.\mathbf{37}$
8	70.56 ± 5.89	69.44 ± 8.78	71.53 ± 8.60	80.94 ± 7.35	73.55 ± 7.28	81.08 ± 7.38	$\mathbf{81}.\mathbf{85}\pm \mathbf{8}.\mathbf{11}$
10	76.02 ± 7.12	70.13 ± 6.95	76.01 ± 5.48	82.99 ± 5.84	76.36 ± 5.92	83.00 ± 5.75	$\mathbf{83}.\mathbf{83}\pm \mathbf{5}.\mathbf{41}$
12	73.21 ± 4.82	70.91 ± 5.50	77.12 ± 5.68	82.16 ± 5.03	75.70 ± 5.33	82.31 ± 5.08	$\mathbf{83}.\mathbf{75}\pm \mathbf{5}.\mathbf{31}$
14	74.10 ± 4.31	70.41 ± 4.66	77.06 ± 4.75	82.20 ± 4.24	76.88 ± 4.63	82.21 ± 4.19	$\mathbf{83}.\mathbf{69}\pm \mathbf{4}.\mathbf{78}$
16	74.85 ± 3.65	72.70 ± 4.04	79.38 ± 4.52	84.05 ± 3.95	79.04 ± 4.27	83.99 ± 3.97	$\mathbf{85}.\mathbf{83}\pm \mathbf{4}.\mathbf{28}$
18	73.28 ± 3.08	71.49 ± 3.00	78.03 ± 3.65	84.61 ± 3.15	79.36 ± 3.47	84.70 ± 3.16	$\mathbf{85}.\mathbf{74}\pm \mathbf{3}.\mathbf{30}$
20	73.83 ± 2.52	71.95 ± 2.76	79.20 ± 3.05	84.08 ± 2.79	78.84 ± 2.76	84.05 ± 2.71	$\mathbf{85}.\mathbf{15}\pm \mathbf{3}.\mathbf{52}$
Avg.	72.0	71.70	76.00	82.36	75.80	82.40	$\mathbf{84}.\mathbf{24}$

Table 4. Clustering accuracy (ACC) on COIL20 data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
4	71.45 ± 14.82	72.35 ± 16.50	73.41 ± 14.85	75.78 ± 17.88	75.33 ± 15.74	75.77 ± 17.88	$\mathbf{81}.\mathbf{31}\pm \mathbf{19}.\mathbf{19}$
6	67.05 ± 10.72	68.87 ± 11.80	69.65 ± 12.61	74.42 ± 13.78	68.04 ± 10.78	74.44 ± 13.78	$\mathbf{77}.\mathbf{30}\pm \mathbf{11}.\mathbf{89}$
8	64.56 ± 64.56	65.39 ± 9.95	64.35 ± 10.93	71.06 ± 11.77	67.36 ± 9.73	70.71 ± 11.75	$\mathbf{71}.\mathbf{43}\pm \mathbf{11}.\mathbf{74}$
10	67.38 ± 9.76	63.27 ± 7.96	66.76 ± 7.47	70.73 ± 10.04	67.07 ± 8.44	70.61 ± 5.75	$\mathbf{73}.\mathbf{39}\pm \mathbf{8}.\mathbf{96}$
12	63.73 ± 7.09	63.19 ± 7.02	66.81 ± 8.30	68.63 ± 8.43	65.81 ± 8.43	69.00 ± 5.08	$\mathbf{71}.\mathbf{50}\pm \mathbf{8}.\mathbf{22}$
14	62.48 ± 6.42	60.01 ± 6.16	65.18 ± 7.57	68.04 ± 8.33	64.21 ± 7.77	68.03 ± 8.18	$\mathbf{83}.\mathbf{69}\pm \mathbf{7}.\mathbf{88}$
16	61.78 ± 5.69	62.18 ± 6.43	65.47 ± 7.25	69.09 ± 7.62	66.03 ± 7.50	68.85 ± 7.66	$\mathbf{70}.\mathbf{31}\pm \mathbf{8}.\mathbf{24}$
18	59.15 ± 6.18	59.68 ± 5.44	63.39 ± 6.86	69.29 ± 6.51	65.84 ± 6.70	69.65 ± 6.59	$\mathbf{70}.\mathbf{85}\pm \mathbf{6}.\mathbf{35}$
20	58.18 ± 5.43	59.11 ± 4.60	63.86 ± 6.24	$\mathbf{68}.\mathbf{68}\pm \mathbf{6}.\mathbf{52}$	63.95 ± 6.00	68.56 ± 6.34	68.19 ± 6.83
Avg.	63.97	63.78	66.54	70.64	67.07	70.63	$\mathbf{72}.\mathbf{63}$

Table 5. Normalized mutual information (NMI) on YALE data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
3	40.18 ± 23.03	28.96 ± 11.71	28.81 ± 12.06	36.08 ± 12.82	37.80 ± 16.40	36.12 ± 12.89	$\mathbf{41}.\mathbf{36}\pm \mathbf{13}.\mathbf{07}$
5	35.72 ± 12.87	38.23 ± 10.25	38.48 ± 10.02	39.37 ± 8.83	40.01 ± 10.61	39.35 ± 8.85	$\mathbf{41}.\mathbf{91}\pm \mathbf{10}.\mathbf{34}$
7	$\mathbf{43}.\mathbf{18}\pm \mathbf{6}.\mathbf{55}$	38.38 ± 6.51	38.17 ± 7.57	39.07 ± 6.84	39.33 ± 6.46	39.38 ± 5.23	42.32 ± 7.39
9	$\mathbf{42}.\mathbf{22}\pm \mathbf{7}.\mathbf{98}$	41.80 ± 3.87	40.56 ± 5.00	38.93 ± 5.05	38.85 ± 4.88	39.18 ± 5.23	40.21 ± 4.35
11	39.80 ± 4.40	41.82 ± 4.45	42.05 ± 4.25	43.88 ± 4.30	44.08 ± 4.62	44.04 ± 4.61	$\mathbf{45}.\mathbf{06}\pm \mathbf{4}.\mathbf{70}$
13	44.13 ± 4.63	44.17 ± 3.03	44.59 ± 3.43	44.24 ± 3.12	44.53 ± 2.80	44.29 ± 3.07	$\mathbf{47}.\mathbf{54}\pm \mathbf{3}.\mathbf{84}$
14	44.34 ± 3.84	43.27 ± 2.92	43.31 ± 3.10	44.21 ± 3.39	44.82 ± 3.02	44.21 ± 3.32	$\mathbf{46}.\mathbf{82}\pm \mathbf{3}.\mathbf{85}$
15	44.48 ± 2.92	43.91 ± 2.90	44.36 ± 2.72	45.37 ± 2.38	45.32 ± 2.62	45.47 ± 2.35	$\mathbf{47}.\mathbf{60}\pm \mathbf{3}.\mathbf{06}$
Avg.	41.76	40.07	40.04	41.40	41.84	41.51	$\mathbf{44}.\mathbf{10}$

Table 6. Clustering accuracy (ACC) on YALE data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
3	62.24 ± 14.91	59.30 ± 8.16	59.12 ± 8.11	61.49 ± 8.20	62.64 ± 11.14	61.91 ± 8.61	$\mathbf{64}.\mathbf{82}\pm \mathbf{10}.\mathbf{06}$
5	50.24 ± 10.66	54.15 ± 8.92	53.78 ± 8.27	54.95 ± 7.83	54.82 ± 9.04	54.84 ± 7.82	$\mathbf{56}.\mathbf{84}\pm \mathbf{8}.\mathbf{79}$
7	49.43 ± 6.26	47.40 ± 6.33	46.92 ± 6.90	47.21 ± 6.81	46.65 ± 6.14	47.44 ± 6.62	$\mathbf{49}.\mathbf{87}\pm \mathbf{7}.\mathbf{32}$
9	44.40 ± 7.02	$\mathbf{44}.\mathbf{87}\pm \mathbf{4}.\mathbf{30}$	43.12 ± 4.87	42.48 ± 5.45	42.15 ± 4.74	42.81 ± 5.49	43.68 ± 4.98
11	39.12 ± 4.80	41.37 ± 5.06	41.74 ± 4.65	43.43 ± 5.00	43.07 ± 5.29	43.50 ± 5.12	$\mathbf{44}.\mathbf{19}\pm \mathbf{5}.\mathbf{20}$
13	40.41 ± 4.81	41.18 ± 3.73	41.11 ± 4.10	40.76 ± 3.83	41.05 ± 3.61	40.78 ± 3.91	$\mathbf{43}.\mathbf{57}\pm \mathbf{4}.\mathbf{17}$
14	39.46 ± 4.26	39.05 ± 4.21	38.27 ± 3.72	39.92 ± 4.21	40.03 ± 3.75	39.94 ± 4.22	$\mathbf{41}.\mathbf{97}\pm \mathbf{4}.\mathbf{33}$
15	38.52 ± 3.30	38.72 ± 3.51	38.67 ± 3.21	40.25 ± 3.39	39.52 ± 3.14	40.44 ± 3.24	$\mathbf{41}.\mathbf{86}\pm \mathbf{3}.\mathbf{37}$
Avg.	45.41	45.76	45.34	46.31	46.24	46.46	$\mathbf{48}.\mathbf{35}$

Table 7. Normalized mutual information (NMI) on ORL data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
5	66.24 ± 12.00	67.18 ± 12.07	68.81 ± 11.16	68.58 ± 12.77	66.51 ± 13.18	68.97 ± 13.01	$\mathbf{68}.\mathbf{98}\pm \mathbf{12}.\mathbf{29}$
10	70.22 ± 6.63	73.59 ± 5.90	$\mathbf{73}.\mathbf{72}\pm \mathbf{7}.\mathbf{10}$	72.11 ± 6.70	71.95 ± 5.92	72.39 ± 6.59	73.29 ± 6.55
15	68.46 ± 4.21	75.23 ± 5.01	76.14 ± 5.18	75.26 ± 5.62	75.67 ± 5.27	75.26 ± 5.62	$\mathbf{76}.\mathbf{54}\pm \mathbf{5}.\mathbf{42}$
20	69.87 ± 4.75	74.21 ± 4.34	74.44 ± 4.78	75.24 ± 4.25	75.49 ± 3.76	75.46 ± 4.19	$\mathbf{76}.\mathbf{00}\pm \mathbf{4}.\mathbf{08}$
25	71.13 ± 3.48	75.51 ± 2.69	75.88 ± 3.13	76.03 ± 3.29	76.10 ± 3.17	76.06 ± 3.12	$\mathbf{76}.\mathbf{91}\pm \mathbf{3}.\mathbf{22}$
30	71.03 ± 2.81	75.34 ± 3.12	75.55 ± 2.81	74.60 ± 2.67	$\mathbf{75}.\mathbf{89}\pm \mathbf{2}.\mathbf{78}$	74.69 ± 2.65	75.88 ± 2.80
35	71.07 ± 1.82	75.07 ± 2.23	74.96 ± 2.06	74.46 ± 1.87	75.85 ± 2.18	74.52 ± 1.91	$\mathbf{75}.\mathbf{96}\pm \mathbf{2}.\mathbf{35}$
40	71.45 ± 2.06	75.05 ± 1.90	75.26 ± 1.82	74.54 ± 1.87	75.40 ± 1.91	74.54 ± 1.91	$\mathbf{76}.\mathbf{07}\pm \mathbf{1}.\mathbf{77}$
Avg.	69.93	73.90	74.35	73.85	74.11	73.99	$\mathbf{74}.\mathbf{95}$

Table 8. Clustering accuracy (ACC) on ORL data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
5	67.32 ± 14.91	68.12 ± 12.11	68.76 ± 12.31	68.70 ± 12.27	67.36 ± 12.72	68.97 ± 13.01	$\mathbf{69}.\mathbf{30}\pm \mathbf{12}.\mathbf{15}$
10	62.72 ± 10.66	$\mathbf{66}.\mathbf{11}\pm \mathbf{8}.\mathbf{02}$	65.85 ± 9.86	64.05 ± 7.87	64.59 ± 7.22	64.46 ± 7.82	65.42 ± 7.50
15	56.19 ± 5.80	63.55 ± 6.85	64.84 ± 6.96	63.99 ± 7.44	64.59 ± 7.32	63.99 ± 7.44	$\mathbf{65}.\mathbf{89}\pm \mathbf{7}.\mathbf{38}$
20	55.29 ± 6.25	61.21 ± 5.83	61.56 ± 6.21	62.44 ± 5.87	62.67 ± 5.30	62.52 ± 5.57	$\mathbf{62}.\mathbf{80}\pm \mathbf{5}.\mathbf{73}$
25	54.15 ± 4.50	60.58 ± 3.98	61.01 ± 4.84	61.31 ± 4.66	61.16 ± 4.96	61.32 ± 4.80	$\mathbf{61}.\mathbf{91}\pm \mathbf{5}.\mathbf{22}$
30	52.52 ± 4.29	58.57 ± 4.67	58.88 ± 4.52	58.07 ± 4.42	$\mathbf{59}.\mathbf{91}\pm \mathbf{4}.\mathbf{08}$	58.38 ± 4.28	59.50 ± 4.20
35	51.30 ± 3.20	57.83 ± 3.62	57.22 ± 3.46	56.95 ± 3.28	$\mathbf{58}.\mathbf{79}\pm \mathbf{3}.\mathbf{58}$	57.05 ± 3.21	58.35 ± 4.06
40	50.68 ± 3.43	56.57 ± 3.39	56.73 ± 3.15	55.88 ± 3.32	57.20 ± 3.46	55.88 ± 3.37	$\mathbf{57}.\mathbf{36}\pm \mathbf{3}.\mathbf{38}$
Avg.	56.27	61.57	61.86	61.42	62.03	61.57	$\mathbf{62}.\mathbf{57}$

Table 9. Normalized mutual information (NMI) on Georgia data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
5	67.05 ± 11.33	59.40 ± 11.79	63.00 ± 12.27	60.93 ± 11.93	64.46 ± 10.83	60.99 ± 11.94	$\mathbf{68}.\mathbf{18}\pm \mathbf{9}.\mathbf{19}$
10	$\mathbf{67}.\mathbf{82}\pm \mathbf{7}.\mathbf{48}$	60.25 ± 8.93	61.24 ± 8.51	57.48 ± 10.50	61.59 ± 10.12	57.63 ± 10.67	65.91 ± 9.15
15	64.64 ± 5.32	60.57 ± 4.39	62.46 ± 4.85	61.89 ± 5.12	64.02 ± 5.72	61.99 ± 5.02	$\mathbf{68}.\mathbf{55}\pm \mathbf{5}.\mathbf{08}$
20	67.12 ± 4.30	60.60 ± 3.71	62.58 ± 3.91	60.98 ± 3.81	64.44 ± 3.18	61.08 ± 3.82	$\mathbf{68}.\mathbf{91}\pm \mathbf{2}.\mathbf{93}$
25	66.30 ± 3.31	59.31 ± 2.73	61.35 ± 2.62	61.33 ± 3.22	64.83 ± 2.98	61.32 ± 3.68	$\mathbf{69}.\mathbf{44}\pm \mathbf{2}.\mathbf{87}$
30	66.01 ± 3.13	60.26 ± 2.56	63.20 ± 2.25	60.52 ± 3.04	64.61 ± 2.97	60.47 ± 2.99	$\mathbf{69}.\mathbf{61}\pm \mathbf{2}.\mathbf{87}$
35	65.10 ± 2.13	59.93 ± 2.33	63.3 ± 1.80	59.21 ± 2.21	63.27 ± 2.37	59.20 ± 2.22	$\mathbf{68}.\mathbf{70}\pm \mathbf{1}.\mathbf{95}$
40	66.06 ± 2.20	59.58 ± 2.34	62.84 ± 1.82	58.61 ± 2.38	63.57 ± 1.98	58.62 ± 2.48	$\mathbf{69}.\mathbf{18}\pm \mathbf{1}.\mathbf{69}$
45	66.17 ± 1.35	59.99 ± 1.75	62.92 ± 1.55	58.22 ± 1.90	62.92 ± 1.66	58.25 ± 1.99	$\mathbf{69}.\mathbf{07}\pm \mathbf{1}.\mathbf{41}$
50	66.36 ± 1.32	59.05 ± 1.56	62.11 ± 1.51	58.19 ± 1.33	63.18 ± 1.24	58.19 ± 1.25	$\mathbf{69}.\mathbf{18}\pm \mathbf{1}.\mathbf{13}$
Avg.	66.26	59.90	62.50	59.74	63.69	59.77	$\mathbf{68}.\mathbf{47}$

Table 10. Clustering accuracy (ACC) on Georgia data set.

k	K-Means	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
5	68.73 ± 11.50	66.68 ± 10.96	69.52 ± 11.15	68.00 ± 10.33	69.76 ± 10.50	67.68 ± 10.47	$\mathbf{73}.\mathbf{12}\pm \mathbf{10}.\mathbf{04}$
10	61.71 ± 8.56	57.79 ± 8.48	59.25 ± 8.44	55.73 ± 9.09	59.07 ± 8.94	55.83 ± 9.23	$\mathbf{61}.\mathbf{97}\pm \mathbf{9}.\mathbf{39}$
15	55.38 ± 6.27	53.33 ± 4.41	55.05 ± 5.38	55.07 ± 5.41	56.08 ± 6.15	55.21 ± 5.38	$\mathbf{59}.\mathbf{40}\pm \mathbf{6}.\mathbf{31}$
20	55.14 ± 5.07	50.55 ± 4.58	52.30 ± 4.60	50.22 ± 4.45	54.93 ± 4.36	50.37 ± 4.40	$\mathbf{57}.\mathbf{50}\pm \mathbf{4}.\mathbf{29}$
25	51.82 ± 4.40	46.8¡¤ ± 3.59	48.50 ± 3.42	48.25 ± 4.38	52.10 ± 4.09	48.311 ± 4.30	$\mathbf{55}.\mathbf{52}\pm \mathbf{4}.\mathbf{06}$
30	49.67 ± 4.12	45.20 ± 3.49	48.31 ± 3.13	45.82 ± 3.48	50.17 ± 3.82	45.68 ± 3.33	$\mathbf{54}.\mathbf{18}\pm \mathbf{3}.\mathbf{67}$
35	47.80 ± 3.28	43.78 ± 3.12	47.09 ± 2.92	42.57 ± 3.03	47.19 ± 3.31	42.53 ± 3.02	$\mathbf{52}.\mathbf{17}\pm \mathbf{2}.\mathbf{96}$
40	47.88 ± 3.28	41.93 ± 2.95	45.35 ± 2.81	40.47 ± 3.26	46.27 ± 3.02	40.41 ± 3.45	$\mathbf{51}.\mathbf{52}\pm \mathbf{2}.\mathbf{90}$
45	47.39 ± 2.26	41.07 ± 2.52	43.93 ± 2.44	38.53 ± 2.49	44.34 ± 2.50	38.58 ± 2.59	$\mathbf{50}.\mathbf{40}\pm \mathbf{2}.\mathbf{50}$
50	46.18 ± 2.19	38.94 ± 2.22	42.14 ± 2.40	37.24 ± 1.92	43.78 ± 2.32	37.26 ± 1.90	$\mathbf{49}.\mathbf{62}\pm \mathbf{2}.\mathbf{45}$
Avg.	53.17	48.61	51.14	48.19	52.37	48.19	$\mathbf{56}.\mathbf{54}$

Table 11. Normalized mutual information (NMI) on Mnist data set.

k	GNMF	HNMF	GSNMF	HGLNMF	GNTD	UGNTD	HGSNMF
2	58.89 ± 23.94	57.08 ± 24.38	42.94 ± 12.91	57.34 ± 24.27	67.11 ± 18.27	53.17 ± 36.19	$\mathbf{71}.\mathbf{18}\pm \mathbf{28}.\mathbf{49}$
4	53.12 ± 12.67	55.64 ± 14.51	53.70 ± 12.26	55.61 ± 14.04	57.17 ± 11.60	56.32 ± 11.42	$\mathbf{68}.\mathbf{09}\pm \mathbf{13}.\mathbf{31}$
6	45.17 ± 4.90	49.23 ± 5.85	48.53 ± 6.11	48.72 ± 5.99	49.20 ± 5.01	56.27 ± 7.69	$\mathbf{61}.\mathbf{57}\pm \mathbf{7}.\mathbf{43}$
7	47.63 ± 6.82	45.51 ± 4.82	47.48 ± 4.48	46.88 ± 5.46	47.05 ± 5.14	55.69 ± 7.31	$\mathbf{59}.\mathbf{52}\pm \mathbf{6}.\mathbf{56}$
8	48.55 ± 3.86	48.32 ± 4.38	49.76 ± 4.87	47.22 ± 3.83	47.38 ± 3.38	57.30 ± 6.01	$\mathbf{61}.\mathbf{25}\pm \mathbf{3}.\mathbf{97}$
10	47.01 ± 4.39	45.06 ± 2.21	46.30 ± 3.65	44.46 ± 2.81	45.07 ± 4.08	56.11 ± 4.73	$\mathbf{57}.\mathbf{92}\pm \mathbf{3}.\mathbf{59}$
Avg.	50.06	50.14	48.12	50.04	52.16	55.81	$\mathbf{63}.\mathbf{26}$

Table 12. Clustering accuracy (ACC) on Mnist data set.

k	GNMF	HNMF	GSNMF	HGLNMF	GNTD	UGNTD	HGSNMF
2	80.07 ± 28.05	87.96 ± 13.17	80.50 ± 17.55	88.13 ± 12.96	$\mathbf{92}.\mathbf{51}\pm \mathbf{5}.\mathbf{32}$	82.73 ± 17.28	91.57 ± 13.32
4	70.89 ± 11.28	74.17 ± 12.67	71.51 ± 16.93	65.99 ± 24.92	$\mathbf{75}.\mathbf{26}\pm \mathbf{10}.\mathbf{35}$	67.74 ± 11.50	71.32 ± 26.19
6	57.26 ± 5.48	60.41 ± 7.06	57.58 ± 7.37	59.36 ± 7.64	60.50 ± 6.74	44.47 ± 21.53	$\mathbf{65}.\mathbf{06}\pm \mathbf{9}.\mathbf{44}$
7	57.52 ± 7.66	54.99 ± 5.84	54.82 ± 3.97	55.54 ± 5.69	55.68 ± 5.24	56.34 ± 7.57	$\mathbf{62}.\mathbf{74}\pm \mathbf{7}.\mathbf{98}$
8	45.34 ± 2.08	55.63 ± 5.18	56.84 ± 5.27	49.11 ± 15.91	55.21 ± 4.34	$\mathbf{58}.\mathbf{01}\pm \mathbf{6}.\mathbf{34}$	56.78 ± 19.12
10	50.72 ± 5.42	48.21 ± 3.68	48.28 ± 6.05	48.02 ± 4.00	47.93 ± 4.53	51.01 ± 6.48	$\mathbf{56}.\mathbf{41}\pm \mathbf{5}.\mathbf{09}$
Avg.	60.30	63.56	61.59	60.86	60.05	68	$\mathbf{67}.\mathbf{31}$

From these experimental results in Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, Table 10, Table 11 and Table 12, we have the following conclusions.

(1) The clustering performance of the proposed HGSNMF method on all of the data was better than that of the other algorithms in most cases, which shows that the HGSNMF method can find more discriminative information for data. For COIL20, YALE, ORL, Georgia, and Mnist, the average clustering ACC of the HGSNMF method was more than 1.99%, 1.89%, 0.54%, 3.37%, and 7.21% higher than the second-best method, respectively, and the average clustering NMI of the HGSNMF method was more than 1.84%, 2.26%, 0.6%, 2.21%, and 7.25% higher than the second-best method, respectively.

(2) The HGSNMF method was better than the HNMF method in most cases, because the $L_p$ smoothing constraint could combine the merits of isotropic and anisotropic diffusion to yield smooth and more accurate solutions.

(3) The HGSNMF method was also better than the GSNMF method in most cases, which indicates that the hypergraph regularization term can discover the underlying geometric information better than the simple graph regularization term. The HGSNMF method did not perform as well as the GSNMF method for some given results. In the ORL dataset in Table 8, the HGSNMF method had a lower clustering accuracy metric than the GSNMF method when classes 30 and 35 were selected. This is because the hyperparameter $\alpha$ was selected from $\{500,1000\}$, which increased the error of the objective function and, therefore, yielded a lower accuracy.

(4) The HGSNMF method was also better than the GNTD and UGNTD method in most cases for the Mnist date set, which indicates that the hypergraph regularization term can discover the underlying geometric information better than the simple graph regularization term. The HGSNMF method did not perform as well as the GNTD and UGNTD on some given results, because the tensor maintained the internal structure of the high-dimensional data well.

For the different numbers of clusters on the YALE and Geogria data sets, we examined the computation time based on NMF, GNMF, HNMF, GSNMF, and HGSNMF. In these experiments, we chose the aforementioned identical conditions, including parameters and iteration times. From Table 13 and Table 14, it can be observed that the NMF had the shortest computation time, because it had no regularization term. The HNMF, GSNMF, and HGSNMF extended the GNMF by adding hypergraph regularization, the $L_p$ smoothing constraint, and the above two terms, respectively; the computation times of the HNMF, GSNMF, and HGSNMF were more than the GNMF. However, the computation time of the HGSNMF was less than the HNMF in most cases, thus showing the computational advantage of the HGSNMF using the $L_p$ smoothing constraint. In the YALE data set, the computation time of the HGSNMF was smaller than the GSNMF, thereby indicating that the hypergraph regularization term sped up the convergence of the proposed HGSNMF method.

Table 13. Comparisons of computation time in the YALE.

k	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
3	$\mathbf{0}.\mathbf{88}$	1.15	2.51	3.14	4.08	3.15
11	$\mathbf{5}.\mathbf{22}$	9.21	20.75	21.31	35.62	14.77
13	$\mathbf{10}.\mathbf{29}$	17.63	41.12	42.18	67.89	28.70
14	$\mathbf{12}.\mathbf{13}$	20.64	49.80	46.75	80.06	35.45
15	$\mathbf{13}.\mathbf{39}$	22.52	55.35	54.18	96.94	43.62

Table 14. Comparisons of computation time in the Georgia.

k	NMF	GNMF	HNMF	GSNMF	HGLNMF	HGSNMF
10	$\mathbf{9}.\mathbf{19}$	25.39	47.79	35.77	69.7	45.07
15	$\mathbf{16}.\mathbf{94}$	40.36	94.57	67.58	134.67	88.01
20	$\mathbf{24}.\mathbf{57}$	56.96	123.30	89.73	183.88	120.07
25	$\mathbf{40}.\mathbf{84}$	90.97	233.04	151.94	312.03	201.96
30	$\mathbf{55}.\mathbf{86}$	114.43	343.62	196.97	429.94	214.84

4.4. Parameters Selection

There are three parameters in our proposed HGSNMF algorithm: the regularization parameters $\alpha ,\mu ,$ and p. When $\alpha ,\mu$, and p are 0, the proposed HGSNMF method reduces to the NMF [13]; when $\mu$ and p are 0, the proposed HGSNMF method reduces to the HNMF [14]. In the experiments, we set $k=5$ for all graph-based and hypergraph-based methods for all data sets. To test the effect of each varying parameter, we fixed the other varying parameters as described Section 4.3.

Firstly, we adjusted the parameter $\alpha$ for the GNMF, GSNMF, HNMF, and HGSNMF methods. In the HGSNMF, for $k=4$, we set $\mu =100$ and $p=1.1$. In the HGSNMF, for $k=8$, we set $\mu =1000$ and $p=0.3$ for the COIL20 data set; for $k=11$ and $k=14$, we set $\mu =1000$ and $p=0.5$ for the YALE data set; for $k=20$, we set $\mu =1$ and $p=1.5$; for $k=40$, we set $\mu =100$ and $p=1.5$ for the ORL data set; for $k=15$, we set $\mu =500$ and $p=0.6$; for $k=20$, we set $\mu =1000$ and $p=0.5$ for the Geogria data set. Figure 2 and Figure 3 demonstrates the accuracy and the normalized mutual information variations with respect to $\alpha$ for four data sets.

Figure 2. Performance comparison of GNMF, HNMF, GSNMF, and HGSNMF for varying the parameter $\alpha$. Each column indicates the COIL20 and YALE data sets.

Figure 3. Performance comparison of GNMF, HNMF, GSNMF, and HGSNMF for varying the parameter $\alpha$. Each column indicates the for ORL and Georgia data sets.

As can be seen from Figure 2 and Figure 3, the HGSNMF performed better than the other algorithms in most cases. We can see that the performance of the HGSNMF was relatively stable with respect to the parameter $\alpha$ for some data sets.

Next, we adjusted the parameter $\mu$ for the GSNMF and HGSNMF for four data sets, and we set $\alpha =100$ and $p=1.7$ in the GSNMF. In the HGSNMF, for $k=6$, we set $\alpha =1000$ and $p=1.1$; for $k=12$, we set $\alpha =1000$ and $0.5$ for the COIL20 data set; for $k=3$, we set $\alpha =500$ and $p=0.1$; for $k=15$, we set $\alpha =500$, $\alpha =1000$, and $p=0.5$ for the YALE data set; for $k=15$, we set $\alpha =10$, $p=0.1$; for $k=25$, we set $\mu =1$ and $p=1.2$ for the ORL data set; for $k=5$ and $k=30$, we set $\mu =1000$ and $p=0.6$ for the Georgia data set. As can be seen from Figure 4 and Figure 5, the HGSNMF performed better than the GSNMF in most cases for the four data sets, and the performance of the HGSNMF was stable with respect to the parameter $\mu$ for some data sets.

Figure 4. Performance comparison of GSNMF and HGSNMF for varying parameter $\mu$. Each column indicates the COIL20 and YALEdata sets.

Figure 5. Performance comparison of GSNMF and HGSNMF for varying parameter $\mu$. Each column indicates the ORL and Georgia data sets.

Finally, we considered the variation of the parameter p. In the HGSNMF, for $k=8$, we set $\alpha =\mu =1000$; for $k=10$, we set $\alpha =1000$ and $\mu =500$ for the COIL20 data set; for $k=3$, we set $\alpha =0.1$ and $\mu =500$; for $k=5$, we set $\alpha =0.1$ and $\mu =1000$ for the YALE data set; for $k=5$, we set $\alpha =\mu =10$, for $k=15$, and we set $\alpha =10$ and $\mu =1$ for the ORL data set; for $k=35$ and $k=45$, we set $\alpha =500$ and $\mu =1000$ for the Georgia data set. As shown in Figure 6 and Figure 7, the performance of the HGSNMF was relatively stable and very good with respect to the parameter p varying from 0.1 to 2 for some data sets.

Figure 6. Performance comparison of GSNMF, and HGSNMF for varying parameter p for COIL20 and YALEdata sets.

Figure 7. Performance comparison of GSNMF, and HGSNMF for varying parameter p forORL and Georgia data sets.

For the Mnist data set, different classes were arbitrarily selected for clustering, and the clustering effects of the seven methods were compared. For the experiments, the graph regularization term parameter was set to 100 in the GNMF, GSNMF, GNTD, and UGNTD, and the hypergraph regularization parameter was set to 100 in the HNMF, HGLNMF, and HGSNMF; µ was also set to 100 in the GSNMF, HGLNMF, and HGSNMF, and the parameter p was set to 1.7 in the GSNMF and HGSNMF. In the GTND and UGNTD, the core tensor was the same as in the experiments in Section 4.3. From Figure 6 and Figure 7, it is clear that our proposed HGSNMF method clustered better than the other methods in most cases with the same selection of parameters on the Mnist data set.

Figure 8 compares performance of GNMF, HNMF, GSNMF, HGLNMF, GNTD, UGNTD, and HGSNMF for number of clusters k for Mnist data set.

Figure 8. Performance comparison of GNMF, HNMF, GSNMF, HGLNMF, GNTD, UGNTD, and HGSNMF for number of clusters k for Mnist data set.

4.5. The Converage Analysis

As described in Section 3, the convergence of the proposed HGSNMF method has been proven theoretically. In this section, we analyze the convergence of the proposed method through experiments. Figure 9 shows the convergence curves of our HGSNMF method for three data sets. As can be seen from Figure 9, the objective function was monotonically decreasing and tended to converge after 300 iterations.

Figure 9. The relative residuals versus the number of iterations for HGSNMF for four data sets.

5. Conclusions

In this paper, we proposed a hypergraph-regularized $L_p$ smooth constrain NMF method for data representation by incorporating the hypergraph regularization term and $L_p$ smoothing constraint into NMF. The hypergraph regularization term can better capture the intrinsic geometry structure of high-dimension space data more comprehensively than a simple graph; the $L_p$ smoothing constraint may produce a smooth and more accurate solution to the optimization problem. We presented the updating rulers and proved the convergence of our HGSNMF method. Experimental results for five real-world data sets show, as follows, for the COIL20,YALE, ORL, Georgia, and Mnist, the average clustering ACC of our proposed HGSNMF method was more than $1.99\%$, $1.89\%$, $0.54\%$, $3.37\%$, and $7.25\%$ higher than the seconded-best method, respectively, and the average clustering NMI of our proposed HGSNMF method was more than $1.84\%$, $2.26\%$, $0.6\%$, $2.21\%$, and $7.21\%$ higher than the seconded-best method, respectively. Thus, our proposed method can achieve a better clustering effect than other state-of-the-art methods in most cases.

Our HGSNMF method has some limitations: it only considers the hypergraph with high similarity among multiple data points and the smoothness of the basis matrix, and other constraints such as sparse, multiple graphs, and hypergraphs can be considered. The HGSNMF method was only applied to the clustering problem of images. In the future, we can apply hyperspectral mixing, recommended systems, and other aspects. In addition, because the vectorization of the matrix will destroy the internal structure of the data, we will extend to consider nonnegative tensor decomposition in the future.