Robust Exponential Graph Regularization Non-Negative Matrix Factorization Technology for Feature Extraction

Abstract

Graph regularized non-negative matrix factorization (GNMF) is widely used in feature extraction. In the process of dimensionality reduction, GNMF can retain the internal manifold structure of data by adding a regularizer to non-negative matrix factorization (NMF). Because Ga NMF regularizer is implemented by local preserving projections (LPP), there are small sample size problems (SSS). In view of the above problems, a new algorithm named robust exponential graph regularized non-negative matrix factorization (REGNMF) is proposed in this paper. By adding a matrix exponent to the regularizer of GNMF, the possible existing singular matrix will change into a non-singular matrix. This model successfully solves the problems in the above algorithm. For the optimization problem of the REGNMF algorithm, we use a multiplicative non-negative updating rule to iteratively solve the REGNMF method. Finally, this method is applied to AR, COIL database, Yale noise set, and AR occlusion dataset for performance test, and the experimental results are compared with some existing methods. The results indicate that the proposed method is more significant.

1. Introduction

In recent years, technologies such as computer vision and pattern recognition have been widely used in all walks of life. In many application scenarios, the dimension of the data matrix that needs to be calculated is very high. For example, a photo of 100 × 100 pixels has 10,000 dimensions in the computer, which enormously increases the difficulty of data processing. For dealing with the problem of the “dimensionality curse” [1], many dimensionality reduction algorithms have been proposed, for instance, principal component analysis (PCA) [2] and linear discriminant analysis (LDA) [3], in the literature [4]. The linear dimensionality reduction algorithm has the advantages of being simple to operate and easy to analyze, but it is no longer effective when the data have non-linear structure. However, in reality, high-dimensional data are usually non-linear structures. When using linear dimensionality reduction methods to extract features from high-dimensional data, we often cannot get a good effect.

For dealing with high-dimensional non-linear data, manifold learning algorithms show an ideal result. Scholars have paid attention to the fact that in practical applications, the neighborhood relationship of data plays an important role, considering that different datasets have different manifold structures. The structure of the data is embedded in the projected function, which can achieve more efficient dimensionality reduction. Examples such as Laplacian Eigenmaps (LE) [5,6], local linear embedding (LLE) [7,8], local preserving projections (LPP) [9,10], neighborhood preserving embedding (NPE) [11,12] and supervised low-rank embedded regression (SLRER) [13] are all classical manifold learning algorithms. LPP retains the local neighborhood information between sample points on the database by constructing the neighborhood map of the sample points. After projection, the nearest neighbor relationship of sample data in high-dimensional space is still maintained in the low-dimensional subspace.

In addition to manifold learning algorithms, matrix factorization can also achieve the purpose of dimensionality reduction. The matrix decomposition method hopes to seek out two or more low-dimensional matrices so that their product can approximate the original matrix. Typical matrix factorization technologies consist of singular value decomposition (SVD) [14], NMF [15], etc. The NMF method is based on the principle of physiology that the overall perception of human beings is composed of local perceptions (purely additive) and is introduced into data processing to form global data by accumulating various parts of the data [16,17]. The idea of the NMF algorithm is to represent the whole by the part. NMF is a partial-based linear representation of non-negative data. Due to the strong constraint of NMF on the non-negativity of the factor matrices, the NMF algorithm has better interpretability and is more in line with human cognition. For any matrix A, NMF can discover two non-negative factor matrices U and V, so that A is equal to the product of U and V. Therefore, NMF can decompose a matrix into the form of the product of left and right non-negative matrices. Different from SVD, the matrices obtained by SVD can be positive or negative, whereas the matrices obtained by NMF are both non-negative, and NMF only allows addition but not subtraction.

Nevertheless, traditional NMF methods have some problems, such as poor robustness to noise and outliers, lack of discriminative information, etc. In recent years, the improvement of the NMF algorithm has become a research hotspot. Lu et al. proposed low-rank non-negative factorization (LRNF) [18], which learns low-rank representations of data and performs non-negative matrix factorization at the same time, eliminating the effect of noisy data on dimensionality reduction. On the basis of LRNF, Lu et al. introduced structural inconsistency and then proposed a new model, i.e., structurally incoherent low-rank NMF (SILR-NMF) [19]. SILR-NMF is able to learn the global structure of samples and introduced sample label information so as to ensure that the clean data points of different categories shall be independent as far as possible. The semi-supervised NMF (SSNMF) method proposed in reference [20] merges the label information and sample matrix into the traditional NMF framework. Robust semi-supervised NMF (RSSNMF) [21] adopts the L2,1 norm as the loss function and incorporates discriminant information into extra conditions. The algorithm is better to accommodate noise and outliers. A semi-supervised non-negative matrix factorization method [22] is proposed for dissimilarity and similarity regularization, which can better utilize label information by introducing a pair of complementary regularizers for matrix factorization.

However, the above NMF algorithms do not take the geometric structure into account in the initial data space when performing matrix decomposition. However, the geometric structure inside the data plays an important part in practical applications. For the sake of improving the recognition rate of NMF, a form of graph regularization is introduced. GNMF [23] combines NMF with the objective function of LPP and adds a regularization parameter on the basis of the original NMF. According to GNMF, robust manifold NMF (RMNMF) [24] is proposed, which adopts the L1 norm to increase the constraints on the noise matrix. Yi et al. [25] propose NMF with locality constrained adaptive graph (NMF-LCAG), in which two local constraints are used to adaptively optimize the graph composition. NMF-LCAG can learn the weight matrix of the graph and the low-dimensional properties of the samples at the same time. The combination of NMF and graph embedding can greatly improve the dimensionality reduction effect and bring both into full play, maintaining the geometric structure of data while exploiting a parts-based linear representation of non-negative information.

Most of the current non-negative matrix factorization algorithms for graph embedding do not consider the problem of matrix singularity, therefore, graph embedding often causes SSS problems [26]. In the past few decades, the analysis method according to matrix index to solve the SSS problems has received extensive concern, for example, exponential locality preserving projection (ELPP) [27], exponential elasticity preserving projection (EEPP) [28], exponential local discriminant embedding (ELDE) [29], exponential discriminant analysis (EDA) [30], exponential semi-spervised discriminant embedding (ESDE) [31], Krylov-EDA [32], exponential marginal Fisher analysis (EMFA) [33], etc. The exponential operation of matrices is also related to much scientific computing work and is widely used in nuclear magnetic resonance spectroscopy, control theory, Markov chain analysis, and other fields. For example, in boundary Fisher analysis, Guo et al. [34] transformed the inverse matrix into a matrix exponent inverse matrix for training support vector machines (SVM) and successfully solved the matrix singularity problem in the process of data optimization. Ivanovs et al. [35] analyzed the number of zero points of the determinant composed of the matrix index of the Markov addition process (MAP) with the unilateral jump, and the theoretical model obtained was widely used in the financial field. MEDLPP [36] introduced the matrix index into DLPP to solve its SSS problems and made corresponding improvements to shorten the running time. Reference [37] proposed the exponential form framework of the dimensionality reduction algorithm, and it pointed out that the index of the matrix is non-singular, so the method based on the matrix index can effectively solve the SSS problems. Many results show that the matrix index discriminant analysis method often has a stronger discriminative ability than the original method.

To sum up, in GNMF, LPP will have the SSS problem, which greatly affects the dimensionality reduction effect. To handle the aforementioned problem, we propose a novel method, i.e., a robust exponential graph regularization NMF algorithm (REGNMF). In REGNMF, the matrix exponent is added to the objection function so that the original Laplacian matrix is not singular. The optimal solution of the model is obtained by the non-greedy iterative method of Lagrangian, which successfully solves the SSS problems. Experiments with this algorithm on multiple databases show that its image feature extraction effect is better than some other algorithms.

The main contributions of this article are as follows:

(1)

A new robust graph embedding algorithm, i.e., REGNMF, for unsupervised subspace learning is proposed in this article. On the basis of Euclidean distance and exponential Laplacian matrix as robustness criteria, we introduce an iterative algorithm to address optimization problems.

(2)

Different from the traditional GNMF algorithm, REGNMF can not only successfully solve the SSS problems by introducing the matrix exponent but gain a part-based representation of data. Test sample points are mapped into a novel subspace by learning the base matrix without being affected by singular matrices.

(3)

REGNMF improves the robust capacity of existing NMF algorithms based on graph embedding. In the contrast to existing methods, REGNMF can recode part-based geometric information of samples to classify. Extensive experiments show that our proposed methods runs well and outperforms existing algorithms on most occasions, especially on noisy and corrupted databases.

The remainder of the paper is organized as follows: For the convenience of understanding, Section 2 describes some related work. Section 3 clarifies some details of REGNMF, including the objective function, optimization process, etc. Section 4 gives the operation results of relevant algorithms on different databases. Section 5 summarizes the article.

2. Related Work

Suppose $X=[x_1,x_2,..,x_n]\in R^{m\times n}$ is a sample dataset of the original space, $Y=[y_1,y_2,\cdots ,y_n]\in R^{d\times n}$ is the projection dataset, and $d<n$ is the subspace dimension. Then we can obtain the formula as follows:

$$\begin{array}{c}\hfill y_i=V^T{x}_i,i\in (1,2,\cdots ,n)\end{array}$$

(1)

2.1. Matrix Exponent [27]

The exponential operation of matrices is also related to much scientific computing work and is widely used in nuclear magnetic resonance spectroscopy, control theory, Markov chain analysis, and other fields. In this section, we give the definition of the matrix exponent and its properties. Suppose D is a square matrix of $n\times n$, the matrix exponent formula of D is defined below:

$$\begin{array}{c}\hfill exp\left(D\right)=\sum _{k=0}^{\infty }\frac{D^k}{k!}=I+D+\frac{D^2}{2!}+\cdots +\frac{D^k}{k!}+\cdots ,\end{array}$$

(2)

where I indicates an identity matrix of n × n. According to the Taylor series expansion theorem, we can convert the matrix index into the form of infinite matrix series. By proving the convergence and divergence of infinite matrix series, we get that ${\sum }_{k=0}^{\infty }\frac{D^k}{k!}$ is convergent. In other words, $exp\left(D\right)$ is a definite matrix. Matrix exponents have the following properties:

(1)

$exp\left(D\right)$ is a full rank matrix;

(2)

If the matrices M and N are commutative, that is $MN=NM$, then $exp(M+N)=exp\left(M\right)·exp\left(N\right)$;

(3)

For any matrix M, ${\left(exp\left(M\right)\right)}^{-1}$ exists, and ${\left(exp\left(M\right)\right)}^{-1}=exp(-M)$;

(4)

Suppose that T is a non-singular matrix, then $exp\left(T^{-1}DT\right)=T^{-1}exp\left(D\right)T$;

(5)

Assuming that $a_1,a_2,\cdots ,a_n$ are the eigenvectors of D, and the corresponding to eigenvalues are ${\beta }_1,{\beta }_2,\cdots ,{\beta }_n$, then $a_1,a_2,\cdots ,a_n$ still are the eigenvectors corresponding to the eigenvalues $e^{{\beta }_1},e^{{\beta }_2},\cdots ,e^{{\beta }_n}$ of matrix $exp\left(D\right)$.

2.2. Graph Regularization Non-Negative Matrix Factorization (GNMF) [23]

By combining NMF with a geometry-based regularizer, GNMF learns the geometric structure between sample points while projecting.

The goal of NMF is to calculate two non-negative matrices U and V so that the product of two factor matrices is as approximate as possible to X, i.e., $X\approx U{V}^T$. The loss after matrix factorization is represented by Euclidean distance, so the calculation formula of NMF is

$$\begin{array}{c}\hfill \underset{U\ge 0,V\ge 0}{min}{∥X-U{V}^T∥}^2.\end{array}$$

(3)

The regularizer part of GNMF is implemented by LPP. LPP is a linear approximation of non-linear Laplacian characteristic projection. The ultimate aim of LPP is to obtain a projection matrix V that can map high-dimensional sample points into a low-dimensional subspace. LPP requires that the points with the nearest neighborhood relationship in the original high-dimensional space are still maintained after mapping into low-dimensional space. The calculated function of the LPP is

$$\begin{array}{c}\hfill min\frac{1}{2}\sum _{i,j}{(v_i-v_j)}^2{W}_{ij}.\end{array}$$

(4)

The W is a graph containing information about the neighborhood of the dataset

$$\begin{array}{c}\hfill \begin{array}{c}\hfill W_{ij}=\left\{\begin{array}{cc}exp(-{∥x_i-x_j∥}^2/t),\hfill & x_j\in N_k\left(x_i\right)\hfill \\ 0,\hfill & otherwise,\hfill \end{array}\right.\end{array}\end{array}$$

(5)

where $N_k\left(x_i\right)$ represents the set of K nearest neighbors of $x_i$, and t is a real number. After calculation, it can be simplified to

$$\begin{array}{c}\hfill \underset{V}{min}Tr\left(V^{T}LV\right),\end{array}$$

(6)

where $L=D-W$, D is a diagonal matrix, and the elements on the diagonal are the sum of each column in the weight matrix W, i.e., $D_{ii}={\sum }_j{W}_{ij}$. $Tr(·)$ represents the trace of matrix.

Compared with traditional NMF, GNMF still maintains the underlying geometry of the original data space when performing dimensionality reduction. GNMF integrates NMF and LPP into a framework, and the calculation formula is as follows:

$$\begin{array}{c}\hfill \underset{U\ge 0,V\ge 0}{min}{∥X-U{V}^T∥}^2+\lambda Tr\left(V^{T}LV\right),\end{array}$$

(7)

where $\lambda >0$ is the regularization parameter.

2.3. Low Rank Non-Negative Factorization (LRNF) [18]

Because NMF is based on Euclidean distance, it has poor robustness to data containing noise and corruption. For the purpose of decreasing the sensitivity of NMF to noise and occlusion, LRNF utilizes low rank to learn the global structure of the sample points. In the problem of image classification, compared with the local information contained in noise data, the global geometry structure of the sample is provided with more robustness.

LRNF divides original data X into clean data A and noise data E. The low-rank constraint on matrix A can learn the global structure of the original matrix X. Therefore, A can replace X and act as a matrix that needs to be decomposed. We use the L1 norm to realize the sparsity of the noise matrix and propose the objective function of LRNF as below:

$$\begin{array}{c}\hfill \underset{A,E,U\ge 0,V\ge 0}{min}{∥A-U{V}^T∥}^2+\alpha {∥Z∥}_{*}+\beta {∥E∥}_{1}s.t.X=A+E,\end{array}$$

(8)

where Z is a low-rank matrix with a block diagonal structure, $\alpha$ and $\beta$ are two regularization parameters, and both of them are greater than zero.

3. Robust Exponential Graph Regularization Non-Negative Matrix Factorization

In this section, we first describe the reason for our proposed method, then detail the principle of the REGNMF algorithm and obtain an optimized solution by the Lagrange multiplier iterative method.

3.1. The Motivation of REGNMF

NMF as a non-linear and non-negative constraint matrix factorization dimensionality reduction method has the advantages of interpretability, simple calculation, physical meaning, large-scale data processing, and small storage space, so it is extensively applied to blind source separation, pattern recognition, time series segmentation, document clustering [38], recommender systems [39], gene detection, and other fields [40]. Hence, the study of NMF has a profound impact.

NMF measures the approximation of the original matrix and decomposed matrix based on Euclidean distance, which indirectly results in a strong level of sensitivity of NMF to noise and outliers. How to increase the robustness of NMF is the primary problem we have to face. There are many ways to increase robustness, such as low rank representation (LRR) [41], which learns the global structure of the samples through low rank and dictionary; LRNF utilizes low rank to increase the robustness of the original NMF; latent low rank representation (LatLRR) [42] divides original data into observed and unobserved data; literature [43] separates clean data from noisy data; sparse preserving projection (SPP) [44] uses sparsity to filter out redundant information; and MEDLPP [36] introduces matrix exponents to solve the SSS problems. In the NMF of graph embedding, although the internal structure of the data is considered, according to the above analysis, it can be seen that graph embedding will cause SSS problems, so the combination of graph embedding and NMF requires a method to increase the robustness of the algorithm.

The small sample size is also a problem that researchers need to consider. When the samples are insufficient, how to improve the dimensionality reduction effect has attracted extensive attention. There are many solutions to the SSS problems, such as feature transfer, metric learning, and others. The introduction of a matrix exponent is also an effective solution. Compared with other methods, matrix exponent is more convenient and concise.

For example, as a classical supervised matrix dimensionality reduction method, LDE’s [45] key idea is to maximize the similarity between similar sample points and the divergence between different sample points. For overcoming the boundedness of the LDE algorithm, the LDE algorithm with matrix exponent, namely ELDE [29], emerged as the times required. In the extension algorithm of LDE, researchers added label information and proposed a semi-supervised version of LDE, named SDE [46]. Although SDE has a high discriminative ability, without regularization, SDE may encounter the SSS problems when the number of samples is insufficient. For the purpose of addressing the above problems, Dornaika et al. proposed the exponential SDE (ESDE) [31] method by exploiting the property of the matrix exponent. Except for solving the SSS problems, ESDE expands the gap between different categories of samples to increase the discrimination ability.

According to the above analysis, the matrix index can solve the SSS problems. For the sake of dealing with SSS problems of NMF in graph embedding, we use a form of matrix index to transform the possible singular matrix into a non-singular one and make the best of the information of the null space to increase the robustness and discriminative power of the algorithm.

3.2. Problem Formulation for REGNMF

In order to deal with the problems in real life and production, we need to obtain certain data for analysis. However, in many scenarios, due to the difficulties in data collection and the high cost of sample labeling, the data we obtain are not always sufficient or effective, but often a small amount of data with little effective information, which makes the scale of training samples small and will lead to overfitting of the model. Therefore, the SSS problems have proved to be one of the most important exploration interests in the field of artificial intelligence.

We know that graph embedding will cause the SSS problems. Although the traditional GNMF method based on Euclidean distance considers the manifold structure of the sample, the problem of small sample size will also occur in the procedure of matrix dimensionality reduction. In the above analysis, we know that the matrix exponent is an effective means to address SSS problems. Therefore, we adopt the form of introducing the matrix exponent to solve the SSS problems in GNMF. Based on this, we make improvements on the basis of the original GNMF, introduce the matrix exponent, and propose the REGNMF method. The objective function of our proposed method is displayed as follows:

$$\begin{array}{c}\hfill \mathcal{L}=\underset{U\ge 0,V\ge 0}{min}{∥X-U{V}^T∥}^2+\lambda Tr\left(V^{T}exp\left(L\right)V\right),\lambda >0.\end{array}$$

(9)

In the above formulation, matrix X is composed of training samples arranged in columns. An additional condition $U,V\ge 0$ restricts two matrix factors U and V to both non-negative matrices: U is a base matrix, and V is a coefficient matrix. The product of U and V is used to approximate X. The parameter $\lambda$ is a regularization parameter to avoid over-fitting in the proceeding of data dimensionality reduction. The expression $V^{T}exp\left(L\right)V$ adds a matrix exponent to the objective function of the LPP algorithm; and L is a Laplacian matrix of sample data, i.e., $L=D-W$.

The Laplacian matrix in traditional LPP may be singular. When a matrix exponent is added, according to the properties of the matrix exponent, it can be seen that $exp\left(L\right)$ is a full-rank matrix. Therefore, $exp\left(L\right)$ must be a non-singular matrix, which greatly solves the SSS problems in LPP. The first item in model (9) represents the reconstruction error of the decomposed matrix and the original matrix. In practical applications, the smaller the reconstruction error is, the more decomposed matrices retain the information of the original data. The second term indicates the manifold structure of samples introduced by a regularizer, and the matrix exponent is mixed to increase the robust ability of our method.

Therefore, in the progress of data dimensionality reduction, REGNMF not only maintains the internal geometric structure between sample points, but reduces the sensitivity to noise and outlier data. The next subsection shows how to solve model (9) with an iterative algorithm.

3.3. The Optimal Solution

As both U and V in the objective function (9) of REGNMF are not convex, we cannot expect to find an algorithm to get a global minimum solution; instead we use a gradient descent method [47] to obtain local minimum values, so we can obtain the following rules:

$$\begin{array}{c}\hfill u_{ik}\leftarrow u_{ik}+{\varphi }_{ik}\frac{\partial \mathcal{L}}{\partial u_{ik}},v_{jk}\leftarrow v_{jk}+{\psi }_{jk}\frac{\partial \mathcal{L}}{\partial v_{jk}},\end{array}$$

(10)

where $\varphi$ and $\psi$ represent the learning rate, which determines the step size of each gradient descent, and when the choice of $\varphi$ and $\psi$ are both small enough, it can be guaranteed that each iteration is decreasing until the objective function converges to a stationary point. Thus, we set ${\varphi }_{ik}=-\frac{u_{ik}}{2{\left(U{V}^{T}V\right)}_{ik}}$, $v_{jk}=-\frac{v_{jk}}{2{(V{U}^{T}U+\lambda exp\left(L\right)V)}_{jk}}$. Then the update rules for $\varphi$ and $\psi$ are

$$\begin{array}{cc}\hfill u_{ik}+{\varphi }_{ik}\frac{\partial \mathcal{L}}{\partial u_{ik}}& =u_{ik}-\frac{u_{ik}}{{\left(2U{V}^{T}V\right)}_{ik}}\frac{\partial \mathcal{L}}{\partial u_{ik}}\hfill \\ \hfill & =u_{ik}-\frac{u_{ik}}{{\left(2U{V}^{T}V\right)}_{ik}}(-2{\left(XV\right)}_{ik}+2{\left(U{V}^{T}V\right)}_{ik})=u_{ik}\frac{{\left(XV\right)}_{ik}}{{\left(U{V}^{T}V\right)}_{ik}}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & v_{jk}+{\psi }_{jk}\frac{\partial \mathcal{L}}{\partial v_{jk}}=v_{jk}-\frac{v_{jk}}{2{(V{U}^{T}U+\lambda exp\left(L\right)V)}_{jk}}(-2{\left(X^{T}U\right)}_{jk}\hfill \\ \hfill & +2{\left(V{U}^{T}U\right)}_{jk}+2{\left(\lambda exp\left(L\right)V\right)}_{jk})=v_{jk}\frac{{\left(X^{T}U\right)}_{jk}}{{(V{U}^{T}U+\lambda exp\left(L\right)V)}_{jk}}\hfill \end{array}$$

(12)

According to the convergence proof of NMF in the paper of Lee and Seung [17], the objective function in formulation (7) is not increasing under the update rule in formulations (11) and (12). Our proof of convergence basically follows the thought of the original NMF in Lee and Seung’s paper, which will not be repeated here. However, many studies have shown that the multiplication algorithm of the original NMF cannot guarantee convergence to a stable point. In order to make our algorithm converge, we further demand that each column vector in the matrix U (or V) has a Euclidean distance of length 1. The matrix V (or U) updates automatically to prevent U from changing. The implementation is as follows:

$$\begin{array}{c}\hfill u_{ik}\leftarrow \frac{u_{ik}}{\sqrt{{\sum }_i{u}_{ik}^2}},v_{jk}\leftarrow v_{jk}\sqrt{{\sum }_i{u}_{ik}^2}.\end{array}$$

(13)

This strategy is also adopted in GNMF, and it is still the same after adding the matrix exponent. After the multiplicative update process has converged, this paper fits the length of Euclidean distance of every column vector in matrix U to 1 and tunes up matrix V to ensure that U is invariant. We prove the convergence of our proposed REGNMF through experiments, as displayed in Figure 1. It can be seen that on different databases, when the number of algorithmic iterations to 200 times, the reconstruction error of the objective function is almost unchanged. Therefore, 200 is used as the number of iterations of the experiment in this paper.

According to our derivation process, the following solution algorithm (Algorithm 1) is obtained.

Algorithm 1 REGNMF Algorithm.

Input:           Training set X, subspace dimensions d, the number of iterations iter, the current iteration number s, the regularization term coefficient $\lambda$, the matrices U and V, the weight matrix W, and the Laplacian matrix L. Initialization:           The number of sample rows m and sample columns n, $U=0$, $V=0$, iter = 200, $W=L=0$, $s=0$;           1. Use random functions to generate U and V factor matrices, $U=rand(m,d)$, $V=rand(n,d)$;           2. Use the K-nearest neighbor algorithm to select the neighbor points of the $x_i$ to construct a neighborhood graph W;           3. According to $L=D-W$, construct the Laplacian matrix L;           4. When s <= iter, loop:                    ➀ Iterate and Update U:                          $u_{ik}\leftarrow u_{ik}\frac{{\left(XV\right)}_{ik}}{{\left(U{V}^{T}V\right)}_{ik}}$;                    ➁ Iterate and Update V:                          $v_{jk}\leftarrow v_{jk}\frac{X^{T}U}{{(V{U}^{T}U+\lambda exp\left(L\right)V)}_{jk}}$;                    ➂ $s=s+1$;           5. If s > iter: end the loop;           6. Normalize matrices U and V:                         $u_{ik}\leftarrow \frac{u_{ik}}{\sqrt{{\sum }_i{u}_{ik}^2}},v_{jk}\leftarrow v_{jk}\sqrt{{\sum }_i{u}_{ik}^2}$. Output: base matrix U and coefficient matrix V

4. Experiment

For the purpose of testing the ability of REGNMF’s feature extraction and its robustness to noise and outliers, REGNMF, GNMF, NMF, LPP, ELPP, LRR, and LRNF are used to extract main features on the AR, COIL, and Yale databases with random noise and AR database with occlusion for recognition experiments. We use MATLAB to carry out our experiments, the device processor used in the experiment is Intel Core i5-7200U, 64-bit Windows operating system, and the running memory is 4G. During the experiments, we use the kNN method for classification. In this paper, we fix the parameter k (number of nearest neighbor nodes of each data instance) in the kNN method to 10, so that the graph structure constructed by the model is consistent each time, thus avoiding the influence of irrelevant parameters on the model performance.

4.1. The AR Database Experiment

The AR face database includes more than 4000 images of 70 men and 56 women, totaling 126 people. The characteristics of these images are that the faces in the front view have different facial expressions, lighting conditions, and occlusion (sunglasses, etc.). Each image is 50 × 40 pixels. Figure 2 shows a partial image from the AR database. From the AR database, we randomly select 24 images in each category as the training set. The recognition rate curve of diverse face recognition algorithms under different feature dimensions is displayed in Figure 3.

4.2. The COIL Database Experiment

The COIL database consists of 20 objects, each with 72 images, which are collected from distinct angles of the object. Each image is 128 × 128 pixels. A partial image from the COIL database is shown in Figure 4. From the COIL database, we randomly select 60 images in each class as the training set and the remaining 12 images as the test set. Figure 5 shows the comparison of the recognition rates of each algorithm under different feature dimensions. We know that the number of samples in the COIL database is less than the sample dimension, and we can see from Figure 5 that our model is effective and performs far better than other models, which indicates that REGNMF successfully solves the SSS problem.

4.3. Robustness Test for Random Pixel Destruction

In order to further evaluate the robust capability of the REGNMF algorithm to noise and outliers, this paper adds Gaussian noise with a density of 0.2 and salt and pepper noise with a density of 0.1, respectively, to the Yale database. Figure 6 shows, respectively, a Yale clean image and after adding different noise Yale images. The Yale dataset includes 165 images in 15 categories, and each category contains 11 images from different lighting, expressions, and shooting angles, and the resolution of each image is 80 × 100. In this paper, different train sample numbers (2, 3, 4, 5, 6) are casually chosen as train sets to test the robustness of each algorithm when reduced to the same dimension (six dimensions).

Table 1. Face recognition rate of each algorithm under different noise densities.

Methods	“Salt and Pepper” Noise (Density = 0.1)					Gaussian Noise (Density = 0.2)
	2	3	4	5	6	2	3	4	5	6
LRR	69.63	73.33	80.00	82.22	78.67	60.00	65.83	71.43	78.89	74.67
LPP	40.74	40.74	35.24	43.33	50.67	37.78	51.67	41.90	55.56	56.00
ELPP	49.30	51.67	54.76	61.11	69.33	47.04	50.00	52.86	61.11	63.33
NMF	65.93	70.00	79.05	82.22	73.33	64.44	67.50	73.33	81.11	80.00
GNMF	67.41	71.67	82.86	79.05	77.33	58.52	77.50	70.48	82.22	73.33
RMNMF	71.85	72.5	74.67	76.67	77.14	63.70	68.33	77.14	78.89	78.67
LRNF	72.59	76.67	76.19	81.11	74.67	70.37	68.33	79.05	74.44	73.33
REGNMF	74.07	76.67	83.81	85.56	81.33	75.56	75	80.95	83.33	82.67

The bolded part indicates the recognition rate of our proposed method.

gives the face recognition rate of each algorithm. It can be proved that our model can perfectly solve the SSS problem.

4.4. Robustness Test of Continuous Pixel Occlusion

For the sake of testing the sensitivity of the REGNMF method to occlusion data, we add the 5 × 5 and 10 × 10 occlusion blocks to the AR database for the face recognition rate experiment. Figure 7a,b are the images after adding different occlusion blocks to the AR database. This paper selects 24 samples as the training set and then assesses the robustness of each algorithm under different feature dimensions. Figure 8 and Figure 9, respectively, show the face recognition rates of each algorithm under different occlusions.

4.5. Analysis of Results

(1)

For the noiseless data, Figure 2 and Figure 4 show the variation of each algorithm with the feature dimension for the AR and COIL databases. It can be seen that in most cases, the effect of REGNMF is significantly better than other algorithms, and the experiment on the COIL database indicates that the SSS problems are successfully solved.

(2)

In the experiment of adding noise, as shown in Table 1, the effect of the REGNMF method is far more effective than other algorithms under different noise densities, and the classification accuracy of image recognition is about 1–4% higher than other algorithms.

(3)

In the experiment with occlusion, it is clear from Figure 8 and Figure 9 that the REGNMF algorithm has better robustness and a higher face recognition rate, so it is more discriminative. In most cases, the classification accuracy of REGNMF and GNMF is much higher than that of other algorithms. This is because REGNMF, RMNMF, and GNMF join a graph regularizer, which takes the structure of the data into account while reducing the dimension. Therefore, the accuracy is higher.

5. Conclusions

Manifold learning and matrix factorization methods are effective in feature extraction, and the GNMF is a generalization based on manifold learning. On this basis, this paper proposes an improved model for the GNMF, i.e., REGNMF. REGNMF integrates the manifold learning model with the NMF model together and still maintains the manifold geometry of data space in the progress of dimension reduction. At the same time, considering the impact of SSS problems caused by matrix singularity on face recognition rate, the author successfully solves the SSS problems by introducing a matrix index to make the Laplacian matrix into a non-singular matrix, which greatly increases the robustness and discrimination of the method. In this paper, the iterative update optimization algorithm is used to derive and certificate the formulation of the proposed REGNMF method, and the optimal solution is obtained. The algorithm is applied to image databases such as AR, Yale Salt and Pepper, Yale Gaussian, and COIL. A large number of experimental results indicate that, in comparison with traditional algorithms, image features extracted by REGNMF have higher recognition rates when used for classification and recognition, especially on datasets with noise and occlusion.