Semi-Supervised Ridge Regression with Adaptive Graph-Based Label Propagation

Abstract

In order to overcome the drawbacks of the ridge regression and label propagation algorithms, we propose a new semi-supervised classification method named semi-supervised ridge regression with adaptive graph-based label propagation (SSRR-AGLP). Firstly, we present a new adaptive graph-learning scheme and integrate it into the procedure of label propagation, in which the locality and sparsity of samples are considered simultaneously. Then, we introduce the ridge regression algorithm into label propagation to solve the “out of sample” problem. As a consequence, the proposed SSSRR-AGLP integrates adaptive graph learning, label propagation and ridge regression into a unified framework. Finally, an effective iterative updating algorithm is designed for solving the algorithm, and the convergence analysis is also provided. Extensive experiments are conducted on five databases. Through comparing the results with some well-known algorithms, the effectiveness and superiority of the proposed algorithm are demonstrated.

1. Introduction

Least square regression (LSR) is a mathematical optimization algorithm that seeks the best matching function of data by minimizing the square of error [1,2,3,4]. Since the advent of least square regression, a large number of LSR-based methods have been proposed, such as weighted LSR [1], partial LSR [2], local LSR [3], kernel LSR [4], support vector machine (SVM) [5], non-negative least squares (NNLS) [6,7] and so on. Moreover, a series of methods based on LSR have been successfully and efficiently applied to face recognition, speech recognition, image retrieval, and so on [8,9,10,11,12,13,14]. For example, in order to improve the performance of retargeted least squares regression (ReLSR), Wang et al. [8] proposed the groupwise retargeted least squares regression (GReLSR) algorithm, which utilized an additional regularization to restrict the translation values of ReLSR so that similar values are within the same class. For the sake of solving the device diversity problem in crowdsourcing system, Zhang et al. [9] introduced a linear regression (LR) approach to obtain the uniform received signal strength (RSS) values. In [10], an elastic-net regularized linear regression (ENLR) framework was developed, in which two particular strategies were proposed to enlarge the margins of different classes. To obtain the orthogonal basis functions for extreme learning machine (ELM) and improve the locality preserving power of feature space, Peng et al. [11] first imposed the orthogonal constraint on the output weight matrix of ELM and then formulated an orthogonal extreme learning machine (OELM) model. To force the samples in the same class to have similar soft target labels, Yuan et al. [12] designed a constrained least square regression (CLSR) model for multi-category classification. In [13], the authors proposed a new semi-supervised learning model named the semi-supervised graph learning retargeted least squares regression model (SSGLReLSR), which integrated the linear square regression and graph construction into a unified framework. Based on the asymptotic bias and variance, a new method for multivariate predictor variables was proposed [14]. To further improve the robustness and effectiveness of LSR, many discriminative LSR methods were developed recently. For example, Xiang et al. [15] utilized the ε-dragging technique to design a general framework of discriminative least square regression (DLSR), and Zhang et al. [16] introduced the retargeted LSR by learning transformed regression. Moreover, in [17], a unified least square framework was constructed to formulate many component analysis methods and their corresponding regularized and kernel extensions versions. In addition, some traditional dimension reduction techniques are also regarded as LSR theory framework [18], such as principal component analysis (PCA) [19], linear discriminant analysis (LDA) [20], locality preserving projection (LPP) [21] and so on.

LSR is an unbiased estimation method, which is very sensitive to noise. In addition, the LSR is also unstable when the sample size is less than the dimension [22,23]. For those reasons, the ridge regression (RR) was proposed by adding a regularization item into the LSR model to improve the performance and reduce the computational complexity [22,23]. The RR is an improved least square estimation, which can be regarded as a biased estimation regression for collinear data analysis. Experimental results showed that the algorithm was effective and robust in “pathological data” [24]. Especially, its performance is more reliable than the LSR in practical applications. Over the decades, many improved methods have been studied [25,26,27,28,29,30,31,32,33,34,35]. The authors of [29] studied a dual version of the ridge regression procedure, which can perform non-linear regression by constructing linear regression function in high dimensional feature space. Xue et al. [30] presented a local ridge regression (LRR) algorithm to effectively solve the illumination variation and partial occlusion problems of facial images. To deal with the singular problem in the extreme learning machine learning algorithms, [31] proposed an extreme learning machine ridge regression (ELMRR) learning algorithm with ridge parameter optimization. For reducing the computation time and retaining statistical optimality, Zhang et al. [32] suggested a decomposition-based scalable approach to perform kernel ridge regression (KRR) [33], which randomly divided a dataset of size N into m subsets with equal size. The KRR calculated an independent estimator for each subset, and then averaged the local solutions into a global predictor. In order to improve the training efficiency with large-scale data, [34] proposed an accelerator for kernel ridge regression algorithms based on data partition (PP-KRR). To address the limitation of LSR method which ignored the correlation among samples, Wen et al. [18] presented the inter-class sparsity based discriminative least square regression (ICS_DLSR) algorithm for multi-class classification. In order to improve the performance of dictionary learning, a locality-constrained and label embedding dictionary learning (LCLE-DL) [35] algorithm was proposed by considering both the locality of samples and label information.

The aforementioned approaches are supervised algorithms, which can make full use of labeled sample information, but they cannot utilize unlabeled samples information adequately. However, the labeled samples are finite since they are needed to be obtained artificially [36]. Moreover, the cost of manually labeled samples is too high due to the usage of manpower, material resources and energy. Hence, it is difficult to satisfy the requirement of for real-life sample [37]. Conversely, the unlabeled samples can easily be collected from the Internet, web chatting, digital camera of surveillance and so on [38]. As a result, it is necessary to utilize the information of unlabeled samples to improve the performance of ridge regression algorithm [39,40]. Rwebangira et al. [39] extended the local linear regression to the local linear semi-supervised regression (LLSSR) by adding manifold regularization. To reduce the distributed error and enlarge the number of data subsets using unlabeled data, Chang et al. [40] provided an error analysis for distributed semi-supervised learning with kernel ridge regression (DSKRR) based on a divide-and-conquer strategy. Although these semi-supervised ridge regression algorithms utilized the labeled and unlabeled data simultaneously, the distribution relationships between the labeled and unlabeled data were not considered at all.

Considering the computational efficiency and effectiveness, label propagation (LP) has attracted much attention in the study of the graph-based semi-supervised learning [41,42]. The core idea of LP is to propagate the information of labeled data to unlabeled data by constructing a weighted undirected graph. In the past few years, many LP algorithms such as gaussian fields and harmonic functions (GFHF) [43] and local and global consistency (LGC) [44] have been proposed. These methods can employ both labeled and unlabeled samples during training processing, but their performances heavily depended on the underlying geometrical structure of the original data distribution. In addition, they fail to build an explicit classifier for the testing samples or new coming samples. Therefore, these methods cannot directly obtain the label information of the testing samples. In order to deal with the first problem, many sophisticated graph construction algorithms have been studied in recent years [45,46,47]. To some extent, they alleviated the limitations of traditional k nearest neighbor or ε ball graphs, but their graph construction process was independent of the subsequent LP task. In other words, the graph structure was fixed in the process of LP.

In this paper, a novel semi-supervised classification method named semi-supervised ridge regression with adaptive graph-based label propagation (SSRR-AGLP) is developed to overcome the drawbacks of existing algorithms. First, inspired by the great success of sparse representation or sparse code based classification [48,49], an adaptive graph-based label propagation algorithm is presented to construct a graph dynamically. During the graph learning, both the locality and sparsity of data are considered simultaneously to optimize the graph structure for improving the performance of label prediction. Second, in order to utilize the predicted label information of unlabeled samples adequately and deal with the “out of sample” problem, the ridge regression algorithm is introduced into SSRR-AGLP. Finally, a simple and effective iterative optimization algorithm is designed to solve the proposed algorithm. To evaluate the performance of the proposed SSRR-AGLP, five benchmark facial image databases (Yale, ORL, Extended YaleB, CMU PIE and AR) are employed in this work. By comparing the experimental results of our SSRR-AGLP with some well-known related methods, the effectiveness and superiority of the proposed method can be justified. The flowchart of the proposed approach is illustrated in Figure 1.

The remainder of the paper is organized as follows. Least square regression (LSR), ridge regression (RR) and label propagation (LP) are reviewed briefly in Section 2. The proposed Semi-supervised ridge regression with adaptive graph-based label propagation (SSRR-AGLP) is described in Section 3 concretely. The experimental results and analysis are illustrated in Section 4. Finally, the conclusions and further work are presented in Section 5.

2. Related Works

In this section, the least square regression (LSR), ridge regression (RR) and label propagation (LP) algorithms are reviewed briefly.

2.1. Least Square Regression

Least squares regression (LSR) plays a very important role in machine learning, which seeks the best matching function of data by minimizing the sum of squares of errors [1,2]. The error is the difference between the predicted value and the real value. Suppose that $X=[x_1,x_2,...,x_n]\in R^{d\times n}$ is a dataset, where n is the sample number, d is the feature number of each sample and each sample has a corresponding label vector which is represented by $y\in R^{c\times 1}$, where c is the number of classes.

For the LSR model, $Y=[y_1,y_2,...,y_n]\in R^{c\times n}$ is defined to represent the class indicator matrix, where $y_i$ is the label vector of the i-th sample. If the i-th $(i=1,2,...,n)$ sample belongs to j-th $(j=1,2,...,c)$ class, then the label vector of the i-th sample is $y_i=[0,...,0,1,0,...,0]\in R^{c\times 1}$, in which only the j-th element equals to one. The optimization problem of LSR is formulated as:

$$\underset{W,b}{\min }\frac{1}{2}{\displaystyle \sum _{i=1}^n||W^T{x}_i+b-y_i|{|}_2^2}$$

(1)

where $W\in R^{d\times c}$ and $b={[b_1,b_2,...,b_c]}^T\in R^{c\times 1}$ represent the weight matrix and the bias vector, respectively. The objective of LSR is to find an optimal transformation matrix for minimizing the value of the error function as Equation (1). As described in [50], the error function can be reformulated as:

$$\underset{\overline{W}}{\min }\frac{1}{2}{\displaystyle \sum _{i=1}^n||{\overline{W}}^T{\overline{x}}_i-y_i|{|}_2^2}$$

(2)

where $\overline{W}=[{\overline{w}}_1,{\overline{w}}_2,...,{\overline{w}}_n]\in R^{(d+1)\times n}$ denotes a matrix and ${\overline{w}}_k=(w_k;b_k)\in R^{(d+1)\times 1}$ denotes the k-th column vector of $\overline{W}$. ${\overline{x}}_i=(x_i;1)\in R^{(d+1)\times 1}$ is also a vector. The optimal $\overline{W}$ can be obtained by minimizing the following function:

$$J(\overline{W})\underset{\overline{W}}{=\min }\frac{1}{2}||{\overline{W}}^T\overline{X}-Y|{|}_2^2$$

(3)

where $\overline{X}=[{\overline{x}}_1,{\overline{x}}_2,...,{\overline{x}}_n]\in R^{(d+1)\times n}$. Setting the derivative $\overline{W}$ to zero, we can obtain the optimal transformation matrix as follows:

$$\overline{W}={(\overline{X}{\overline{X}}^T)}^{-1}\overline{X}{Y}^T$$

(4)

where ${(\cdot )}^{-1}$ denotes the matrix inverse operation.

2.2. Ridge Regression

From Equation (4), we can see that when the sample number n is less than the feature number d $(n\prec d)$, the matrix $\overline{X}{\overline{X}}^T$ is a singular matrix, which will lead to the non-uniqueness of the solution. To overcome the shortcoming of LSR, a regularized LSR method named ridge regression (RR) [22] has been proposed, which integrates l₂-regularization into LSR. The objective function of RR is defined as follows:

$$\underset{W}{\min }\frac{1}{2}{\displaystyle \sum _{i=1}^n||W^T{x}_i-y_i|{|}_2^2}+\lambda ||W|{|}_2^2$$

(5)

where $\lambda >0$ is a regularization parameter.

Taking the derivative of Equation (5) with respect to $W$ and then setting it to zero, we have the matrix $W$:

$$W={(X{X}^T+\lambda I)}^{-1}X{Y}^T$$

(6)

2.3. Label Propagation

Suppose there is a sample set $X=[x_1,x_2,...,x_l,x_{l+1},...,x_u]\in R^{d\times n}$ from c class, in which the first $l(l<n)$ samples $X_l=[x_1,x_2,...,x_l]\in R^{l\times n}$ are labeled and the rest $n-l$ samples $X_u=[x_{l+1},x_{l+2},...,x_n]\in R^{(n-l)\times n}$ are unlabeled. Let $Y=[y_1,y_2,...,y_n]\in {\{0,1\}}^{c\times n}$ be a label matrix. Specfically, we set $y_{ij}=1$, if $x_i$ is a labeled sample belonging to the j-th class, otherwise $y_{ik}=0\hspace{0.17em}(k\ne j)$. The core purpose of the label propagation is to estimate the labels of the unlabeled data. We define G as a weighted undirected graph, in which each node corresponds to a data sample in X. The weight of edge between x_i and x_j is defined as

$$W_{ij}=\{\begin{array}{l}\exp (-\frac{||x_i-x_j|{|}_2^2}{2{\sigma }^2}), \mathrm{if} x_j\in N_k(x_i) \mathrm{or} x_i\in N_k(x_j)\\ 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{otherwise}\end{array}$$

(7)

where $N_k(x_i)$ is the set of k-nearest neighbors of x_i and σ is a parameter which determines the decay rate of heat kernel function. Let $F=[f_1,f_2,...,f_n]\in R^{c\times n}$ be a predicated label matrix, in which $f_i\in R^c$ is the corresponding predicated label vector of x_i. In terms of the LP theory [43,44], the predicated labels of the nodes in graph G is estimated as

$$\min {\displaystyle \sum _{i,j=1}^n||f_i-f_j|{|}_2^2{W}_{ij}}+\mu {\displaystyle \sum _{i=1}^n||f_i-y_i|{|}_2^2}$$

(8)

where $\mu$ is a balance parameter which controls the discrepancy between the predicated label vector and true label vector for the labeled samples. From Equation (7), we can find that the weighted undirected graph in the LP is constructed in advance and remains unchanged during the label propagation procedure. Furthermore, seen from the objective function in Equation (8), the LP can propagate the label information of the labeled samples to the unlabeled ones in the training dataset. However, since the LP fails to provide an explicit classifier, it suffers from the “out-of-samples” problem.

3. The Proposed Method

In this section, we first propose the semi-supervised ridge regression with adaptive graph-based label propagation (SSRR-AGLP) algorithm, which integrates adaptive graph learning, label propagation and ridge regression into a unified framework. Next, an optimization method based on iterative updating rules is designed to solve the objective function of SSRR-AGLP. Then, the convergence analysis of the optimization algorithm is also provided. Finally, the classification criterion for the testing samples is given.

3.1. Objective Function of SSRR-AGLP

In the proposed method, we first divide the dataset into training set and testing set, and then the training set is subdivided into labeled sample set and unlabeled sample set. Suppose the training sample set is $X=[x_1,x_2,...,x_l,x_{l+1},...,x_n]=[X_l,X_u]\in R^{d\times n}$, where $X_l=[x_1,x_2,...,x_l]\in R^{l\times d}$ represents the labeled samples of the training set and $X_u=[x_{l+1},x_{l+2},...,x_{l+u}]\in R^{d\times (n-l)}$ represents unlabeled samples of the training set. In addition, n represents the number of training samples and d represents the feature dimension of the samples. $Y=[y_1,y_2,...,y_l,y_{l+1},...,y_n]\in {\{0,1\}}^{c\times n}$ is the labeled matrix of training samples, where $y_i\in R^{c\times 1}(1\le i\le n)$ represents the label vector of the i-th sample, c is the total number of classes, and $y_{ij}$ is the j-th element in $y_i$. If x_i is a labeled sample and belongs to the j-th class, then $y_{ij}=1$, otherwise $y_{ij}=0$. If x_i is an unlabeled sample, all elements in $y_i$ are 0, that is, $\forall i>l, y_i=0\in R^{c\times 1}$.

The first aim of the proposed SSRR-AGLP is to utilize the label information of labeled samples to predict the label of unlabeled samples. Therefore, the LP algorithm is adopted in this work. However, different from the traditional LP algorithm, a relationship graph between samples is first constructed by the reconstructive coefficient of samples. The objective function is formulated as follows:

$$\begin{array}{l}\underset{S}{\min }\hspace{0.17em}\phi (S)=||X-XS|{|}_2^2\\ s.t.\hspace{1em}S\ge 0\end{array}$$

(9)

where $S=[s_1,s_2,\cdots ,s_n]\in R^{n\times n}$ is the weight matrix of graph, $s_i={[s_{i1},s_{i2},...,s_{in}]}^T\in R^{n\times 1}$ is the reconstruction coefficient vector of sample x_i, and its element value denotes the weight of an edge between the sample x_i and other samples in the graph. In order to make the reconstruction coefficients physically more meaningful, the non-negative constraint for the reconstruction coefficients is introduced in our model. The non-negativity constraint can enhance the discriminability of the reconstruction coefficients, so that the sample will more likely be reconstructed by the samples from the same clustering. The connection relationship between sample pairs in the graph is represented by reconstructive coefficients. For instance, if a coefficient of the sample pair x_j and x_i is non-zero, then there exists an edge between them in the graph and the weight of edge S_ij is set as the coefficient corresponding to x_j. Thus, the local information of data can be exploited by selecting the nearest neighbors of a sample. As mentioned above, the locality and sparsity constraint term for S can be defined as follows:

$$\underset{S}{\min }\hspace{0.17em}\varphi (S)=||E\odot S|{|}_1$$

(10)

where $||\cdot |{|}_1$ is the l₁-norm of a matrix and $\odot$ is the element-wise multiplication. $E=[e_{ij}]\in R^{n\times n}$ is the local adaptation matrix, in which the element $e_{ij}$ is defined as follows:

$$e_{ij}=\exp (\frac{||x_i-x_j|{|}_2^2}{{\sigma }^2})$$

(11)

From Equation (11), we can clearly find that a smaller $e_{ij}$ indicates x_i is more similar to x_j and vice versa. Hence, minimizing Equation (10) with respect to S will assign small or nearby zero to reconstructive coefficients of samples which are far from x_i. This means that if two samples are distant from each other, they are unlikely to be connected in the graph.

Combining Equation (9) with (10), the adaptive graph model in the label propagation is generated as follows:

$$\begin{array}{l}\underset{S}{\min }\hspace{0.17em}\phi (S)=||X-XS|{|}_2^2+\alpha ||E\odot S|{|}_1\\ s.t. S\ge 0\end{array}$$

(12)

where $\alpha >0$ is a balance parameter which controls the importance of the locality and sparsity constraint term.

Subsequently, let $F=[f_1,f_2,...,f_n]\in R^{c\times n}$ denote the prediction label matrix, where $f_i\in R^{c\times 1}$ is the column vector that denotes the probability of the sample x_i belonging to each class. For example, the largest value of f_ij means the highest probability of the sample x_i belonging to j-th class. In terms of the LP theory [37,38], the nearby or similar samples and the samples from the same global cluster should share similar labels. Therefore, the objective function of LP is defined as follows:

$$\underset{F}{\min }\hspace{0.17em}\phi (F)={\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n||f_i-f_j|{|}_2^2{R}_{ij}}}$$

(13)

where $R=(S+S^T)/2$ denotes the weight matrix. Moreover, in order to make the predicted labels and true labels of labeled samples be close as possible, we introduce the penalty constrain term as:

$$\underset{F}{\min }\hspace{0.17em}\vartheta (F)={\displaystyle \sum _{i=1}^n||f_i-y_i|{|}_2^2{u}_{ii}}$$

(14)

where U is a selection diagonal matrix and element u_ij is defined as follows:

$$u_{ii}=\{\begin{array}{l}+\infty , \mathrm{if} x_i \mathrm{is} \mathrm{a} \mathrm{labeled} \mathrm{sample}\\ 0\hspace{1em}, \mathrm{otherwise}\end{array}$$

(15)

Combining Equations (12)–(14), the adaptive graph label propagation model is defined as follows:

$$\begin{array}{l}\underset{F,\hspace{0.17em}S}{\min }\hspace{0.17em}\phi (F,S)=||X-XS|{|}_2^2+\alpha ||E\odot S|{|}_1+\beta {\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n||f_i-f_j|{|}_2^2{R}_{ij}}}+{\displaystyle \sum _{i=1}^n||f_i-y_i|{|}_2^2{u}_{ii}}\\ s.t.\hspace{1em}F\ge 0,S\ge 0\end{array}$$

(16)

where $\beta >0$ is a balance parameter which controls the importance of the label propagation term.

The second aim of our proposed SSRR-AGLP is to make full use of the predicted labels of samples to learn a classifier function. Thus, the ridge regression model is introduced into our method and the objective function of RR is:

$$\underset{W}{\min }\hspace{0.17em}\phi (W)=||F-W^{T}X|{|}_2^2+\gamma ||W|{|}_2^2$$

(17)

where $\gamma >0$ is a balance parameter which prevents the RR model from over-fitting.

Finally, combining Equation (16) with (17), the objective function of the SSRR-AGLP is formulated as follows:

$$\begin{array}{l}\underset{F,S,W}{\min }\hspace{0.17em}\epsilon (F,S,W)=||X-XS|{|}_2^2+\alpha ||E\odot S|{|}_1+\beta {\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n||f_i-f_j|{|}_2^2{R}_{ij}}}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\displaystyle \sum _{i=1}^n||f_i-y_i|{|}_2^2{u}_{ii}}+||F-W^{T}X|{|}_2^2+\gamma ||W|{|}_2^2\\ s.t. F\ge 0,W\ge 0,S\ge 0\end{array}$$

(18)

From Equation (18), it clearly can be seen that the proposed model combines the adaptive graph, label propagation algorithm and ridge regression algorithm together to solve the following problems:

(a) By introducing the LP algorithm into our method, it can solve the problem that the traditional RR algorithm cannot make use of unlabeled information. In addition, it can learn a classifier to deal with the “out-of-sample” problem.

(b) By integrating the adaptive graph learning and the LP algorithm into a unified framework, it can break the defect of traditional LP algorithm which needs to construct a graph in advance.

3.2. Optimization Solution

In Equation (18), there are three variables, the transformation matrix W, prediction label matrix F and weight matrix S, in the objective function. Unfortunately, the objective function of our algorithm is not convex in all these three variables together, so the global optimal solution cannot be given directly. In order to solve this problem, in this subsection, an iterative optimization solution is proposed, which is performed by fixing two variables and updating another variable.

3.2.1. Fix Transformation Matrix W and Prediction Label Matrix F to Solve Weight Matrix S

Removing the items that are not related to the matrix S in Equation (18), the optimization problem of the variables S can be obtained as follows:

$$\begin{array}{l}\underset{S}{\min }\hspace{0.17em}\epsilon (S)=||X-XS|{|}_2^2+\alpha ||E\odot S|{|}_1+\beta {\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n||f_i-f_j|{|}_2^2{R}_{ij}}}\\ s.t. S>0\end{array}$$

(19)

Through a series of algebraic formulations, Equation (19) is simplified to

$$\begin{array}{l}\underset{S}{\min }\hspace{0.17em}\epsilon (S)=tr(X^{T}X-2S^T{X}^{T}X+S^T{X}^{T}XS)+\alpha tr(ES)+\beta tr(QS)\\ s.t. S>0\end{array}$$

(20)

where $Q={[q_{ij}]}_{n\times n}\in R^{n\times n}$ and the element $q_{ij}$ is defined as $q_{ij}=||f_i-f_j|{|}_2^2$.

To solve the above problem, we need to introduce the Lagrange multiplier matrix $\psi$. The Lagrange function of Equation (20) is:

$$\eta (S,\psi )=tr(X^{T}X-2S^T{X}^{T}X+S^T{X}^{T}XS)+\alpha tr(ES)+\beta tr(QS)+tr(\psi S)$$

(21)

Setting the derivative with respect to S to zero, we obtain

$$\frac{\partial \eta (S,\psi )}{\partial S}=-2X^{T}X+2X^{T}XS+\alpha E+\beta Q+\psi =0$$

(22)

According to the KKT condition ${\psi }_{ij}{S}_{ij}=0$ [51], we obtain

$${(-2X^{T}X+2X^{T}XS+\alpha E+\beta Q)}_{ij}{S}_{ij}=0$$

(23)

According to Equation (23), the updating rule of S is

$$S_{ij}=:S_{ij}\frac{{[X^{T}X]}_{ij}}{{[X^{T}XS+\frac{\alpha }{2}E+\frac{\beta }{2}Q]}_{ij}}$$

(24)

3.2.2. Fix Weight Matrix S and Transformation Matrix W to Solve Prediction Label Matrix F

Removing the items that are not related to the matrix F in Equation (18), the optimization problem of the variable F can be obtained as follows:

$$\begin{array}{l}\underset{F}{\min }\hspace{0.17em}\epsilon (F)=||F-W^{T}X|{|}_2^2+\beta {\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n||f_i-f_j|{|}_2^2{R}_{ij}}}+{\displaystyle \sum _{i=1}^n||f_i-y_i|{|}_2^2{u}_{ii}}\\ s.t. F>0\end{array}$$

(25)

Through a series of algebra formulations, the Equation (25) is simplified to

$$\begin{array}{l}\underset{F}{\min }\hspace{0.17em}\epsilon (F)=tr(F{F}^T-2P^{T}X{F}^T+P^{T}X{X}^{T}P)+\beta tr(F(D-R)F^T)+tr((F-Y)U{(F-Y)}^T)\\ s.t. F>0\end{array}$$

(26)

where D is a diagonal matrix with its entries being $D_{ii}={\displaystyle {\sum }_j^n{R}_{ij}}$.

To solve the problem of Equation (26), we also need to introduce the Lagrange multiplier matrix $\Lambda$. The Lagrange function of Equation (26) is:

$$\begin{array}{l}\eta (F,\Lambda )=tr(F{F}^T-2W^{T}X{F}^T+W^{T}X{X}^{T}W)+\beta tr(F(D-R)F^T)\\ \hspace{1em}\hspace{1em}\hspace{1em}+tr((F-Y)U{(F-Y)}^T)+tr(\Lambda F)\end{array}$$

(27)

Setting the derivative with respect to F to zero, we obtain

$$\frac{\partial \eta (F,\Lambda )}{\partial F}=2F-2W^{T}X+2\beta F(D-R)+2(F-Y)U+\Lambda =0$$

(28)

According to the KKT condition ${\Lambda }_{ij}{W}_{ij}=0$ [51], we obtain

$${(2F-2W^{T}X+2\beta F(D-R)+2(F-Y)U)}_{ij}{\delta }_{ij}=0$$

(29)

According to Equation (29), the updating rule of F is

$$F_{ij}=:F\frac{{[W^{T}X+\beta FR+YU]}_{ij}}{{[F+\beta FD+FU]}_{ij}}$$

(30)

3.2.3. Fix Weight Matrix S and Prediction Label Matrix F to Solve Transformation Matrix W

Removing the items that are not related to the matrix W in Equation (18), the optimization problem of the variable W can be obtained as follows:

$$\begin{array}{l}\underset{W}{\min }\hspace{0.17em}\epsilon (W)=||F-W^{T}X|{|}_2^2+\gamma ||W|{|}_2^2\\ s.t.\hspace{1em}W\ge 0\end{array}$$

(31)

Through a series of algebra formulations, the Equation (31) is simplified to

$$\begin{array}{l}\underset{W}{\min }\hspace{0.17em}\epsilon (W)=tr(F{F}^T-2W^{T}X{F}^T+W^{T}X{X}^{T}W)+\gamma tr(W^{T}W)\\ s.t.\hspace{1em}W\ge 0\end{array}$$

(32)

To solve the above problem, we need to introduce the Lagrange multiplier matrix θ. The Lagrange function of Equation (32) is:

$$\eta (W)=tr(F{F}^T-2W^{T}X{F}^T+W^{T}X{X}^{T}W)+\gamma tr(W^{T}W)+tr(\theta W)$$

(33)

The partial derivation of Equation (33) with respect to W is

$$\frac{\partial \eta (W,\theta )}{\partial W}=-2X{F}^T+2X{X}^{T}W+2\gamma W+\theta$$

(34)

Setting the derivative is equal to zero, we obtain

$$\frac{\partial \eta (W,\theta )}{\partial W}=-2X{F}^T+2X{X}^{T}W+2\gamma W+\theta =0$$

(35)

According to the KKT condition ${\theta }_{ij}{W}_{ij}=0$ [51], we obtain

$${(-2X{F}^T+2X{X}^{T}W+2\gamma W)}_{ij}{\theta }_{ij}=0$$

(36)

According to Equation (36), the updating rule of W is:

$$W_{ij}=:W_{ij}\frac{{[X{F}^T]}_{ij}}{[X{X}^{T}W+\gamma W]}$$

(37)

3.2.4. The Optimization Algorithm

In summary, we provide the primary optimization procedure of the proposed algorithm in Algorithm 1.

Algorithm 1. The algorithm to solve the objective function of SSRR-AGLP

Input: the training set $X=[x_1,x_2,...,x_l,x_{l+1},\dots x_n]=[X_l,X_u]\in R^{d\times n}$, the label matrix of training set $Y=[y_1,y_2,...,y_l,y_{l+1},...,y_n]\in {\{0,1\}}^{c\times n}$ 1: Initialization: parameters $\alpha \mathrm{and} \beta =1 \mathrm{and} \gamma =0.01$, matrices $W_t\in R^{d\times c},F_t\in R^{c\times n} \mathrm{and} S_t\in R^{n\times n}$ are an arbitrary nonnegative matrix, t = 0 2: According to Equations (11) and (15), the diagonal matrices $E\in R^{n\times n}$ and $U\in R^{n\times n}$ are calculated respectively. 3: Repeat steps 3–9 until convergence conditions 4: According to $S_t$, calculate $R_t=\frac{(S_t+S_t^T)}{2}$, and then calculate matrix $D_t$ where ${(D_{ii})}_t={\displaystyle {\sum }_{j=1}^n{(R_{ij})}_t}$ 5: According to $F_t$, calculate $Q_t$ where ${(q_{ij})}_t=||{(f_i)}_t-{(f_j)}_t|{|}_2^2$ 6: Update $S_{t+1}$ to $S_{t+1}=:S_t\odot \frac{X^{T}X}{X^{T}X{S}_t+\frac{\alpha }{2}E+\frac{\beta }{2}{Q}_t}$ 7: Update $F_{t+1}$ to $F_{t+1}=:F_t\odot \frac{W^{T}X+\beta FR+YU}{F+\beta FD+FU}$ 8: Update $W_{t+1}$ to $W_{t+1}=:W_t\odot \frac{X{F}_t^T}{X{X}^T{W}_t+\gamma W_t}$ 9: Update $t=:t+1$

Output: Predicted label matrix F, weight matrix S and transformation matrix W

Clearly, the updating of S, F and W is calculated alternately in each iteration of Algorithm 1 which indicates the process of the graph, label propagation and classifier learning are jointly implemented in our proposed SSRR-AGLP. In addition, the predicted label matrix (F) and the transformation matrix (W) of RR are mutually affected in each iteration, which makes both the classifiers and predicted labels more accurate in our algorithm.

3.3. Classification Criterion

Given a testing sample $x_{test}$, its predicted label vector $f={[f_1,f_2,...,f_c]}^T\in R^{c\times 1}$ is computed by follows:

$$f=W^T{x}_{test}$$

(38)

Then, we adopt f to assign the single class label for the testing data and the rule is:

$$\mathrm{identity}(x_{test})=\underset{i}{\arg \max }{f}_i,\hspace{1em}i=1,2,...,c$$

(39)

3.4. Convergence Analysis

The convergence of the updating rules in Equations (24), (30) and (37) is analyzed in this section. Similar to [50], the definition of the auxiliary function is first given:

Definition 1.

$\vartheta (u,u\prime )$is an auxiliary function for$\varphi (u)$if conditions$\vartheta (u,u\prime )\ge \varphi (u)$and$\vartheta (u,u\prime )=\varphi (u)$are satisfied.

The auxiliary function plays a very important role in the following lemma.

Lemma 1.

If$\vartheta$is an auxiliary function of$\varphi$, then it is non-increasing with the following updating formula.

$$u^{t+1}=\underset{i}{\arg \min }\vartheta (u,u^{t+1})$$

(40)

Proof. 

$$\varphi (u^{t+1})\le \vartheta (u^{t+1},u^t)\le \vartheta (u^t,u^t)=\varphi (u^t)$$

(41)

Firstly, we present the updating rule for S in Equation (24) which is exactly the updating in Equation (40) with a proper auxiliary function. Considering any element S_ij in S, ${\varphi }_{ij}(S_{ij})$ denotes the part of the objective function of SSRR-AGLP, which is only relevant to S_ij, as follows:

$${\varphi }_{ij}(S_{ij})={[-2S^T{X}^{T}X+S^T{X}^{T}XS+\alpha ES+\beta QS]}_{ij}$$

(42)

$${\varphi }_{ij}^{\prime }(S_{ij})={[-2X^{T}X+2X^{T}XS+\alpha E+\beta Q]}_{ij}$$

(43)

$${\varphi }_{ij}^{\prime \prime }(S_{ij})={[2X^{T}X]}_{ii}$$

(44)

where ${\varphi }_{ij}^{\prime }(S_{ij})$ and ${\varphi }_{ij}^{\prime \prime }(S_{ij})$ are the first-order and second-order derivatives of the objective function with respect to S_ij. □

Lemma 2.

The function${\vartheta }_{ij}^{}(S_{ij},S_{ij}^t)$is an auxiliary function for${\varphi }_{ij}^{}(S_{ij})$, which is formulated as:

$${\vartheta }_{ij}^{}(S_{ij},S_{ij}^t)={\varphi }_{ij}^{}(S_{ij}^t)+{\varphi }_{ij}^{\prime }(S_{ij}^t)(S_{ij}-S_{ij}^t)+\frac{[X^{T}XS+\frac{\alpha }{2}E+\frac{\beta }{2}Q]}{S_{ij}^t}{(S_{ij}-S_{ij}^t)}^2$$

(45)

Proof. 

We first generate the Taylor series expansion of ${\varphi }_{ij}^{}(S_{ij})$

$$\begin{array}{l}{\varphi }_{ij}^{}(S_{ij})={\varphi }_{ij}^{}(S_{ij}^t)+{\varphi }_{ij}^{\prime }(S_{ij}^t)(S_{ij}-S_{ij}^t)+{\varphi }_{ij}^{″}(S_{ij}^t){(S_{ij}-S_{ij}^t)}^2\\ ={\varphi }_{ij}^{}(S_{ij}^t)+{\varphi }_{ij}^{\prime }(S_{ij}^t)(S_{ij}-S_{ij}^t)+{\varphi }_{ij}^{″}(S_{ij}^t){(S_{ij}-S_{ij}^t)}^2\end{array}$$

(46)

According to Equation (45), ${\vartheta }_{ij}^{}(S_{ij},S_{ij}^t)\ge {\varphi }_{ij}^{}(S_{ij})$ is equivalent to

$$\frac{{[X^{T}XS+\frac{\alpha }{2}E+\frac{\beta }{2}Q]}_{ij}}{S_{ij}^t}\ge {[X^{T}X]}_{ii}$$

(47)

Then, we obtain

$${[X^{T}XS]}_{ij}={\displaystyle \sum _{l=1}^n{(X^{T}X)}_{il}{S}_{lj}}\ge {(X^{T}X)}_{ii}{S}_{ij}$$

(48)

Thus, Equation (47) holds and ${\vartheta }_{ij}^{}(S_{ij},S_{ij}^t)\ge {\varphi }_{ij}^{}(S_{ij})$. Furthermore, we can see that ${\vartheta }_{ij}^{}(S_{ij},S_{ij}^t)={\varphi }_{ij}^{}(S_{ij})$.

Secondly, we indicate the updating rule for F in Equation (30) which is exactly updating in Equation (40) with a proper auxiliary function. Considering any element F_ij in F, ${\varphi }_{ij}(F_{ij})$ is used to show the part of the objective function of SSRR-AGLP which is only relevant to F_ij, as follows:

$${\varphi }_{ij}(F_{ij})={[F{F}^T-2P^{T}X{F}^T+\beta F(D-R)F^T+FU{F}^T-2YU{F}^T]}_{ij}$$

(49)

$${\varphi }_{ij}^{\prime }(F_{ij})={[2F-2P^{T}X+2\beta F(D-R)+2FU-2YU]}_{ij}$$

(50)

$${\varphi }_{ij}^{\prime \prime }(F_{ij})={[2I+2\beta (D-R)+2U]}_{jj}$$

(51)

where ${\varphi }_{ij}^{\prime }(F_{ij})$ and ${\varphi }_{ij}^{\prime \prime }(F_{ij})$ are the first-order and second-order derivatives of the objective function with respect to F_ij. □

Lemma 3.

The function${\vartheta }_{ij}^{}(F_{ij},F_{ij}^t)$is an auxiliary function for${\varphi }_{ij}^{}(F_{ij})$, which is defined as:

$${\vartheta }_{ij}^{}(F_{ij},F_{ij}^t)={\varphi }_{ij}^{}(F_{ij}^t)+{\varphi }_{ij}^{\prime }(F_{ij}^t)(F_{ij}-F_{ij}^t)+\frac{[F+\beta FD+FU]}{F_{ij}^t}{(F_{ij}-F_{ij}^t)}^2$$

(52)

Proof. 

First, the Taylor series expansion of ${\varphi }_{ij}^{}(F_{ij})$ is:

$$\begin{array}{l}{\varphi }_{ij}^{}(F_{ij})={\varphi }_{ij}^{}(F_{ij}^t)+{\varphi }_{ij}^{\prime }(F_{ij}^t)(F_{ij}-F_{ij}^t)+{\varphi }_{ij}^{″}(F_{ij}^t){(F_{ij}-F_{ij}^t)}^2\\ ={\varphi }_{ij}^{}(F_{ij}^t)+{\varphi }_{ij}^{\prime }(F_{ij}^t)(F_{ij}-F_{ij}^t)+{\varphi }_{ij}^{″}(F_{ij}^t){(F_{ij}-F_{ij}^t)}^2\end{array}$$

(53)

According to Equation (52), ${\vartheta }_{ij}^{}(F_{ij},F_{ij}^t)\ge {\varphi }_{ij}^{}(F_{ij})$ is equivalent to

$$\frac{{[F+\beta FD+FU]}_{ij}}{F_{ij}^t}\ge {[I+\beta (D-R)+U]}_{ij}$$

(54)

Then, we have

$${[\beta FD]}_{ij}=\beta {\displaystyle \sum _{l=1}^N{F}_{il}^t{D}_{lj}}\ge \beta F_{ij}^t{D}_{jj}\ge \beta F_{ij}^t(D_{jj}-R_{jj})$$

(55)

$${[FU]}_{ij}={\displaystyle \sum _{l=1}^N{F}_{il}^t{U}_{lj}}\ge F_{ij}^t{U}_{jj}$$

(56)

Thus, Equation (54) holds and ${\vartheta }_{ij}^{}(F_{ij},F_{ij}^t)\ge {\varphi }_{ij}^{}(F_{ij})$. Furthermore, we can see that ${\vartheta }_{ij}^{}(F_{ij},F_{ij}^t)={\varphi }_{ij}^{}(F_{ij})$.

Subsequently, we describe the updating rule for W in Equation (37), which is correctly updated in Equation (40) with a proper auxiliary function. For any element W_ij in W, ${\varphi }_{ij}(W_{ij})$ is adopted to represent the part of the objective function of SSRR-AGLP, which is only relevant to W_ij. It is defined as:

$${\varphi }_{ij}(W_{ij})={[-2W^{T}X{F}^T+W^{T}X{X}^{T}W+\gamma W^{T}W]}_{ij}$$

(57)

$${\varphi }_{ij}^{\prime }(W_{ij})={[-2X{F}^T+2X{X}^{T}W+2\gamma W]}_{ij}$$

(58)

$${\varphi }_{ij}^{\prime \prime }(W_{ij})={[2X{X}^T+2\gamma I]}_{jj}$$

(59)

where ${\varphi }_{ij}^{\prime }(W_{ij})$ and ${\varphi }_{ij}^{\prime \prime }(W_{ij})$ are the first-order and second-order derivatives of the objective function with respect to W_ij. □

Lemma 4.

The function

$${\vartheta }_{ij}^{}(W_{ij},W_{ij}^t)={\varphi }_{ij}^{}(W_{ij}^t)+{\varphi }_{ij}^{\prime }(W_{ij}^t)(W_{ij}-W_{ij}^t)+\frac{[X{X}^{T}W+\gamma W]}{W_{ij}^t}{(W_{ij}-W_{ij}^t)}^2$$

(60)

is an auxiliary function for${\varphi }_{ij}^{}(W_{ij})$.

Proof. 

The Taylor series expansion of ${\varphi }_{ij}^{}(W_{ij})$ is:

$$\begin{array}{l}{\varphi }_{ij}^{}(W_{ij})={\varphi }_{ij}^{}(W_{ij}^t)+{\varphi }_{ij}^{\prime }(W_{ij}^t)(W_{ij}-W_{ij}^t)+{\varphi }_{ij}^{″}(W_{ij}^t){(W_{ij}-W_{ij}^t)}^2\\ ={\varphi }_{ij}^{}(W_{ij}^t)+{\varphi }_{ij}^{\prime }(W_{ij}^t)(W_{ij}-W_{ij}^t)+{\varphi }_{ij}^{″}(W_{ij}^t){(W_{ij}-W_{ij}^t)}^2\end{array}$$

(61)

From Equation (60), we can find that ${\vartheta }_{ij}^{}(W_{ij},W_{ij}^t)\ge {\varphi }_{ij}^{}(W_{ij})$ is equivalent to

$$\frac{{[X{X}^{T}W+\gamma W]}_{ij}}{W_{ij}^t}\ge {[X{X}^T+\gamma I]}_{ii}$$

(62)

Then, we have

$${[X{X}^{T}W]}_{ij}={\displaystyle \sum _{l=1}^d{(X{X}^T)}_{il}^t{W}_{lj}}\ge {(X{X}^T)}_{ii}^t{W}_{ij}$$

(63)

Thus, Equation (62) holds and ${\vartheta }_{ij}^{}(W_{ij},W_{ij}^t)\ge {\varphi }_{ij}^{}(W_{ij})$. Furthermore, we can see that ${\vartheta }_{ij}^{}(W_{ij},W_{ij}^t)={\varphi }_{ij}^{}(W_{ij})$. □

Finally, in order to demonstrate the convergence of the updating rules in Equations (24), (30) and (37), we give the following theorem about the three updating rules:

Theorem 1.

For$S\ge 0$,$F\ge 0$and$W\ge 0$, the objective function in Equation (14) is non-increasing with the updating rules in Equations (24), (30) and (37).

Proof of Theorem 1.

Replacing ${\vartheta }_{ij}^{}(F_{ij},F_{ij}^t)$ in Equation (40) with Equation (45), we have

$$S_{ij}^{t+1}=S_{ij}^t-S_{ij}^t\frac{{\varphi }_{ij}^{\prime }(S_{ij})}{2{[X^{T}XS+\frac{\alpha }{2}E+\frac{\beta }{2}Q]}_{ij}}=S_{ij}^t\frac{{[X^{T}X]}_{ij}}{{[X^{T}XS+\frac{\alpha }{2}E+\frac{\beta }{2}Q]}_{ij}}$$

(64)

Similarly, replacing ${\vartheta }_{ij}^{}(F_{ij},F_{ij}^t)$ in Equation (40) by Equation (52), we obtain

$$F_{ij}^{t+1}=F_{ij}^t-F_{ij}^t\frac{{\varphi }_{ij}^{\prime }(F_{ij})}{2{[F+\beta FD+FU]}_{ij}}=F_{ij}^t\frac{{[W^{T}X+\beta FR+YU]}_{ij}}{{[F+\beta FD+FU]}_{ij}}$$

(65)

In the same way, replacing ${\vartheta }_{ij}^{}(W_{ij},W_{ij}^t)$ in Equation (40) by Equation (60), there is

$$W_{ij}^{t+1}=W_{ij}^t-W_{ij}^t\frac{{\varphi }_{ij}^{\prime }(W_{ij})}{2{[X{X}^{T}W+\gamma W]}_{ij}}=F_{ij}^t\frac{{[X{F}^T]}_{ij}}{{[X{X}^{T}W+\gamma W]}_{ij}}$$

(66)

Clearly, Equations (45), (52) and (60) are auxiliary functions for ${\varphi }_{ij}^{}$. Therefore, ${\varphi }_{ij}^{}$ is non-increasing under these updating rules described in Equations (24), (30) and (37). □

4. Experiment and Analysis

To evaluate the performance of the proposed SSRR-AGLP for classification, we test it on five facial image databases (Yale [52], ORL [53], Extended YaleB [54], AR [55] and CMU PIE [56]). The detailed information of the five databases utilized in our experiments is shown in

Table 1. Detailed information of databases.

Database	Size	Samples	Classes	Per Class
Yale	32 × 32	165	15	11
ORL	32 × 32	400	40	10
Extended YaleB	32 × 32	2432	38	64
AR	32 × 32	1400	100	14
CMU PIE	32 × 32	1632	68	24

and some sample images from them are displayed in Figure 2. In all experiments, 50% samples of each subject are selected randomly for training and the remaining samples are used for testing in each database. Specifically, a part of the samples is randomly selected as labeled data in each training set, and the numbers of selected training and testing sets for experiments are shown in

Table 2. Detailed information of sample selection for different databases.

Database	Training Sample Size		Test Sample Size
	Labeled Samples	Unlabeled Samples
Yale	3	3	5
ORL	3	2	5
Extended YaleB	18	14	32
AR	4	3	7
CMU PIE	6	6	12

.

Moreover, to justify the superiority of the proposed SSRR-AGLP, we compare it with some well-known algorithms, such as traditional k-nearest neighbor (KNN) [57], least squares regression (LSR) [1], ridge regression (RR) [22], inter-class sparsity based discriminative least square regression (ICS_DLSR) [18] and the locality-constrained and label embedding dictionary learning (LCLE-DL) [35].

4.1. Parameter Setting

According to Section 3, there are three parameters, i.e., α, β and γ, that need to be determined in the objective function of the proposed SSRR-AGLP. The best values of these parameters are tuned by searching the grid {0.0001, 0.001, 0.01, 0.1, 1, 10, and 100} in an alternate manner. We first fix parameter γ as 0.01, and the influences of the parameters α and β on different datasets are illustrated in Figure 3. From these results, the tendencies of α and β on the accuracy rates increase first and then decrease for all the databases, which indicates that locality constraint term and label propagation can improve the performance of SSRR-AGLP. Moreover, when the values of parameters are set between 1 and 10, the performance of our SSRR-AGLP is insensitive to the parameter values. Then, we fix parameters α and β with best values, and the results of our algorithm under various values of parameter γ on the different databases are listed in

Table 3. The average accuracy (%) and standard deviation (%) of different values of parameter γ for different databases.

Database	Yale	ORL	Extended YaleB	AR	CMU PIE
γ = 0.0001	89.61 ± 5.20	88.73 ± 4.68	92.08 ± 5.69	91.00 ± 4.39	92.96 ± 3.57
γ = 0.001	91.27 ± 5.79	90.00 ± 3.16	94.22 ± 6.72	92.66 ± 5.45	95.51 ± 3.41
γ = 0.01	91.60 ± 5.83	90.25 ± 3.75	94.40 ± 6.87	92.73 ± 5.06	95.88 ± 3.55
γ = 0.1	91.00 ± 4.80	89.95 ± 3.18	94.24 ± 5.18	91.93 ± 4.86	95.37 + 3.87
γ = 1	90.52 ± 4.69	88.76 ± 3.87	93.13 ± 5.60	91.00 ± 4.24	94.19 + 3.46
γ = 10	89.87 ± 5.64	87.54 ± 3.25	92.16 ± 5.09	90.54 ± 3.69	93.27 ± 3.23
γ = 100	88.76 ± 5.78	86.90 ± 3.63	91.40 ± 4.28	89.79 ± 4.50	92.35 ± 3.08

. From this table, we can find the proposed SSRR-AGLP reaches the best performance when the parameter γ is set as 0.01 for all the databases. Moreover, when the parameters are set between 0.001 and 0.1, the performance of our SSRR-AGLP is insensitive to the parameter value of γ. Finally, we can find the best parameter values for our model are {α = 1, β = 10 and γ = 0.01} on Yale database, {α = 1, β = 1 and γ = 0.01} on ORL database, {α = 10, β = 1 and γ = 0.01} on Extended YaleB database, {α = 1, β = 1 and γ = 0.01} on AR database and {α = 10, β = 1 and γ = 0.01} on CMU PIE database. Therefore, according to the experimental results, we can set the parameters α and β as relatively larger values (1 and 10), and set the parameter γ as a small value (from 0.001 to 0.1) for the application tasks.

4.2. Experimental Results and Analysis

To fairly compare the performances of our approach and other algorithms, the average accuracy rates and standard deviations over 10 random training sample selection procedures of different algorithms are listed in

Table 4. The average accuracy (%) and standard deviation (%) of different methods for different databases.

Method	Yale	ORL	Extended YaleB	AR	CMU PIE
KNN	84.93 ± 7.20	73.05 ± 2.68	75.42 ± 6.98	69.04 ± 6.93	90.07 ± 6.51
LSR	88.27 ± 7.97	84.75 ± 1.60	87.51 ± 7.22	82.66 ± 6.70	91.57 ± 5.41
RR	89.07 ± 7.82	85.75 ± 2.30	88.33 ± 9.23	86.77 ± 4.72	92.44 ± 5.03
ICS_DLSR	90.00 ± 2.45	87.60 ± 1.98	92.04 ± 0.81	90.93 ± 1.44	92.71 + 0.75
LCLE_DL	90.40 ± 3.00	87.65 ± 3.11	91.93 ± 0.62	90.91 ± 1.14	93.91 + 1.34
SSRR-AGLP	91.60 ± 5.83	90.25 ± 3.75	94.40 ± 6.87	92.73 ± 5.06	95.88 ± 3.55

. From Table 4, we can get the following observations: (1) KNN is sensitive to noise; its accuracy rate is lower than other methods. (2) The performance of RR is better than that of LSR, which indicates that RR can avoid the over-fitting problem effectively. (3) The accuracy rate of ICS_DLSR is higher than that of LSR and RR. This is because that the inter-class sparsity regularization term can improve discriminative ability of transformation matrix adequately. (4) By exploring the locality structure of the learned dictionary, the performance of LCLE_DL is better than that of RR and ICS_DLSR in most cases, which implies that the locality of data can improve the performance effectively. (5) The performance of the proposed SSRR-AGLP is consistently superior to other algorithms for five face image databases. It is because SSRR-AGLP not only employs both the labeled and unlabeled data to train the classifiers, but also takes advantage of the locality and sparsity of data for adaptive graph construction.

In the second experiment, to verify the performances of the proposed SSRR-AGLP under different numbers of labeled samples, we select 50% samples as the training set for each facial database, in which different numbers of samples are selected as labeled samples for each training set.

Table 5. The average results of the proposed SSRR-AGLP with different numbers of labeled samples for five facial image databases.

Database	Accuracy (%)
Yale	88.80 ± 5.56(2/4)	91.60 ± 5.83(3/3)	95.20 ± 4.88(4/2)
ORL	83.25±2.92(2/3)	90.25 ± 3.75(3/2)	92.35 ± 2.16(4/1)
Extended YaleB	92.07 ± 4.20(12/20)	92.38 ± 4.23(16/16)	94.40 ± 6.87(18/14)
AR	90.37 ± 8.07(3/4)	92.73 ± 5.06(4/3)	94.27 ± 4.93(5/2)
CMU PIE	85.78 ± 7.95(4/8)	95.88 ± 3.55(6/6)	93.82 ± 4.58(8/4)

Note that the two numbers in parentheses are the corresponding the quantities of labeled samples and unlabeled samples in training set.

lists the best average accuracy rates obtained by SSRR-AGLP with varied numbers of labeled samples. We can see that with the increase in labeled samples, the performance of the proposed SSRR-AGLP is improved gradually.

In the third experiment, the predicted labels of SSRR-AGLP are compared with the traditional label propagation algorithms to evaluate the performance of SSRR-AGLP. Specifically, the number of the selected label samples for each database is set as the same as in the first experiment, and the accuracy of the predicted labels by SSRR-AGLP and LP is listed in

Table 6. The label propagation accuracies (%) and standard deviations (%) of the proposed SSRR-AGLP and traditional LP for five facial image databases.

Database	LP	SSRR-AGLP
Yale	89.56 ± 5.89	97.33 ± 2.30
ORL	88.13 ± 2.38	90.25 ± 3.81
Extended YaleB	77.39 ± 8.30	95.55 ± 5.15
AR	78.00 ± 9.27	97.63 ± 3.40
CMU PIE	88.55 ± 9.75	91.74 ± 6.83

. It can be seen that the performance of SSRR-AGLP is superior to that of LP due to the cooperation of adaptive graph leaning and label propagation in it. In other words, compared with the traditional LP which predicts the labels by constructing a graph in advance, adaptive graph learning in SSRR-AGLP can improve the predicted performance of SSRR-AGLP. Besides, according to the predicted labels of traditional LP algorithm, we utilize the ridge regression (RR) algorithm (denoted as LP + RR) for classification and the results for the five databases are listed

Table 7. The average classification accuracies (%) and standard deviations (%) of RR, RR based on the predicted labels of traditional LP and SSRR-AGLP for five facial image databases.

Database	RR	LP + RR	SSRR-AGLP
Yale	89.07 ± 7.82	89.20 ± 6.89	91.60 ± 5.83
ORL	85.75 ± 2.30	87.55 ± 2.34	90.25 ± 3.75
Extended YaleB	88.33 ± 9.23	91.89 ± 7.89	94.40 ± 6.87
AR	86.77 ± 4.72	88.44 ± 4.07	92.73 ± 5.06
CMU PIE	92.44 ± 5.03	92.95 ± 5.18	95.88 ± 3.55

. From Table 7, we can see that the performance of LP + RR is better than that of the RR algorithm, but still worse than that of SSRR-AGLP. This indicates that combining both of the LP and RR algorithms into a uniform framework can provide a benefit for classification tasks.

Finally, Figure 4 displays the convergence curves of SSRR-AGLP for five facial image databases. In this figure, the x-axis and y-axis denote the number of iterations and the value of the objective function, respectively. We can see that the proposed iterative updating algorithm converges very fast (usually within 20 iterations).

5. Conclusions

In this paper, we present a semi-supervised ridge regression algorithm which combines ridge regression with graph learning and label propagation into a unified framework. Compared with other approaches, the proposed SSRR-AGLP not only can adaptively construct graph based on the locality and sparsity of data, but also overcome the “out-of-sample” problem. Moreover, we design an effective iterative updating algorithm to solve the proposed framework and the convergence analysis is also provided accordingly. Extensive experiments indicate the effectiveness and superiority of the proposed SSRR-AGLP.

From the experimental results, the performance of the proposed SSRR-AGLP is affected by the parameter values. Therefore, how to extend our framework to a parameter-free approach is one focus in our future research. Furthermore, we only take one distance measurement, i.e., the exponential function in Equation (9), to characterize the local information of input data in this study. Hence, introducing more distance measurements (such as Euclidean distance, inner-product and so on) into our SSRR-AGLP so that the local geometrical structure of data can be better exploited is also future work.