Relational Fisher Analysis: Dimensionality Reduction in Relational Data with Global Convergence ^†

Abstract

Most of the dimensionality reduction algorithms assume that data are independent and identically distributed (i.i.d.). In real-world applications, however, sometimes there exist relationships between data. Some relational learning methods have been proposed, but those with discriminative relationship analysis are lacking yet, as important supervisory information is usually ignored. In this paper, we propose a novel and general framework, called relational Fisher analysis (RFA), which successfully integrates relational information into the dimensionality reduction model. For nonlinear data representation learning, we adopt the kernel trick to RFA and propose the kernelized RFA (KRFA). In addition, the convergence of the RFA optimization algorithm is proved theoretically. By leveraging suitable strategies to construct the relational matrix, we conduct extensive experiments to demonstrate the superiority of our RFA and KRFA methods over related approaches.

1. Introduction

In some applications, such as pattern recognition and data mining, dimensionality reduction methods are often used since they can reduce space-time complexity, denoise, and make the model more robust. Principal component analysis (PCA)  [1,2,3] and linear discriminant analysis (LDA) [2,4,5,6,7] are two typical linear algorithms. Following them, researchers have proposed many variants, such as kernel PCA [8], generalized discriminant analysis (GDA) [9], and linear discriminant analysis for robust dimensionality reduction (RLDA) [10].

For nonlinear dimensionality reduction problems, manifold learning provides an effective solution. By supposing that data are located on a low-dimensional manifold, data samples observed in high-dimensional space can be represented in a low-dimensional space. Some representative manifold learning algorithms are ISOMAP [11], locally linear embedding (LLE) [12] and Laplacian Eigenmaps (LE) [13].

From the algorithmic perspective, algorithms mentioned above can be categorized as global methods or local methods. Global methods learn the low-dimensional representations by using global information of data. PCA and LDA are all global methods. The global methods are often effective and efficient, such that they are widely used in many real world applications. However, when dealing with non-linear data, using the global method cannot capture the genuine distribution of data very well. Local methods using the manifold learning idea, such as LLE and LE, pay special attention to the intrinsic structure of data. Nevertheless, most of these manifold learning methods disregard label information when recovering the low-dimensional manifold structure, in that they are inherently unsupervised.

Although the above-mentioned methods are defined from different perspectives of dimensionality reduction, graph embedding provide a unified framework for understanding and comparing them [14]. Furthermore, by integrating label information in the computation of intrinsic and penalty graphs within the graph embedding framework, a supervised dimensionality reduction method called marginal Fisher analysis (MFA) is proposed [14].

In traditional dimensionality reduction algorithms as described above, a data distribution assumption is generally applied that data are independent and identically distributed (i.i.d.). However, in real-world applications, there are often certain relativity or links between certain data, for instance, geometrical or semantic similarity, links among web pages, citation relations between scientific papers. Relationships usually indicate that these related samples are likely to have similarities or belonging to the same class. Nevertheless, although some dimensionality reduction methods consider to preserve the locality of data [15,16,17,18], the useful relationships are often simply ignored during the learning process of most existing dimensionality reduction methods.

Recently, relational learning has often been used in practical applications, for instance, web mining [19] and social network analysis [20]. In addition, relational information is also considered in social network discovery, document classification, sequential data analysis and semi-supervised graph embedding [21,22,23].

In the domain of dimensionality reduction, some algorithms have already been proposed in which the relational information is integrated into the representation learning process. In [24], Duin et al. propose the relational discriminant analysis (RDA). In RDA, relationships among data are measured by the Euclidean distance between objects and prototypes or support objects of each class. However, RDA uses mean squared error as the objective function, and cannot perform well on multi-class learning problems. In [25], Li et al. propose the probabilistic relational PCA (PRPCA) to build a probabilistic model associated with PCA and relational learning. RDA and PRPCA effectively integrate relational learning into the dimensionality reduction algorithms. Nevertheless, how to better inject relationships into traditional dimensionality reduction models is still worth exploring.

Recently, a few deep relational learning algorithms have been proposed. Specifically, Gao et al. design a deep learning model based on relational network for hyperspectral image few-shot classification [26], Chen et al. apply local relation learning for face forgery detection [27], and Cho et al. develop a weakly supervised anomaly detection method via context-motion relational learning [28]. In addition, some relational learning methods are used in the one-shot [29] or zero-shot [30] learning scenarios. However, many real-world applications have only very few data. Hence, shallow relational learning algorithms are still needed to be proposed and utilized.

In this paper, we propose a novel and general framework for dimensionality reduction, called relational Fisher analysis (RFA) [31]. Besides the intrinsic and penalty graph in the graph embedding framework, we further construct a relational graph which captures the relational information encoded within data. Through this graph, the proposed RFA takes into account the impact of the relational information in the presentation learning process. An effective iterative trace ratio algorithm is proposed to optimize RFA. Futhermore, we use the kernel trick to extend RFA to its kernelized version—KRFA. Additiionally, we theoretically prove that the optimization algorithm of RFA converges. To evaluate the effectiveness of RFA, we conduct extensive experiments in many real-world applications. The results demonstrate that the proposed RFA outperforms most of the classic dimensionality reduction algorithms on the datasets we use. The effectiveness of KRFA is also tested.

This paper is based on one of our previous conference papers [31], with significant improvements. For concreteness, we propse the KRFA algorithm and add more exclusive experiments with comparison to the related approaches. The rest of this paper is organized as follows: In Section 2, we introduce several related ideas, including graph embedding, trace ratio problem and relational learning, which are highly relevant to our work. In Section 3, we focus on our proposed method RFA, including the notation, the formulation and the iterative optimization method of RFA. Section 4 includes the proof of the convergence of RFA and we present how to extend RFA to a kernel version—KRFA. In Section 5, we compare our methods RFA and KRFA to other commonly used dimensionality reduction methods with extensive experiments, which demonstrate the effectiveness of our proposed methods. Finally, we summarize this paper in Section 6.

2. Related Work

In this section, we first briefly introduce some traditional dimensionality reduction methods, and then offer a detailed description about relevant ideas including graph embedding, trace ratio problem and relational learning, respectively. Finally, we specify how those ideas are used in this work.

2.1. Traditional Dimensionality Reduction Methods

Some basic ideas of traditional dimensionality reduction methods are presented in this subsection, such as PCA, LDA as well as several locality based manifold learning methods including LLE and LE. Advantages and drawbacks of these methods are also presented in this part.

2.1.1. PCA

The main idea of PCA [1] is to seek projection directions with maximal variances of the low-dimensional embeddings. It effectively extract and retain the principle components of the original data. However, as PCA is an unsupervised dimensionality reduction method, low-dimensional embeddings obtained from this method cannot perfectly maintain the discrimination between data of different classes.

2.1.2. LDA

LDA [7], well known as a supervised dimensionality reduction method, aims to seek projection directions to minimize the intraclass scattering and maximize the interclass scattering for the low-dimensional embeddings. However, for LDA, if the dimensionality of data is far greater than the data size, the intraclass scattering matrix may suffer from the singularity problem and thus it influences the solution of this dimensionality reduction algorithm. Furthermore, since the rank of the interclass scattering matrix is at most $C-1$, the number of available projection directions of LDA is at most $C-1$, where C is the number of classes.

2.1.3. Manifold Learning Methods

PCA and LDA are all global methods which use global information to project the original data into a subspace and obtain the low-dimensional data representations. However, for highly nonlinear data structure, these linear methods cannot learn the nonlinear relationships between data and thus the results are not ideal. By assuming that the high-dimensional data have a low-dimensional manifold structure, manifold learning algorithms can nonlinearly map the high-dimensional data onto their low-dimensional manifold. Among manifold learning methods, local geometric information-based methods, such as LLE, LE and local preserving projection (LPP) [32] are widely used. The ideas behind them are as follows.

LLE [12] preserves the linear reconstruction characteristics in a local neighborhood of each datum. Hence, the low-dimensional embeddings obtained by LLE presents the local geometrical structure of the data manifold. LE [13] preserves the similarities of the neighboring data points based on an adjacency matrix and a graph Laplacian matrix. However, for LLE and LE, as the nonlinear mapping function between the high-dimensional and low-dimensional spaces is not learned, we cannot easily obtain the low-dimensional representations of new data. To the end, LPP [32] performs a linear approximation of LE, and successfully overcomes its drawback as mentioned above.

2.2. Graph Embedding

In [14], Yan et al. show that some commonly used dimensionality reduction algorithms could be transformed into a unified framework despite their different motivations, and the unified framework is called graph embedding. This framework derives a low-dimensional feature space, which preserves the adjacency relationship between sample pairs. The general objective function of this framework is presented in Equation (1), where $\mathbf{W}$ denotes the similarity matrix of the undirected weighted graph $\mathbf{G}=\{\mathbf{X},\mathbf{W}\}$ and $\mathbf{B}$ is the constraint matrix defined to avoid a trivial solution of the objective function,

$$\begin{array}{c}\hfill {\mathbf{V}}^{\ast }=\underset{{\mathbf{V}}^T{\mathbf{S}}_B\mathbf{V}=\mathbf{I}}{\mathrm{argmin}}{\mathbf{V}}^T{\mathbf{S}}_W\mathbf{V}=\underset{\mathbf{V}}{\mathrm{argmin}}\frac{{\mathbf{V}}^T{\mathbf{S}}_W\mathbf{V}}{{\mathbf{V}}^T{\mathbf{S}}_B\mathbf{V}},\end{array}$$

(1)

where ${\mathbf{S}}_W$ and ${\mathbf{S}}_B$ are matrices constructed with respect to $\mathbf{X}$, $\mathbf{W}$ and $\mathbf{B}$, respectively.

We note that this unified framework graph embedding also provides a new idea for researchers to propose new dimensionality reduction algorithms. In particular, Yan et al. propose a novel dimensionality reduction method by defining an intrinsic graph which characterizes the intraclass compactness and a penalty graph which characterizes the interclass separability in the graph embedding framework, and call it marginal Fisher analysis (MFA).

2.3. Trace Ratio Problems

As presented in the above subsection, within the context of graph embedding, the dimensionality reduction methods can be viewed as trying to obtain the transformation matrix $\mathbf{W}$ that makes $Tr\left({\mathbf{W}}^T{\mathbf{S}}_p\mathbf{W}\right)$ maximum and $Tr\left({\mathbf{W}}^T{\mathbf{S}}_l\mathbf{W}\right)$ minimum. This is often formulated as a trace ratio optimization problem, that is ${max}_{\mathbf{W}}Tr\left({\mathbf{W}}^T{\mathbf{S}}_p\mathbf{W}\right)/Tr\left({\mathbf{W}}^T{\mathbf{S}}_l\mathbf{W}\right)$ [33]. Generally, there are two kinds of solutions for this problem: (1) Simplifying the problem into a ratio trace problem: ${max}_{W}Tr\left[{\left({\mathbf{W}}^T{S}_l\mathbf{W}\right)}^{-1}\left({\mathbf{W}}^T{S}_p\mathbf{W}\right)\right]$, then using generalized eigenvalue decomposition (GED) to obtain the transformation matrix $\mathbf{W}$; (2) Directly optimizing the objective function through an iterative procedure, with each step presented as a trace difference problem: $Tr\left[\left({\mathbf{W}}^T({\mathbf{S}}_p-{\lambda }^n{\mathbf{S}}_l)\mathbf{W}\right)\right]$. However, for the first solution, the optimization of ratio trace formulation may deviate from the original objective, which results in a closed-form but inexact solution and may subsequently lead to uncertainty in subsequent classification or clustering problems. For the second solution, Wang et al. [33] propose an efficient iterative procedure by solving the trace difference problem in each iterative step, named iterative procedure (ITR). It is proven that ITR could converge to the optimal solution and solve the trace ratio problem. With the orthogonal assumption on the projection matrix, objective function of the ITR optimization can be defined as

$$\begin{array}{c}\hfill \mathbf{W}=\underset{{\mathbf{W}}^T\mathbf{W}=\mathbf{I}}{\mathrm{argmax}}\frac{\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{S}}_p\mathbf{W}\right)}{\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{S}}_t\mathbf{W}\right)}.\end{array}$$

(2)

In [34], Nie et al. address the graph-based feature selection framework using the iterative process of the trace ratio problem. In [35], Zhong et al. analyze the iterative procedures for the trace ratio problem and prove necessary and sufficient conditions of the existence of the optimal solution of trace ratio problems, which are that there is a sequence $\{{\lambda }_1^{\ast },{\lambda }_2^{\ast },\dots ,{\lambda }_n^{\ast }\}$ that converges to ${\lambda }^{\ast }$ as $n\to +\infty$, where ${\lambda }^{\ast }$ is the optimal value of Equation (2). Based on these previous works, we also formulate RFA as a trace ratio problem and theoretically prove the convergence of its optimization algorithm.

2.4. Relational Learning

In many real-world applications, data generally share some kinds of relations, such as geometrical or semantic similarity, links or citations. This relation information encoded inside data provides valuable evidence for some issues, such as classification and retrieval. To the end, relational learning is generally integrated into the representation learning models.

In [36], Duin et al. prove that it is possible to use only proximity measure (distances or similarities) to represent the samples rather than mapping the feature vectors to the low-dimensional space. In addition, they propose a proximity description-based dimensionality reduction method called relational discriminant analysis (RDA) in [24]. Instead of data, RDA uses similarities to a subset of objects in the training data as features. In this case, dimensionality reduction can be conducted either by selection methods (such as random selection [37], systematic selection [38]), or by feature extraction methods (such as multi-dimensional scaling [39], Sammon mapping [40] and Niemann mapping [41]).

In [25], Li et al. model the covariance of data with the relationships between instances and propose a Gaussian latent variable model which successfully integrates relational information into the dimensionality reduction process, called probabilistic relational PCA (PRPCA). In PRPCA, relational information is defined by the relevance between data samples. We take the scientific paper citation as an example. If there is a quoting between the papers, it means that these papers most likely have similar topics. To take the inter-influence between cited papers into account, Li et al. further construct a matrix $\mathsf{\Phi }={\mathsf{\Delta }}^{-1}$, which satisfy the condition that similar instances often have a lower probability density at the latent space. To the end, PRPCA, based on the relational covariance $\mathsf{\Phi }$, successfully applies the relational information to the dimensionality reduction algorithms.

Relational learning is also commonly used for data mining, information retrieval and other machine learning-related applications. Paccanaro et al. [42] propose a method, called linear relational embedding, for the distributed representations of data, where data consist of the relationship of concepts. Wang et al. [43] utilize the characteristic that existing relations between items are often useful in recommendation systems and propose a model called relational collaborative topic regression (RCTR), which expand the traditional CTR model by integrating feedback information, item content information and relational information. Xuan et al. [44] propose a nonparametric relational topic model using stochastic processes instead of fixed-dimensional probability distributions.

Based on the classical works mentioned above, we propose a general and effective dimensionality reduction framework named relational Fisher analysis (RFA). This framework uses graph embedding [14] as theoretical foundation and integrates the relational information [24,25] encoded inside data into the dimensionality reduction process. Besides the intrinsic graph and the penalty graph as defined using graph embedding, we further construct a relational graph based on the existing relationships between data, which enables the desired low-dimensional space to preserve the intrinsic information, reduce the penalty information and further learn and preserve the relational information among the data samples. In addition, through the derivation and equivalent transformation operations, the objective function of our proposed method can be transformed into the trace ratio form for optimization. Based on a systematic analysis of two optimization method for trace ratio problems [33], we propose a novel iterative algorithm which uses the value of the trace ratio as criterion for the algorithmic convergence. In addition, by further introducing the ITR-Score defined in [34] into the iterative process, optimal projection directions are learned, which improves the effectiveness of the proposed RFA model.

3. Methodology

In this section, we first present some notations used in our work. The iterative steps, the optimization method and the proof of global convergence of RFA are then introduced in detail.

3.1. Notation

Matrices are represented in uppercase bold letters, for instance, $\mathbf{A}$, while vectors are represented in boldface lowercase letters, for instance, $\mathbf{a}$, and ${\mathbf{a}}_i$ is the ith element of $\mathbf{a}$. ${\mathbf{A}}_{i\ast }$ and ${\mathbf{A}}_{\ast j}$ denote the ith row and jth column of a matrix $\mathbf{A}$; therefore, the element of the ith row and jth column of the matrix is represented by ${\mathbf{A}}_{ij}$. The trace of $\mathbf{A}$ is defined by $\mathrm{tr}\left(\mathbf{A}\right)$ and the transpose of $\mathbf{A}$ is defined by ${\mathbf{A}}^T$. In addition, $|{\mathbf{A}}_{ij}|$ is the absolute value of ${\mathbf{A}}_{ij}$, ${\parallel \mathbf{A}\parallel }_F$ is the Frobenius norm of $\mathbf{A}$. If $\mathbf{A}$ is positive definite, we have $\mathbf{A}\succ 0$, while it is positive semi-definite (psd), we have $\mathbf{A}⪰0$.

In a learning task that contains multiple classes of data, we usually have dataset $\{\{{\mathbf{X}}_{\ast i},{\mathbf{y}}_i\}\in {\Re }^D\times {\Re }^1,i=1,2,\dots ,N\}$, where each ${\mathbf{X}}_{\ast i}$ represents a sample and ${\mathbf{y}}_i\in \{1,2,\dots ,C\}$ is the class of that sample, $C⩾2$ is the total number of classes and N is the total number of samples. For a linear dimensionality reduction task, we hope to find a projection matrix $\mathbf{W}$ and obtain the d-dimensional representation ${\mathbf{Z}}_{\ast i}$ of ${\mathbf{X}}_{\ast i}$ by ${\mathbf{Z}}_{\ast i}=\mathbf{W}\ast {\mathbf{X}}_{\ast i}$, where ${\mathbf{Z}}_{\ast i}\in {\Re }^d,i=1,2,\dots ,N$, $d<D$ is the dimensionality of the output.

3.2. Formulation of RFA

As discussed in Section 2, graph embedding has already been proven to be a general framework for dimensionality reduction. However, there are two shortages of graph embedding. First, graph embedding does not obtain and preserve the relational information between data. Second, the graph embedding framework needs to be solved by generalized eigenvalue decomposition, which is only an approximate approach. Inspired by graph embedding, we propose a new dimensionality reduction framework called RFA, which integrates relationships among data into the dimensionality reduction model and can alleviate the two problems mentioned above. Formulation of the proposed RFA is described as follows.

We use $\mathbf{R}\in {\Re }^{N\times N}$ to denote the relational matrix. The dimensionality reduction framework RFA is modeled as

$$\begin{array}{ccc}\hfill & & \mathcal{L}=\underset{\mathbf{W}}{min}\frac{\mathrm{tr}\left({\mathbf{W}}^T\mathbf{X}{\mathbf{L}}_I{\mathbf{X}}^T\mathbf{W}\right)}{\mathrm{tr}\left({\mathbf{W}}^T\mathbf{X}{\mathbf{L}}_P{\mathbf{X}}^T\mathbf{W}\right)}+\lambda \mathrm{tr}\left({\mathbf{W}}^T\mathbf{X}\mathbf{R}{\mathbf{X}}^T\mathbf{W}\right),\hfill \end{array}$$

(3)

where ${\mathbf{L}}_I$ and ${\mathbf{L}}_P$ define the intrinsic and penalty graphs, respectively, and $\lambda ⩾0$ is a hyperparameter. Specifically, we only consider undirected graph and assume that ${\mathbf{L}}_I$, ${\mathbf{L}}_P$ and $\mathbf{R}$ are symmetric and psd. Based on this formulation and these assumptions, the generality of RFA can be explained from the following two points:

(1) If $\lambda =0$, our algorithm can be simplified to a basic graph embedding model, so that some commonly used dimensionality reduction algorithms can be regarded as special cases of RFA;

(2) Otherwise, if $\mathcal{L}$ only contains relational information, RFA can be considered to use relational learning to reduce the dimensionality of data. For instance, the MDS algorithm is a special RFA algorithm under this condition.

3.3. Optimization of RFA

We reformulate Problem (3) as

$$\begin{array}{c}\hfill \mathcal{L}=\underset{\mathbf{W}}{min}\frac{\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{S}}_I\mathbf{W}\right)}{\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{S}}_P\mathbf{W}\right)}+\lambda \mathrm{tr}\left({\mathbf{W}}^T{\mathbf{S}}_R\mathbf{W}\right),\end{array}$$

(4)

where ${\mathbf{S}}_I=\mathbf{X}{\mathbf{L}}_I{\mathbf{X}}^T$, ${\mathbf{S}}_P=\mathbf{X}{\mathbf{L}}_P{\mathbf{X}}^T$ and ${\mathbf{S}}_R=\mathbf{X}\mathbf{R}{\mathbf{X}}^T$.

As $\mathbf{R}$ is psd, ${\mathbf{S}}_R$ is as well. We suppose ${\mathbf{S}}_R=\mathbf{U}\mathsf{\Lambda }{\mathbf{U}}^T$. We have

$$\begin{array}{c}\hfill \mathcal{L}=\underset{\mathbf{V}}{min}\frac{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_I\mathbf{V}\right)}{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)}+\lambda \mathrm{tr}\left({\mathbf{V}}^T\mathbf{V}\right),\end{array}$$

(5)

where $\mathbf{W}=\mathbf{U}{\mathsf{\Lambda }}^{-1/2}\mathbf{V}$, ${\tilde{\mathbf{S}}}_I={\mathsf{\Lambda }}^{-1/2}{\mathbf{U}}^T{\mathbf{S}}_I\mathbf{U}{\mathsf{\Lambda }}^{-1/2}$ and ${\tilde{\mathbf{S}}}_P={\mathsf{\Lambda }}^{-1/2}{\mathbf{U}}^T{\mathbf{S}}_P\mathbf{U}{\mathsf{\Lambda }}^{-1/2}$.

Furthermore,

$$\begin{array}{c}\hfill \frac{\partial \mathcal{L}}{\partial \mathbf{V}}=\frac{2\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right){\tilde{\mathbf{S}}}_I\mathbf{V}-2\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_I\mathbf{V}\right){\tilde{\mathbf{S}}}_P\mathbf{V}}{\mathrm{tr}{\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)}^2}+2\lambda \mathbf{V}.\end{array}$$

(6)

We let $\frac{\partial \mathcal{L}}{\partial \mathbf{V}}=\mathbf{0}$. We have

$$\begin{array}{c}\hfill \frac{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right){\tilde{\mathbf{S}}}_I\mathbf{V}-\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_I\mathbf{V}\right){\tilde{\mathbf{S}}}_P\mathbf{V}}{\mathrm{tr}{\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)}^2}=-\lambda \mathbf{V}.\end{array}$$

(7)

Equation (7) can be rewritten as

$$\begin{array}{c}\hfill ({\tilde{\mathbf{S}}}_I-\frac{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_I\mathbf{V}\right)}{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)}{\tilde{\mathbf{S}}}_P)\mathbf{V}=-\lambda \mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)\mathbf{V}.\end{array}$$

(8)

We let $\eta =\frac{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_I\mathbf{V}\right)}{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)}$ and $\tilde{\lambda }=-\lambda \mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)$. We obtain

$$\begin{array}{c}\hfill ({\tilde{\mathbf{S}}}_I-\eta {\tilde{\mathbf{S}}}_P)\mathbf{V}=\tilde{\lambda }\mathbf{V}.\end{array}$$

(9)

It can be seen from Equation (9) that the columns of matrix $\mathbf{V}$ are the eigenvectors of ${\tilde{\mathbf{S}}}_I-\eta {\tilde{\mathbf{S}}}_P$, where $\eta$ is a parameter related to $\mathbf{V}$.

Without loss of generality, we assume ${\mathbf{V}}^T\mathbf{V}={\mathbf{I}}_d$, where ${\mathbf{I}}_d$ is an identity matrix. Hence, we have the following constrained trace ratio problem [33,45]:

$$\begin{array}{c}\hfill \tilde{\mathcal{L}}=\underset{{\mathbf{V}}^T\mathbf{V}={\mathbf{I}}_d}{min}\frac{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_I\mathbf{V}\right)}{\mathrm{tr}\left({\mathbf{V}}^T{\tilde{\mathbf{S}}}_P\mathbf{V}\right)}.\end{array}$$

(10)

Problem (10) can be solved with the iterative method similar to that in [33,45]. The specific steps are as follows:

(1) Removing the null space of ${\mathbf{S}}_t={\tilde{\mathbf{S}}}_I+{\tilde{\mathbf{S}}}_P$ [46]. We assume that ${\mathbf{S}}_t=\tilde{\mathbf{U}}\tilde{\mathsf{\Lambda }}{\tilde{\mathbf{U}}}^T$, where $\tilde{\mathsf{\Lambda }}$ is a diagonal matrix and $\tilde{\mathbf{U}}$ contains the eigenvectors of ${\mathbf{S}}_t$ corresponding to nonzero eigenvalues. Therefore, Formula (10) can be transformed to

$$\begin{array}{c}\hfill \tilde{\mathcal{L}}=\underset{{\tilde{\mathbf{W}}}^T\tilde{\mathbf{W}}={\mathbf{I}}_d}{min}\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_I\tilde{\mathbf{W}}\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_P\tilde{\mathbf{W}}\right)},\end{array}$$

(11)

where $\mathbf{V}=\tilde{\mathbf{U}}\tilde{\mathbf{W}},\tilde{\mathbf{W}}\in {\Re }^{r\times d}$, ${\widehat{\mathbf{S}}}_I={\tilde{\mathbf{U}}}^T{\tilde{\mathbf{S}}}_I\tilde{\mathbf{U}}$ and ${\widehat{\mathbf{S}}}_P={\tilde{\mathbf{U}}}^T{\tilde{\mathbf{S}}}_P\tilde{\mathbf{U}}$. We can further rewrite the problem (11) as

$$\begin{array}{c}\hfill \tilde{\mathcal{L}}=\underset{{\tilde{\mathbf{W}}}^T\tilde{\mathbf{W}}={\mathbf{I}}_d}{min}\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_I\tilde{\mathbf{W}}\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_T\tilde{\mathbf{W}}\right)},\end{array}$$

(12)

where ${\widehat{\mathbf{S}}}_T={\tilde{\mathbf{U}}}^T({\tilde{\mathbf{S}}}_I+{\tilde{\mathbf{S}}}_P)\tilde{\mathbf{U}}={\widehat{\mathbf{S}}}_I+{\widehat{\mathbf{S}}}_P$. Since ${\widehat{\mathbf{S}}}_T$ is positive definite, for any orthonormal matrix $\tilde{\mathbf{W}}$, Problem (12) satisfies that the denominator is positive.

(2) Efficient iterative optimization. The original trace ratio problem (12) can be rewritten as a trace difference problem:

$$\begin{array}{c}\hfill {\tilde{\mathbf{W}}}^{\ast }=\underset{{\tilde{\mathbf{W}}}^T\tilde{\mathbf{W}}={\mathbf{I}}_d}{\mathrm{argmin}}\mathrm{tr}\left({\tilde{\mathbf{W}}}^T({\widehat{\mathbf{S}}}_I-\tilde{\eta }{\widehat{\mathbf{S}}}_T)\tilde{\mathbf{W}}\right),\end{array}$$

(13)

where $\tilde{\eta }$ is a parameter which can be calculated in the iterative process. In the iterative process, we first randomly initialize the target matrix $\tilde{\mathbf{W}}$ to be an arbitrary orthogonal matrix as ${\tilde{\mathbf{W}}}_0\in {\Re }^{r\times d}$, and then calculate ${\tilde{\eta }}_0=\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}_0^T{\widehat{\mathbf{S}}}_I{\tilde{\mathbf{W}}}_0\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}_0^T{\widehat{\mathbf{S}}}_T{\tilde{\mathbf{W}}}_0\right)}$. By using the calculated ${\tilde{\eta }}_0$, we can obtain ${\tilde{\mathbf{W}}}_1$ by solving Problem (13). In the end, through several iterations, we obtain ${\tilde{\mathbf{W}}}_T$, where T is the number of iterations and it’s satisfied $|{\tilde{\eta }}_T-{\tilde{\eta }}_{T-1}|<ϵ$ ($ϵ={10}^{-5}$ is used in our experiments). Then, ${\tilde{\mathbf{W}}}_T$ is the optimal solution of Problem (12). In next section, we prove that RFA owns the global convergence. In order to improve the effectiveness of our method, we select some superior projection directions for each ${\tilde{\mathbf{W}}}_t$, as performed in [47]. Our selection criterion is

$$\begin{array}{c}\hfill {\tilde{\mathbf{W}}}_t=\underset{\tilde{\mathbf{W}}\in \Phi }{\mathrm{argmin}}\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_I\tilde{\mathbf{W}}\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_T\tilde{\mathbf{W}}\right)},\end{array}$$

(14)

where $\Phi$ is a set of $r\times d$ matrices with columns formed by eigenvectors of ${\widehat{\mathbf{S}}}_I-{\tilde{\eta }}_{t-1}{\widehat{\mathbf{S}}}_T$. We use the eigenvectors corresponding to d smallest ITR-score [48] to initialize the selection.

Algorithm 1 specifically describes the iterative procedure of Problem (12).

Algorithm 1 Optimization of Problem (12)

1: Initialization: Initialize $\tilde{\mathbf{W}}$ as an orthonormal matrix ${\tilde{\mathbf{W}}}_0\in {\Re }^{r\times d}$ and Let ${\tilde{\eta }}_0=0$. 2: Iterations: 3: for $t=0$ to MaxIt do 4:      (1) Compute ${\tilde{\eta }}_t$ as $$\begin{array}{c}\hfill {\tilde{\eta }}_t=\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}_{t-1}^T{\widehat{\mathbf{S}}}_I{\tilde{\mathbf{W}}}_{t-1}\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}_{t-1}^T{\widehat{\mathbf{S}}}_T{\tilde{\mathbf{W}}}_{t-1}\right)}.\end{array}$$ 5:      (2) Solve the eigenvalue decomposition problem: $$\begin{array}{c}\hfill ({\widehat{\mathbf{S}}}_I-\tilde{\eta }{\widehat{\mathbf{S}}}_T){\widehat{\mathbf{W}}}_{\ast i}={\tilde{\lambda }}_i{\widehat{\mathbf{W}}}_{\ast i}.\end{array}$$ 6:      (3) Compute $$\begin{array}{c}\hfill s_i=\frac{\mathrm{tr}\left({\widehat{\mathbf{W}}}_{\ast i}^T{\widehat{\mathbf{S}}}_I{\widehat{\mathbf{W}}}_{\ast i}\right)}{\mathrm{tr}\left({\widehat{\mathbf{W}}}_{\ast i}^T{\widehat{\mathbf{S}}}_T{\widehat{\mathbf{W}}}_{\ast i}\right)},i=1,\dots ,r.\end{array}$$ 7:      (4) Use ${\left\{{\widehat{\mathbf{W}}}_{\ast i}\right\}}_{i=1}^r$ to initialize ${\tilde{\mathbf{W}}}_t$ and solve the following problem: $$\begin{array}{c}\hfill {\tilde{\mathbf{W}}}_t^{\ast }=\underset{\tilde{\mathbf{W}}\in \Phi }{\mathrm{argmin}}\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_I\tilde{\mathbf{W}}\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_T\tilde{\mathbf{W}}\right)},\end{array}$$ where $\Phi$ is a set of matrices with columns formed by ${\left\{{\widehat{\mathbf{W}}}_{\ast i}\right\}}_{i=1}^r$. 8:      if $|{\tilde{\eta }}_t-{\tilde{\eta }}_{t-1}|<ϵ$ ($ϵ={10}^{-5}$ is used in our experiments) then 9:           Break. 10:      end if 11: end for 12: Output:${\tilde{\mathbf{W}}}_t$.

4. Global Convergence of RFA and Extensions

In this section, we first prove that RFA can converge to the global optimal solution, and then, we apply the kernel trick to RFA for nonlinear relational dimensionality reduction.

4.1. Global Convergence of RFA

Theorem 1 states the convergence of RFA.

Theorem 1.

The RFA algorithm converges to a global optimal solution of Problem (3).

Proof. 

Considering that all the formulas, from Problem (3) to Problem (12) in the previous section, are all transformed equivalently, we only prove the convergence of Problem (12) here. Specifically, we show that $\tilde{\mathcal{L}}$ of Problem (12) has a lower bound, which gradually decreases with the iterative process.

We can easily see that for any $\tilde{\mathbf{W}}$, it satisfies that $0\le \frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_I\tilde{\mathbf{W}}\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}^T{\widehat{\mathbf{S}}}_T\tilde{\mathbf{W}}\right)}\le 1$, so that the lower bound of $\tilde{\mathcal{L}}$ is 0.

Next, we prove that the objective value of Problem (12) gradually decreases with the iterative process of RFA. Defining

$$\begin{array}{c}\hfill {\tilde{\eta }}_t=\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}_{t-1}^T{\widehat{\mathbf{S}}}_I{\tilde{\mathbf{W}}}_{t-1}\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}_{t-1}^T{\widehat{\mathbf{S}}}_T{\tilde{\mathbf{W}}}_{t-1}\right)},\end{array}$$

(15)

we then have

$$\begin{array}{c}\hfill \mathrm{tr}\left({\tilde{\mathbf{W}}}_{t-1}^T({\widehat{\mathbf{S}}}_I-{\tilde{\eta }}_t{\widehat{\mathbf{S}}}_T){\tilde{\mathbf{W}}}_{t-1}\right)=0.\end{array}$$

(16)

However,

$$\begin{array}{c}\hfill {\tilde{\mathbf{W}}}_t=\underset{{\tilde{\mathbf{W}}}^T\tilde{\mathbf{W}}={\mathbf{I}}_d}{\mathrm{argmin}}\mathrm{tr}\left({\tilde{\mathbf{W}}}^T({\widehat{\mathbf{S}}}_I-{\tilde{\eta }}_t{\widehat{\mathbf{S}}}_T)\tilde{\mathbf{W}}\right).\end{array}$$

(17)

Therefore,

$$\begin{array}{c}\hfill \mathrm{tr}\left({\tilde{\mathbf{W}}}_t^T({\widehat{\mathbf{S}}}_I-{\tilde{\eta }}_t{\widehat{\mathbf{S}}}_T){\tilde{\mathbf{W}}}_t\right)\le 0,\end{array}$$

(18)

and

$$\begin{array}{c}\hfill {\tilde{\eta }}_{t+1}=\frac{\mathrm{tr}\left({\tilde{\mathbf{W}}}_t^T{\widehat{\mathbf{S}}}_I{\tilde{\mathbf{W}}}_t\right)}{\mathrm{tr}\left({\tilde{\mathbf{W}}}_t^T{\widehat{\mathbf{S}}}_T{\tilde{\mathbf{W}}}_t\right)}\le {\tilde{\eta }}_t.\end{array}$$

(19)

Thus, we prove that ${\tilde{\eta }}_t$ gradually decreases with the iterative process, and Theorem 1 holds. □

According to Theorem 1, we can see that RFA can converge to the global optimal solution. In addition, for given intrinsic and penalty graphs and the relational matrix, the computational complexity of the RFA algorithm is only $\Theta \left(N\right)$, where N is the total number of data. That also illustrates the efficiency of our algorithm.

4.2. Kernel Extension

In this subsection, we apply kernel trick to our RFA method and present the kernel RFA (KRFA) method, which can be used to nonlinear dimensionality reduction problems.

The KRFA optimization problem is basically the same as Problem (3), except that the data point ${\mathbf{X}}_{\ast i}$ needs to be mapped to a reproducing kernel Hilbert space and obtain $\varphi \left({\mathbf{X}}_{\ast i}\right)$, where $\varphi (·)$ denotes the mapping function. In addition, the corresponding intrinsic and penalty graphs and the relational matrix should also be mapped to the reproducing kernel Hilbert space. The kernel function $\mathbf{K}({\mathbf{X}}_{\ast i},{\mathbf{X}}_{\ast j})=\varphi {\left({\mathbf{X}}_{\ast i}\right)}^T\varphi \left({\mathbf{X}}_{\ast j}\right)$.

We suppose the projection matrix $\mathbf{W}=\mathsf{\Phi }\mathsf{\Gamma }$. We have ${\mathbf{W}}^T\mathsf{\Phi }={\mathsf{\Gamma }}^T\mathbf{K}$.

Normalizing and centering the data in the high-dimensional feature space, we substitute $\mathbf{K}$ with

$$\widehat{\mathbf{K}}=\mathbf{K}-{\mathbf{1}}_N\mathbf{K}-\mathbf{K}{\mathbf{1}}_N+{\mathbf{1}}_N\mathbf{K}{\mathbf{1}}_N.$$

(20)

In this way, the learning task of RFA is able to be described as

$$\begin{array}{c}\hfill {\mathcal{L}}_{\widehat{\mathbf{K}}}=\underset{\mathsf{\Gamma }}{min}\frac{\mathrm{tr}\left({\mathsf{\Gamma }}^T\widehat{\mathbf{K}}{\tilde{\mathbf{L}}}_I{\widehat{\mathbf{K}}}^T\mathsf{\Gamma }\right)}{\mathrm{tr}\left({\mathsf{\Gamma }}^T\widehat{\mathbf{K}}{\tilde{\mathbf{L}}}_P{\widehat{\mathbf{K}}}^T\mathsf{\Gamma }\right)}+\lambda \mathrm{tr}\left({\mathsf{\Gamma }}^T\widehat{\mathbf{K}}\tilde{\mathbf{R}}{\widehat{\mathbf{K}}}^T\mathsf{\Gamma }\right),\end{array}$$

(21)

where ${\tilde{\mathbf{L}}}_I$, ${\tilde{\mathbf{L}}}_P$ and $\tilde{\mathbf{R}}$ are the intrinsic graph, the penalty graph and the relational matrix, respectively. The distance of the samples in the reproducing kernel Hilbert space can be calculated by

$$D(x_i,x_j)=\sqrt{\mathbf{K}(x_i,x_i)+\mathbf{K}(x_i,x_j)-2\mathbf{K}(x_i,x_j)}.$$

(22)

With reference to the derivation and transformation procedure of RFA, KRFA can be eventually transformed into

$${\tilde{\mathcal{L}}}_{\mathbf{K}}=\underset{{\mathbf{V}}_{\mathbf{K}}^T{\mathbf{V}}_{\mathbf{K}}={\mathbf{I}}_d}{min}\frac{\mathrm{tr}\left({\mathbf{V}}_{\mathbf{K}}^T{\tilde{\mathbf{S}}}_I^{\mathbf{K}}{\mathbf{V}}_{\mathbf{K}}\right)}{\mathrm{tr}\left({\mathbf{V}}_{\mathbf{K}}^T{\tilde{\mathbf{S}}}_P^{\mathbf{K}}{\mathbf{V}}_{\mathbf{K}}\right)}.$$

(23)

As KRFA’s optimization process is similar to RFA, we still use iterative methods to solve it.

5. Experiments

In this section, extensive experiments are conducted to validate the effectiveness of our RFA and KRFA methods. For the linear case, we conduct experiments on document analysis, handwritten digits recognition, face recognition and webpage classification problems. For KRFA, we test its performance on several benchmark datasets. The results of comparative experiments are presented below.

5.1. Performance of RFA

To evaluate the performance of RFA, we selected several related dimensionality reduction methods for comparison with RFA. These methods were LDA, MFA, RDA, and PRPCA, respectively. Among them, LDA, MFA and RDA are supervised methods, while PRPCA is unsupervised. We note that, due to the flexibility in the design of the relational matrix, RFA could be either global or local. We describe this detail in the following part.

We followed MFA to construct the intrinsic and penalty graphs [14]. The numbers of nearest neighbors for constructing the intrinsic graph ($k_i$) and the penalty graph ($k_p$) were set to 5 and 20, respectively, for all the datasets.

We applied RFA to document understanding, face recognition and several other recognition tasks. In order to test the performance of RFA on document recognition tasks, we used the Ibn Sina ancient Arabic document dataset [49], USPS handwritten digits dataset (http://www.cs.nyu.edu/∼roweis/data.html, accessed on 19 October 2023), two handwritten digits datasets (Optdigits and Pendigits) and one English letter dataset (Letter) from the UCI machine learning repository [50]. For face recognition tasks, we used two face datasets [51,52,53,54], the CMU PIE (http://www.face-rec.org/databases/, accessed on 19 October 2023) and the YaleB (http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html, accessed on 19 October 2023) datasets. At the same time, we used some UCI datasets, including Shuttle, Thyroid, Vowel and Waveform21, to evaluate RFA.

In our experiments, we used the classical graph Laplacian matrix, $\mathbf{L}=\mathbf{D}-\mathbf{M}$, to define the relational matrix, whose weight matrix $\mathbf{M}$ is shown as below:

$${\mathbf{M}}_{ij}=\left\{\begin{array}{c}1,\mathrm{if}{\mathbf{x}}_i\in {\mathcal{N}}_k\left({\mathbf{x}}_j\right)\mathrm{or}{\mathbf{x}}_j\in {\mathcal{N}}_k\left({\mathbf{x}}_i\right),\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.$$

(24)

where ${\mathcal{N}}_k\left({\mathbf{x}}_i\right)$ is the set consisting of the k nearest neighborForof ${\mathbf{x}}_i$. For each dataset, the value of k was selected based on 5-fold cross-validation. Moreover, $\mathbf{D}$ is the diagonal degree matrix with ${\mathbf{D}}_{ii}={\sum }_j{\mathbf{M}}_{ij}$.

We note that the Laplacian weight matrix formed by the above formula contains the relationship information between the sample and a certain number of its neighbors, which allows the iintegration of the local relationships between data into the supervised representation learning algorithm. At the same time, as a general model, the relationship matrix $\mathbf{R}$ in RFA can be of various forms. For example, $\mathbf{R}$ can be the centralization matrix $\mathbf{H}={\mathbf{I}}_N-\frac{1}{N}\mathbf{1}{\mathbf{1}}^T$, where N denotes the data size and $\mathbf{1}$ is a column vector of length N with all ones.

For the PRPCA algorithm, we performed this experiment using the codes provided by the authors. For the RDA algorithm, we randomly selected the prototypes [24]. For algorithms other than RDA, we used the 1-nearest neighbor classifier to evaluate the classification performance of them.

5.1.1. Comparison on Hand-Writing Datasets

We first tested the performance of RFA on the hand-writing datasets. For clarity, the details of the used datasets are shown in

Table 1. Statistics of the used datasets.

Dataset	Classes	Samples	Dimensions
Ibn Sina	174	20,688	200
USPS	10	9298	256
Pendigits	10	5620	16
Optdigits	10	10,992	64
Letter	26	20,000	16

.

Ibn Sina dataset: This dataset [55] is an ancient manuscript dataset, one image of which is shown in Figure 1, and we tried to identify the Arabic subwords on this dataset. In the experiment, we used a 50-page manuscript as the training set and 10 pages as the test set. We extracted the square-root velocity (SRV) representation [56] of the Arabic subwords. Then, we removed the outlier classes, including the classes that had less than 10 samples. Finally, we obtained a 174-class Arabic subword dataset with 17,543 samples for training and 3125 samples for test.

In this experiment, we set the number of the nearest neighbors – k of the RFA to 8 and compared RFA with the LDA and MFA dimensionality reduction algorithms. The classification accuracies after using these three dimensionality reduction methods to map the data to different dimensionalities are shown in Figure 2. We can see that RFA is far better than the LDA algorithm and slightly better than the MFA algorithm. At the same time, when the dimensionality is from 50 to $C-1$ (C is the number of classes), the correct rate of the MFA algorithm has a certain fluctuation and tends to decline, while the classification performance of our RFA algorithm is relatively stable, which means that our algorithm is more robust than MFA.

USPS dataset: The USPS is an U.S. post handwritten digits dataset that contains 7291 training data and 2007 test data from 10 classes and the dimensionality of the data features is 256. Some handwritten digits in the USPS dataset are shown in Figure 3.

In this experiment, we set the number of the nearest neighbors – k of RFA to 24. The classification accuracies obtained by RFA and the compared algorithms are shown in Figure 4. Because the LDA subspace has a maximum of $C-1$ dimension (C is the number of classes), LDA is only presented with a black star in the figure. We can see that when the dimension is 9, RFA obtains a comparative results with LDA and MFA. When the dimension increases, the results of RFA are always optimal. At the same time, the classification accuracy obtained by RDA is low.

Figure 5 presents 3D visualization of the learned data representations by RFA, which shows the effect of RFA to obtain better classification boundaries between the classes. In Figure 6, we show the 2D projections of data learned by both RFA and MFA, to further show the effectiveness of RFA. It is easy to see that the samples processed by RFA are less likely to overlap at the boundary, indicating that compared to MFA, RFA preserves more properties that help distinguish the samples.

In addition, the robustness of RFA is tested with respect to $k_i$ and $k_p$ (used to construct the intrinsic and penalty graphs). From Figure 4, it can be seen that RFA obtained the best result when the subspace dimension was 35, with parameter settings $k_i$ = 5, $k_p$ = 20 and k = 24. We fixed k and one of $k_i$ and $k_p$ to obtain the results when another parameter took different values. Figure 7 and Figure 8 show that RFA is very robust.

We also selected three document recognition-related datasets from the UCI machine learning repository to further test the effectiveness of RFA. They are Optdigits, Pendigits and Letter. Optdigits was preprocessed by NIST [57] programs to obtain 5620 instances in 8 × 8 dimensions. The Pendigits dataset contains a large number of preprocessed 16-dimensional samples written by 44 different authors, including 7494 training examples and 3498 test samples. Letter consists of 20,000 handwritten characters written by 20 fonts from 26 capital letters in the English alphabet. We used the 5-fold cross-validation for these experiments.

Table 2. Classification results obtained on the Optdigits dataset. The best results are highlighted in boldface.

Dimension	LDA	MFA	PRPCA	RDA	RFA
9	0.9219 ± 0.0117	0.9657 ± 0.0020	0.9687 ± 0.0053	0.7407 ± 0.0048	0.9662 ± 0.0017
15	-	0.9845 ± 0.0039	0.9690 ± 0.0087	0.7407 ± 0.0017	0.9875 ± 0.0031
20	-	0.9836 ± 0.0041	0.9696 ± 0.0075	0.7399 ± 0.0054	0.9891 ± 0.0025
25	-	0.9849 ± 0.0030	0.9696 ± 0.0071	0.7425 ± 0.0042	0.9891 ± 0.0035
30	-	0.9819 ± 0.0062	0.9698 ± 0.0071	0.7415 ± 0.0053	0.9875 ± 0.0048
35	-	0.9795 ± 0.0051	0.9703 ± 0.0067	0.7404 ± 0.0066	0.9879 ± 0.0036
40	-	0.9783 ± 0.0058	0.9710 ± 0.0079	0.7402 ± 0.0056	0.9875 ± 0.0032
50	-	0.9781 ± 0.0026	0.9701 ± 0.0071	0.7407 ± 0.0043	0.9877 ± 0.0023

,

Table 3. Classification results obtained on the Pendigits dataset.

Dimension	LDA	MFA	PRPCA	RDA	RFA
9	0.9843 ± 0.0020	0.9929 ± 0.0013	0.9797 ± 0.0027	0.9203 ± 0.0046	0.9939 ± 0.0006
13	-	0.9940 ± 0.0026	0.9604 ± 0.0028	0.8856 ± 0.0052	0.9950 ± 0.0018
15	-	0.9945 ± 0.0018	0.9887 ± 0.0008	0.9212 ± 0.0050	0.9962 ± 0.0013

and

Table 4. Classification results obtained on the Letter dataset.

Dimension	LDA	MFA	PRPCA	RDA	RFA
9	0.9108 ± 0.0070	0.9570 ± 0.0014	0.9230 ± 0.0018	0.3699 ± 0.0064	0.9580 ± 0.0018
13	0.9108 ± 0.0070	0.9725 ± 0.0022	0.9604 ± 0.0028	0.3727 ± 0.0065	0.9753 ± 0.0022
15	0.9575 ± 0.0023	0.9596 ± 0.0018	0.9567 ± 0.0022	0.3715 ± 0.0087	0.9695 ± 0.0011

show the classification accuracy and standard deviation obtained on these three datasets and the boldface results are the best ones. We can see that RFA performs consistently better than other compared methods.

5.1.2. Comparison on Face Datasets

Here, we tested RFA on the face recognition problems. The PIE and YaleB datasets were used. The details of these two datasets are shown in

Table 5. Statistics of the face recognition datasets.

Dataset	Classes	Samples	Feature Dimension
PIE	68	11,554	1024
YaleB	38	2414	1024

. For the corresponding experimental settings, we set the number of the nearest neighbors – k on the PIE dataset to 8 and k on the YaleB dataset to 18.

Considering that RDA cannot perform well on multi-class classification problems, we used LPP instead as a compared method in these experiments. We used the 5-fold cross-validation to decide the value of parameter k for graph construction in LPP. We performed the experiments in different low-dimensional spaces on the PIE and YaleB datasets, and the experimental results are shown in

Table 6. Classification results obtained on the PIE dataset.

Dimension	LDA	MFA	PRPCA	LPP	RFA
9	0.9021 ± 0.0070	0.9173 ± 0.0045	0.7736 ± 0.0113	0.8156 ± 0.0089	0.9198 ± 0.0026
15	0.9430 ± 0.0038	0.9526 ± 0.0040	0.8810 ± 0.0063	0.8930 ± 0.0054	0.9540 ± 0.0021
20	0.9528 ± 0.0031	0.9592 ± 0.0027	0.9089 ± 0.0062	0.9122 ± 0.0044	0.9603 ± 0.0030
25	0.9597 ± 0.0040	0.9617 ± 0.0028	0.9224 ± 0.0056	0.9262 ± 0.0042	0.9621 ± 0.0040
30	0.9613 ± 0.0037	0.9639 ± 0.0042	0.9299 ± 0.0041	0.9320 ± 0.0031	0.9644 ± 0.0027
35	0.9623 ± 0.0025	0.9643 ± 0.0024	0.9342 ± 0.0037	0.9352 ± 0.0040	0.9660 ± 0.0024
40	0.9633 ± 0.0019	0.9638 ± 0.0021	0.9370 ± 0.0049	0.9383 ± 0.0036	0.9655 ± 0.0021
50	0.9639 ± 0.0012	0.9643 ± 0.0029	0.9394 ± 0.0052	0.9411 ± 0.0034	0.9656 ± 0.0024
C-1(67)	0.9641 ± 0.0048	0.9640 ± 0.0034	0.9356 ± 0.0045	0.9437 ± 0.0035	0.9654 ± 0.0037

and

Table 7. Classification results obtained on the YaleB dataset.

Dimension	LDA	MFA	PRPCA	LPP	RFA
9	0.7788 ± 0.0199	0.8355 ± 0.0190	0.3675 ± 0.0378	0.4859 ± 0.0437	0.8347 ± 0.0212
15	0.8774 ± 0.0092	0.9031 ± 0.0071	0.4793 ± 0.0572	0.5911 ± 0.0367	0.9039 ± 0.0096
20	0.9209 ± 0.0136	0.9273 ± 0.0104	0.5460 ± 0.0426	0.6276 ± 0.0171	0.9279 ± 0.0067
25	0.9395 ± 0.0189	0.9370 ± 0.0126	0.5427 ± 0.0377	0.6392 ± 0.0345	0.9387 ± 0.0109
30	0.9507 ± 0.0157	0.9490 ± 0.0081	0.5518 ± 0.0399	0.6736 ± 0.0219	0.9495 ± 0.0079
35	0.9582 ± 0.0132	0.9503 ± 0.0093	0.5481 ± 0.0385	0.6740 ± 0.0109	0.9503 ± 0.0116
40	-	0.9511 ± 0.0102	0.2084 ± 0.3698	0.6997 ± 0.0059	0.9569 ± 0.0146
50	-	0.9482 ± 0.0101	0.3698 ± 0.1481	0.7142 ± 0.0123	0.9532 ± 0.0089
C-1(37)	0.9594 ± 0.0094	0.9511 ± 0.0122	0.5306 ± 0.2147	0.6864 ± 0.0102	0.9548 ± 0.0107

.

We can see from the experimental results that RFA performs very well. Although LPP is an effective dimensionality reduction method for face recognition, RFA is significantly better than LPP. Moreover, RFA obtains comparable results with LDA and MFA. These results demonstrate the effectiveness of RFA in the face recognition applications.

Additionally, convergence of RFA is verified on these two datasets. As illustrated in Figure 9 and Figure 10, the value of $\eta$ (trace ratio) decreases through the iterative procedures until it reaches the global optimal value ${\eta }^{\ast }$ on both of the two datasets, which clearly shows the convergence of RFA.

5.1.3. Comparison on Other UCI Datasets

To evaluate the generalization ability of RFA, we conducted experiments on UCI datasets of other fields. The details of the used datasets are shown in

Table 8. Statistics of the UCI datasets.

Dataset	Classes	Samples	Dimensions
Shuttle	7	14,516	9
Thyroid	3	215	5
Vowel	11	990	10
Waveform21	3	5000	21

. For the corresponding experimental settings of these four dataset, we set the number of the nearest neighbors – k to 15. For the fairness of comparison, the subspace dimension of each method was set to $C-1$.

The results shown in

Table 9. Classification results obtained on several UCI datasets.

Dimension	LDA	MFA	PRPCA	RDA	RFA
Shuttle	0.9975 ± 0.0009	0.9979 ± 0.0012	0.9959 ± 0.0037	0.9162 ± 0.0132	0.9981 ± 0.0003
Thyroid	0.9395 ± 0.0265	0.9488 ± 0.0303	0.9488 ± 0.0195	0.9209 ± 0.0265	0.9581 ± 0.0255
Vowel	0.9859 ± 0.0109	0.9869 ± 0.0085	0.9859 ± 0.0090	0.3505 ± 0.0349	0.9879 ± 0.0077
Waveform21	0.8184 ± 0.0119	0.8258 ± 0.0197	0.8150 ± 0.0118	0.5972 ± 0.0279	0.8278 ± 0.0175

demonstrate the advantage of RFA over the related approaches. It is very effective in a wide range of applications.

5.1.4. Comparison on Document Classification and Webpage Classification Problems

As a general dimensionality reduction framework, the relational matrix can be constructed with different strategies. In the previous sections, we considered the relationship between samples based on their class labels or similarity. However, in some complicated problems, relationships may presented in other forms. For example, as indicated in [25], if there is a reference relationship between two papers, they are likely to have the same topic. However, due to the sparse nature of the bag-of-words representation, the similarity between these two papers may be very low. Thus, to further testify RFA, we designed a relational matrix based on the citation relevance between data samples to model RFA, and tested its effectiveness on document classification and webpage classification problems.

For this experiment, we used two datasets, citeseer and WebKB (https://linqs-data.soe.ucsc.edu/public/lbc/, accessed on 19 October 2023). We note that the WebKB dataset contains four subsets: Cornell, Texas, Washington and Wisconsin, and we show the experimental results of these four subsets separately. Each dataset contains bag-of-words representation of documents or webpages and citation links between the instances. Citeseer contains 3312 scientific documents from 6 different classes, and there are 4732 citation relation between the documents. WebKB consists of 877 webpages from 5 different classes, and there are 1608 page links within this dataset. We adopted the same strategy as in PRPCA to construct the relational matrix:

(1) Constructing the adjacent graph $\mathbf{A}$ according to the relevance between data samples. If there was a citation or link between sample i and j, then ${\mathbf{A}}_{ij}=1$; else, ${\mathbf{A}}_{ij}=0$.

(2) Letting ${\tilde{\mathbf{D}}}_{ii}={\sum }_j{\mathbf{A}}_{ij}={\left(\mathbf{A}\mathbf{A}\right)}_{ij}$, then $\mathbf{B}=\mathbf{A}\mathbf{A}-\tilde{\mathbf{D}}$,

$${\mathbf{B}}_{ij}=\left\{\begin{array}{c}{\left(\mathbf{A}\mathbf{A}\right)}_{ij}=\sum _{k=1}^N{\mathbf{A}}_{ij}{\mathbf{A}}_{kj},\mathrm{if}\mathrm{i}\ne \mathrm{j},\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.$$

(25)

(3) Defining $\mathbf{G}=2\mathbf{A}+\mathbf{B}$ as the relational matrix in RFA.

We took PRPCA as the baseline method in this part. Experimental results are illustrated in Figure 11; we can see that RFA achieves comparable results with PRPCA in all these five datasets and is even better than PRPCA on some of the datasets.

5.2. Performance of KRFA

To evaluate the efficiency of KRFA, we tested its performance on several benchmark datasets from the UCI machine learning repository. The details of these datasets are shown in

Table 10. Statistics of the UCI datasets.

Dataset	Classes	Samples	Dimensions
Ecoli	8	336	8
Satimage	7	6435	36
Vehicle	4	846	18
Waveform40	3	5000	40
Wine	3	178	13

. For the corresponding experimental settings, we set the number of the nearest neighbors – k of these five dataset to 15. To avoid the singular value issue, we adopted KPCA to retain 98% of the variance before formally performing KMFA and KRFA. We used Gaussian kernel in the experiment and for the fairness of the comparison, the subspace dimension of each method was set to $C-1$.

Table 11. Classification accuracy and standard deviation obtained on several UCI datasets.

Dimension	KMFA	RFA	KRFA
Ecoli	0.4200 ± 0.0431	0.4230 ± 0.0423	0.4511 ± 0.0648
Satimage	0.7749 ± 0.0345	0.3915 ± 0.0834	0.8182 ± 0.0343
Vehicle	0.7694 ± 0.0152	0.6808 ± 0.0489	0.7824 ± 0.0187
Waveform40	0.7996 ± 0.0136	0.8216 ± 0.0097	0.8256 ± 0.0062
Wine	0.9716 ± 0.0350	0.9716 ± 0.0286	0.9773 ± 0.0239

shows the comparison results obtained by KRFA, KMFA and RFA.

As shown in Table 11, the proposed KRFA obtained a comparable and even better result than KMFA. Furthermore, experimental results of KRFA were all better than RFA on the used datasets. That superiority can be especially reflected on the Satimage dataset. The performance of RFA on Satimage was unsatisfactory. However, KRFA conducted effective nonlinear dimensionality reduction and thus obtained good result on the following classification problem. These two points clearly demonstrate the nonlinear dimensionality reduction ability of KRFA.

6. Conclusions

In this paper, we propose a novel and general framework named relational Fisher analysis (RFA) which integrates relational information into the dimensionality reduction models. RFA can be effectively optimized with an iterative method based on trace ratio. For nonlinear dimensionality reduction, we adopt kernel trick to RFA and design its kernel version named KRFA. Extensive experiments demonstrate that RFA and KRFA outperform other related dimensionality reduction algorithms in most cases. In future work, we plan to extend this research in the following aspects: (1) Exploiting efficient relationship metric for different relational data to further test the effectiveness of the proposed RFA model; (2) Further extending the formulation of RFA for semi-supervised learning; and (3) Extending RFA for tensor representation learning and applying it to tensor analysis problems.