Next Article in Journal
Extreme Multiclass Classification Criteria
Previous Article in Journal
Minimizing the Duration of Repetitive Construction Processes with Work Continuity Constraints
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Multi Similarity Metric Fusion in Graph-Based Semi-Supervised Learning

by
Saeedeh Bahrami
1,
Alireza Bosaghzadeh
1,* and
Fadi Dornaika
2,3
1
Department of Artificial Intelligence, Faculty of Computer Engineering, Shahid Rajaee Teacher Training University (SRTTU), Tehran 16788-15811, Iran
2
Faculty of Computer Engineering, University of the Basque Country, 20018 San Sebastian, Spain
3
Ikerbasque, Foundation for science, 48013 Bilbao, Spain
*
Author to whom correspondence should be addressed.
Computation 2019, 7(1), 15; https://doi.org/10.3390/computation7010015
Submission received: 30 January 2019 / Revised: 22 February 2019 / Accepted: 25 February 2019 / Published: 7 March 2019
(This article belongs to the Section Computational Engineering)

Abstract

:
In semi-supervised label propagation (LP), the data manifold is approximated by a graph, which is considered as a similarity metric. Graph estimation is a crucial task, as it affects the further processes applied on the graph (e.g., LP, classification). As our knowledge of data is limited, a single approximation cannot easily find the appropriate graph, so in line with this, multiple graphs are constructed. Recently, multi-metric fusion techniques have been used to construct more accurate graphs which better represent the data manifold and, hence, improve the performance of LP. However, most of these algorithms disregard use of the information of label space in the LP process. In this article, we propose a new multi-metric graph-fusion method, based on the Flexible Manifold Embedding algorithm. Our proposed method represents a unified framework that merges two phases: graph fusion and LP. Based on one available view, different simple graphs were efficiently generated and used as input to our proposed fusion approach. Moreover, our method incorporated the label space information as a new form of graph, namely the Correlation Graph, with other similarity graphs. Furthermore, it updated the correlation graph to find a better representation of the data manifold. Our experimental results on four face datasets in face recognition demonstrated the superiority of the proposed method compared to other state-of-the-art algorithms.

1. Introduction

In machine learning problems, representing the manifold of the data is a key point, and it is a cornerstone in many semi-supervised learning methods. One of the most useful tools for representing an intrinsic structure of the data is the use of a graph [1]. In addition, the graph can encode pairwise similarities between the samples. Since the goal is to find an appropriate similarity metric for the data space, it is hard to define a fixed and single similarity metric. Multi-metric fusion refers to a type of fusion that combines different metrics [2,3,4,5]. Using different similarity metrics and considering their connection can improve and robustify the performance of learning tasks [2,4,5].
In semi-supervised learning, graphs can be used in many tasks, such as classification [6,7], clustering [8,9,10], dimension reduction [11,12], and label propagation (LP) [13,14], to mention a few. In a semi-supervised learning context, very often data comprise two subsets of labeled and unlabeled samples. One main task of semi-supervised learning is graph-based LP. This aims to propagate the label of labeled data to unlabeled ones over the graph [15]. LP has different applications in community detection [16], image segmentation [17], clustering [18], and classification [19] tasks. Although most of the algorithms use one single graph as an input of the LP algorithms, such as the Zhou method [20], flexible manifold embedding (FME) [21], local and global consistency (LGC) [22], and Gaussian fields and harmonic function (GFHF) [23], exploiting various similarity graphs can enhance the performance of LP process (graph-fusion methods) [15,24,25,26,27,28], thereby creating multiple similarity graphs where each contains complementary information of data and fusing them together can lead to a better representation of data.
The manifestation of the Internet and technology caused the growth of data in multiple forms as video, image, audio, text, etc. [29,30]. The key is how to integrate them to benefit complementary information from each view. Therefore, graph fusion methods have been proposed as one of the approaches to fuse different views. While different algorithms for graph fusion have been designed in three levels of graph integration (early integration [31], late integration [15], and intermediate integration [24,25,26,27,28,32,33]), a common solution is to construct different graphs based on multiple views called intermediate integration and then fuse the graphs either linearly via dynamic graph fusion LP (DGFLP) [25], sparse multiple graph integration (SMGI) [24], multi-view LGC [32], and deep graph fusion [33]), or nonlinearly with similarity network fusion (SNF) [27], nonlinear graph fusion (NGF) [26], and multi-modality dynamic LP (MDLP) for semi-supervised multi-class multi-label [28]. Furthermore, some methods attempt to learn different metrics based on different feature descriptors [2,3,4,5], whereas, in some cases, the variation of multi-view leads to misalignment in feature descriptors [34]. So, determining the discriminant and reliable feature descriptor is a key point [35]. To solve this problem, some algorithms build different types of similarity graphs based on an available view [24] and merge them to be used in the LP process.
Constructing an appropriate single similarity graph that represents the exact manifold of data is not a trivial task. Some research provides different similarity graphs by adopting different parameters in the graph construction scheme. Moreover, this approach depends on the parameters of the similarity graph and the way they are adjusted.
The data space, which is known as the feature space is the main source of information about the data samples. Although, label space has some information that can be integrated with the data space, which makes the graph construction more robust to noises and outliers [31], very few works have used this information in the fusion process [25,28,31]. The joint use of the label space and feature space can boost the performance of LP [25,28,31].
The contributions of this paper are as follows: We proposed a new multi-metric algorithm that integrates graph fusion and LP in a unified framework. With respect to this, we extended the FME LP method to fuse multiple similarity graphs based on a single feature descriptor of data. Since the relevance of all graphs was equal, the proposed method did not seek to weigh the individual metrics. In addition, to further improve the performance, we integrated the information in the label space with the feature space. Exploiting the label space was achieved by constructing a graph based on the correlation between the label vectors (both available and predicted ones).
The rest of the paper is outlined as the follows: Section 2 briefly explains multi-metric method, and Section 3 describes the proposed method in detail. Our experimental results are reported in Section 4, and Section 5 provides the discussion and conclusion.

2. An Overview on Multi-Metric Fusion

A metric or distance function refers to a function that explains a distance between pair-wise samples. A similarity metric defines a similarity function that measures how two samples are similar or related. Depending on the adopted feature descriptor, a metric can well discriminate between the samples, however, finding an individual proper metric is difficult. Multi-metric methods proposed exploiting different metrics corresponding to different views of data [5]. Moreover, even a simple metric like k-Nearest Neighbors (k-NN) graph has parameters (e.g., number of neighbors) to tune where correct tuning of the parameters can have an important effect on the results.
Recently proposed methods in semi-supervised learning fused several graphs in order to maximize inter-class scattering and minimize the variation of intra-class [36]. This strategy also achieves the complementary information from each view and enhances the performance of the classification [24,25,28].
In Reference [24], the authors proposed an SMGI algorithm that integrates a set of k-NN graphs with different values of K (neighborhood parameter) linearly with some noisy graphs. This method is adopted for a semi-supervised learning task. They assigned unequal weight to each graph and sparsely selected the relevant graphs from the noisy ones. They only used one type of similarity graph with different parameters and do not employ label space in its fusion approach.
In Reference [28], the authors proposed an MDLP algorithm which utilized multiple views (each sample has multiple descriptors) in semi-supervised learning. A feature descriptor was associated with each view. Two types of k-NN graphs were then constructed from each descriptor. A local k-NN graph which considered the similarity of each sample to its local neighbors and the global k-NN graph focused on the similarity of each sample to all other samples. In the fusion process, the information of label space integrated with these two graphs.
In Reference [25] the authors proposed a DGFLP method that fused the information of data and label spaces together. The algorithm allocates dynamic unequal weights to each constructed graph, while uses a fixed weight for the information of the label space.
Another semi-supervised multi-graph fusion is presented in [32]. It extended the LGC [22] method to the multi-view case (MLGC). They constructed one graph for each adopted feature descriptor and then combined them linearly with equal weights. This approach used only one type of similarity metric and, furthermore, it ignored employing label space information in its fusion process.
In our previous work [33], we concentrated on the multi-view fusion scenario, where the goal was to merge different available information of data samples. A graph was constructed based on each feature and then they were merged by summing them linearly to obtain the final graph.

3. Proposed Method

Consider that we have a dataset X = [x1, x2, …, xN] that contains N samples whose dimension is equal to d. Suppose we have C classes index by c = 1, 2, …, C, then Y ∈ RN×C is the initial binary label matrix, where y(i,c) represents the probability value of sample ith belonging to the class c. In semi-supervised learning, few data samples have labels, but the majority of them have no labels. The objective is to predict the labels of unlabeled data via a graph which propagates the labels of labeled data to their nearby unlabeled data. The output of LP is the prediction label matrix (F).
A graph is a data structure with three components G = (V,E,W). V denotes the nodes of the graph, E represents the edges on the graph, and W ∈ RN×N is the similarity matrix. The similarity value between the two nodes i and j, is given by the entry w(i,j) of the similarity matrix. Since we do not have precise prior knowledge about the appropriate similarity graph, we choose a strategy that combines several different graphs.
As an example, consider the k-NN graph construction technique, a graph construction method that makes an edge and connects each sample to its K nearest neighbors. The weight of each edge can be computed by different algorithms and the most common one is called the Gaussian kernel [15,37,38] function as:
w i j = exp ( x i x j 2 2 σ 2 )
where, xi and xj represents the node i and j, respectively, and σ is the width of the Gaussian kernel. The value of σ can be defined either manually or adaptively [39]. The rows of the affinity matrix are then normalized to have sum equals to one adopting the Equation (2). Finally, the matrix is made symmetric using Equation (3).
w i j = w i j j = 1 N w i j
W = 1 2 ( W + W T )
Let L denote the graph Laplacian matrix, which is obtained by L = DW. D is a degree matrix, where D ( i , i ) = j = 1 N w ( i , j ) represents its diagonal elements. Table 1 shows all the notations that were used in the paper.

3.1. Review of Flexible Manifold Embedding (FME)

FME [21] is an LP framework that was proposed in 2010. This method can be used in both semi-supervised and unsupervised learning settings. The objective function of FME is given by:
argmin F , Q , b { Trace ( F T L F ) + Trace ( F Y ) T U ( F Y ) + μ ( Q 2 + γ X T Q + 1 b T F 2 ) } ,
where L, F, and Y are the Laplacian matrix, the prediction label matrix, and the initial label matrix, respectively. U is the diagonal matrix such that the diagonal elements for labeled nodes are 1 and for unlabeled nodes are 0. In unsupervised learning, as there is no predefined label, the second term is eliminated by setting all elements of U to 0. μ and γ are the trade-off parameters. X is the training data matrix, Q is the projection matrix that maps feature space to label space, b is the bias vector, and 1 is a vector that each element of it is one. The projection matrix Q ∈ Rd×C and the bias vector b ∈ RC×1 are used for estimating the labels of unseen data.
In Equation (4), the first term of the cost function represents the smoothness assumption and the second term is the prediction error for the labeled data. The last term is the error of the projection from the data space into the label space. The outputs of the algorithm are the prediction label matrix (F), the projection matrix (Q), and the bias vector (b). This method uses a projection matrix to map data space to label space through a linear regression function. Optimal F, Q, and b can be obtained using the closed-form solution as (see Reference [21] for more details):
F = ( U + L + μ γ ( B I ) T ( B I ) + μ A T A ) 1 U Y B = H c X T A + ( 1 n ) 11 T H c = I ( 1 n ) 11 T A = γ ( γ X H c X T + I ) 1 X H c
Q = A F
b = 1 n ( F T 1 W T X 1 ) ,
where n is the total number of samples.
Regarding this method, we extended it to multi-metric fusion algorithm, which integrates multiple graphs and incorporates the label space into the fusion process.

3.2. Multi Similarity Metric Fusion

According to our available view, we build M different similarity graphs (denoted as Wi for the ith metric) based on M different metrics. Corresponding to each graph, we have a Laplacian matrix (i.e., Li). With respect to [15,28], we suppose that the manifold structure of data is attained by a linear integration of multiple graphs as i = 1 M W i . Similar to this idea in our proposed method, we extended the smooth manifold term of FME as:
i = 1 M Trace ( F T L i F ) .
If we consider L data = i = 1 M L i , then we have:
Trace ( F T ( i = 1 M L i ) F ) : = Trace ( F T L data F ) ,
where, Ldata is the Laplacian fusion of the graphs constructed based on different metrics. By replacing this equation in the first term of the FME algorithm, our cost function is obtained as:
argmin F , Q , b { Trace ( F T L data F ) + Trace ( F Y ) T U ( F Y ) + μ ( Q 2 + γ X T Q + 1 b T F 2 ) } ,
where the first term fuses M metrics from an available feature descriptor. The last two terms are the same as the last two terms in the FME algorithm.

3.3. Incorporating Label Space Information

A feature descriptor, which is known as the data space, extracts information of data. The works of Reference [25,28,31] demonstrate that there is some hidden information in the label space which is ignored. As the similarity of two vectors can be described by the correlation of them, so by calculating the correlation of predicted labels, we can define a new similarity measure in the label space. The concept of it is that if two samples have a slight similarity in the data space, it is possible that they have a strong similarity in the label space. Here, the label space can affect the similarity between them.
The main key is how to merge the label space with the data space. We proposed a new similarity metric based on label space. In fact, we considered the label space as a new view of the data that could be described by a new similarity metric. The nodes of the graph were the samples and the weight of each edge was computed by the Pearson correlation measure, as shown in Equation (11). For two samples, xi and xj with the label vectors fi and fj, the similarity between them was calculated as:
correlation ( f i , f j ) : = q = 1 c ( f q i f ¯ i ) ( f q j f ¯ j ) q = 1 c ( f q i f ¯ i ) 2 q = 1 c ( f q j f ¯ j ) 2 ,
where, f ¯ i and f ¯ j were the average of vectors fi and fj.
Afterward, we omitted negative values obtained by correlation measure and for each sample, the K highest similar samples were chosen as its K neighbors. We called this new similarity metric the correlation graph. To incorporate the label space graph with the data space graph, we linearly added it to the graphs obtained by other metrics with equal weights as:
L ^ : = L data + L Corr ,
where LCorr is the Laplacian of the correlation graph. Since the predicted labels changed in each iteration, the correlation graph changed, too. After each iteration of LP, the obtained L ^ would be used (instead of Ldata) in the cost function of the proposed multi-view FME of Equation (10).
The framework of our proposed method is depicted in Figure 1. From the available views, M graphs were constructed. Also, a correlation graph was created based on the initial label matrix Y. Then, all the graphs became integrated by a linear fusion with equal weights and were fed to FME LP method. The FME algorithm estimated the label matrix F, Q projection matrix, and the b bias vector. In the next iteration of fusion, the correlation graph updated based on the correlation of predicted matrix F. This process was repeated through a given number of iterations. In the last iteration, the projection matrix and the bias vector could be used for estimating the labels of unseen data, which were not available at first.
We summarize the procedure of our proposed method in Algorithm 1. Moreover, Figure 2 shows the flowchart of the proposed method.
Algorithm 1. The proposed method.
  Input: Feature from one view X;
  Initial label matrix Y = [Y1, Yu];
  Parameters μ and γ.
  Output: Predicted label matrix F, projection matrix Q, and bias vector b.
1. Construct M different graphs from the available view.
2. Compute the Laplacian matrices of the graphs.
3. Initialize the soft label matrix F = Y.
4. for t = 1: Max iteration.
5. Generate the correlation graph based on F by Equation (11).
6. Fuse the M + 1 Laplacian graph to obtain L ^ adopting Equation (12).
7. Feed L ^ to FME and calculate a new soft label matrix F.
8. Reinitialize the part of F matrix corresponding to the labeled samples.
9. end for.
10. To calculate the labels of unseen samples, use the projection matrix Q and the bias vector b to predict the labels of them.

4. Experimental Results

4.1. Experimental Setup

We evaluated the proposed method using four real-world face datasets, PF01 [40], Extended Yale (http://vision.ucsd.edu/~leekc/ExtYaleDatabase/ExtYaleB.html), PIE (http://www.ri.cmu.edu/projects/project_418.html), and the subsets of FERET [41], and compared the performance results with other recently proposed graph fusion methods. The details of each dataset are reported in Table 2. The experiments were conducted on a PC equipped with an i5-4300U 1.9 GHz 2.5 GHz CPU and 8 GB of RAM.
We extracted three features from the face images. Two Deep features are extracted from the pre-trained VGG Face model [42] and Local Binary Pattern (LBP) image [43] was the handcrafted feature. For deep features, we used two fully-connected layers of VGG Face model, namely FC6 and FC7. For the handcrafted feature, we applied LBP on the image, and the output was reshaped to form a single vector.
In the graph construction step, we adopted two graphs, the K-nearest neighbor graph and the adaptive k-NN graph, both with the Gaussian kernel to weight the edges. To keep the k-NN graphs as sparse as possible, we set the number of neighbors K = 3. For the k-NN graph, the sigma of the Gaussian kernel was empirically set to σ = 0.05, and for the adaptive one, it would be selected automatically, according to:
σ = 1 N i = 1 N dis ( x i , x i K )
where x i K is the Kth closest node to sample xi, and dis(xi, xj) is the Euclidian distance between nodes xi and xj. For FME parameters, we selected μ = 103 and γ = 10−3. Since our method was an iterative process, we empirically set Algorithm 1’s number of iterations to 10. We stress the fact that we did not optimize any of the aforementioned parameters because their optimum values depended on the datasets.
In Section 4.2, we assessed the effect of using multiple similarity graphs into the FME framework, and in Section 4.3, we compared the proposed method with other state-of-art graph fusion algorithms. In Section 4.4, we demonstrated the CPU-time of our proposed method with other methods.

4.2. Comparison with Individual Graphs

In this section, the performance of our multi-metric fusion method was compared with that of individual graphs using the same FME method. The average accuracy over 10 random combinations of labeled and unlabeled data was reported in Table 3. Since we assumed that the whole dataset was available at first and no unseen samples exist, we only used the transductive part of the FME algorithm, i.e., the classification adopted the predicted label matrix F. In this experiment, for each dataset, we considered two cases: one labeled sample per class and two labeled samples per class.
The results depicted in Table 3 showed that integrating two similarity graphs from data space and the correlation graph of label space (our proposed method) improved the performance of the basic FME algorithm when used with one individual graph. As the number of samples in the FERET dataset was small and the number of classes was large, thereby, the accuracy of opting one/two sample/samples per class was already high, so our proposed method had a slight effect on it. Whenever our labeled data increased, the accuracy slowly altered since no more information augmented to the data. We can observe this situation in the columns corresponding to two labeled samples.
To illustrate the influence of our fusion method, we opted 17 samples from the first 8 classes of PF01 face database. In the feature extraction step, we fed the images into the VGG Face deep network and extracted the information in layer 6, which was a fully connected layer (we called it FC6). One sample in each class was selected as the labeled data and the rest as unlabeled data. The data were placed close to each other, such that the samples of each class were alongside each other. Figure 3a,b exhibit the similarity between only the unlabeled samples based on the FC6 descriptor obtained by Adaptive k-NN graph and k-NN graph, respectively, both with K = 6. Before the LP, the similarity values of unlabeled data are zeros, so the similarity graph of the unlabeled part was plotted with the blue color that presents the 0 value on Figure 3c. After LP by FME, we used the labels of unlabeled samples to find the similarity between the nodes, Figure 3d is the correlation graph between the labels after 10 iterations of LP. Figure 3e shows the fusion graph associated with three graphs. As we observed, the fusion graph can amplify the strong similarity values and weaken the slight similarity values. Furthermore, we observed that the edge weight of inter-class nodes had high values (red colors), whereas the intra-class nodes had very small values (blue color), which is desirable behavior.

4.3. Comparison with Other Methods

In this part, we compared our proposed method with four state-of-the-art methods, named MLGC [32], MDLP [28], DGFLP [25], and SMGI [24]. To have a fair comparison, we used the same graphs as before for all methods. According to Reference [24], the parameters of SMGI were tuned as the maximum number of iteration to 100, λ1 = 0.01, λ2 = 0.1, and stopping condition was 1e−4. For the other algorithms, the optimum parameter was set to those which provide the highest accuracy on the FERET dataset. For MLGC, the optimal number of iterations was set to 20 from the range of Reference [1,25] and λ = 0.1 from the set of [0.01,0.1,0.5,1], and for DGFLP the number of optimal iterations was set to 20 from the range of Reference [1,25]. Finally, for MDLP, we set the parameters as K = 10, α = 0.3, λ = 1 among the set of K = [6,10,15,20], α, λ = [0.01,0.1,0.5,1] and the number of iterations set to six from the Reference [1,20].
Similar to the previous experiment, we set the number of labeled samples per class to one and two. The obtained results based on VGG Face-FC7 and VGG Face-FC6 descriptors were reported in Table 4 and for LBP feature descriptor in Table 5. With respect to the results of Table 4 and Table 5, we observed that the proposed method outperformed other fusion techniques adopting either deep or handcrafted features. Compared to the algorithms that use the correlation between the labels (i.e., DGFLP and MDLP), the proposed method obtained higher accuracy, which showed that the explicit use of the correlation information as a graph could improve the performance. Moreover, we observed that DGFLP algorithm had higher accuracy compared to MDLP. While both used the correlation graph, the former adopted unequal weights for the fusion of the graphs. As we mentioned, our method omitted the small and negative values that were obtained from the correlation of labels in order to remove the noisy values. Furthermore, it built a graph based on the correlation between the labels hence, its performance outperformed other methods.
Moreover, we compared the confusion matrix of our proposed method with DGFLP method. For this test, we used the Extended Yale dataset which had 28 classes with LBP features and selected 1 sample in each class as the labeled data and the rest as the unlabeled data. In Figure 4, the horizontal axis demonstrates the target labels and the vertical axis shows the predicted labels. The warm color represents the high accuracy value and the cool color represents the low value of accuracy. Figure 4a is the confusion matrix of DGFLP method, and Figure 4b is the confusion matrix of our proposed method. As we observed, the confusion matrix of our proposed method was more uniform compared to that of DGFLP and had similar performance across different classes.

4.4. CPU-Time and Computational Complexity

We also measured the running time of our method and that of other methods and plotted it in Figure 5. It is depicted from the average of 10 CPU elapsed time on PIE dataset based on the VGG Face-FC6 descriptor. The vertical axis demonstrates the time of CPU in seconds, and the horizontal axis shows the name of each method. As we can see, our method with 45.23 s had the greatest CPU time rather than other methods, but due to our accuracy results reported in Table 4, this computational time worth to be tolerated.
Moreover, since the computational complexity of FME is O(N3), the computational complexity of the proposed method will be O(T·N3) where T is the number of the adopted iterations.

5. Discussion and Conclusions

In this paper, we created multiple similarity metrics based on one feature descriptor and then fused the similarity graphs with equal weights to apply in the LP framework. We also used a new graph construction named correlation graph, based on the label space. In fact, we extended the FME algorithm into the multi-metric fusion method, to fuse multiple graphs based on the available feature and combining with the graph of the label space, to improve the accuracy of label propagation task. The experimental results on four face datasets showed that our multi-metric fusion method increased the accuracy compared to the use of a single feature. Indeed, we used the train part of the FME algorithm that assumed all the data were available at first. In the results, our method outperformed than other rival multi-graph fusion methods.
As a future work, we will try to assign unequal weight to each graph due to the available information of each one. Moreover, we envision the use of more than one view of data and the extraction of multiple feature descriptors to benefit multi-view information in graph fusion. Furthermore, the use of inductive setting of FME algorithm to estimate the labels of unseen data can be adopted to extend the proposed method to large scale databases. As we mentioned before, label propagation has different applications in community detection [16], image segmentation [17], and clustering [18], and classification [19] tasks. Therefore, in our future works, we will focus on applying the proposed method in the aforementioned applications. Moreover, since the proposed scenario of extracting information from the labels and creating a correlation graph is not limited to a specific label propagation method, any label propagation algorithm can benefit from the proposed scenario.

Author Contributions

Conceptualization, A.B. and F.D.; Methodology, A.B. and F.D.; Validation, S.B. and A.B.; Formal Analysis, A.B. and F.D.; Investigation, A.B., and F.D.; Data Curation, S.B. and A.B.; Writing-Original Draft Preparation, S.B., A.B., and F.D.; Writing-Review and Editing, S.B., A.B., and F.D.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Bashir, Y.; Aslam, A.; Kamran, M.; Qureshi, M.; Jahangir, A.; Rafiq, M.; Bibi, N.; Muhammad, N. On forgotten topological indices of some dendrimers structure. Molecules 2017, 22, 867. [Google Scholar] [CrossRef] [PubMed]
  2. Lu, J.; Hu, J.; Tan, Y.-P. Discriminative deep metric learning for face and kinship verification. IEEE Trans. Image Process. 2017, 26, 4269–4282. [Google Scholar] [CrossRef] [PubMed]
  3. Mirmahboub, B.; Mekhalfi, M.L.; Murino, V. Person re-identification by order-induced metric fusion. Neurocomputing 2018, 275, 667–676. [Google Scholar] [CrossRef]
  4. Zhang, H.; Huang, T.S.; Nasrabadi, N.M.; Zhang, Y. Heterogeneous Multi-Metric Learning for Multi-Sensor Fusion. In Proceedings of the 14th International Conference on Informatio Fusion (FUSION), Chicago, IL, USA, 5–8 July 2011; pp. 1–8. [Google Scholar]
  5. Zhang, L.; Zhang, D. Metricfusion: Generalized metric swarm learning for similarity measure. Inf. Fusion 2016, 30, 80–90. [Google Scholar] [CrossRef]
  6. Bai, J.; Xiang, S.; Pan, C. A graph-based classification method for hyperspectral images. IEEE Trans. Geosci. Remote Sens. 2013, 51, 803–817. [Google Scholar] [CrossRef]
  7. Boiman, O.; Shechtman, E.; Irani, M. In defense of nearest-neighbor based image classification. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, AK, USA, 23–28 June 2008; pp. 1–8. [Google Scholar]
  8. Hartigan, J.A.; Wong, M.A. Algorithm as 136: A k-means clustering algorithm. J. R. Stat. Soc. 1979, 28, 100–108. [Google Scholar] [CrossRef]
  9. Vesanto, J.; Alhoniemi, E. Clustering of the self-organizing map. IEEE Trans. Neural Netw. 2000, 11, 586–600. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  10. Von Luxburg, U. A tutorial on spectral clustering. Stat. Comput. 2007, 17, 395–416. [Google Scholar] [CrossRef] [Green Version]
  11. Zhang, L.; Qiao, L.; Chen, S. Graph-optimized locality preserving projections. Pattern Recognit. 2010, 43, 1993–2002. [Google Scholar] [CrossRef]
  12. Zhou, G.; Lu, Z.; Peng, Y. L1-graph construction using structured sparsity. Neurocomputing 2013, 120, 441–452. [Google Scholar] [CrossRef]
  13. Tang, J.; Hong, R.; Yan, S.; Chua, T.-S.; Qi, G.-J.; Jain, R. Image annotation by k nn-sparse graph-based label propagation over noisily tagged web images. ACM Trans. Intell. Syst. Technol. 2011, 2, 14:1–14:16. [Google Scholar] [CrossRef]
  14. Wang, F.; Zhang, C. Label propagation through linear neighborhoods. IEEE Trans. Knowl. Data Eng. 2008, 20, 55–67. [Google Scholar] [CrossRef]
  15. Gong, C.; Tao, D.; Maybank, S.J.; Liu, W.; Kang, G.; Yang, J. Multi-modal curriculum learning for semi-supervised image classification. IEEE Trans. Image Process. 2016, 25, 3249–3260. [Google Scholar] [CrossRef] [PubMed]
  16. Zhang, X.-K.; Ren, J.; Song, C.; Jia, J.; Zhang, Q. Label propagation algorithm for community detection based on node importance and label influence. Phys. Lett. A 2017, 381, 2691–2698. [Google Scholar] [CrossRef]
  17. Breve, F. Interactive image segmentation using label propagation through complex networks. Expert Syst. Appl. 2019, 123, 18–33. [Google Scholar] [CrossRef]
  18. Seyedi, S.A.; Lotfi, A.; Moradi, P.; Qader, N.N. Dynamic graph-based label propagation for density peaks clustering. Expert Syst. Appl. 2019, 115, 314–328. [Google Scholar] [CrossRef]
  19. Cui, B.; Xie, X.; Hao, S.; Cui, J.; Lu, Y. Semi-supervised classification of hyperspectral images based on extended label propagation and rolling guidance filtering. Remote Sens. 2018, 10, 515–533. [Google Scholar] [CrossRef]
  20. Chapelle, O.; Schölkopf, B.; Zien, A. Semi-Supervised Learning; MIT Press: Cambridge, MA, USA, 2006; Chapter 11; ISBN 978-0-262-03358-9. [Google Scholar]
  21. Nie, F.; Xu, D.; Tsang, I.W.-H.; Zhang, C. Flexible manifold embedding: A framework for semi-supervised and unsupervised dimension reduction. IEEE Trans. Image Process. 2010, 19, 1921–1932. [Google Scholar] [PubMed]
  22. Zhou, D.; Bousquet, O.; Lal, T.N.; Weston, J.; Schölkopf, B. Learning with local and global consistency. In Proceedings of the 18th Conference on Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada, 13–16 December 2004; pp. 321–328. [Google Scholar]
  23. Zhu, X.; Ghahramani, Z.; Lafferty, J.D. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), Washington, DC, USA, 21–24 August 2003; pp. 912–919. [Google Scholar]
  24. Karasuyama, M.; Mamitsuka, H. Multiple graph label propagation by sparse integration. IEEE Trans. Neural Netw. Learn. Syst. 2013, 24, 1999–2012. [Google Scholar] [CrossRef] [PubMed]
  25. Lin, G.; Liao, K.; Sun, B.; Chen, Y.; Zhao, F. Dynamic graph fusion label propagation for semi-supervised multi-modality classification. Pattern Recognit. 2017, 68, 14–23. [Google Scholar] [CrossRef]
  26. Tong, T.; Gray, K.; Gao, Q.; Chen, L.; Rueckert, D. Alzheimer’s Disease Neuroimaging, I. Multi-modal classification of alzheimer’s disease using nonlinear graph fusion. Pattern Recognit. 2017, 63, 171–181. [Google Scholar] [CrossRef]
  27. Wang, B.; Mezlini, A.M.; Demir, F.; Fiume, M.; Tu, Z.; Brudno, M.; Haibe-Kains, B.; Goldenberg, A. Similarity network fusion for aggregating data types on a genomic scale. Nat. Methods 2014, 11, 333:1–333:8. [Google Scholar] [CrossRef] [PubMed]
  28. Wang, B.; Tsotsos, J. Dynamic label propagation for semi-supervised multi-class multi-label classification. Pattern Recognit. 2016, 52, 75–84. [Google Scholar] [CrossRef]
  29. Muhammad, N.; Bibi, N.; Qasim, I.; Jahangir, A.; Mahmood, Z. Digital watermarking using hall property image decomposition method. Pattern Anal. Appl. 2018, 21, 997–1012. [Google Scholar] [CrossRef]
  30. Muhammad, N.; Bibi, N. Digital image watermarking using partial pivoting lower and upper triangular decomposition into the wavelet domain. IET Image Process. 2015, 9, 795–803. [Google Scholar] [CrossRef]
  31. Li, S.; Liu, H.; Tao, Z.; Fu, Y. Multi-view graph learning with adaptive label propagation. In Proceedings of the IEEE International Conference on Big Data (Big Data), Boston, MA, USA, 11–14 December 2017; pp. 110–115. [Google Scholar]
  32. An, L.; Chen, X.; Yang, S. Multi-graph feature level fusion for person re-identification. Neurocomputing 2017, 259, 39–45. [Google Scholar] [CrossRef]
  33. Saeedeh, B.; Bosaghzadeh, A. Deep graph fusion for graph based label propagation. In Proceedings of the 10th Conference on Machine Vision and Image Processing (MVIP), Isfahan, Iran, 22–23 November 2017; pp. 149–153. [Google Scholar]
  34. Zhao, R.; Ouyang, W.; Wang, X. Person re-identification by salience matching. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 2528–2535. [Google Scholar]
  35. Zhao, R.; Ouyang, W.; Wang, X. Unsupervised salience learning for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 23–28 June 2013; pp. 3586–3593. [Google Scholar]
  36. Zhang, Y.; Zhang, H.; Nasrabadi, N.M.; Huang, T.S. Multi-metric learning for multi-sensor fusion based classification. Inf. Fusion 2013, 14, 431–440. [Google Scholar] [CrossRef]
  37. Wang, B.; Tu, Z.; Tsotsos, J.K. Dynamic label propagation for semi-supervised multi-class multi-label classification. In Proceedings of the IEEE International Conference on Computer Vision, Sydney Conference Centre, Darling Harbour, Sydney, 1–8 December 2013. [Google Scholar]
  38. Cortes, C.; Mohri, M. On transductive Regression. In Proceedings of the Conference on Neural Information Processing Systems (NIPS), Hyatt Regency Vancouver, Vancouver, BC, Canada, 3–8 December 2007. [Google Scholar]
  39. Yang, B.; Chen, S. Sample-dependent graph construction with application to dimensionality reduction. Neurocomputing 2010, 74, 301–314. [Google Scholar] [CrossRef]
  40. Bang, S.; Kim, D.; Choi, S. Asian Face Image Database PF01; Intelligent Multimedia Lab, University of Science and Technology: Pohang, Korea, 2001. [Google Scholar]
  41. Phillips, P.J.; Moon, H.; Rauss, P.; Rizvi, S.A. The feret evaluation methodology for face-recognition algorithms. In Proceedings of the Conference on IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, USA, 17–19 June 1997. [Google Scholar]
  42. Parkhi, O.M.; Vedaldi, A.; Zisserman, A. Deep face recognition. In Proceedings of the BMVC, Swansea, UK, 7–10 September 2015; p. 6. [Google Scholar]
  43. Ojala, T.; Pietikainen, M.; Maenpaa, T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 2002, 24, 971–987. [Google Scholar] [CrossRef] [Green Version]
Figure 1. The framework of the proposed method. The available view of data is demonstrated in green color. Different graphs are created according to available data. The brown-colored graph is built on the view of the label space. All the graphs are fused. The resulted graph is given to the flexible manifold embedding (FME) algorithm. In the final iteration, the projection matrix and bias vector are used for predicting the labels of the unseen data.
Figure 1. The framework of the proposed method. The available view of data is demonstrated in green color. Different graphs are created according to available data. The brown-colored graph is built on the view of the label space. All the graphs are fused. The resulted graph is given to the flexible manifold embedding (FME) algorithm. In the final iteration, the projection matrix and bias vector are used for predicting the labels of the unseen data.
Computation 07 00015 g001
Figure 2. The flowchart of the proposed method.
Figure 2. The flowchart of the proposed method.
Computation 07 00015 g002
Figure 3. The similarity matrix of unlabeled data from the PF01 dataset with eight classes and one sample from each class selected as the labeled data. (a) The similarity matrix of Adaptive k-NN graph based on VGG Face-FC6 with K = 6. (b) The similarity matrix of k-NN graph based on VGG Face-FC6 with K = 6 and σ = 0.05. (c) The similarity matrix of correlation graph based on VGG Face-FC6 with K = 6, which is before LP. (d) The similarity matrix of the correlation graph after 10 iterations of LP. (e) The similarity matrix of the fused graph.
Figure 3. The similarity matrix of unlabeled data from the PF01 dataset with eight classes and one sample from each class selected as the labeled data. (a) The similarity matrix of Adaptive k-NN graph based on VGG Face-FC6 with K = 6. (b) The similarity matrix of k-NN graph based on VGG Face-FC6 with K = 6 and σ = 0.05. (c) The similarity matrix of correlation graph based on VGG Face-FC6 with K = 6, which is before LP. (d) The similarity matrix of the correlation graph after 10 iterations of LP. (e) The similarity matrix of the fused graph.
Computation 07 00015 g003aComputation 07 00015 g003b
Figure 4. Confusion matrix of our proposed method and DGFLP method on the Extended Yale dataset on LBP features, where (a) is the confusion matrix of DGFLP method and (b) is the confusion matrix of our proposed method.
Figure 4. Confusion matrix of our proposed method and DGFLP method on the Extended Yale dataset on LBP features, where (a) is the confusion matrix of DGFLP method and (b) is the confusion matrix of our proposed method.
Computation 07 00015 g004
Figure 5. The mean of runtime on PIE dataset for the SMGI, MDLP, MLGC, DGFLP, and our proposed method based on VGG Face-FC6.
Figure 5. The mean of runtime on PIE dataset for the SMGI, MDLP, MLGC, DGFLP, and our proposed method based on VGG Face-FC6.
Computation 07 00015 g005
Table 1. List of mathematical notations.
Table 1. List of mathematical notations.
NotationDescription
NNumber of samples
MNumber of metrics
X = [x1, x2, …, xN]Data matrix
CNumber of classes
dSample dimension
YInitial binary label matrix
FPrediction label matrix
WSimilarity matrix
LLaplacian matrix
L ^ Fusion Laplacian matrix
QProjection matrix
bBias vector
tIteration number
Table 2. Statistics of datasets used in experiments.
Table 2. Statistics of datasets used in experiments.
DatasetSize# of ClassesFeaturesDimension
PF011819107VGG Face-FC74096
VGG Face-FC64096
LBP900
Extended_Yale177428VGG Face-FC74096
VGG Face-FC64096
LBP900
PIE192668VGG Face-FC74096
VGG Face-FC64096
LBP900
FERET1400200VGG Face-FC74096
VGG Face-FC64096
LBP900
Table 3. Mean accuracy and standard deviation of label propagation (LP) for the individual feature and the proposed method. K-NN = k-Nearest Neighbor.
Table 3. Mean accuracy and standard deviation of label propagation (LP) for the individual feature and the proposed method. K-NN = k-Nearest Neighbor.
Dataset# Labeled SamplesFeatureAccuracy (Mean ± STD )
k-NN graphAdaptive k-NN GraphProposed Method
PF011FC789.63 ± 7.9789.07 ± 8.3992.42 ± 2.86
FC690.85 ± 7.1689.54 ± 8.9993.99 ± 0.85
LBP51.79 ± 11.6449.16 ± 11.5456.19 ± 8.97
2FC794.16 ± 1.0993.94 ± 194.36 ± 1.19
FC694.74 ± 1.0494.22 ± 1.1194.75 ± 0.98
LBP63.77 ± 6.8261.08 ± 6.9865.47 ± 5.86
Extended_Yale1FC740.33 ± 20.8640.33 ± 20.9443.5 ± 19.34
FC644.67 ± 22.7644.32 ± 22.950.38 ± 19.27
LBP91.32 ± 5.1791.69 ± 5.6694.25 ± 1.52
2FC754.24 ± 2254.13 ± 21.9955.35 ± 21.68
FC657.01 ± 22.9956.81 ± 23.0658.68 ± 21.77
LBP96.01 ± 2.396.17 ± 2.0796.73 ± 1.93
PIE1FC777.36 ± 9.8277.17 ± 9.7979.04 ± 8.21
FC676.52 ± 11.4375.73 ± 11.779.77 ± 8.81
LBP45 ± 7.7144.56 ± 7.1646.79 ± 7.36
2FC787.57 ± 6.4187.14 ± 6.4688.35 ± 5.5
FC686.24 ± 7.1985.6 ± 7.2687.21 ± 5.9
LBP60.82 ± 6.1559.22 ± 6.2661.64 ± 5.99
FERET1FC798.65 ± 0.1398.63 ± 0.1398.83 ± 0.08
FC698.83 ± 0.1498.87 ± 0.1299.03 ± 0.13
LBP8.41 ± 6.327.8 ± 5.888.62 ± 6.24
2FC798.96 ± 0.2798.97 ± 0.2998.99 ± 0.3
FC699.05 ± 0.3199.1 ± 0.2599.13 ± 0.26
LBP15.67 ± 7.2614.26 ± 6.6516.04 ± 6.76
Table 4. Mean accuracy and standard deviation of the accuracy of the proposed method compared with other methods based on VGG Face-FC7 and VGG Face-FC6 features. DGFLP = dynamic graph fusion label propagation; SMGI = sparse multiple graph integration; MLGC = multi-view local and global consistency; MDLP = multi-modality dynamic label propagation.
Table 4. Mean accuracy and standard deviation of the accuracy of the proposed method compared with other methods based on VGG Face-FC7 and VGG Face-FC6 features. DGFLP = dynamic graph fusion label propagation; SMGI = sparse multiple graph integration; MLGC = multi-view local and global consistency; MDLP = multi-modality dynamic label propagation.
DatasetMethodAccuracy (Mean ± STD)
FC7FC6
PF01
1 labeled sample
SMGI80.33 ± 13.0178.12 ± 15.87
MLGC80.01 ± 12.8477.83 ± 15.69
DGFLP80.54 ± 13.378.14 ± 15.72
MDLP76.29 ± 17.0774.71 ± 19.12
Proposed method92.42 ± 2.8693.99 ± 0.85
PF01
2 labeled samples
SMGI88.12 ± 2.487.31 ± 3.35
MLGC87.79 ± 2.3987.07 ± 3.39
DGFLP88.33 ± 2.7287.4 ± 3.59
MDLP85.55 ± 1.8285.42 ± 2.81
Proposed method94.36 ± 1.1994.75 ± 0.98
Extended_Yale
1 labeled sample
SMGI37.62 ± 19.0839.28 ± 18.94
MLGC37.11 ± 17.9339.56 ± 17.98
DGFLP37.93 ± 19.14 39.8 ± 19.21
MDLP29.21 ± 19.5531.24 ± 20.2
Proposed method43.5 ± 19.3450.38 ± 19.27
Extended_Yale
2 labeled samples
SMGI48.87 ± 18.3251.48 ± 18.64
MLGC47.72 ± 17.6850.87 ± 18.12
DGFLP49.57 ± 18.6151.96 ± 19.02
MDLP42.6 ± 18.5645.57 ± 19.01
Proposed method55.35 ± 21.6858.68 ± 21.77
PIE
1 labeled sample
SMGI67.71 ± 9.0763.03 ± 12.48
MLGC66.46 ± 8.7262.12 ± 12.62
DGFLP68.96 ± 9.1763.59 ± 12.22
MDLP63.01 ± 10.5460.04 ± 12.83
Proposed method79.04 ± 8.2179.77 ± 8.81
PIE
2 labeled samples
SMGI77.65 ± 6.2674.57 ± 7
MLGC76.16 ± 6.2473.47 ± 6.95
DGFLP79.06 ± 6.6575.54 ± 7.46
MDLP73.39 ± 6.7170.11 ± 7.95
Proposed method88.35 ± 5.587.21 ± 5.9
FERET
1 labeled sample
SMGI98.38 ± 0.2598.56 ± 0.24
MLGC98.25 ± 0.398.52 ± 0.31
DGFLP98.43 ± 0.13 98.55 ± 0.13
MDLP96.32 ± 0.7297.12 ± 0.4
Proposed method98.83 ± 0.0899.03 ± 0.13
FERET
2 labeled samples
SMGI98.71 ± 0.2998.9 ± 0.32
MLGC98.49 ± 0.3998.82 ± 0.31
DGFLP98.65 ± 0.4398.8 ± 0.37
MDLP97.01 ± 0.4997.5 ± 0.86
Proposed method98.99 ± 0.399.13 ± 0.26
Table 5. Mean accuracy and standard deviation of the proposed method compared with other methods on LBP feature descriptor.
Table 5. Mean accuracy and standard deviation of the proposed method compared with other methods on LBP feature descriptor.
Accuracy (Mean ± STD)
SMGIMLGCDGFLPMDLPProposed Method
PF0147.66 ± 10.9446.55 ± 11.3747.82 ± 11.117.47 ± 11.2556.19 ± 8.97
1 labeled sample
PF0156.77 ± 7.16 55.83 ± 7.8256.9 ± 7.2313.7 ± 10.5865.47 ± 5.86
2 labeled samples
Extended_Yale73.48 ± 10.5258.29 ± 1180.77 ± 9.9812.34 ± 20.3194.25 ± 1.52
1 labeled sample
Extended_Yale80.05 ± 4.7965.18 ± 6.8786.73 ± 5.217.01 ± 11.6196.73 ± 1.93
2 labeled samples
PIE36 ± 5.934 ± 5.5136.09 ± 5.5816.28 ± 5.1246.79 ± 7.36
1 labeled sample
PIE49.84 ± 7.8548.06 ± 7.7250.65 ± 7.6330.4 ± 9.8661.64 ± 5.99
2 labeled samples

Share and Cite

MDPI and ACS Style

Bahrami, S.; Bosaghzadeh, A.; Dornaika, F. Multi Similarity Metric Fusion in Graph-Based Semi-Supervised Learning. Computation 2019, 7, 15. https://doi.org/10.3390/computation7010015

AMA Style

Bahrami S, Bosaghzadeh A, Dornaika F. Multi Similarity Metric Fusion in Graph-Based Semi-Supervised Learning. Computation. 2019; 7(1):15. https://doi.org/10.3390/computation7010015

Chicago/Turabian Style

Bahrami, Saeedeh, Alireza Bosaghzadeh, and Fadi Dornaika. 2019. "Multi Similarity Metric Fusion in Graph-Based Semi-Supervised Learning" Computation 7, no. 1: 15. https://doi.org/10.3390/computation7010015

APA Style

Bahrami, S., Bosaghzadeh, A., & Dornaika, F. (2019). Multi Similarity Metric Fusion in Graph-Based Semi-Supervised Learning. Computation, 7(1), 15. https://doi.org/10.3390/computation7010015

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop