Structure fusion based on graph convolutional networks for semi-supervised classification

Suffering from the multi-view data diversity and complexity for semi-supervised classification, most of existing graph convolutional networks focus on the networks architecture construction or the salient graph structure preservation, and ignore the the complete graph structure for semi-supervised classification contribution. To mine the more complete distribution structure from multi-view data with the consideration of the specificity and the commonality, we propose structure fusion based on graph convolutional networks (SF-GCN) for improving the performance of semi-supervised classification. SF-GCN can not only retain the special characteristic of each view data by spectral embedding, but also capture the common style of multi-view data by distance metric between multi-graph structures. Suppose the linear relationship between multi-graph structures, we can construct the optimization function of structure fusion model by balancing the specificity loss and the commonality loss. By solving this function, we can simultaneously obtain the fusion spectral embedding from the multi-view data and the fusion structure as adjacent matrix to input graph convolutional networks for semi-supervised classification. Experiments demonstrate that the performance of SF-GCN outperforms that of the state of the arts on three challenging datasets, which are Cora,Citeseer and Pubmed in citation networks.


Introduction
As a efficient representation of data distribution, graph plays a important role for describing the intrinsic structure of data. Therefore, many existing works have constructed the significant theory and method depending on the graph structure of data in pattern recognition, such as graph cut building energy function for semantic segmentation task [1], graph-based learning system constructing the accurate recommendations for the interaction of the different objects [2] [3], graph modeling molecules bioactivity for drug discovery [4] [5] and graph simulating the link connection of citation network for the different group classification [4][5] [6]. In fact, we usually observe objects and their relationship (this relationship is defined as the objects of structure, which often can be described by graph.) from multiple views, which provide the more abundant and complete information for object recognition. Learning on multi-graph (multiple observation structure) can effectively mine multiple relationship to discriminate the different object. Existing learning methods on multi-graph trend to tow ways. One is structure fusion [7] [8] [9][10] [11] [12][13] [14] [15] [16] [17] or diffusion on tensor product graph [18] [19] [20] [21] [22] [23] based on the complete data, which include each view obser-vation data. Another is graph convolutional networks for the salient graph structure preservation [6] based on the incomplete data, which lost some view observation data.
For example, link relationship can be extracted by application necessary in citation networks, but it can not be described by the corresponding observation data computation.
In other words, these link relationship exists, while the corresponding support data lost. Therefore, the method based on graph convolutional networks usually ignores the complete complementary of the different observation structure based on the incomplete multi-view data. To analysis this issue, we attempt to construct structure fusion based on graph convolutional networks for classification. Figure 1 shows the overall flow diagram of structure fusion based on graph convolutional networks (SF-GCN).
The inspiration of SF-GCN comes from Multi-GCN in the literature [6], but there are tow points difference comparison with Multi-GCN. One is that SF-GCN considers the inequality of multiple structures, while Multi-GCN only equally deal with their relationship. The other is that SF-GCN focuses on the contributions of all nodes structure in the fusion structure, while Multi-GCN only emphasises on the salient structure of the part nodes. From the classification sense,the strong and weak links between nodes both considered for complementing structure can more fit to the intrinsic structure of the data for classification.
Our contributions can be summarized as following. (a)We present a novel structure fusion based on graph convolutional networks (SF-GCN) that discriminates the different classes by optimizing the linear relationship of multiple observation structure with balancing the specificity loss and the commonality loss. (b) In three citation datasets with document sparse feature and document link relationship, the proposed SF-GCN outperforms the state of the arts for semi-supervised classification. (c) Our model is generalized the different multi-graph fusion methods for evaluating the performance of the proposed SF-GCN.

Related Works
In this section, we mainly review recent related works about structure fusion and graph neural network.

Structure fusion
Structure fusion initially proposed in [7]can merge multiple structures for shape classification. In the follow-up works, the extend methods can be divided into three categories according to the different fusion ways. The first kind of methods try to find the optimized linear relationship of multiple observation structure based on the different manifold learning method [8] or statistics model analysis [9]. The second kind of methods attempt to mine the nonlinear relationship of heterogeneous feature structure based on the global feature [10] [11] or the local feature encoding [12]. The third kind of methods can capture the dynamic changes of multiple structures for semi-supervised classification [13]or the structure propagation way for zero-shot learning [14] [15] [16] [17].
From above mention, existing methods emphasis on the completeness of data and their relationship based on data project, while graph convolutional networks focus on the transformation and evolution of data structure by deep learning frameworks. Therefore, we expect to draw support from structure fusion based on structure metric and graph convolutional networks for processing the incomplete view data, and find evolution law of the the fusion structure with the consideration of their specificity and commonality.

Graph neural network
Graph neural networks can discover the potential data relationship by the computation based on graph nodes and links. Especially, the computation is defined as convolution for graph data, and graph convolution networks (GCN) have become a promising direction in pattern recognition. In terms of the different node representation, graph convolution networks include spectral-based GCN and spatial-based GCN. Spectralbased GCN can define graph Fourier transform based on graph Laplacian matrix for projecting graph signal into the orthonormal space. The difference of these methods is the selection of the filter, which may be the learned parameters set [24], Chebyshev polynomial [4], or the first-order Chebyshev polynomial [25] [26]. Spatial-based GCN regards images as a special graph with a pixel describing a node. To avoid the storage of all states, these methods have present the improved training strategies, such as subgraph training [27]or stochastically asynchronous training [28]. Furthermore, some complex networks architecture can utilize gating unite to control the selection of node neighborhood [29], or design two graph convolution networks with the consideration of the local and global consistency on graph [30] , or adjust the receptive field of node on graph by hyper-parameters [31].
Because spectral-based GCN can explicitly construct the learning model on the graph structure,which can easily be separated from GCN architecture. Therefore, this point provides a way for processing multiple structures, which may be incremental. In this paper, we focus on the important role of graph (structure) from multi-view data, and attempt to mine the plentiful information from multiple structures for spectralbased GCN inputting. To the best of our knowledge,existing structure fusion methods usually construct the optimizing function for feature projection, in which feature data and the corresponding structure jointly participate in computation. Because of the possible loss and the structure preservation of multi-view data, we expect to build a novel structure fusion by structure metric, in which the optimizing function only involves multiple structures for avoiding the negative effect of the data lost. Simultaneously, multiple structures have each specificity and their commonality. Therefore,we also anticipate that a novel structure fusion can be constrained by these characteristics of multiple structures. Figure 2 demonstrates the internal mechanism of structure fusion in SF-GCN. First, we construct the specificity loss based on spectral embedding method with the consideration of multiple structure linear relationship. Second, we measure the commonality loss between multiple structures based on distance metric in Grassman manifold. Finally, we jointly exploit the structure fusion based on two losses, and input GCN for classification.

Specificity loss of multiple structures
Given an object set with m multi-view, we can use graph G i to describe the observation distribution of data on each view. Therefore, the graph G i is the representation of the observation structure and G = {G i |i = 1, 2, ..., m} can indicate multiple structures of data from multi-view observations. Because multiple structures detail the same object set, each G i includes the same vertex set V , or the possible different edges set E i . If W i is the adjacency matrix of G i and is the numerical expression of the structure in ith view. In terms of spectral embedding, we can obtain the following optimization function on the embedding matrix Y i ∈ R n×k (n is the number of samples, and k is the dimension of embedding space) of each view.
Therefore, L i can still describe the characteristic of structure on graph G i . We can compute the embedding matrix Y i by optimizing equation (1), which is equivalent to a eigenvalue solution problem. When all eigenvalues are solved, eigenvectors corresponding to the smallest eigenvalues can build the embedding matrix Y i , which can project the original nodes into the low dimensional spectral space [32]. We can regard as the specificity loss of structure on graph graph G i , and then we can reformulate the specificity loss of multiple structures as follow.
Where, Y is the embedding matrix of multiple structures in graph G and closely approximates Y i . Suppose fusion structure W is the linear combination of W i , then L and L i have the same linear relationship L = m i=1 β i L i , in which β i is the linear coefficient to encode the importance of multiple structures.

Commonality loss of multiple structures
To measure the commonality loss of multiple structures, we need metric the distance between Laplacian matrix L i and L. According to the solvation of the equation (1), we can obtain the equation (3) for describing the internal connection between embedding matrix Y i and the corresponding Laplacian matrix L i .
Where, λ i is diagonal matrix, in which diagonal values is the smallest eigenvalues of L i . Y i can be explained a subspace for preserving the smaller variance of the column in L i , that is for reserving the bigger variance of the column in the structure W i . In other words, Y i can keep the more discrimination of the data. Similarly, Y has the same sense in multiple observation structure. Therefore, we can replace the distance between Laplacian matrix L i and L by the distance between Y i and Y for indirectly computing the commonality loss of multiple structures. This point is consistent with the specificity loss of the assumption, which is that Y i approximates Y between each view and multi-view.
In terms of Grassmann manifold theory [33][34], the orthonormal matrix Y i ∈ R n×k can be regard as the column of Y i spanning an unique subspace, which can be project into an unique point on Grassmann manifold G(n, k). Similarly, Y also can be mapped into an unique point on this Grassmann manifold. Therefore, the principle angles {θ j } k j=1 between these subspaces can represent the distance between Y i and Y . Furthermore, this distance can be reformulate as following [35].
In multiple structures, we can define the commonality loss Loss c as the distance be-

Structure fusion by structure metric losses
As two structure metric losses, specificity loss can balance the contribution of the structure in each view, while commonality loss can consider the similarity of multiple structures in multi-view. These structure metric losses can both constrain the linear of multiple structures. Therefore, we combine these structure metric losses as a total loss for encoding the importance of multiple structures. The total loss can be reformulated as following.
Where, α is regularization parameter. From equation (6), we can construct the object optimization function as following.
In commonality loss, constant term km can not influence the loss trend change, so we may remove this term for conveniently computing. Equation (7) is reformulated as equation (8) for balancing parameter {β i } m i=1 between 0 and 1.
Equation (8) is a nonconvex optimization problem, we can solve this problem by Y and γ alternated optimization. If γ is fixed, equation (8) can be transformed as a eigenvalue solving problem as following.
Equation (9)is equivalent to a eigenvalue solution problem. When all eigenvalues of M are solved, eigenvectors corresponding to the smallest eigenvalues can build the fusion embedding matrix Y . If Y is fixed, equation (8) can be converted into a quadratic programming problem as following.

Graph convolutional networks
In terms of the multiplication of convolution in the Fourier domain, graph convolution is defined as the the multiplication between the signal s ∈ R n and the filter g η [24].
Furtherly, graph convolution can also be approximated by 1 th -order Chebyshev polynomials [25] as following.  (1) 4: for 1 < t < T do 5: Computing the spectral embedding fusion Y of multiple structures in multi-view by equation (9) 6: Updating the linear relationship γ of multiple structures by equation (10) 7: end for Where, U is the eigen-decomposition of the normalized Laplacian L = I−D −1/2 W D −1/2 (I is the identity matrix; D is the degree matrix of graph G);L = I −D −1/2WD−1/2 (D andW respectively are the rescaled degree and adjacent matrix byW = W + I); T k expresses the Chebyshev polynomials; η = η 0 = −η 1 .
Fusion structure W can directly be input into the above graph convolutional networks. The forward propagation based on two layers of graph convolutional networks can be indicated as following.
Where, Z is the output of networks;S is the representation matrix of each nodes;Θ 0 and Θ 1 respectively are the 1 th and 2 th layer filter parameters; ReLU and sof tmax are the different type of activation function located in the various layers.

Experiments
For evaluating the proposed SF-GCN, we carry out the experiments from four aspects. Firstly, we conduct the comparing experiment between the proposed SF-GCN and the baseline methods, which include graph convolutional networks (GCN) [25] with the combination view and Multi-GCN [6]. Secondly,we utilize the different multigraph fusion methods for analyzing the intrinsic mechanism of the proposed SF-GCN.
Thirdly, we show the experimental results between the proposed SF-GCN and the state of the art methods for the node classification in citation networks. Finally, we implement the proposed SF-GCN method of the lost structure for demonstrating the importance of the complete structure.

Datasets
We use the paper-citation networks of the citation networks in experiments. The three popular datasets usually utilized in node classification respectively are Cora, Citeseer and Pubmed. Cora dataset has 7 classes that involve 2708 the grouped publication about machine learning and their undirected graph. Citeseer dataset includes 6 classes that have 3327 scientific papers and their undirected graph. In these datasets, each publication stands for a node of the related graph and is represented by one-hot vector, each element of which can indicate the presence and absence state of a word in the learned directory. Pubmed dataset has 3 classes that contain 19717 diabetes-related publications and their undirected graph. In this dataset, each paper (each node of the related graph) can be described by a term frequency-inverse document frequency (TF-IDF) [36]. Table 1 shows the statistics of these datasets. To obtain the structure of the second view from publication description, we normalize the cosine similarity between these publication. If these similarity is greater than 0.8, we produce an edge for the corresponding to nodes in the citation network. This configuration is same in the literature [6].

Experimental configuration
In experiments, we follow the configuration in GCN [25], in which we train a twolayer GCN for maximum of 200 epochs and test model in 1000 labeled samples. Moreover, we select the same validation set of 500 labeled sample for hyper-parameter optimization (dropout rate for all layers, number of hidden units and learning rate).
In proposed SF-GCN, we initially set the linear relationship {β i } m i=1 of multiple structures and regularization parameter α as 0.5, and then update these parameters in iteration optimization. The iteration time T of the algorithm is 5 according it's the convergence degree in fact.

Comparison with the base-line methods
The proposed method (SF-GCN) can be constructed based on GCN [25], and attempt to mine the different structure information for completing the intrinsic structure in multi-view data. Therefore, two base-line methods (GCN and Multi-GCN can find and capture the different structure information from the different consideration.) is involved for processing multi-view data based on GCN. GCN for multi-view [25] can concatenate the different structure to build a sparse block-diagonal matrix where each block corresponding to the different structure (the adjacent matrix of different graph).
Multi-GCN [6] can preserve the significant structure of the different structure by manifold ranking. In contrast with these base-line methods, the proposed method (SF- GCN) can not only enhance the common structure, but also retain the specific structure by structure fusion. Table 2 shows that the classification performance of SF-GCN outperforms that of the base-line methods and the least improvement of SF-GCN respectively is 0.7% for Cora, 2.1% for Citeseer and 0.6% for PubMed. However, GCN for multi-view is not superior to GCN for single-view, and it demonstrates that information mining of multiview data is a key point for node classification. Therefore, SF-GCN attempt to mine the structure information from multi-view data for this purpose and obtain the better performance.  [22] and propagation [16] [14] of the different structure can also describe the complex relationship of the various structure, and become the important part of structure fusion. Therefore, we can define fusion structure W by the propagation fusion (PF) of the different structure as follow.
The propagation fusion can exchange and interact the relationship information between the various structures, and mine the neighbour relationship of multiple structures. However,this propagation can effect on the clustering performance of the original structure by high-order iteration multiplication. Therefore, we only consider zero-order (for example SF) and first-order (for instance PF) multiplication, that is structure propagation fusion (SPF) as follow.
For evaluating structure fusion generalization, we compare structure fusion based graph convolutional networks (SF-GCN), propagation fusion based graph convolutional networks (PF-GCN) and structure propagation fusion based graph convolutional networks (SPF-GCN).In Table 3, we observe that the performance of SPF-GCN is better than that of other method, and the least improvement of SPF-GCN respectively is 0.2% for Cora, 0.1% for Citeseer and 0.7% for PubMed, while the performance of SP is superior to that of PF-GCN, and the improvement of SF-GCN respectively is 0.6% for Cora, 0.9% for Citeseer and 0.2% for PubMed Therefore, PF and SF both are benefit for further mining the structure information and the role of SF is more important than that of PF.

Comparison with the state-of-the-arts
Because graph convolutional networks and structure fusion are basic ideas for constructing the proposed method SPF-GCN, we analyze six related state-of-the-arts methods for evaluating SPF-GCN. These methods include two categories. One is node neighbour information exploiting for GCN, and another is node information fusion based on GCN.
Node neighbour information exploiting attempts to capture the distribution structure of the node neighbour for obtain the stable graph structure representation. For example, graph attention networks(GAT) can specify different weights to different nodes in a neighborhood [37]; stochastic training of graph convolutional networks (StoGCN) allows sampling an arbitrarily small neighbor size [38]; deep graph infomax(DGI) can maximize mutual information between different level subgraph centered around nodes of interest (the different way for considering neighbour information) [39].
Node information fusion tries to mine the information from multi-view node description or multiple structures for complementing the difference of multi-view data.
For instance, large-scale learnable graph convolutional networks (LGCN) can fuse neighbouring nodes feature by ranking selection to transform graph data into grid-like structures in 1-D format [40]; dual graph convolutional networks (DGCN) can consider local and global consistency for fusion different views graph of raw data [30]; Multi-GCN can extract and select the significant structure form multi-view structure by manifold ranking [6].
The proposed method SPF-GCN belongs to node information fusion method, and the difference compared with the above methods focuses on the complementary of multiple structures by mining their commonality, specificity and interactive propagation. Table 4 shows SPF-GCN outperforms other state-of-the-art methods except DGCN in Cora and PubMed datasets. Although SPF-GCN and DGCN reach to the same performance in Cora and PubMed datasets, SPF-GCN can preserve the higher computation efficient of the original GCN because of the separable computation between structure fusion and GCN.

Incomplete structure influence
Structure fusion can capture the complementary information of multiple structures, and this complementary information can supply an efficient way for incomplete structure influence. The main reason of the incomplete structure may be because of noise and data loss in practical situation. For evaluating the performance of the proposed methods under the condition of the incomplete structure, we design a experiment in all datasets. In semi-supervised classification, the distribution structure of test datasets is more important than that of train datasets, and can assure the performance of classification because of the transfer relation of structure between train and test datasets.
Therefore, we delete the some structure of test datasets to destroy this transfer relation for simulating incomplete structure.
In the details, we proportionally set the adjacency matrix(graph structure from the original dataset) of elements (corresponding to test datasets) to zero from 10% to 60%, and then respectively implement GCN for multi-view [25], DGCN [30], SF-GCN and SPF-GACN methods in all dataset. In figure 3, we select structure loss degree from We can observe that the proposed SF-GCN and SPF-GACN is more stable and robust with incomplete degree increasing of structure than GCN for multi-view and DGCN.
In this situation, the performance of SPF-GACN is better than that of SF-GACN, while the performance of GCN outperforms that of DGCN in Cora datasets, and the performance of DGCN is superior to that of GCN in Citeseer and PubMed datasets. The details of this reason can be analyzed in section 4.7.

Experimental results analysis
In our experiments, we compare the proposed method with eight methods, which contain two kinds of base-line methods (Multi-GCN [6], GCN [25] for multi-view, GCN [25] for view1 and view2 in section 4.3), two kinds of structure fusion generalization methods (PF-GCN and SF-GCN in section 4.4), and six kinds of the state-ofthe-art methods(GAT [37], StoGCN [38], DGI [39], LGCN [40], DGCN [30] and Multi-GCN [6] in section 4.5). These methods can utilize the graph structure mining based graph convolutional networks for semi-supervised classification by the different ways.
In contrast to other methods, the proposed SF-GCN and SPF-GCN methods focus on the complementary relationship of multiple structures by the consideration of their commonality and specificity. Moreover, the proposed SPF-GCN method not only cap-ture the optimization distribution of fusion structure, but also emphasize on the interactive propagation between the different structures. From the above experiments, we can observe several points as following.
• The performance of SF-GCN is superior to that of the base-line methods (Multi-GCN [6], GCN [25] for multi-view, GCN [25] for view1 and view2 in section 4.3). GCN [25] constructs a general graph convolutional architecture by the first-order approximation of spectral graph convolutions for greatly improving the computation efficiency of graph convolutional networks, and also provides a feasible deep mining frameworks for effective semi-supervised classification.
For using multiple structures, GCN for multi-view can input a sparse blockdiagonal matrix, each block of which corresponding to the different structure.
Therefore, the relationship of each block (the different structure) is ignored for GCN, and this point leads to the poor performance (in some times, the performance of GCN for multi-view is worse than that of GCN for view1) of GCN for multi-view. In contrast, Multi-GCN [6] can capture the relationship of the different structure to preserve the significant structure of merging subspace. However, Multi-GCN [6] neglects the optimizing fusion relationship of the different structure, while the proposed SF-GCN focuses on finding these relationship by jointly considering the commonality and specificity loss of multiple structure for obtaining the better performance of semi-supervised classification.
• SPF-GCN shows the best performance in structure fusion generalization experiments, whereas the performance of SF-GCN is better than that of SP-GCN. The can compromise these constrains for balancing the incomplete structure information by optimizing the weight of multiple structures, and also connect the different structure for complementing the different information. Therefore, the proposed SPF-GCN obtains the best performance in experiments.
• The proposed SPF-GCN expect to mine the commonality and the specificity of multiple structures. The commonality describes the similarity characteristic of structures by Grassmann manifold metric, while the specificity narrates the difference characteristic of structures by spectral embedding. In the proposed method, the specificity is constructed based on the commonality. Therefore, we only execute the ablation experiment for preserving the commonality loss by deleting the specificity loss from the total loss. This experiment obtain the following performance, that is 82.6% in Cora, 71.5% in Citeseer and 78.9% in PubMed. These results obviously are worse than the performance of the proposed SF-GCN and SPF-GCN, which can balance the commonality and specificity for mining the suited weight of multiple structures.

Conclusion
We have proposed structure fusion based on graph convolutional networks (SF-GCN) to address the multi-view data diversity and complexity for semi-supervised classification. SF-GCN can not only adapt spectral embedding to preserve the specificity of structure, but also model the relationship of the different structure to find the 6. Acknowledgements