Extreme Learning Machine for Multi-Label Classiﬁcation

: Extreme learning machine (ELM) techniques have received considerable attention in the computational intelligence and machine learning communities because of the signiﬁcantly low computational time required for training new classiﬁers. ELM provides solutions for regression, clustering, binary classiﬁcation, multiclass classiﬁcations and so on, but not for multi-label learning. Multi-label learning deals with objects having multiple labels simultaneously, which widely exist in real-world applications. Therefore, a thresholding method-based ELM is proposed in this paper to adapt ELM to multi-label classiﬁcation, called extreme learning machine for multi-label classiﬁcation (ELM-ML). ELM-ML outperforms other multi-label classiﬁcation methods in several standard data sets in most cases, especially for applications which only have a small labeled data set.


Introduction
Multi-label classification deals with one object which possibly belongs to multiple labels simultaneously, which are common in real-world applications, such as text categorization, scene and video annotation, bioinformatics, and music emotion classification [1].For example, a sunrise image could be labeled with "sun," "sky" and "sea" at the same time in semantic scene classification.Formally [2], let X " R d denote the d-dimensional input space, and Y " ty 1 , y 2 , . . ., y m u denote the label space consisting of m possible class labels.The task of multi-label learning is to learn a function: h : X Ñ 2 Y from the multi-label training set D " tpx i , Y i q |1 ď i ď nu .For each multi-label example (x i , Y i ), x i P X is a d-dimensional feature vector (x i1 , x i2 , . . ., x id ) T and Y i Ď Y is the relevant label with x i .For any unseen instance x i P X , the multi-label classifier h p¨q predicts h pxq Ď Y as the set of proper labels for x.
Multi-label classification has attracted a lot of attention in the past few years [3][4][5][6].Nowadays, there are two main ways to construct various discriminative multi-label classification algorithms: problem transformation and algorithm adaptation.The key philosophy of problem transformation methods is to fit the data to the algorithm, while the key philosophy of algorithm adaptation methods is to fit the algorithm to the data [2].
A problem transformation strategy tackles a multi-label learning problem by transforming it into multiple independent binary or multi-class sub-problems, constructing a sub-classifier for each sub-problem using an existing technique, and then assembling all sub-classifiers into an entire multi-label classifier.It is convenient and fast to implement a problem transformation method due to the number of existing techniques and their free software.Representative algorithms include Binary Relevance [7], AdaBoost.MH [8], Calibrated Label Ranking [3], Random k-labelsets [9], etc.
An algorithm adaptation strategy tackles multi-label learning problem by adapting popular learning techniques to deal with multi-label data.Representative algorithms include Multi-Label k-Nearest Neighbor (ML-kNN) [10], Multi-Label Decision Tree (ML-DT) [11], Ranking Support Vector Machine (Rank-SVM) [12], Backpropagation for Multi-Label Learning (BP-MLL) [13], etc.The basic idea of ML-kNN is to adapt k-nearest neighbor techniques to deal with multi-label data, where a maximum a posteriori (MAP) rule is utilized to make predictions by reasoning with the labeling information embodied in the neighbors.The basic idea of BP-MLL is to adapt feed-forward neural networks to deal with multi-label data, where the error back propagation strategy is employed to minimize a global error function capturing label correlations.
In the multi-labeled setting, classes belonging to one instance are often related to each other.The performance of the multi-label learning system is poor if it ignores the relationships between the different labels of each instance.Therefore, the famous Rank-SVM defines the margin over hyper planes for relevant-irrelevant label pairs, which explicitly characterizes label correlations of an individual instance.Rank-SVM achieves great accuracy.Unfortunately, Rank-SVM has a high computational cost, which limits its usability for many applications.Therefore, it is still necessary to build some novel efficient multi-label algorithms.
Recently, Huang et al. [14][15][16] proposed a novel learning algorithm for single-hidden layer feedforward neural networks called extreme learning machine (ELM).The single-hidden layer feedforward neural networks have been widely applied in machine learning [17][18][19][20][21][22] and ELM represents one of the recent successful approaches in machine learning.Compared with traditional computational intelligence techniques, ELM exhibits better generalization performance at a much faster learning speed and with fewer human interventions.ELM techniques have received considerable attention in computational intelligence and machine learning communities, in both theoretic study and applications [23][24][25][26][27][28][29].ELM provides solutions to regression, clustering, feature learning, binary classification and multiclass classifications, but not to multi-label learning, which is a harder task than traditional binary and multi-class problems.Therefore, a thresholding method-based ELM is proposed in this paper to adapt ELM to multi-label classification, called ELM-ML (Extreme Learning Machine for Multi-Label classification).Experiments on eight multi-label datasets show that the performance of ELM-ML is superior to some other well-established multi-label learning algorithms including Rank-SVM, ML-kNN, BP-MLL and Multi-Label Naïve Bayes (MLNB) in most cases, especially for applications which only have a small labeled data set.

A Brief Review of ELM
This section briefly reviews the standard ELM [14].ELM was originally proposed for single hidden-layer feedforward neural networks (SLFNs).For N arbitrary distinct samples (x i , t i ), where x i " rx i1 , x i2 , . . ., x id s T P R d and t i " rt i1 , t i2 , . . ., t im s T P R m , standard SLFNs with r N hidden nodes and activation function g pxq are mathematically modeled as where w i " rw i1 , w i2 , . . ., w iN s T is the weight vector connecting the ith hidden node and the input nodes, β i " rβ i1 , β i2 , . . ., β im s T is the weight vector connecting the ith hidden node and the output nodes, and b i is the thresholding of the ith hidden node.w i ¨xj denotes the inner product of w i and x j .
If a SLFNs with r N hidden nodes with activation function g pxq can approximate these N samples with zero error, it means that ř r N j"1 o j ´tj " 0, i.e., there exist β i , w i and b i such that ÿ r N i"1 The above N equations can be written compactly as where H is called the hidden layer output matrix of the SLFN; the ith column of H is the ith hidden node output with respect to inputs x 1 , x 2 , . . ., x N .It has been proven [15] that if the activation function g is infinitely differentiable in any interval, the hidden layer parameters can be randomly generated.Therefore, Formula (3) becomes a linear system and the output weights β are estimated as where H † is the Moore-Penrose generalized inverse of the hidden layer output matrix H. Thus, ELM randomly generated hidden node parameters and then analytically calculated the hidden layer output matrix H and the output weights β.This avoids any long training procedure where a hidden layer of SLFNs need to be tuned.Compared with traditional computational intelligence techniques, ELM provides better generalization performance at a much faster learning speed and with fewer human interventions.

ELM-ML
In this section, we will describe our multi-label classification algorithm, called an extreme learning machine for multi-label classification (ELM-ML).
From the standard optimization method point of view [15], ELM with multi-output nodes can be formulated as Minimize : Formula (7) tends to reach not only the smallest training error but also the smallest norm of output weights, where 1 ď i ď N. N is the number of training samples.ξ i " " ξ i,1 , . . ., ξ i,m ‰ T is the training error vector of the m output nodes with respect to the training sample x i .C is a user-specified parameter and provides a trade-off between the distance of the separating margin and the training error.The predicted class label of a given testing sample is the index number of the output node which has the highest output value.Formula (7) provides a solution to multi-class classifications.Multi-label learning is a harder task than traditional multi-class problems, which is a special case of multi-label classification.One sample belongs to several related labels simultaneously, so we cannot simply regard the index number of the highest output value as a predicted class for a given testing sample.A proper thresholding function th pxq should be set.Naturally, the predicted class labels for a given testing sample are those index numbers for output nodes which have higher output value than the predefined thresholding.
Setting th pxq as a constant function is a common multi-label thresholding method.One straightforward choice is to use zero as the calibration constant [7,8].An alternative choice for the calibration constant is 0.5, when the multi-label learned model f px, yq represents the posterior probability of y being a proper label of x [10,11,23].Another popular approach, called experimental thresholding [30], consists in testing different values as thresholds on a validation set and choosing the value which maximizes the effectiveness.Of all the threshold calibration strategies above, experimental thresholding seems to be the most reasonable, but it is time-consuming and labor-intensive.We believe that the thresholding function th pxq should be learned from instances.That is to say, different instances should correspond to different thresholdings in multi-label learning model.A novel method would be to consider the thresholding function th pxq as a regression problem for the training data.In this paper, we use an ELM algorithm with a single output node to solve this regression problem.Overall, the proposed ELM-ML algorithm has two phases: multi-class classifier-based ELM with multi-outputs and thresholding function learning-based ELM with single outputs.The pseudo-code of ELM-ML is summarized in Figure 1.

Experiments
Firstly, we compare the proposed thresholding strategy in this paper with the constant thresholding strategy and strategy in Rank-SVM [12].Secondly, we compare the performance of

Experiments
Firstly, we compare the proposed thresholding strategy in this paper with the constant thresholding strategy and strategy in Rank-SVM [12].Secondly, we compare the performance of different multi-label classification algorithms, including our algorithm ELM-ML, Rank-SVM, MLNB, BP-MLL [13] and ML-kNN [10] on eight multi-label classification data sets.Before presenting our experimental results, we briefly introduce eight benchmark data sets and six multi-label performance measures.

Datasets
In order to verify the performance of thresholding strategy and different multi-label classification algorithms, a wide variey of data sets have been tested in our simulations, which are of small/large sizes, low/high dimensions, and small/large labels.These data sets [31] include diversified multi-label classification cases, which cover four distinct domains: text, scene, music and biology.
To characterize the properties of the multi-label data sets, several useful multi-label indicators can be utilized.The most natural way to measure the degree of multi-labeledness is Label Cardinality , the average number of labels per sample.Accordingly, Label Density(LD) normalizes label cardinality by the number of possible labels in the label space: LD pDq "  1, the data sets cover a different range of cases whose characteristics are diversified.These data sets are categorized into three groups, such as small data sets, medium data sets and large data sets, according to the amount of training samples.Significantly, there are not only low or high dimensions but also small or large labels in any group.

Evaluation Measures
With the aim of fair and honest evaluation, performance of the multi-label learning algorithms should be tested on a broad range of metrics instead of only on the one being optimized.In this paper, we chose six evaluation criteria suitable for classification: Subset Accuracy, Hamming Loss, Accuracy, Precision, Recall and F1.These measures are defined as follows [2].
Assume a test data set of size n to be S = {(x 1 , Y 1 ), . . ., (x i , Y i ), . . ., (x n , Y n )} and h p¨q be the learned multi-label classifier.A common practice in multi-label learning is to return a real-valued function f px, yq.For a unseen instance x, the real-valued output f px, yq on each label should be calibrated against the thresholding function output th pxq.

Hamming Loss
The hamming loss evaluates the fraction of misclassified instance-label pairs, i.e., a relevant label is missed or an irrelevant label is predicted.
Hamming Loss phq " where ∆ stands for the symmetric difference between two sets.

‚ Subset Accuracy
The subset accuracy evaluates the fraction of correctly classified examples, i.e., the predicted label set is identical to the ground-truth label set.Intuitively, subset accuracy can be regarded as a multi-label counterpart of the traditional accuracy metric, and tends to be overly strict especially when the size of the label space is large.Subset Accuracy phq "

Accuracy, Precision, Recall and F1
Accuracy phq " Precison phq " Recall phq " F1 phq " 2 ˆPrecision phq ˆRecall phq Precision phq `Recall phq (13) Here, Y i and h px i q correspond to the ground-truth and predicted label set for x i , respectively.Obviously, except for the first metrics, the larger the last five metric value, the better the system's performance is.

Thresholding Function
We compare three thresholding strategies: the proposed thresholding strategy in ELM-ML, thresholding strategy in Rank-SVM and constant strategy.To achieve an optimal constant thresholding parameter Ct, we tested different values as thresholdings on the validation set and chose the value which maximizes the effectiveness.More specifically, hamming loss and subset accuracy are regarded as criteria to tune constant parameter Ct, respectively.Constant parameter Ct is tuned from ´1 to 1 with interval step rate δ " 0.1.Rank-SVM employs the stacking-style procedure to set the thresholding function th pxq [12].We apply multi-class ELM algorithm providing scores for each sample, then use three thresholding strategies, called ELM-ML, ELM-Rank-SVM and ELM-Constant, to predict labels for each sample.
In experiments, our computational platform is a 32-bit HP workstation with Interl ® (Santa Clara, CA, USA) Core TM i3-2130 CPU (4CPUs) except for on TMC2007-500 data set and TMC2007 data set, because the amount of training data for these two data sets is very large, and for which the computational platform is a 64-bit HP server with Interl ® Core TM i7-4700MQ CPU (8CPUs).
All results are detailed in Tables 2-4.The best performance is highlighted in boldface.As seen in Tables 2-4, the proposed thresholding strategy in ELM-ML achieves the highest performance among them in most cases.It is worthwhile to note that we tune parameter Ct very carefully on each data set in a constant thresholding strategy.The hamming loss and subset accuracy measures are regarded as criteria to tune parameter Ct.That is to say, when we verified these three thresholding strategies using hamming loss, hamming loss was regarded as a criterion to tune parameter Ct; when we verified these three thresholding strategies using subset accuracy, subset accuracy was regarded as a criterion to tune parameter Ct.Unfortunately, a constant thresholding strategy holds only a slight advantage in the subset accuracy criterion for the scene data set.Meanwhile, the constant thresholding strategy outperforms others for the hamming loss criterion only on the Enron and CAL500 data sets.Training times and testing times are listed in Table 4.We only compare running time of the thresholding strategy in ELM-ML and the thresholding strategy in Rank-SVM, because the constant thresholding is time-consuming.Obviously, ELM-ML achieves overwhelming performance.In conclusion, the proposed thresholding strategy in ELM-ML is effective and efficient.

Multi-Label Algorithms
We also compare the performances of different multi-label classification algorithms, including ELM-ML, Rank-SVM, BP-MLL, MLNB and ML-kNN on eight multi-label classification data sets.We downloaded Matlab code [32] of BP-MLL, MLNB and ML-kNN .We developed the ELM-ML algorithm in Matlab and chose a sigmoid activation function.Hidden nodes were set to r N = 1000.We accept their recommended parameter settings.The best parameters of BP-MLL, MLNB and ML-kNN reported in the literature [10,13,33], were used.For BP-MLL, the learning rate is fixed at 0.05, the number of hidden neurons is 20% of the number of input neurons, the training epochs is set to be 100 and the regularization constant is fixed to be 0.1.For MLNB, the fraction of remaining features after PCA is set to the moderate value of 0.3.For ML-kNN, the Laplacian estimator s = 1 and k = 10 are used.The number of iterations is fixed at 100.For Rank-SVM developed in Matlab, a Gaussian kernel is tested, where kernel parameter γ and cost parameter C need to be chosen appropriately for each data set.In our experiments, the Hamming Loss measure is regarded as a criterion to tune two parameters.To achieve an optimal parameter combination (γ, C), we use a similar tuning procedure as in [9].The optimal parameters of Rank-SVM on each data set are shown in Table 5. Due to the large size of the label space, Rank-SVM failed to calculate the results on CAL500.On the other hand, Rank-SVM, BP-MLL and MLNB did not output experimental results for TMC2007 and TMC2007-500.The amounts of training data of TMC2007 and TMC2007-500 are huge, which leads to high computational complexity.BP-MLL needs more iteration, which is time-consuming.Therefore, there are no results on these three data sets or parts of them.The detailed experimental results are shown in Tables 6-12.The best performance among the five comparing algorithms is highlighted in boldface.From Tables 6-11, ELM-ML obtains the best performances in all six criteria on a small data set.Genbase, Emotions and CAL500 are all small-size training data regardless of the size of labels and feature dimensions.That is to say, the proposed ELM-ML is more suitable to solve those applications for which a large amount of labeled data is difficult to obtain.ELM-ML tends to achieve better results than other well-established multi-label algorithms when only a small amount of labeled training data is available.However, ELM-ML is inferior to others on medium data sets, whereas Rank-SVM and ML-kNN work well.It is interesting that if a large number of labeled data are obtained easily, ELM-ML presents the best performance.
Furthermore, training times and testing times are listed in Table 12.ELM-ML achieves the best testing time and ML-kNN obtains the best training time except on the Scene and TMC2007-500 data sets.

Conclusions
In this paper, we present an ELM-ML algorithm to solve multi-label classification.ELM is regarded as a recent successful approach to machine learning, because ELM requires a significantly lower computational time for training a learning model and provides better generalization performance with less human intervention.However, ELM does not provide a solution to multi-label classifications.A post-processing step, threshold calibration strategies, should be used to predict the label set of a given sample.A novel method would be to consider the thresholding function th pxq as a regression problem for training data with class labels.In this paper, we first use an ELM algorithm with multi-output nodes to train a learning model returning a real-valued function, then use the ELM algorithm with a single output node to learn a thresholding function.Experiments on eight diverse benchmark multi-label datasets show that ELM-ML is effective and efficient.

Entropy 2016 ,
18, 225 4 of 12 learning model.A novel method would be to consider the thresholding function ℎ( ) as a regression problem for the training data.In this paper, we use an ELM algorithm with a single output node to solve this regression problem.Overall, the proposed ELM-ML algorithm has two phases: multi-class classifier-based ELM with multi-outputs and thresholding function learning-based ELM with single outputs.The pseudo-code of ELM-ML is summarized in Figure 1.
1 |y| ¨LC pDq.Table 1 describes these eight benchmark data sets, in which #Training and #Test means the numbers of training examples and test examples, respectively.As shown in Table

Table 1 .
Information for eight benchmark data sets.

Table 2 .
Hamming loss for three thresholding strategies on eight data sets (hamming loss Ó).

Table 3 .
Subset sccuracy for three thresholding strategies on eight data sets (subset accuracy Ò).

Table 4 .
Computation time of non-constant thresholding strategies.

Table 5 .
The optimal γ and C values for each data set.

Table 12 .
Computation time of four algorithms.