Semi-Supervised Clustering via Constraints Self-Learning

Abstract

So far, most of the semi-supervised clustering algorithms focus on finding a suitable partition that well satisfies the given constraints. However, insufficient supervisory information may lead to over-fitting results and unstable performance, especially on complicated data. To address this challenge, this paper attempts to solve the semi-supervised clustering problem by self-learning sufficient constraints. The essential motivation is that constraints can be learned from the local neighbor structures within appropriate feature spaces, and sufficient constraints can directly divide the data into clusters. Hence, we first present a constraint self-learning framework. It performs an expectation–maximization procedure iteratively between exploring a discriminant space and learning new constraints. Then, a constraint-based clustering algorithm is proposed by taking advantage of sufficient constraints. Experimental studies on various real-world benchmark datasets show that the proposed algorithm achieves promising performance and outperforms the state-of-the-art semi-supervised clustering algorithms.

1. Introduction

In most real-world applications, the ground truth is expensive to generate for the whole dataset. Consequently, semi-supervised learning with partial background information has drawn more and more attention in the machine learning and data mining communities [1,2,3]. Recently, semi-supervised clustering has become a significant research topic due to the emergence of unlabeled big data of which only a few instances can be labeled by experts. The typical partial background information is formed as must-links, which specify the pairs of instances assigned to the same cluster, and cannot-links, which specify the pairs assigned to different clusters.

In the recent decade, many popular unsupervised clustering methods have been reformed into a semi-supervised way by adapting constraints as prior supervised knowledge, such as K-means [2,4,5,6]. The aim of a semi-supervised clustering algorithm is to find clusters satisfying as many constraints as possible. Its advantages have been validated and reported by many real-world applications [7,8]. However, most of the current methods focus on finding a suitable partition that can well enforce the given constraints, without paying attention to the intrinsic data structure itself. Meanwhile, insufficient constraints cannot yet well handle the non-Gaussian data [9], especially the manifold structure data. For the complex manifold data, the common way is to find a good distance metric, which is equivalent to finding a good feature space transformation [10]. The spectral clustering methods can be treated as a kind of space transformation way for cluster finding. Spectral clustering has been extended to the semi-supervised way by enforcing the constraints [11,12,13]. However, their performance becomes unstable when the prior supervised knowledge is scarce [14,15]. To address this challenge, this paper attempts to solve the semi-supervised clustering problem by self-learning sufficient constraints. We first present a constraint self-learning framework to iteratively explore more pairwise constraints from the local structure of various discriminant feature spaces. Technically speaking, the iterative framework of constraints self-learning and discriminant feature spaces exploring is a general expectation–maximization (EM) procedure. Its iteration alternates between performing the expectation (E) step, which exploits a discriminant feature space well enforcing the current constraints, and a maximization (M) step, which learns more constraints from the local structure of the data, i.e., nearest neighbors in the space explored in the E step. To take advantage of the augmented pairwise constraints, we then propose a two-step constraint-based clustering algorithm, i.e., greedy searching pure must-link sub-clusters and merging sub-clusters. It tries to find a near-optimal partition of the data by minimizing the cost of violating the constraints. The proposed method has the following advantages:

• The proposed method can learn partial discriminant feature spaces with the partial pairwise constraints, i.e., discriminant spaces for the labeled point. And, an approximately optimal discriminant feature space can be finally explored by constraints self-learning iteratively.
• This method allows self-learning more constraints by local neighbors in the Euclidean distance from the partial discriminant spaces. Even for complicated data, promising must-link and cannot-link constraints can be self-learned.
• The proposed constraint-based clustering algorithm is effective, which only uses the constraint information to conduct clustering. As prior knowledge of the number of clusters is unnecessary, it can be further used for detecting the outliers in a large dataset.

2. Related Works

As partial prior information existed in most of the real-world datasets [16], semi-supervised clustering has drawn a lot of attention in the past decade [17]. The semi-supervised clustering algorithms partition the data into groups enforcing the given constraints. The earlier algorithms tried to strictly satisfy all the constraints, such as semi-supervised variants of K-means [4,5,18]. Usually, domain knowledge is heuristic rather than exact [4], so it would be better to explore a soft partition way to satisfy most of the constraints [19,20].

Distance metric learning algorithms have also been incorporated into semi-supervised clustering, such as relevant component analysis [21], and Information Theoretic Metric Learning [22]. Currently, the dividing line between the constraint-based and metric-based algorithms is rather fuzzy. Bilenko et al. [6] proposed the MPCK-Means, incorporating both metric learning and the use of pairwise constraints in a uniform framework via an expectation–maximization process. Finding a good distance metric for clustering is equivalent to finding a good feature space transformation [10], such as spectral clustering. The most direct way is to manipulate the affinity matrix with the given pairwise constraints. For example, Lu et al. [11] proposed to update the affinity matrix by propagating the pairwise constraints information over the original affinity matrix.

Like unsupervised clustering, constrained algorithms suffer a lot when the intrinsic clusters have very different scales and densities [23]. Especially, the performance becomes unstable when the prior supervised knowledge is scarce [14]. To address this issue, a few works try to augment the constraints from existing knowledge. Yi et al. [24] casted the pairwise similarity matrix reconstruction into a matrix completion problem, which results in a convex optimization problem. Wang et al. [14] proposed a Self-Taught Spectral Clustering method by augmenting the given constraint set. And, their constraint augmentation procedure is realized by exploiting the low-rank property for matrix completion. However, these matrix completion-based methods do not explore the problem from the structure of data distribution itself and are very time-consuming [25]. So, it is not suitable for non-Gaussian data and high-dimensional manifold data. Klein et al. [26] proposed a constraints propagating algorithm by updating the proximity matrix with the shortest path searching. Its disadvantage is that the distance metric is learned in the original feature space and the process can be treated as somehow a density searching problem. Here, we will address the semi-supervised clustering problem by self-learning sufficient constraints from various explored discriminant spaces. Each discriminant space will be optimal for the current must-link and cannot-link constraints.

3. Materials and Methods

Assume we have N data instances, $X=[x_1,x_2,\cdots ,x_N]$, $x_i\in {\mathcal{R}}^m$, which can be mapped into K ground truth clusters. Let M be the must-link matrix where $M_{ij}=1$ implies $x_i$ and $x_j$ should be in the same cluster, and C be the cannot-link matrix where $C_{ij}$ implies $x_i$ and $x_j$ should be in different clusters. And, ${\mathsf{\Omega }}^M$ and ${\mathsf{\Omega }}^C$ are the ground truth constraint matrices; then, the sum of ${\mathsf{\Omega }}^M$ and ${\mathsf{\Omega }}^C$ should be an all-ones matrix. Assume the algorithm assigns each instance $x_i$ a label $l_i$. So, the goal of clustering can be formulated as minimizing an objective function, defined as the sum of costs incurred by violating any constraint entries of ${\mathsf{\Omega }}^M$ and ${\mathsf{\Omega }}^C$.

$$\mathcal{J}=\sum _{{\mathsf{\Omega }}_{ij}^M=1}w·\mathbb{1}[l_i\ne l_j]+\sum _{{\mathsf{\Omega }}_{ij}^C=1}w·\mathbb{1}[l_i=l_j],$$

(1)

where $\mathbb{1}[·]$ is the indicator function $\mathbb{1}\left[true\right]=1$ and $\mathbb{1}\left[false\right]=0$, and w is the penalty cost.

The perfect result of clustering is to encode every instance to a cluster enforcing ${\mathsf{\Omega }}^M$ and ${\mathsf{\Omega }}^C$, i.e., $\mathcal{J}=0$. In the context of semi-supervised clustering, the constraint matrices M and C can be viewed as incomplete and a sparse observation of the ground truth. One possible way for better semi-supervised clustering is to achieve complete or approximately complete constraint matrices $M^{*}$ and $C^{*}$.

$${\mathsf{\Omega }}^M\approx M^{*}=\mathcal{F}\left(M\right),{\mathsf{\Omega }}^C\approx C^{*}=\mathcal{F}\left(C\right)$$

(2)

Following this, we first propose a framework (CS-PDS) $\mathcal{F}(·)$ to achieve approximately complete constraint matrices $M^{*}$ and $C^{*}$. Then, objective function (1) can be rewritten for semi-supervised clustering as follows:

$${\mathcal{J}}^{*}=\sum _{M_{ij}^{*}=1}w·\mathbb{1}[l_i\ne l_j]+\sum _{C_{ij}^{*}=1}w·\mathbb{1}[l_i=l_j]$$

(3)

Last, a two-step constraint-based clustering algorithm (FC2) is introduced to find a near-optimal partition of the data that minimizes the objective function (3). And the key symbols used in this paper are listed in

Table 1. Summarization of the key symbols.

Operator	Definition
$X=[x_1,x_2,\cdots ,x_N]$, $x_i\in {\mathcal{R}}^m$	N data instances
M and C	must-link and cannot-link matrix
${\mathsf{\Omega }}^M$ and ${\mathsf{\Omega }}^C$	ground truth constraint matrices
$M^{*}$ and $C^{*}$	target constraint matrices

.

3.1. Constraints Self-Learning from Partial Discriminant Spaces (CS-PDS)

Commonly, the discriminant space can provide a good view of the clusters, which makes distinct clusters as far from each other as possible. However, with partial constraint information, we cannot explore a space to have a perfect view of the intrinsic local structure for the whole data instances. Fortunately, a first-step discriminant space can be found to satisfy the partial constraints and be used to learn new constraints from the local structures. Together with the fresh constraints, we can further find another discriminant space to learn more constraints. The discriminant space should enforce constraints better in the structure, i.e., must-linked instances should be drawn much closer; meanwhile, cannot-linked instances should be much sparser. The local neighbors for each constrained instance can be exploited to form new constraints.

The key questions are as follows:

• How do we learn a discriminant space that can well illustrate the intrinsic structure of data in the Euclidean distance with the partial constraint information, i.e., must-links and cannot-links?
• Which kind of local neighborhoods can be used as new constraints?
• How do we tackle the conflict problem between must-link and cannot-link when we augment the constraints?

3.1.1. Solution for the 1st Question: Partial Discriminant Space Exploring

Let $y={[y_1,y_2,\cdots ,y_m]}^T$ be the mapping of x from the original space to the new space, i.e., $y={\omega }^{T}x$. For the must-link constraints, the optimal y should be satisfied by minimizing Formula (4), which means the must-link points should be close to each other.

$$y^{*}=\arg \min \sum _{i,j=1}^N{\parallel y_i-y_j\parallel }^2{M}_{ij}$$

(4)

For the cannot-link constraints, the optimal y should be satisfied by maximizing Formula (5), which means the cannot-link points should be far away from each other.

$$y^{*}=\arg \max \sum _{i,j=1}^N{\parallel y_i-y_j\parallel }^2{C}_{ij}$$

(5)

We firstly define the Laplacian matrix and diagonal matrix of the two constraint matrices as follows, respectively.

$$L^M=D^M-M,D_{ii}^M=\sum _{i\ne j}{M}_{ij}\forall i$$

(6)

$$L^C=D^C-C,D_{ii}^C=\sum _{i\ne j}{C}_{ij}\forall i$$

(7)

Then, the optimization problem can be formalized as maximizing Formula (8).

$$y^{*}=\arg \max {\displaystyle \frac{Y{L}^C{Y}^T}{Y{L}^M{Y}^T}}$$

(8)

Assuming $\omega$ is the projection vector, the new representations of x can be obtained by $y={\omega }^{T}x$. Formula (8) can be rewritten as exploring the optimal projection vector.

$${\omega }^{*}=\arg \max {\displaystyle \frac{{\omega }^{T}X{L}^C{X}^T\omega }{{\omega }^{T}X{L}^M{X}^T\omega }}$$

(9)

Let $B=X{L}^C{X}^T$ be a bonus matrix, which means cannot-link instances will be favored more if they are separated farther in the new space. Let $P=X{L}^M{X}^T$ and constraint ${\omega }^{T}P\omega$ be a constant c. Then, we can rewrite Equation (9) in Lagrangian form as Equation (10).

$${\omega }^{*}=\arg \max ({\omega }^{T}B\omega +\lambda ({\omega }^{T}P\omega -c))$$

(10)

The solution can be obtained by solving the generalized eigenvalue decomposition problem as follows.

$$B\omega =\lambda P\omega$$

(11)

Note that it can suffer from a small sample size problem caused by limited must-link constraints, which makes the matrix $P=X{L}^M{X}^T$ singular. Therefore, in the first several iterations, we have to use another definition of P with Tikhonov regularization as follows.

$$P=X{L}^M{X}^T+\alpha ·I,$$

(12)

where I is the identity matrix. For regularization $\alpha$, we can simply set $\alpha =0.01$ in case of a non-positive definition of P.

3.1.2. Solution for the 2nd Question: Self-Learning

Here, we start the augmentation process from the given constrained instances. The unconstrained instances will be passively linked step by step.

(1)

Constraints learning for each must-link instance. u

Let ${\tau }_u=\{m_1,\dots ,m_{\left|\tau \right|}\}$ be a set in which each instance must-link to instance u, and ${\mu }_u$ is the average distance between each linked instance pair with u. Then, we define ${\pi }_u=\{n_1,\dots ,n_{\left|\pi \right|}\}$, where $\left|\pi \right|\le k$ and ${\pi }_u\cap {\tau }_u=\varnothing$, as the at most k nearest neighbors set of u with $distance(u,n_i)<{\mu }_u$. Finally, we will augment the must-links by making connections between u and the nodes in ${\pi }_u$, and update the must-link matrix M as the following equation.

$$M_{uj}=min\{1-\beta ,M_{uj}+\beta \},\forall j\in {\pi }_u$$

(13)

(2)

Constraints learning for each cannot-link instance. v

Define $\mu$ as the average distance between each must-link instance pair. And, ${\kappa }_v=\{m_1,\dots ,m_{\left|\kappa \right|}\}$ is a set in which each instance is restricted to cannot-link to instance v. Then, let ${\pi }_v=\{n_1,\dots ,n_{\left|\pi \right|}\}$ be the at most k nearest neighbors set of constraint instance v with $distance(v,n_i)<\mu$, and $distance(v,n_i)<\underset{m\in {\kappa }_v}{min}\left\{distance(v,m)\right\}$. At last, we will augment the cannot-links by making connections between ${\kappa }_v$ and ${\pi }_v$, and update the cannot-link matrix C as the following equation.

$$C_{ij}=min\{1-\beta ,C_{ij}+\beta \},\forall i\in {\kappa }_v,\forall j\in {\pi }_u$$

(14)

In Equations (13) and (14), the parameter $\beta$ is used as a flexible learning rate and also an upper limit making the self-learned constraints always less important than the given ones. And, [$0.1,0.2$] is the experiential security setting for $\beta$. We set $\beta =0.2$ for all the experiments carried out in this paper.

3.1.3. Solution for the 3rd Question: Soft Regularization

We augment constraints within different spaces that may cause conflicts, i.e., two instances are labeled as must-link and cannot-link from different spaces. Such situations commonly occur on the points distributed on the edges between different clusters. Here, we introduce a soft regular solution as follows.    

$$M_{ij}=\left\{\begin{array}{cc}0\hfill & \mathrm{i}\mathrm{f}{C}_{ij}=1\hfill \\ min(M_{ij},{\displaystyle \frac{M_{ij}}{M_{ij}+C_{ij}}})\hfill & \mathrm{i}\mathrm{f}0<C_{ij}<1\hfill \\ M_{ij}\hfill & \mathrm{i}\mathrm{f}{C}_{ij}=0\hfill \end{array}\right.$$

(15)

$$C_{ij}=\left\{\begin{array}{cc}0\hfill & \mathrm{i}\mathrm{f}{M}_{ij}=1\hfill \\ min(C_{ij},{\displaystyle \frac{C_{ij}}{M_{ij}+C_{ij}}})\hfill & \mathrm{i}\mathrm{f}0<M_{ij}<1\hfill \\ C_{ij}\hfill & \mathrm{i}\mathrm{f}{M}_{ij}=0\hfill \end{array}\right.$$

(16)

It can be drawn from the above formulas, in case of conflict, that constraint affiliation of the instance is a weight and the sum equals one (i.e., $M_{ij}+C_{ij}=1$).

Few constraints cannot project the data in a perfect space, but just a space satisfying the current constraints. However, the imperfect space is enough for learning local neighbor structures. The local structure in the projected space can be used to form new constraints. And, more self-learned constraints can make the projection more perfect step by step. As shown in the first line of Figure 1, we illustrate the must-link constraints self-learned step by step. It can be seen that, in only a few steps, all the iris points in one category are connected by must-links. The first algorithm (CS-PDS) provides a practical way to find a set of reasonable must-link and cannot-link constraints.

3.1.4. The Algorithm

Given a set of data points X, a given must-link constraint matrix M, and a given cannot-link constraint matrix C, the algorithm performs a moderate number of EM iterations between (E) partial discriminant space exploiting and (M) constraints self-learning. Pseudo-code for the algorithm is presented in Algorithm 1. The outputs are two approximately complete constraint matrices, which can be further used for clustering.

Parameter settings and convergence. The parameter k is the number of neighbors to construct the neighbor set $\pi$, and $k\in [1,3]$ is the experiential security setting for various datasets, which is based on the empirical performance or dataset characteristics. For the loop in the algorithm, we use the percentage $P_{full}$ of non-zero values in the matrix ($M+C$) and a maximum iteration time, Loops. For the experiments, we first set $P_{full}=20\%$ and $Loops=50$.

Illustration. As shown in Figure 1, we take the IRIS data as a toy example to show the effectiveness of CS-PDS by visualizing the must-link matrix and its connection graph step by step. We can see that with the initial constraints, CS-PDS can explore an almost perfect constraint matrix within sixteen steps. As shown by the connection graph of the last iteration, the must-link matrix leads to a significantly good partition.

Algorithm 1 Constraints Self-learning from Partial Discriminant Spaces (CS-PDS)

Require:  $X={\left\{x_i\right\}}_i^N$, $k,M,C$ Ensure:  Augmented constraint matrices $M^{*}$ and $C^{*}$ 1:   repeat 2:         //   E-step: partial discriminant space exploiting 3:         Find partial discriminant space by optimizing 4:                  ${\omega }^{*}=\arg \max ({\omega }^{T}B\omega +\lambda ({\omega }^{T}P\omega -c))$; 5:         Transfer X into new feature space Y by $Y={\omega }^{T}X$; 6:         //   M-step: constraints self-learning 7:         for each point u having a must-link do 8:               Create set ${\tau }_u$ and ${\pi }_u$, and let $n_j\in {\pi }_u$; 9:               Update M with constraints $\langle u,n_j\rangle$ by Equation (13); 10:        end for 11:        for each point v having a cannot-link do 12:              Create set ${\kappa }_v$ and ${\pi }_v$, and let $m_i\in {\kappa }_v$ and $n_j\in {\pi }_v$; 13:              Update C with constraints $\langle m_i,n_j\rangle$ by Equation (14); 14:        end for 15:        Regularize the matrices M and C by Equations (15) and (16); 16:  until Convergence 17:  return $M^{*}=M,C^{*}=C$;

3.2. Finding Clusters on Constraints

To take advantage of the augmented constraints, we propose an effective two-step algorithm for finding clusters that only relies on constraint matrices. For convenience, we first illustrate some definitions used in the algorithm. For each point $x_i$, a must-link degree $d_i$ is defined as the number of must-links of point $x_i$. $I_h^C=\left\{x_j\right|C(x_i,j)\ne 0\}$ is defined as the point set in which each point has at least one cannot-link constraint to a sub-cluster ${\mathcal{K}}_h$. $M_i$ (or $C_i$) is the point set in which each point must-link (or cannot-link) to point $x_i$. $|·|$ denotes the number of elements of the point set. And, $|{\mathcal{K}}_i,{\mathcal{K}}_j{|}_{Must\text{-}link}$ and $|{\mathcal{K}}_i,{\mathcal{K}}_j{|}_{Cannot\text{-}link}$ denote the number of must-links and cannot-links between sub-clusters ${\mathcal{K}}_i$ and ${\mathcal{K}}_j$, respectively. Pseudo-code for the algorithm is presented in Algorithm 2.

Step one. The first step is a straightforward greedy searching procedure that forms a set of pure must-link sub-clusters ${\left\{{\mathcal{K}}_h\right\}}_{h=1}^{|{\mathcal{K}}_h|}$ without any cannot-links itself. It begins the greedy searching from the point with the largest must-link degree every time. Step 5 initializes a new sub-cluster ${\mathcal{K}}_h$ with the must-link points of $x_i$ and forms the cannot-link point set $I_h^C$ with its cannot-link points. Each iteration (Steps 7 to 15) will greedily search new points that only exist must-links to the current points of sub-cluster ${\mathcal{K}}_h$; meanwhile, we make sure ${\mathcal{K}}_h$ does not contain any cannot-links among its members. Moreover, Steps 10–11 eliminate the point from ${\mathcal{K}}_h$ in case most of its must-link points have cannot-link constraints to ${\mathcal{K}}_h$. Each iteration, from Steps 3 to 17, will form a pure must-link sub-cluster.

Step two. The second step is to merge these subsets into k clusters. The repulsion r between two sub-clusters is defined as the ratio of the number of cannot-links and must-links between the two sub-clusters. We first calculate the repulsion r between each pair of sub-clusters, and then merge the sub-clusters into k clusters one by one from the lowest repulsion. In case of an isolated point, we assign it to the nearest cluster.

Proof. 

The algorithm (FC2) gives a near-optimal solution for minimizing the objective function (3). It can be proved in a straightforward way. In the first step, the formed pure must-link sub-clusters ${\left\{{\mathcal{K}}_h\right\}}_{h=1}^{|{\mathcal{K}}_h|}$ guarantee that ${\sum }_{C_{ij}^{*}=1}w·\mathbb{1}[l_i=l_j]=0$. According to the definition of repulsion r, the merging step minimizes the penalty cost of ${\sum }_{M_{ij}^{*}=1}w·\mathbb{1}[l_i\ne l_j]$ with the lowest cost of ${\sum }_{C_{ij}^{*}=1}w·\mathbb{1}[l_i=l_j]$.    □

Algorithm 2 Finding Clusters on Constraints (FC2)

Require:  $M^{*},C^{*},X={\left\{x_i\right\}}_{i=1}^N$ Ensure:  Clusters: ${\left\{{\mathcal{K}}_h\right\}}_{h=1}^k$ 1:   Calculate the must-link degree $d_i$ for each point $x_i$; 2:   //   First step: searching pure must-link subsets 3:   while  $X\ne \varnothing$  do 4:         Find the point $x_i$ with the largest degree from X; 5:         Initialize ${\mathcal{K}}_h=\left\{x_i\right\}\cup \left\{x_j\right|M(x_i,j)\ne 0\}$ 6:                   ${\mathcal{I}}_h^C=\left\{x_j\right|C(x_i,j)\ne 0\}$; 7:         for each point $x_i\in {\mathcal{K}}_h$ do 8:               Find sets $M_i=\left\{x_j\right|M(x_i,j)\ge 0.5\}$ 9:                        $C_i=\left\{x_j\right|C(x_i,j)\ge 0.5\}$; 10:              if $|M_i\cap I_h^C| \ge |M_i\cap {\mathcal{K}}_h|$ then 11:              ${\mathcal{K}}_h={\mathcal{K}}_h-x_i$; 12:              else 13:              $I_h^C=I_h^C\cup C_i$ and ${\mathcal{K}}_h=({\mathcal{K}}_h\cup M_i)-I_h^C$; 14:              end if 15:         end for 16:         $X=X-{\mathcal{K}}_h,h=h+1$; 17:   end while 18:   //   Second step: merge sub-clusters ${\left\{{\mathcal{K}}_h\right\}}_{h=1}^{|{\mathcal{K}}_h|}$ 19:   for pair $\left\{({\mathcal{K}}_i,{\mathcal{K}}_j)\right|{\mathcal{K}}_i,{\mathcal{K}}_j\in {\left\{{\mathcal{K}}_h\right\}}_{h=1}^{|{\mathcal{K}}_h|},{\mathcal{K}}_i\ne {\mathcal{K}}_j\}$ do 20:         Calculate $r_{ij}={\displaystyle \frac{|{\mathcal{K}}_i,{\mathcal{K}}_j{|}_{Cannot\text{-}link}}{|{\mathcal{K}}_i,{\mathcal{K}}_j{|}_{Must\text{-}link}}}$; 21:   end for 22:   Merge ${\left\{{\mathcal{K}}_h\right\}}_{h=1}^{|{\mathcal{K}}_h|}$ into k clusters from lowest ${\left\{r_{ij}\right\}}_{i,j=1}^{|{\mathcal{K}}_h|}$.

4. Experiments

In this section, we empirically validate the effectiveness of the proposed algorithm with a comparison to various state-of-the-art algorithms.

4.1. Datasets and Experiments Setup

We chose three commonly used datasets from the UCI Archive including Iris, Wine, and Segmentation, the Protein dataset from the literature [27], and randomly sampled subsets from Digits {3, 8, 9} and Letters {I, J, L} handwritten character recognition datasets as suggested by Basu et al. [6]. These datasets come from different domains with various numbers of instances. The statistics of these datasets are listed in

Table 2. Statistics of datasets used in the experimental evaluation.

Name	#Instances	#Features	#Clusters
Protein	116	20	6
Iris	150	4	3
Wine	178	13	3
Letters	227	16	3
Digits	317	16	3
Segmentation	2310	19	7

.

We compared the proposed semi-supervised clustering algorithm to the following eight algorithms, including three label-constrained algorithms, four pairwise-constrained algorithms, and the unsupervised spectral clustering as the baseline. The three label-constrained algorithms are (a) Flexible Constrained Spectral Clustering (FCSC) [12], (b) Constrained Spectral Clustering, and (c) Group with Bias (GrBias) [28,29]. The four pairwise-constrained algorithms include (d) Constrained K-Means clustering with background knowledge (C-K-Means) [4], (e) the Metric Pairwise-Constrained K-Means algorithm (MPCK-Means) [6], (f) the Information Theoretic Metric Learning algorithm (ITML) [22], and (g) Self-Taught Spectral Clustering (STSC) [14]. As mentioned earlier, both label and pairwise constraints can be formed as must-links and cannot-links. However, pairwise-constrained clustering poses more of a challenge for finding clusters from data. So, we conducted separate comparison studies on these state-of-the-art semi-supervised clustering algorithms with different types of constraints.

4.2. Results

As mentioned earlier, both label and pairwise constraints can be formed as must-links and cannot-links. However, pairwise-constrained clustering poses more of a challenge for finding clusters from data. So, we conducted separate comparison studies on these state-of-the-art semi-supervised clustering algorithms with different types of constraints. In order to examine the effectiveness of constraints for clustering, spectral clustering is applied as the baseline method without any constraints. In our experiments, we will report the Rand Index (RI) of all algorithms against the number of pairwise constraints and the percentage of known labels.

We first conducted comparison experiments with the label-constrained semi-supervised clustering algorithms. For each dataset, we randomly generated a set of labeled instances and then applied all algorithms to the dataset with the same label constraints. The process was repeated ten times with different label constraints. The average ARI and standard deviation are reported in Figure 2. Pairwise-constrained clustering is a much more difficult problem than the former one because the categories are uncleared. The number of must-link and cannot-link constraints are randomly generated and applied to all the algorithms at the same time. By executing the procedure ten times, we reported the average ARI and standard deviation in Figure 3.

As expected, pairwise constraints enhance the clustering performance in an effective way. We can observe from Figure 3 that the performance of a traditional method (i.e., C-K-Means) does not keep increasing when the number of constraints becomes larger. The reason is that inappropriate constraints may confuse the K-Means method. Combined with the metric learning method, the MPCK-Means and ITML methods perform more stably. However, the performance of these methods is still very low with insufficient constraints. The results indicate our method achieves steadily increasing performance against the number of constraints and outperforms the competing algorithms in most cases.

We can see that the Self-Taught (S-T) method also produces good performance, especially on the Wind dataset. In contrast, it employs a matrix completion approach to augment the given constraints. However, the matrix completion approach is very time-consuming and unstable.

Table 3. Average running time (in seconds).

Name	FC2	MPCK -Means	C- K-Means	ITML	S-T
Protein	0.098	1.705	0.016	1.962	29.48
Iris	0.225	2.765	41.83	1.410	53.58
Wine	0.170	3.710	0.017	2.123	77.50
Letters	0.352	5.521	12.12	2.473	156.6
Digits	4.704	0.907	0.161	1.500	2021
Segment	33.57	2.246	9.470	7.613	N/A

summarizes the average running time for the six datasets of recovering a different number of constraints. The running time reported for our FC2 algorithm includes the CS-PDS algorithm. And, N/A means that the clustering task cannot be completed by the corresponding algorithm within 24 h. The Self-Taught (S-T) method takes more time than ours to obtain comparable performance. It is impractical for large data, such as the Segmentation dataset. We can draw a conclusion from Table 3 that the proposed algorithm is practical for large data and efficient.

The algorithm of self-learning sufficient constraints is effective. Figure 4 shows the graph distance matrices of the original space and the space of the last iteration for three example datasets. The number of initial constraints are 50, 100, and 100, respectively. If the self-learned constraints are effective and correct, they can help to find a discriminant space to separate the points. It can be seen that obvious blocks can be found from the last explored discriminant space, which indirectly indicates that the self-learned constraints are effective. All the above results demonstrate that our algorithm can yield good performance with an insufficient number of constraints.

5. Conclusions

Insufficient supervisory information sometimes confuses the semi-supervised clustering and misleads to unstable performance. In this paper, we solved the semi-supervised clustering problem from a new point of view that perfects the supervisory information in order to cluster the data. We first designed a self-learning framework to iteratively explore two approximately perfect constraint matrices. Then, a constraint-based clustering algorithm was presented to find a near-optimal partition for the data. Experimental results justified the effectiveness of our algorithm. Extending the algorithm to outliers detection is the primary avenue for future research work. One drawback of our algorithm is the efficiency of large datasets. In the case of a large dataset, it may take a long time to handle the clustering procedure. This is also one of our further research issues. The possible way is parallel performing the clustering procedure and employing it on GPU devices.