Adaptive Clustering via Symmetric Nonnegative Matrix Factorization of the Similarity Matrix

Abstract

The problem of clustering, that is, the partitioning of data into groups of similar objects, is a key step for many data-mining problems. The algorithm we propose for clustering is based on the symmetric nonnegative matrix factorization (SymNMF) of a similarity matrix. The algorithm is first presented for the case of a prescribed number k of clusters, then it is extended to the case of a not a priori given k. A heuristic approach improving the standard multistart strategy is proposed and validated by the experimentation.

1. Introduction

Clustering can be applied to a variety of different kinds of documents as long as a distance measure can be assigned among the data objects. Generally, a similarity function assigns to pairs of closely related objects with a higher similarity than to objects which are only weakly related. A clustering technique classifies the data into groups, called clusters, in such a way that objects belonging to a same cluster are more similar to each other than to objects belonging to different clusters.

In this paper we assume that the data objects are n distinct points ${\mathit{p}}_1,\dots ,{\mathit{p}}_n$ in ${\mathbb{R}}^d$ and denote by k, with $1<k\ll n$, the number of clusters. As usual in this context, we consider the similarity expressed through the Gaussian kernel [1]

$$e_{i,r}=exp\left(-\frac{\parallel {\mathit{p}}_i-{\mathit{p}}_r{\parallel }_2^2}{\sigma \mu }\right),\mathrm{where}\mu =\underset{j,s}{max}{\parallel {\mathit{p}}_j-{\mathit{p}}_s\parallel }_2^2,$$

(1)

and $\sigma >0$ is a parameter based on the scale of the points.

The nonnegative symmetric matrix $A\in {\mathbb{R}}_{+}^{n\times n}$, whose elements are

$$a_{i,r}=d_i^{-1/2}{e}_{i,r}{d}_r^{-1/2},\mathrm{where}{d}_i=\sum _{s=1}^n{e}_{i,s},\mathrm{for}i,r=1,\dots ,n,$$

(2)

is called the similarity matrix.

Among the many different techniques devised to solve the clustering problem, we chose a method which performs a dimensionality reduction through the Nonnegative Matrix Factorization (NMF) of matrix A. NMF finds an approximate factorization of A by means of two low-rank, nonnegative matrices. In its basic form, the problem is set as follows: given a matrix $A\in {\mathbb{R}}_{+}^{m\times n}$ and an integer $k\ll n$, we look for two low-rank matrices $W\in {\mathbb{R}}_{+}^{m\times k}$ (the basis matrix) and $H\in {\mathbb{R}}_{+}^{n\times k}$ (the coefficient matrix), such that the product $W{H}^T$ approximates A according to

$${\displaystyle \underset{W,H\ge O}{min}f(W,H),\mathrm{with}f(W,H)=\frac{1}{2}{\parallel A-W{H}^T\parallel }_F^2,}$$

(3)

where the subscript F denotes the Frobenius norm. Actually, if the nonnegativity condition was not imposed, the minimum of $f(W,H)$ could be obtained by truncating the Singular Value Decomposition $A=U\Sigma V^T$. Unfortunately, some entries of U and V would be negative, and this would prevent interpretation of the factorization in terms of clustering [1]. In fact, since each column of A can be approximately represented by a linear combination with coefficients in H of columns of W, thanks to the nonnegativity feature, the elements of H, properly normalized, could be seen as the probabilities of assignment of the points to the different clusters. The clustering can be obtained by assigning the ith point ${\mathit{p}}_i$ to the jth cluster corresponding to the largest coefficient $h_{i,j}$ in the ith row of H.

The problem of computing NMF was first proposed in [2] for data analysis and, from then on, it has received much attention. The first algorithm to compute NMF was proposed under the name of multiplicative updating rule in [3], but its slow convergence suggested looking for more efficient algorithms. Many of them belong to the class of Block Coordinate Descent (BCD) methods [4,5,6,7,8], but other iterative methods for nonlinear optimization have also been suggested, based on some gradient descent procedure. To improve the convergence rate, a quasi-Newton strategy has also been considered, coupled with nonnegativity (see, for example, the Newton-like algorithm of [9]). Open source libraries for computing NMF are available—see, for example, [10], where several methods proposed in related works are implemented. Over the years, many variants of the basic problem (3) have been considered. The case of a symmetric matrix A has been deeply investigated, but other features have also been taken into consideration. For example, sparsity and/or regularizing constraints can be easily embedded [5], classification labels can be incorporated through a supervisor [11], and extensions to tensor nonnegative factorization have been studied [12].

Since our goal was to obtain clustering through the symmetric matrix A, we restricted our analysis to the symmetric NMF problem.

When a clustering problem is proposed, the dimension of the clustering, that is, the number k of the clusters, can be either given a priori or looked for. In this paper we consider both cases:

-

problem ${\mathcal{P}}_{kfix}$, with an a priori assigned rank k of matrix W. The matrix W is computed using NMF and the clustering is obtained.

-

${\mathcal{P}}_{k\mathit{v}ar}$, where an interval ${\mathcal{J}}_k=[k_{min},k_{max}]$ is assigned. The number k varies in ${\mathcal{J}}_k$ and the best k, according to a given criterium, has to be found.

The algorithm solving ${\mathcal{P}}_{kfix}$ is a constitutive brick of a larger procedure for ${\mathcal{P}}_{k\mathit{v}ar}$. When dealing with ${\mathcal{P}}_{k\mathit{v}ar}$, the factorization of A must be performed for each k in ${\mathcal{J}}_k$, and a validity index must be provided to compare the clusterings obtained with different k to decide which is the best one.

Our contribution consists in devising an algorithm which implements a heuristic approach by improving a multistart policy. The algorithm, especially suitable when working in a multitasking/multiprocessor environment, lets the computation efforts be concentrated on the more promising runs, without any a priori knowledge. Each run applies a BCD technique known as alternating nonnegative least squares, which replaces the BPP method used in [13] by the GCD method of [7]. This choice, as shown in a previous paper [14], leads to a saving in the computational cost of the single run. The suggested heuristics enjoys the same effectiveness of the standard implementation of a multistart approach, with a considerable reduction of the cost. Its only drawback is more complicated coding.

The paper is organized as follows: Section 2 is devoted to recall the alternating nonnegative least squares procedure for NMF with both general matrices and symmetric matrices, constituting the basis of our proposed algorithm. Section 3 describes the validity indices used for comparison criteria. Next, in Section 4, the algorithm for problem ${\mathcal{P}}_{kfix}$ is presented, with a priority queue implementation to reduce the computational time. The extension of the algorithm to problem ${\mathcal{P}}_{k\mathit{v}ar}$, described in Section 5, requires particular attention to the scale parameter $\sigma$ chosen for the construction of A by using (1) and (2). The experiments are described and commented on in Section 6.

Notations: the $(i,j)$th element of a matrix M is denoted by ${\left(M\right)}_{i,j}$ or by the lowercase $m_{i,j}$. The notation $M\ge O$ means that all the elements of M are nonnegative. Several parameters enter in the description of the algorithm. In the text, where they are introduced, the notation “(Section 6)” appears, meaning that the values assigned to them in the experiments are listed at the beginning of Section 6.

2. The Alternating Nonnegative Least Squares for NMF

Problem (3) is nonconvex and finding its global minimum is NP-hard [15]. In general, nonconvex optimization algorithms guarantee only the stationarity of the limit points, and only local minima can be detected. Most iterative algorithms for solving (3) are based on a simple block nonlinear Gauss–Seidel scheme [4,5]. An initial $W_0\in {\mathbb{R}}_{+}^{m\times k}$ is chosen, and the sequences

$$\left\{\begin{array}{c}{H}_{\nu }=\underset{H\ge 0}{\mathrm{argmin}}\frac{1}{2}{\parallel A-W_{\nu -1}{H}^T\parallel }_F^2,\hfill \\ W_{\nu }=\underset{W\ge 0}{\mathrm{argmin}}\frac{1}{2}{\parallel A^T-H_{\nu }{W}^T\parallel }_F^2,\hfill \end{array}\right.$$

(4)

are computed for $\nu =1,2,\dots$, until a suitable stopping condition is satisfied, such as by checking whether a local minimum of the objective function $f(W,H)$ has been sufficiently well-approximated. The convergence of this scheme, called Alternating Nonnegative Least Squares (ANLS), follows from [16].

Although the original problem (3) is nonconvex, the two subproblems of (4) are convex. Let us denote by $\phi \left(X\right)$ the objective function to be minimized in both cases,

$$\phi \left(X\right)=f(Y,X)=\frac{1}{2}{\parallel B-Y{X}^T\parallel }_F^2,$$

(5)

where $B=A$ and $Y=W_{\nu -1}$ for the first subproblem in (4), and $B=A^T$ and $Y=H_{\nu }$ for the second subproblem in (4). Problem (5) can be dealt with by applying aprocedure for nonnegatively constrained least squares optimization, such as an Active-Set-like method [13,17,18,19] or an iterative inexact method as a gradient descent method modified to satisfy the nonnegativity constraints. We have performed in [14] a preliminary ad hoc experimentation, comparing an Active-Set-like method (the one called BPP and coded as Algorithm 2 in [13]) with the Greedy Coordinate Descent method, called GCD in [7]. The results have shown that in our context, GCD is faster and equally reliable. GCD, specifically designed for nonnegative constrained least squares problems, implements a selection policy of the elements updated during the iteration, based on the largest decrease of the objective function. We give here a brief description of GCD (the corresponding code can be found as Algorithm 1 in [7]).

The gradient of $\phi \left(X\right)$ is

$$G\left(X\right)=\left(X{Y}^T-B^T\right)Y=XQ-B^{T}Y,\mathrm{where}Q=Y^{T}Y.$$

GCD computes a sequence of matrices $X_j$, $j=1,2,\dots$, until termination. Initially, $X_0$ is set equal to $H_{\nu -1}$ for the first subproblem in (4) and to $W_{\nu -1}$ for the second subproblem in (4) (with $H_0=O$, which ensures that $G\left(X_0\right)\le O$). At the jth iteration, matrix $X_j$ is obtained by applying a single coordinate correction to the previous matrix $X_{j-1}$ according to the rule $X_j=X_{j-1}+s{\mathit{e}}_r{\mathit{e}}_i^T$, where r and i are suitably selected indices and ${\mathit{e}}_r$ and ${\mathit{e}}_i$ are the rth and ith canonical vectors of compatible lengths. The scalar s is determined by imposing that $\phi \left(X_j\right)$, as a function of s, is the minimum on the set

$$S=\{s\mathrm{such}\mathrm{that}{\left(X_{j-1}\right)}_{r,i}+s\ge 0\}.$$

Denoting $g_{r,i}={\left(G\left(X_{j-1}\right)\right)}_{r,i}$ and $q_{i,i}$ the ith principal element of Q, the value which realizes the minimum of $\phi \left(X_j\right)$ on S is

$${\displaystyle \widehat{s}=-\frac{g_{r,i}}{q_{i,i}}\mathrm{if}\frac{g_{r,i}}{q_{i,i}}\le {\left(X_{j-1}\right)}_{r,i}\mathrm{and}\widehat{s}=-{\left(X_{j-1}\right)}_{r,i}\mathrm{otherwise}.}$$

In correspondence, the objective function is decreased by

$${\displaystyle {\theta }_{r,i}^{\left(j\right)}=\phi \left(X_{j-1}\right)-\phi \left(X_j\right)=-g_{r,i}\widehat{s}-\frac{1}{2}{q}_{i,i}{\widehat{s}}^2.}$$

A natural choice for indices $(r,i)$ would be the one that maximizes ${\theta }_{r,i}^{\left(j\right)}$ on all the pairs $(r,i)$, but it would be too expensive. So it is suggested to proceed by rows. For a fixed $r=1,\dots ,n$, the index i is chosen as the one that maximizes ${\theta }_{r,i}^{\left(j\right)}$ on $i\in [1,k]$ and the $(r,i)$th element of $X_{j-1}$ is updated. Then a new index i is detected, and so on, until a stopping condition is met, such as

$$\underset{i}{max}{\theta }_{r,i}^{\left(j\right)}<{\eta }_1{\theta }_{max},$$

where ${\eta }_1$ is a preassigned tolerance (Section 6) and ${\theta }_{max}={max}_{r,i}{\theta }_{r,i}^{\left(1\right)}$ is the largest possible reduction of the objective function that can be expected when a single element is modified in $X_0$.

The Symmetric Case

In our case, the similarity matrix $A\in {\mathbb{R}}_{+}^{n\times n}$ is symmetric; hence, the general NMF problem (3) is replaced by the symmetric NMF (SymNMF) problem

$${\displaystyle \underset{W\ge O}{min}f\left(W\right),\mathrm{with}f\left(W\right)=\frac{1}{2}{\parallel A-W{W}^T\parallel }_F^2,\mathrm{with}W\in {\mathbb{R}}_{+}^{n\times k}.}$$

(6)

Since A may generally be indefinite, while $W{W}^T$ is positive semidefinite, $W{W}^T$ represents a good factorization of A if A has enough nonnegative eigenvalues. When this happens, we are led to believe that W naturally captures the cluster structure hidden in A [1,9]. Thanks to the nonnegativity, the largest entry in the ith row of W is assumed as the clustering assignment of the ith point—that is, ${\mathit{p}}_i$ belongs to the jth cluster if $w_{ij}={max}_{r=1,\dots ,k}{w}_{ir}$. This interpretation furnishes a clustering $\Pi$ induced by the matrix W.

In [9], this approach, which efficiently solves the clustering problem, is shown to be competitive with the widely used spectral clustering techniques, which perform a dimensionality reduction by means of the eigenvectors corresponding to the leading eigenvalues of A.

Problem (6) has a fourth-order nonconvex objective function. Following [1], we suggest solving it through a nonsymmetric penalty problem of the form

$$\underset{W,H\ge O}{min}{f}_{\alpha }(W,H),\mathrm{with}{f}_{\alpha }(W,H)=\frac{1}{2}\left(\parallel A-W{H}^T{\parallel }_F^2+\alpha {\parallel W-H\parallel }_F^2\right),$$

(7)

$\alpha$ being a positive parameter which acts on the violation of the symmetry. Problem (7) can be tackled by using any of the techniques proposed in the literature for solving the general NMF problem (3). For example, algorithm ANLS can be applied by alternating the solution of the two subproblems

$$\left\{\begin{array}{c}{H}_{\nu }=\underset{H\ge 0}{\mathrm{argmin}}\frac{1}{2}\parallel \left[\begin{array}{c}A\\ \sqrt{{\alpha }_{\nu }}{W}_{\nu -1}^T\end{array}\right]-\left[\begin{array}{c}{W}_{\nu -1}\\ \sqrt{{\alpha }_{\nu }}{I}_k\end{array}\right]H^T{\parallel }_F^2,\hfill \\ W_{\nu }=\underset{W\ge 0}{\mathrm{argmin}}\frac{1}{2}\parallel \left[\begin{array}{c}A\\ \sqrt{{\alpha }_{\nu }}{H}_{\nu }^T\end{array}\right]-\left[\begin{array}{c}{H}_{\nu }\\ \sqrt{{\alpha }_{\nu }}{I}_k\end{array}\right]W^T{\parallel }_F^2.\hfill \end{array}\right.$$

(8)

In [1], the sequence of penalizing parameters is constructed by setting

$${\alpha }_{\nu }={\beta }_{\nu }maxA,\mathrm{with}{\beta }_0=1,$$

and ${\beta }_{\nu }$ modified according to the geometric progression ${\beta }_{\nu }={\zeta }^{\nu }$ with the fixed ratio $\zeta =1.01$. We suggest, instead, to let ${\beta }_{\nu }$ be modified adaptively, as shown in [14].

Both problems (8) have a form similar to (5), with k rows involving the $\sqrt{{\alpha }_{\nu }}$, appended at the bottom. Hence, each problem is solved by applying GCD with gradient

$$G\left(X\right)=X{Y}^{T}Y-B^{T}Y+\alpha (X-Y)=XQ-{(B+\alpha I_n)}^{T}Y,\mathrm{where}Q=\alpha I_k+Y^{T}Y.$$

The function which computes a single iteration of ANLS by applying GCD and modifying ${\alpha }_{\nu }$ as described in [14], is thus called

$$\left(W_{\nu },H_{\nu },{\alpha }_{\nu }\right)=\mathtt{Sym}\_\mathtt{ANLS}\left(A,W_{\nu -1},H_{\nu -1},{\alpha }_{\nu -1}\right).$$

The whole iterative procedure is organized in the following function, $\mathtt{Sym}\_\mathtt{NMF}$. The stopping condition tests the stabilization of the residuals ${ϵ}_{\nu }={\parallel A-W_{\nu }{W}_{\nu }^T\parallel }_F^2$ by using two parameters: a tolerance ${\eta }_2$ and a maximum number ${\nu }_{max}$ of allowed iterations (Section 6).

function$\mathtt{Sym}\_\mathtt{NMF}\left(A,W_0,H_0,{\alpha }_0,{\nu }_{max}\right)$

ν = 0; ${ϵ}_0={\parallel A-W_0{W}_0^T\parallel }_F^2$

$\mathtt{repeat}$

ν = ν + 1;

{Wν,Hν,αν} = Sym_ANLS (A,Wν−1,Hν−1,αν−1);

${ϵ}_{\nu }={\parallel A-W_{\nu }{W}_{\nu }^T\parallel }_F^2$;

cond = |ϵν − ϵν − 1|/ϵν ≤ η2

$\mathtt{until}$ cond |ν ≥ νmax|

$\mathtt{return}$ (ν, Wν, Hν, αν, cond);

The effective dimension $k_e$ of $\Pi$ can be smaller than the expected k, because void clusters may turn out (see the following function Partition, where $\left\{\right\}$ denotes the void set).

function Partition$\left(W\right)$

${\pi }_r=\left\{\right\}$ for r = 1, …, k;

$\mathtt{for}i=1,\dots ,n$

let j be such that $w_{i,j}={max}_{r=1,\dots ,k}{w}_{i,r}$;

append i to ${\pi }_j$

end for;

$\Pi =[{\pi }_1,\dots ,{\pi }_k]$;

$k_e=k$;

for $r=1,\dots ,k$

if ${\pi }_r=\left\{\right\}$ then discard ${\pi }_r;k_e=k_e-1$

end for;

return $\left(\Pi ,k_e\right)$;

When applying Sym_NMF we expect that, together with the stabilization of the sequence of the residuals ${ϵ}_{\nu }={\parallel A-W_{\nu }{W}_{\nu }^T\parallel }_F^2$, a form of stabilization of the corresponding clustering should also take place. Actually, we have verified experimentally that, in general, the stabilization of the clusterings proceeds irregularly, before the residuals ${ϵ}_{\nu }$ show a substantial reduction. This is the reason why Sym_NMF does not rely on early clustering stabilization, which could produce incorrect results, and exploits the decrease of the residuals.

3. The Validity Indices

Since the solution W of problem (6), computed by starting with different initial approximations $W_0$, may be only a local minimum, tools to evaluate the validity of the clustering $\Pi$ induced by W are needed. Denoting by ${\pi }_j$ the jth cluster, a clustering $\Pi =\{{\pi }_1,\dots ,{\pi }_k\}$, with $k\ge 2$, corresponds to a partitioning of the indices $\{1,\dots ,n\}$ (actually, the number $k_e$ of clusters effectively obtained by applying Partition$\left(W\right)$ can be smaller than the required k). A clustering algorithm generally aims at minimizing some function of the distances of the points of the jth cluster from its centroid defined by

$${\mathit{c}}_j=\frac{1}{n_j}\sum _{i\in {\pi }_j}{\mathit{p}}_i,\mathrm{with}{n}_j=\#{\pi }_j,$$

where the symbol # denotes the cardinality. Over the years, dozens of validity indices have been proposed (see [20] and its references). Most of them take into account the compactness, which measures the intra-cluster distances, and the separability, which measures the inter-cluster distances. A good clustering algorithm should produce small intra-cluster distances and large inter-cluster distances.

3.1. The DB Index

Both compactness and separability are combined by the following Davies-Bouldin index (DB) [21]:

$$\chi =\frac{1}{k_e}\sum _{j=1}^{k_e}{\zeta }_j,\mathrm{where}{\zeta }_j=\underset{r\ne j}{max}\frac{{\gamma }_j+{\gamma }_r}{d_{j,r}},{\gamma }_j=\frac{1}{n_j}\sum _{i\in {\pi }_j}\parallel {\mathit{p}}_i-{\mathit{c}}_j{\parallel }_2\mathrm{and}{d}_{j,r}={\parallel {\mathit{c}}_j-{\mathit{c}}_r\parallel }_2.$$

(9)

A good clustering is associated to a small value of the DB index. In the codes, the function which uses (9) to compute the DB index $\chi$ of the clustering $\Pi$ obtained by Partition$\left(W\right)$, is called

$$\chi =\mathtt{DBindex}\left(\Pi \right).$$

3.2. The DB** Index

When the validity of different clusterings corresponding to different dimensions k must be established, the DB index can still be used, but if the given points can be organized in different subclusters, it may be difficult to establish which one results in being the best one. In [22] another index, called DB**, is proposed to deal with this case. Unlike index DB, index DB** takes into account the compactness behavior of the clustering corresponding to different values of k in order to penalize unnecessary merging and to detect finer partitions.

We assume that a sequence of clusterings ${\Pi }^{\left(h\right)}$ of dimensions $k^{\left(h\right)}$ is given, with $h=1,\dots ,h_{max}$, such that $k^{\left(h\right)}<k^{(h+1)}$. Let ${\mathit{c}}^{\left(h\right)}$ denote the vector of the centroids of ${\Pi }^{\left(h\right)}$, and ${\mathbf{\gamma }}^{\left(h\right)}$ the vector of $k^{\left(h\right)}$ components whose jth component is

$${\gamma }_j^{\left(h\right)}=\frac{1}{n_j^{\left(h\right)}}\sum _{i\in {\pi }_j^{\left(h\right)}}{\parallel {\mathit{p}}_i-{\mathit{c}}_j^{\left(h\right)}\parallel }_2\mathrm{where}{n}_j^{\left(h\right)}=\#{\pi }_j^{\left(h\right)}.$$

For $h=1,\dots ,h_{max}-1$, let ${\mathit{u}}^{\left(h\right)}$ and ${\mathit{v}}^{\left(h\right)}$ be the vectors of $k^{\left(h\right)}$ components whose jth component is

$${\mathit{u}}_j^{\left(h\right)}=\underset{r\ne j}{max}\left({\gamma }_j^{\left(h\right)}+{\gamma }_r^{\left(h\right)}\right)-\underset{r\ne j}{max}\left({\gamma }_j^{(h+1)}+{\gamma }_r^{(h+1)}\right),{\mathit{v}}_j^{\left(h\right)}=\underset{r=h,\dots ,h_{max}-1}{max}{\mathit{u}}_j^{\left(r\right)}.$$

DB** returns a vector ${\mathbf{\chi }}^{**}$ of length $h_{max}-1$ whose hth component is

$${\chi }_h^{**}=\frac{1}{k^{\left(h\right)}}\sum _{j=1}^{k^{\left(h\right)}}{\zeta }_j^{\left(h\right)},\mathrm{where}{\zeta }_j^{\left(h\right)}=\frac{\underset{r\ne j}{max}\left({\gamma }_j^{\left(h\right)}+{\gamma }_r^{\left(h\right)}\right)+{\mathit{v}}_j^{\left(h\right)}}{\underset{r\ne j}{min}{d}_{j,r}^{\left(h\right)}}.$$

The lowest component of ${\mathbf{\chi }}^{**}$ detects the best clustering among the clusterings ${\Pi }^{\left(h\right)}$ with $h\in \{1,\dots ,h_{max}-1\}$.

3.3. The CL Index

Both the DB index and the DB** index base their valuation of the clustering separability on the distances between centroids, according to the rule: the farther the centroids, the more separated the clusters. This rule does not appear profitable in the presence of nonconvex clusters or of clusters that nearly touch each other, or when the densities of the points differ much from cluster to cluster. This can be particularly troublesome when the points are affected by noise or when there are outliers. To check whether the rth and sth clusters nearly touch each other, one should compute their distance; that is, the minimum distance between each pair of points ${\mathit{p}}_i\in {\pi }_r$ and ${\mathit{p}}_j\in {\pi }_s$, with a not negligible increase in computational time. For example, this minimum distance between two clusters is used by the Dunn index [20].

To reduce the burden of this control, we propose a new index called CL, which provides a measure of closeness. First, a small integer p (Section 6) is selected, and for any point ${\mathit{p}}_i$, the indices of its p nearest points are listed in ${\ell }_i$. Using the same notations of the previous subsection, we assume that a sequence of clusterings ${\Pi }^{\left(h\right)}$ of dimensions $k^{\left(h\right)}$ is given, with $h=1,\dots ,h_{max}$. Let ${\pi }_r^{\left(h\right)}$ be the cluster to which ${\mathit{p}}_i$ belongs. We consider the quantity

$$x_i^{\left(h\right)}=\sum _{j\in {\ell }_i,j\notin {\pi }_r^{\left(h\right)}}exp\left(-\mathit{c}\frac{\parallel {\mathit{p}}_i-{\mathit{p}}_j{\parallel }_2^2}{\mu }\right),$$

where c is a constant (Section 6) large enough to make negligible in the sum the contribution of far-away points not belonging to ${\pi }_r^{\left(h\right)}$. Set

$$y_h=\sum _{i=1}^n{x}_i^{\left(h\right)},h=1,\dots ,h_{max}.$$

The sequence $y_h$ can be very oscillating. For this reason, we defined as the CL index the vector ${\psi }_h$ obtained by computing the exponential moving average [23] with smoothing constant 0.1, of the vector $y_h$

$${\psi }_1=y_1,{\psi }_{h+1}={\psi }_h+0.1(y_{h+1}-{\psi }_h),h=1,\dots ,h_{max}-1.$$

This index is large when at least one point ${\mathit{p}}_i$ exists, having some sufficiently close points not belonging to the same cluster of ${\mathit{p}}_i$ in ${\Pi }^{\left(h\right)}$. Typically, the CL index has the following behavior: a zero ${\psi }_h$ means that the clusters of ${\Pi }^{\left(h\right)}$ are well-separated, and this occurs only in the absence of noise. The increasing rate, which is bounded for small values of k, tends towards infinity when k exceeds values for which separated clusters exist. This behavior, together with DB** values, suggests when stopping the increase of k if $k_{max}$ is not assigned.

4. The Clustering Algorithm for Problem ${\mathcal{P}}_{kfix}$

Given a fixed value of k, the algorithm Kfix we propose for problem ${\mathcal{P}}_{kfix}$ searches the minimum of (6). This algorithm will be used in Section 5 for solving problem ${\mathcal{P}}_{k\mathit{v}ar}$. Before describing it, it is necessary to deal with the important issue of how to choose a suitable value of the scale parameter $\sigma$ for the construction of the similarity matrix A according to (1) and (2).

Values of $\sigma$ that are too small or too large would reduce the variability of A, making all the points equally distant, resulting in badly mixed-up clusters. Since we must discriminate between close and far points, an intermediate value between the smallest and the largest distance of the points should be chosen. Clearly, there is a strong correlation between the value of $\sigma$ and the number k of clusters that the algorithm is expected to find: more clusters may have a smaller size and may require a smaller $\sigma$ to separate them. To this aim, a triplet ${\mathbf{\sigma }}^{\left(k\right)}$ of suitable values (Section 6) is detected and tested for any k.

Besides the choice of $\sigma$, the choice of the initial matrix $W_0$ may also be critical, due to the fact that only a local minimum of problem (6) can be expected. It is common practice to tackle this issue by comparing the results obtained by applying Sym_NMF to several matrices $W_0$, randomly generated with uniform distribution between 0 and 1. The number of the starting matrices is denoted q (Section 6).

For any pair $\{\sigma ,W_0\}$ with $\sigma \in {\mathbf{\sigma }}^{\left(k\right)}$, matrix $A\left(\sigma \right)$ is constructed using (1) and (2) and the function Sym_NMF is applied starting with $W_0$. Then, the DB index $\chi$ of the clustering so obtained is computed. A trivial multistart implementation could consist in carrying out, up to convergence, function Sym_NMF for any pair $\{\sigma ,W_0\}$ and choosing, at the end, the best $\chi$.

In the following we present a more sophisticated version, suitable for large problems, which uses a heuristic approach. The run for any pair $\{\sigma ,W_0\}$ is split in segments with a fixed number $\lambda$ of iterations, and the execution scheduling is driven by the values $\chi$ already found. The possibility of discarding the less promising runs is considered. The proposed strategy, which is accomplished by using a priority queue, specializes to the present clustering problem the procedure given in [24], where the global minimization of a polynomial function is dealt with. In [24], the heuristic approach is shown to outperform the multistart one for what regards the computational cost, maintaining the same efficacy.

The items of the priority queue $\mathcal{Q}$ have the form

$$Y=\{r,t,W,H,\alpha ,\xi \},$$

where

-

r is the index which identifies the pair $\{\sigma ,W_0\}$.

-

$t=t^{\left(r\right)}$ is the number of iterations computed until now by Sym_NMF on the item with index r. Initially $t=0$.

-

$W=W^{\left(r\right)}$ and $H=H^{\left(r\right)}$ are the NMF factors of $A\left(\sigma \right)$ computed for the rth item. Initially $W=W_0$, $H=O$.

-

$\alpha ={\alpha }^{\left(r\right)}$ is the penalizing parameter of the rth item. Initially $\alpha =maxA\left(\sigma \right)$.

-

$\xi ={\chi }^{\left(r\right)}+t^{\left(r\right)}/t_{max}$, where ${\chi }^{\left(r\right)}$ is the DB index defined in (9) for the rth item, and $t_{max}$ (Section 6) is the maximum number of allowed iterations for the rth item. Initially $\xi =0$.

The queue is ruled, according to the minimum policy, by the value of $\xi$, allowing to compute first the more promising items, keeping into account also the age of the computation. Initially, the priority queue contains $3q$ items. Also referred in the procedure, but not belonging to the queue, are

-

an array m of length k, whose ith element contains the minimum of the DB indices computed so far for all the items returning $k_e=i$. Initially, the elements of m are set equal to ∞.

-

an array M, containing in position i the partition $\Pi$ corresponding to $m_i$. Initially, the elements of M are empty sets.

The belonging of an element to an item Y is denoted by using the subscript (Y). Moreover, $A_{\left(Y\right)}$ denotes the similarity matrix $A\left(\sigma \right)$ generated with the scale parameter $\sigma$ associated with $r_{\left(Y\right)}$.

After the initialization of the queue, the adaptive process evolves as follows: an item Y is extracted from the queue according to its priority ${\xi }_{\left(Y\right)}$, and the function Sym_NMF is applied. A new item is so built and, if it is recognized as promising by a specific function Control, it is inserted back into $\mathcal{Q}$. Otherwise, if Y is not promising, no new item is inserted back into $\mathcal{Q}$—that is, Y is discarded. Two bounds, $t_{min}$ and $t_{max}$ (Section 6), are used to control the execution flow for each item.

function$\mathtt{Kfix}\left(k,{\mathbf{\sigma }}^{\left(k\right)},q\right)$;

inizialize $\mathit{m},M,\mathcal{Q}=\left\{\right\}$;

compute matrices $A\left({\sigma }_i^{\left(k\right)}\right)$, for $i=1,\dots ,3$, using (1) and (2);

generate randomly $W_0\left(j\right)$, for $j=1,\dots ,q$;

for i = 1 to 3

A = A(σi); α = max A;

$\mathtt{for}j=1\mathtt{to}q$

$W_0=W_0\left(j\right)$;

$r=i+3(j-1)$;

$\mathtt{Enqueue}\left(\mathcal{Q},\{r,0,W_0,O,\alpha ,0\}\right)$;

end for

end for

${\chi }_{min}=\infty$;

while Length ($\mathcal{Q}$) ≥ 1

Y = Dequeue ($\mathcal{Q}$);

{ν,W,H,α,cond} = Sym_NMF (A(Y),W(Y),H(Y),α(Y),λ);

(Π,ke) = Partition (W)

χ = DBindex(Π)

t = t(Y) + ν; ζ = χ + t/tmax;

if Control (t,χ,Π,ke,cond) then Enqueue ($\mathcal{Q}$,{r(Y),t,W,H,α,ζ})

end while

return (M, m);

At the end, the kth partition in M, if not empty, is the solution of the problem for the given k. Other partitions possibly present in M can be of interest, and will be used when Kfix is called for solving ${\mathcal{P}}_{k\mathit{v}ar}$.

The core of the whole procedure is the following Boolean function Control, which verifies whether an item is promising (i.e., must be further processed) or not.

functionControl$(t,\chi ,\Pi ,k_e,\mathit{c}ond)$;

if $\chi <m_{k_e}$ then $m_{k_e}=\chi ;M_{k_e}=\Pi$;

if $k_e<k$ then return $t<2t_{min}$;

if $\mathit{c}ond\left|\right|(t>t_{max})$ then return False;

if $t<t_{min}$ then return True;

return $\chi <m_{k_e}(1+exp(1-t/t_{min}))$

If True is returned, the selected item is enqueued again, otherwise it is discarded. When the number $k_e$ of clusters effectively found is equal to the expected k, and the convergence of Sym_NMF has not been yet reached, the item is enqueued both when $t<t_{min}$ and when $t_{min}<t<t_{max}$, provided that $\chi$ satisfies a bound, which becomes tighter as t increases. When, instead, $k_e<k$, the item is enqueued only if less than $2t_{min}$ iterations have been performed. In any case, the best value of the DB index and the corresponding clustering are maintained for each value of $k_e\le k$.

Remark 1.

(a) The use of the priority queue allows for an easy parallelization of the computation by simultaneously keeping active a number of items most equal to the number of the available processors. The numerical examples of Section 6 have been carried out both with eight active processors and one active processor in order to evaluate the scalability of the algorithm.

(b) The use of an internal validity index is fundamental in our heuristics. Because of its high impact in the algorithm, the choice of the DB index among the many indices listed in [20] has been driven by its low computational time complexity. The DB index, which turned out to be reliable in our experimentation, tends to detect clusters with separated convex envelopes.

5. The Clustering Algorithm for Problem ${\mathcal{P}}_{k\mathit{v}ar}$

For problem ${\mathcal{P}}_{k\mathit{v}ar}$, the dimension k of the clustering is not a priori assigned, but an interval ${\mathcal{J}}_k=[k_{min},k_{max}]$ is only given where k has to be looked for, according to some performance criterium. The problem can be tackled by applying the algorithm designed for problem ${\mathcal{P}}_{kfix}$ with different values of k in the interval, and then by choosing the one which gives the better result. The strategy we propose in the following algorithm Kvar selects the consecutive integers $k=k_{min},\dots ,k_{max}$. For each k, the triple ${\mathbf{\sigma }}^{\left(k\right)}$ of scale parameters is constructed using formula (10), given in the next section, and function Kfix is applied. The array m of the optimal DB values and the array M of the corresponding clusterings $\Pi$ are returned. The indices DB** and CL, described in Section 3, help in choosing the best k and the corresponding clustering. Moreover, if a proper value of $k_{max}$ is not assigned, plots of DB** and CL can indicate when the computation may be stopped.

function$\mathtt{Kvar}\left(k_{min},k_{max}\right)$

initialize m and M;

for $k=k_{min}$ to $k_{max}$

compute triple ${\mathbf{\sigma }}^{\left(k\right)}$;

($M,m$) = Kfix $\left(k,{\mathbf{\sigma }}^{\left(k\right)},q\right)$;

end for;

return ($M,m$)

From (10) we see that the same $\sigma$ appears repeatedly for different values of k. Hence, it is not worthwhile to construct each time the matrix $A\left(\sigma \right)$. Whenever possible, it is better to construct and store beforehand the matrices $A\left(\sigma \right)$ for all the different scale parameters $\sigma$ that are expected to be used.

Another shortcut that immediately comes to mind is to exploit the already computed NMF factorization of an $A\left(\sigma \right)$ for a given dimension k to obtain the NMF factorization of the same matrix for a larger k, instead of computing it from scratch as suggested in Kvar. In [5], these updating algorithms are proposed. We have tried them in our experimentation and have found that, in our clustering context, the improvements on the computational cost are negligible in the face of a more complex code. Hence we do not recommend them.

6. Experimentation

The experimentation has been performed in JAVA on a Macintosh with a 4 GHz Intel 8 core Xeon W processor running MAC OS MOJAVE 10.14.4 on synthetic datasets of points in ${\mathbb{R}}^2$. It intends to prove that the proposed heuristics is reliable. In fact, with regard to a multistart strategy, the proposed heuristics allows for time-saving, but one could expect that during the elimination of the less promising roads, roads leading to final good solutions can also be lost. To verify that this does not happen, the experimentation aims to test the effectiveness of the algorithm.

Initially, an extensive computation has been carried out on selected datasets of points, to determine suitable values for the parameters to be used in the subsequent computation. Namely:

-

the tolerances for GCD and Sym_NMF are set ${\eta }_1={10}^{-3}$ and ${\eta }_2={10}^{-4}$, respectively,

-

the constants for CL index are set $\mathit{p}=4$ and $\mathit{c}=100$,

-

the number of the starting random matrices for Kfix is set $q=8$,

-

the number of iterations for a single segment of Kfix is set $\lambda =10$, hence ${\nu }_{max}=10$,

-

the bounds for Control are set $t_{min}=3\lambda$ and $t_{max}=20\lambda$,

-

for the definition of ${\mathbf{\sigma }}^{\left(k\right)}$ we have set $\widehat{\sigma }=0.04$ and

$$\begin{array}{c}{\sigma }_0^{\left(k\right)}=\widehat{\sigma }\mathrm{for}k\le 5,{\sigma }_0^{\left(k\right)}=\widehat{\sigma }/2\mathrm{for}5<k\le 10,{\sigma }_0^{\left(k\right)}=\widehat{\sigma }/4\mathrm{for}10<k\le 20,\hfill \\ {\sigma }_0^{\left(k\right)}=\widehat{\sigma }/8\mathrm{for}20<k\le 40,{\sigma }_0^{\left(k\right)}=\widehat{\sigma }/16\mathrm{for}k>40.\hfill \end{array}$$

The triplets considered in the experiments, varying k, are

$${\mathbf{\sigma }}^{\left(k\right)}=\{{\sigma }_i^{\left(k\right)}={\sigma }_0^{\left(k\right)}/2^i,i=0,1,2\}.$$

(10)

Regarding the value $\widehat{\sigma }$, we have taken into account that the quantities $\parallel {\mathit{p}}_i-{\mathit{p}}_r{\parallel }_2^2/\mu$ in (1) are not greater than 1. The choice $\widehat{\sigma }=0.04$ makes the lowest value exp$(-1/\sigma )$ of $e_{i,r}$ comparable with the machine precision.

The datasets used in the first two experiments are suggested in [25]. The points are normally distributed with different variances around a given number of centers.

The first experiment concerns the clustering of the three datasets of Figure 1: dataset Subcluster (SC) has five clusters, and four of them are subclusters forming two pairs of clusters, respectively; dataset Skewdistribution (SD) has three clusters with skewed dispersions; dataset Differentdensity (DD) has clusters with different densities. Each dataset is generated with 1000 points.

We applied Kvar to each dataset with k varying in the interval $[2,15]$. Figure 2 shows the piecewise linear plots versus k of the DB index (dashed line), DB** index (solid line), and CL index (dotted line) for the three datasets. The CL plot is normalized in order to fit the size of the figure.

For the dataset SC, the DB index suggests choosing $k=3$, while the DB** index suggests both $k=3$ and $k=5$, with a weak preference for the former. The plot of the CL index, which is zero for $k\le 5$, shows that the clusters in this range of k are well-separated, while for $k\ge 6$ the increase of the plot suggests that some clusters are improperly subdivided.

For both datasets SD and DD, the DB and DB** indexes indicate the same value, 3, for k. The CL plot, which is not zero even for small values of k, reveals the presence of close points in different clusters. The difficulty encountered in the assignment of these points to the clusters suggests that such points could be considered as noise. In any case, using the information gathered from the CL plot, the computation should stop before k becomes too large.

The purpose of the second test is precisely to see how the procedure behaves in the presence of noise. To this aim, the two datasets of five points Well-separated (WS) and Well-separated noise (WSN) of Figure 3 are generated, the latter one with $5\%$ noise added to the former.

We applied Kvar, with k varying in the interval $[2,20]$ and obtained the piecewise linear plots versus k of Figure 4. For dataset WSN, according to the DB index, both $k=5$ and $k=6$ seemed to be acceptable. This uncertainty is explained by the presence of noise which causes spurious clusters to appear. The DB** index correctly suggests $k=5$. The presence of noise is confirmed by the CL index which is not zero for small values of k.

To analyze more thoroughly the performance of our algorithm when subclusters are present, a third experiment was carried out concerning the three datasets of Figure 5. Dataset C17 consists of 1700 points generated around 17 randomly chosen centers; dataset C23, consisting of 2300 points, has 23 clusters of 100 points which can be grouped in several ways in a lower number of clusters; dataset C27, consisting of 2700 points, has 27 clusters of 100 points which can be seen as 9 clusters of 300 points or three clusters of 900 points, depending on the granularity level of interest.

Problems C17, C23, and C27 have been tested with values of k ranging in the intervals $[2,25]$, $[2,30]$, and $[2,35]$, respectively. The resulting piecewise linear plots versus k of DB, DB**, and CL indices are shown in Figure 6. By using the DB** indices, it clearly appears that our algorithm is able to find good clusterings for all three problems. The inspection of the plot of the DB** index gives further information. For example, for problem C17, a local minimum of the DB** index for $k=4$ evidences the clustering shown in Figure 7 on the left. For problem C23, a local minimum of the DB** index for $k=7$ evidences the possible clustering shown in Figure 7 on the right. For problem C27, two local minima in the DB** plot for $k=3$ and $k=9$ evidence the two other expected clusterings.

In all the experiments shown above, the measure CL helps decide the value of $k_{max}$ if not assigned.

The fourth test regards the computational cost of algorithm Kvar when run on one or eight processors. Due to its iterative and heuristic structure, our algorithm is not suitable for a theoretical analysis of the complexity. In order to analyze how the required time increases as a function of the number of the points, the datasets SC, SD, DD, WS, and WSN have been generated with n varying in $\{1000,2000,4000,8000\}$ and k varying in the interval $[2,15]$. Figure 8 shows the plots versus n of the costs in seconds, averaged on all the datasets. The times corresponding to runs with one processor are fitted with a solid line, where those corresponding to runs with eight processors are fitted with a dashed line. The resulting polynomial fits are $1.5\times {10}^{-5}{n}^{2.16}$ for one processor and $1.4\times {10}^{-6}{n}^{2.23}$ for eight processors. This shows that for the considered problems, the time grows slightly more than quadratically as a function of n, but clearly less than cubically, as one could expect. It also shows that the whole procedure has a good degree of scalability.

7. Discussion and Conclusions

In this paper, we considered the symmetric modeling of the clustering problem and proposed an algorithm which applies NMF techniques to a penalized nonsymmetric minimization problem. The proposed algorithm makes use of a similarity matrix A, which depends on the Gaussian kernel function and a scale parameter $\sigma$, varying with k. The Gaussian kernel function was chosen because it is widely used in the clustering problems of points in ${\mathbb{R}}^d$. Of course, other kernels can be used in different environments, with different procedures. We think that our heuristic approach could also be applied to treat other kernels.

As shown in [1,13], the ANLS algorithm with the BPP method handles large datasets more efficiently than other algorithms frequently used for the same problem. Moreover, in [14], it is shown that ANLS with GCD allows for further time reduction with respect to ANLS with BPP.

The proposed heuristic approach, which uses ANLS with GCD, processes multiple instances of the problem and focuses the computation only on the more promising ones, making use of the DB index. In this way, an additional time reduction is achieved with respect to the standard multistart approach. The experimentation, conducted on synthetic problems with different characteristics, evidences the reliability of the algorithm.

The same framework is repeatedly applied with different values of k varying in an assigned interval, allowing us to deal also with the clustering problem when the number of clusters is not a priori fixed. The combined use of the different considered validity indices allows to point out the correct solutions.