Deterministic Coresets for k-Means of Big Sparse Data ^†

Abstract

Let P be a set of n points in ${\mathbb{R}}^d$, $k\ge 1$ be an integer and $\epsilon \in (0,1)$ be a constant. An ε-coreset is a subset $C\subseteq P$ with appropriate non-negative weights (scalars), that approximates any given set $Q\subseteq {\mathbb{R}}^d$ of k centers. That is, the sum of squared distances over every point in P to its closest point in Q is the same, up to a factor of $1\pm \epsilon$ to the weighted sum of C to the same k centers. If the coreset is small, we can solve problems such as k-means clustering or its variants (e.g., discrete k-means, where the centers are restricted to be in P, or other restricted zones) on the small coreset to get faster provable approximations. Moreover, it is known that such coreset support streaming, dynamic and distributed data using the classic merge-reduce trees. The fact that the coreset is a subset implies that it preserves the sparsity of the data. However, existing such coresets are randomized and their size has at least linear dependency on the dimension d. We suggest the first such coreset of size independent of d. This is also the first deterministic coreset construction whose resulting size is not exponential in d. Extensive experimental results and benchmarks are provided on public datasets, including the first coreset of the English Wikipedia using Amazon’s cloud.

1. Background

Given a set of n points in ${\mathbb{R}}^d$, and an error parameter $\epsilon >0$, a coreset in this paper is a small set of weighted points in ${\mathbb{R}}^d$, such that the sum of squared distances from the original set of points to any set of k centers in ${\mathbb{R}}^d$ can be approximated by the sum of weighted squared distances from the points in the coreset. The output of running an existing clustering algorithm on the coreset would then yield approximation to the output of running the same algorithm on the original data, by the definition of the coreset.

Coresets were first suggested by [1] as a way to improve the theoretical running time of existing algorithms. Moreover, a coreset is a natural tool for handling $\mathrm{Big}\mathrm{Data}$ using all the computation models that are mentioned in the previous section. This is mainly due to the merge-and-reduce tree approach that was suggested by [2,3] and is formalized by [4]: coresets can be computed independently for subsets of input points, e.g., on different computers, and then be merged and re-compressed again. Such a binary compression tree can also be computed using one pass over a possibly unbounded streaming set of points, where in every moment only $O\left(\log n\right)$ coresets exist in memory for the n points streamed so far. Here the coreset is computed only on small chunks of points, so a possibly inefficient coreset construction still yields efficient coreset constructions for large sets; see Figure 1. Note that the coreset guarantees are preserved while using this technique, while no assumptions are made on the order of the streaming input points. These coresets can be computed independently and in parallel via M machines (e.g., on the cloud) and reduce the running time by a factor of M. The communication between the machines is also small, since each machine needs to communicate to a main server only the coreset of its data.

In practice, this technique can be implemented easily using the map-reduce approach of modern software for $\mathrm{Big}\mathrm{Data}$ such as [5].

Coresets can also be used to support the dynamic model where points are deleted/inserted. Here the storage is linear in n since we need to save the tree in memory (practically, on the hard drive), however the update time is only logarithmic in n since we need to reconstruct only $O\left(\log n\right)$ coresets that correspond to the deleted/inserted point along the tree. First such coreset of size independent of d was introduced by [6]. See [1,2] for details.

Constrained k-Means and Determining k

Since the coreset approximates every set of k centers, it can also be used to solve the k-means problem under different constraints (e.g., allowed areas for placing the centers) or given a small set of candidate centers. In addition, the set of centers may contain duplicate points, which means that the coreset can approximate the sum of squared distances for $k^{\prime }<k$ centers. Hence, coresets can be used to choose the right number k of centers up to $k^{\prime }$, by minimizing the sum of squared distances plus $f\left(k\right)$ for some function of k. Full opensource is available [8].

2. Related Work

We summarize existing coresets constructions for k-means queries, as will be formally defined in Section 4.

Importance Sampling

Following a decade of research, coreset of size polynomial in d were suggested by [9]. Ref [10] suggested an improved version of size $O(d{k}^2/{\epsilon }^2)$ in which is a special case of the algorithms proposed by [11]. The construction is based on computing an approximation to the k-means of the input points (with no constraints on the centers) and then sample points proportionally to their distance to these centers. Each chosen point is then assigned a weight that is inverse proportional to this distance. The probability of failure in these algorithms reduces exponentially with the input size. Coresets of size $O(dk/{\epsilon }^2)$, i.e., linear in k, were suggested in work of [12], however the weight of a point may be negative or a function of the given query. For the special case $k=1$, Inaba et al. [13], provided constructions of coresets of size $O(1/{\epsilon }^2)$ using uniform sampling.

Projection based coresets. Data summarization which are similar to coresets of size $O(k/\epsilon )$ that are based on projections on low-dimensional subspaces that diminishes the sparsity of the input data were suggested by [14] by improving the analysis of [4]. Recently [15] improves both on [4,14] by applying Johnson-Lindenstrauss Lemma [16] on construction from [4]. However, due to the projections, the resulting summarizations of all works mentioned above are not subset of the input points, unlike the coreset definition of this paper. In particular, for sparse data sets such as adjacency matrix of a graphs, documents-term matrix of Wikipedia, or images-objects matrix, the sparsity of the data diminishes and a single point in the summarization might be larger than the complete sparse input data.

Other type of weak coresets approximates only the optimal solution, and not every k centers. Such randomized coresets of size independent of d and only polynomial in k were suggested by [11] and simplified by [12].

Deterministic Constructions. The first coresets for k-means [2,17] were based on partitioning the data into cells, and take a representative point from each cell to the coreset, as is done in hashing or Hough transform [18]. However, these coresets are of size at least $k/{\epsilon }^{O\left(d\right)}$, i.e., exponential in d, while still providing result which is a sub-set of the the input in contrast to previous work [17]. While, our technique is most related to the deterministic construction that was suggested in [4] by recursively computing k-means clustering of the input points. While the output set has size independent of d, it is not a coreset as defined in this paper, since it is not a subset of the input and thus cannot handle sparse data as explained above. Techniques such as uniform sampling for each cluster yields coresets with probability of failure that is linear in the input size, or whose size depends on d.

m-means is a coreset for k-means? A natural approach for coreset construction, that is strongly related to this paper, is to compute the m-means of the input set P, for a sufficiently large m, where the weight of each center is the number of point in its cluster. If the sum of squared distances to these m-centers is about $\epsilon$ factor from the k-means, we obtain a $(k,\epsilon )$-coreset by (a weaker version of) the triangle inequality. Unfortunately, it was recently proved in [19] that there exists sets such that $m\in k^{\mathsf{\Omega }\left(d\right)}$ centers are needed in order to obtain this small sum of squared distances.

3. Our Contribution

We suggest the following deterministic algorithms:

• An algorithm that computes a $(1+\epsilon )$ approximation for the k-means of a set P that is distributed (partitioned) among M machines, where each machine needs to send only $k^{O\left(1\right)}$ input points to the main server at the end of its computation.
• A streaming algorithm that, after one pass over the data and using $k^{O\left(1\right)}\log n$ memory returns an $O\left(\log n\right)$-approximation to the k-means of P. The algorithm can run “embarrassingly in parallel [20] on data that is distributed among M machines, and support insertion/deletion of points as explained in the previous section.
• Description of how to use our algorithm to boost both the running time and quality of any existing k-means heuristic using only the heuristic itself, even in the classic off-line setting.
• Extensive experimental results on real world data-sets. This includes the first k-means clustering with provable guarantees for the English Wikipedia, via 16 EC2 instances on Amazon’s cloud.
• Open-code for for fully reproducing our results and for the benefit of the community. To our knowledge, this is the first coreset code that can run on the cloud without additional commercial packages.

3.1. Novel Approach: m-Means Is A Coreset for k-Means, for Smart Selection of m

One of our main technical result is that for every constant $\epsilon >0$, there exists an integer $m\in k^{O\left(1\right)}$ such that the m-means of the input set (or its approximation) is a $(k,\epsilon )$-coreset; see Theorem 2. However, simply computing the m-means of the input set for a sufficiently large m might yield m that is exponential in d, as explained by [19] and the related work. Instead, Algorithm 1 carefully selects the right m between k and $k^{O\left(1\right)}$ after checking the appropriate conditions in each iteration.

3.2. Solving k-Means Using k-Means

It might be confusing that we suggest to solve the k-means problem by computing m-means for $m>k$. In fact, most of the coreset constructions actually solve the optimal problem in one of their first construction steps. The main observation is that we never run the coreset on the complete input of n (or unbounded stream of) points, but only on small subsets of size $\sim 2m$. This is since our coresets are composable and can be merged (to $2m$ points) and reduced back to m using the merge-and-reduce tree technique. The composable property follows from the fact that the coreset approximates the sum of squared distances to every k-centers, and not just the k-means for the subset at hand.

3.3. Running Time

Running time that is exponential in k is unavoidable for any $(1+\epsilon )$-approximation algorithm that solves k-means, even in the planar case $(d=2)$ [21] and $\epsilon <0.1$ [4]. Our main contributions is a coreset construction that uses memory that is independent of d and running time that is near-linear in n. To our knowledge this is an open problem even for the case $k=2$. Nevertheless, for large values of $\epsilon$ (e.g., $\epsilon >10$) we may use existing constant factor approximations that takes time polynomial in k to compute our coreset in time that is near-linear in n but also polynomial in k.

In practice, provable $(1+\epsilon )$-approximations for k-means are rarely used due to the lower bounds on the running times above. Instead, heuristics are used. Indeed, in our experimental results we evaluate this approach instead of computing the optimal k-means during the coreset construction and on the resulting coreset itself.

4. Notation and Main Result

The input to our problem is a set $P^{\prime }$ of n points in ${\mathbb{R}}^d$, where each point $p\in P^{\prime }$ includes a multiplicative weight $u\left(p\right)>0$. In addition, there is an additive weight $\rho >0$ for the set. Formally, a weighted set in ${\mathbb{R}}^d$ is a tuple $P=(P^{\prime },u,\rho )$, where $P^{\prime }\subseteq {\mathbb{R}}^d$, $u:P^{\prime }\to [0,\infty )$. In particular, an unweighted set has a unit weight $u\left(p\right)=1$ for each point, and a zero additive weight.

4.1. k-Means Clustering

For a given set $Q=\left\{q_1,\cdots ,q_k\right\}$ of $k\ge 1$ centers (points) in ${\mathbb{R}}^d$, the Euclidean distance from a point $p\in {\mathbb{R}}^d$ to its closest center in Q is denoted by 1.7em$(p,Q)={\min }_{q\in Q}{∥p-q∥}_2$. The sum of these weighted squared distances over the points of P is denoted by

$$\mathrm{cost}(P,Q):=\sum _{p\in P^{\prime }}u\left(p\right)·1.7{em}^2(p,Q)+\rho .$$

If P is an unweighted set, this cost is just the sum of squared distances over each point in $P^{\prime }$ to its closest center in Q.

Let $P_i^{\prime }$ denote the subset of points in $P^{\prime }$ whose closest center in Q is $q_i$, for every $i\in \left[m\right]=\left\{1,\cdots ,m\right\}$. Ties are broken arbitrarily. This yields a partition $\left\{P_1^{\prime },\cdots ,P_k^{\prime }\right\}$ of $P^{\prime }$ by Q. More generally, the partition of P by Q is the set $\left\{P_1,\cdots ,P_k\right\}$ where $P_i=(P_i^{\prime },u_i,\rho /k)$, and $u_i\left(p\right)=u\left(p\right)$ for every $p\in P_i^{\prime }$ and every $i\in \left[k\right]$.

A set $Q_k$ that minimizes this weighted sum $\mathrm{cost}(P,Q)$ over every set Q of k centers in ${\mathbb{R}}^d$ is called the k-means of P. The 1-means $\mu \left(P\right)$ of P is called the centroid, or the center of mass, since

$$\mu \left(P\right)=\frac{1}{{\sum }_{p^{\prime }\in P^{\prime }}u\left(p^{\prime }\right)}\sum _{p\in P^{\prime }}u\left(p\right)·p.$$

We denote the cost of the k-means of P by $\mathrm{opt}(P,k):=\mathrm{cost}(P,Q_k)$.

4.2. Coreset

Computing the k-means of a weighted set P is the main motivation of this paper. Formally, let $\epsilon >0$ be an error parameter. The weighted set $S=(S^{\prime },w,\varphi )$ is a $(k,\epsilon )$-coreset for P, if for every set $Q\subset {\mathbb{R}}^d$ of $\left|Q\right|=k$ centers we have

$$(1-\epsilon )\mathrm{cost}(P,Q)\le \mathrm{cost}(S,Q)\le (1+\epsilon )\mathrm{cost}(P,Q),$$

That is,

$$\begin{array}{c}\hfill (1-\epsilon )\sum _{p\in P}u\left(p\right)1.7{em}^2(p,Q)+\rho \le \sum _{p\in S}w\left(p\right)·1.7{em}^2(p,Q)+\varphi \le (1+\epsilon )\sum _{p\in P}u\left(p\right)·1.7{em}^2(p,Q)+\rho .\end{array}$$

To handle streaming data we will need to compute “coresets for union of coresets”, which is the reason that we assume that both the input P and its coreset S are weighted sets.

4.3. Sparse Coresets

Unlike previous work, we add the constraints that if each point in P is sparse, i.e., has few non-zeroes coordinates, then the set S will also be sparse. Formally, the maximum sparsity $s\left(P\right):={\max }_{p\in P}{∥p∥}_0$ of P is the maximum number of non zeroes entries ${∥p∥}_0=\left|\left\{i\mid p_i\ne 0,i\in \left[d\right]\right\}\right|$ over every point p in P.

In particular, if each point in S is a linear combination of at most $\alpha$ points in P, then $s\left(S\right)\le \alpha ·s\left(P\right)$. In addition, we would like that the set S will be of size independent of both $n=\left|P\right|$ and d.

We can now state the main result of this paper.

Theorem 1

(Small sparse coreset). For every weighted set $P=(P^{\prime },u,\rho )$ in ${\mathbb{R}}^d$, $\epsilon >0$ and an integer $k\ge 1$, there is a $(k,\epsilon )$-coreset $S=(S^{\prime },w,\varphi )$ of size $\left|S\right|=k^{O(1/{\epsilon }^2)}$ where each point in $S^{\prime }$ is a linear combination of $O(1/{\epsilon }^2)$ points from $P^{\prime }$. In particular, the maximum sparsity of $S^{\prime }$ is $s\left(P\right)/{\epsilon }^2$.

By plugging this result to the traditional merge-and-reduce tree in Figure 1, it is straight-forward to compute a coreset using one pass over a stream of points.

Corollary 1.

A $(k,\epsilon \log n)$ coreset $(S,w,\varphi )$ of size $\left|S\right|=\log \left(n\right)·k^{O(1/{\epsilon }^2)}$ and maximum sparsity $s\left(P\right)/{\epsilon }^2$ can be computed for the set P of the n points seen so far in an unbounded stream, using $\left|S\right|·s\left(P\right)/{\epsilon }^2$ memory words. The insertion time per point in the stream is $\log \left(n\right)·2^{{(k/\epsilon )}^{O\left(1\right)}}$. If the stream is distributed uniformly to M machines, then the amortized insertion time per point is reduced by a (multiplicative) factor of M to $(1/M)\log \left(n\right)·2^{{(k/\epsilon )}^{O\left(1\right)}}$. The coreset for the union of streams can then be computed by communicating the M coresets to a main server.

5. Coreset Construction

Our main coreset construction algorithm k$-\mathrm{M}\mathrm{EAN}-\mathrm{C}\mathrm{ORESET}(P,k,\epsilon )$ gets a set P as input, and returns a $(k,\epsilon )$-coreset $(S,w)$; see Algorithm 1.

To obtain running time that is linear in the input size, without loss of generality, we assume that P has $\left|P\right|=k^{O(1/{\epsilon }^2)}$ points, and that the cardinality of the output S is $\left|S\right|\le \left|P\right|/2$. This is thanks to the traditional merge-and-reduce approach: given a stream of n points, we apply the coreset construction only on subsets of size $2·\left|S\right|$ from P during the streaming and reduce them by half. See Figure 1 and e.g., [4,7] for details.

Algorithm Overview

In Line 1 we compute the smallest integer $m=k^t$ such that the cost $\mathrm{opt}(P,m)$ of the m-means of P is close to the cost $\mathrm{opt}(P,mk)$ of the $\left(mk\right)$-means of P. In Line 3 we compute the corresponding partition $\left\{P_1,\cdots ,P_m\right\}$ of P by its m-means $Q_m=\left\{q_1,\cdots ,q_m\right\}$. In Line 5 a $(1,\epsilon )$-sparse coreset $S_i$ of size $O(1/{\epsilon }^2)$ is computed for every $P_i$, $i\in \left[m\right]$. This can be done deterministically e.g., by taking the mean of $P_i$ as explained in Lemma 1 or by using a gradient descent algorithm, namely Frank-Wolfe, as explained in [22] which also preserves the sparsity of the coreset as desired by our main theorem. The output of the algorithm is the union of the means of all these coresets, with appropriate weight, which is the size of the coreset.

The intuition behind the algorithm stems from the assumption that m-means clustering will have lower cost than k-means and this is actually supported by series of previous work [23,24]. In fact our experiments in Section 7 evidence that in practice it actually works better than anticipated by theoretical bounds.

6. Proof of Correctness

The first lemma states the common fact that the sum of squared distances of a set of point to a center is the sum of their squared distances to their center of mass, plus the squared distance to the center (the variance of the set).

Lemma 1.

For every$x\in {\mathbb{R}}^d$

$$\mathrm{cost}(P,x)=\mathrm{cost}(P,\mu \left(P\right))+{∥\mu \left(P\right)-x∥}^2\sum _{p\in P}u\left(p\right).$$

Proof. 

We have

$$\begin{array}{cc}\hfill \mathrm{cost}(P,x)-\rho & =\sum _{p\in P}u\left(p\right){∥p-x∥}^2=\sum _{p\in P}u\left(p\right){∥(p-\mu (P\left)\right)+\left(\mu \right(P)-x)∥}^2\hfill \\ & =\sum _{p\in P}u\left(p\right){∥p-\mu \left(P\right)∥}^2+\sum _{p\in P}u\left(p\right){∥\mu \left(P\right)-x∥}^2+2(\mu \left(p\right)-x)·\sum _{p\in P}u\left(p\right)(p-\mu \left(P\right)).\hfill \end{array}$$

The last term equals zero since $\mu \left(P\right)=\frac{1}{{\sum }_{p^{\prime }\in P}u\left(p^{\prime }\right)}·{\sum }_{p\in P}u\left(p\right)·p$, and thus

$$\begin{array}{cc}\hfill \sum _{p\in P}u\left(p\right)(p-\mu \left(P\right))& =\sum _{p\in P}u\left(p\right)·p-\sum _{p\in P}u\left(p\right)\mu \left(P\right)=\sum _{p\in P}u\left(p\right)·p-\sum _{p\in P}u\left(p\right)·p=0.\hfill \end{array}$$

Hence,

$$\mathrm{cost}(P,x)=\rho +\sum _{p\in P}u\left(p\right){∥p-\mu \left(P\right)∥}^2+\sum _{p\in P}u\left(p\right){∥\mu \left(P\right)-x∥}^2=\mathrm{cost}(P,\mu \left(P\right))+{∥\mu \left(P\right)-x∥}^2\sum _{p\in P}u\left(p\right).$$

 □

The second lemma shows that assigning all the points of P to the closest center in Q yields a small multiplicative error if the 1-mean and the k-means of P has roughly the same cost. If $t=0$, this means that we can approximate $\mathrm{cost}(P,Q)$ using only one center in the query; see Line 1 of Algorithm 1. Note that $(1+2\epsilon )/(1-2\epsilon )\le 1+4\epsilon$ for $\epsilon <1/4$.

Lemma 2.

For every set $Q\subseteq {\mathbb{R}}^d$ of $\left|Q\right|=k$ centers we have

$$\mathrm{cost}(P,Q)\le \underset{q\in Q}{\min }\mathrm{cost}(P,\left\{q\right\})\le \mathrm{cost}(P,Q)·\frac{1+2\epsilon }{1-2\epsilon }+\frac{\mathrm{opt}(P,1)-\mathrm{opt}(P,k)}{(1-2\epsilon )\epsilon }.$$

(1)

Proof. 

Let $q^{*}$ denote a center that minimizes $\mathrm{cost}(P,\{q\left\}\right)$ over $q\in Q$. The left inequality of (1) is then straight-forward since

$$\begin{array}{cc}\hfill \mathrm{cost}(P,Q)-\underset{q\in Q}{\min }\mathrm{cost}(P,\left\{q\right\})& =\sum _{p\in P^{\prime }}\underset{q\in Q}{\min }u\left(p\right){∥p-q∥}^2-\sum _{p\in P^{\prime }}u\left(p\right){∥p-q^{*}∥}^2\hfill \\ \hfill & =\sum _{p\in P^{\prime }}u\left(p\right)\left(\underset{q\in Q}{\min }{∥p-q∥}^2-{∥p-q^{*}∥}^2\right)\le 0.\hfill \end{array}$$

(2)

It is left to prove the right inequality of (1). Indeed, for every $p\in P^{\prime }$, let $q_p\in Q$ denote the closest point to p in Q. Ties are broken arbitrarily. Hence,

$$\underset{q\in Q}{\min }\mathrm{cost}(P,\left\{q\right\})-\mathrm{cost}(P,Q)=\sum _{p\in P^{\prime }}u\left(p\right){∥p-q^{*}∥}^2-\sum _{p\in P^{\prime }}u\left(p\right){∥p-q_p∥}^2.$$

Let $\left\{P_1,\cdots ,P_k\right\}$ denote the partition of P by $Q=\left\{q^1,\cdots ,q^k\right\}$, where $P_i$ are the closest points to $q^i$ for every $i\in \left[k\right]$; see Section 4. For every $p\in P_i^{\prime }$, let $q_p^{*}=\mu \left(P_i\right)$. Hence,

$$\begin{array}{cc}\hfill & \sum _{p\in P_i^{\prime }}u\left(p\right){∥p-q^{*}∥}^2-\sum _{p\in P_i^{\prime }}u\left(p\right){∥p-q_p∥}^2\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & =\sum _{p\in P_i^{\prime }}u\left(p\right){∥p-\mu \left(P_i\right)∥}^2+{∥\mu \left(P_i\right)-q^{*}∥}^2\sum _{p\in P_i^{\prime }}u\left(p\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & -\left(\sum _{p\in P_i^{\prime }}u\left(p\right){∥p-\mu \left(P_i\right)∥}^2+{∥\mu \left(P_i\right)-q^i∥}^2\sum _{p\in P_i^{\prime }}u\left(p\right)\right)\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & =\sum _{p\in P_i^{\prime }}u\left(p\right)\left({∥q_p^{*}-q^{*}∥}^2-{∥q_p^{*}-q_p∥}^2\right),\hfill \end{array}$$

(6)

where in (4) and (5) we substituted $x=q^{*}$ and $x=q_p$ respectively in Lemma 1, and in (6) we use the fact that $q_p^{*}=\mu \left(P_i\right)$ and $q_p=q^i$ for every $p\in P_i$. Summing (6) over $i\in \left[k\right]$ yields

$$\begin{array}{cc}\hfill & \sum _{p\in P^{\prime }}u\left(p\right){∥p-q^{*}∥}^2-\sum _{p\in P^{\prime }}u\left(p\right){∥p-q_p∥}^2=\sum _{p\in P^{\prime }}u\left(p\right)\left({∥q_p^{*}-q^{*}∥}^2-{∥q_p^{*}-q_p∥}^2\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\sum _{p\in P^{\prime }}u\left(p\right)\left({∥(q_p^{*}-\mu \left(P\right))+(\mu \left(P\right)-q^{*})∥}^2-{∥(q_p^{*}-\mu \left(P\right))+(\mu \left(P\right)-q_p)∥}^2\right)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & =\sum _{p\in P^{\prime }}u\left(p\right)\left({∥\mu \left(P\right)-q^{*}∥}^2-{∥\mu \left(P\right)-q_p∥}^2\right)\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill & -2\sum _{p\in P^{\prime }}u\left(p\right)(q_p^{*}-\mu \left(P\right))(q^{*}-q_p).\hfill \end{array}$$

(9)

To bound (8), we substitute $x=q^{*}$ and then $x=q$ in Lemma 1, and obtain that for every $q\in Q$

$$\begin{array}{cc}\hfill \left({∥\mu \left(P\right)-q^{*}∥}^2-{∥\mu \left(P\right)-q∥}^2\right)\sum _{p\in P^{\prime }}u\left(p\right)& =\mathrm{cost}(P,\left\{q^{*}\right\})-\mathrm{cost}(P,\mu \left(P\right))-\left(\mathrm{cost}(P,\{q\left\}\right)-\mathrm{cost}(P,\mu (P\left)\right)\right)\hfill \\ & =\mathrm{cost}(P,\left\{q^{*}\right\})-\mathrm{cost}(P,\left\{q\right\})\le 0.\hfill \end{array}$$

where the last inequality is by the definition of $q^{*}$. This implies that for every $p\in P^{\prime }$,

$${∥\mu \left(P\right)-q^{*}∥}^2-{∥\mu \left(P\right)-q_p∥}^2\le 0.$$

Plugging the last inequality in (8) yields

$$\begin{array}{cc}\hfill & \sum _{p\in P^{\prime }}u\left(p\right){∥p-q^{*}∥}^2-\sum _{p\in P^{\prime }}u\left(p\right){∥p-q_p∥}^2\le -2\sum _{p\in P^{\prime }}u\left(p\right)(q_p^{*}-\mu \left(P\right))(q^{*}-q_p)\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & \le 2\sum _{p\in P^{\prime }}u\left(p\right)∥q_p^{*}-\mu \left(P\right)∥∥q^{*}-q_p∥=\sum _{p\in P^{\prime }}u\left(p\right)·2·\frac{∥q_p^{*}-\mu \left(P\right)∥}{\sqrt{\epsilon }}·\sqrt{\epsilon }∥q^{*}-q_p∥\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \le \sum _{p\in P^{\prime }}u\left(p\right)\left(\frac{{∥q_p^{*}-\mu \left(P\right)∥}^2}{\epsilon }+\epsilon {∥q^{*}-q_p∥}^2\right)\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & =\frac{1}{\epsilon }\sum _{p\in P^{\prime }}u\left(p\right){∥q_p^{*}-\mu \left(P\right)∥}^2+\epsilon \sum _{p\in P^{\prime }}u\left(p\right){∥q^{*}-q_p∥}^2,\hfill \end{array}$$

(13)

where (11) is by Cauchy-Schwartz inequality, and in (12) we use the fact that $2ab\le a^2+b^2$ for every $a,b\ge 0$.

To bound the left term of (13) we use the fact $q_p^{*}=\mu \left(P_i\right)$ and substitute $x=\mu \left(P\right)$, $P=P_i$ in Lemma 1 for every $i\in \left[k\right]$ as follows.

$$\begin{array}{cc}\hfill & \sum _{p\in P^{\prime }}u\left(p\right){∥q_p^{*}-\mu \left(P\right)∥}^2=\sum _{i=1}^k{∥\mu \left(P_i\right)-\mu \left(P\right)∥}^2\sum _{p\in P_i^{\prime }}u\left(p\right)\hfill \\ \hfill & =\sum _{i=1}^k\left(\sum _{p\in P_i^{\prime }}u\left(p\right){∥p-\mu \left(P\right)∥}^2-\sum _{p\in P_i^{\prime }}u\left(p\right){∥p-\mu \left(P_i\right)∥}^2\right)\hfill \\ \hfill & =\sum _{p\in P^{\prime }}u\left(p\right){∥p-\mu \left(P\right)∥}^2-\sum _{i=1}^k\sum _{p\in P_i^{\prime }}u\left(p\right){∥p-\mu \left(P_i^{\prime }\right)∥}^2\le \mathrm{opt}(P,1)-\mathrm{opt}(P,k).\hfill \end{array}$$

(14)

To bound the right term of (13) we use ${(a-b)}^2\le a^2+b^2+2\left|ab\right|\le 2a^2+2b^2$ to obtain

$$\begin{array}{cc}\hfill \sum _{p\in P^{\prime }}u\left(p\right){∥q^{*}-q_p∥}^2& =\sum _{p\in P^{\prime }}u\left(p\right){∥(q^{*}-p)+(p-q_p)∥}^2\hfill \\ & \le \sum _{p\in P^{\prime }}u\left(p\right)\left(2{∥q^{*}-p∥}^2+2{∥p-q_p∥}^2\right)=2·\left(\mathrm{cost}(P,\left\{q^{*}\right\})+\mathrm{cost}(P,Q))\right).\hfill \end{array}$$

Plugging (14) and the last inequality in (13) yields

$$\begin{array}{cc}\hfill \mathrm{cost}(P,\left\{q^{*}\right\})-\mathrm{cost}(P,Q)& =\sum _{p\in P^{\prime }}u\left(p\right){∥p-q^{*}∥}^2-\sum _{p\in P^{\prime }}u\left(p\right){∥p-q_p∥}^2\hfill \\ & \le \frac{\mathrm{opt}(P,1)-\mathrm{opt}(P,k)}{\epsilon }+2·\epsilon \left(\mathrm{cost}(P,\left\{q^{*}\right\})+\mathrm{cost}(P,Q)\right).\hfill \end{array}$$

Rearranging,

$$\begin{array}{cc}\hfill \mathrm{cost}(P,\left\{q^{*}\right\})& \le \mathrm{cost}(P,Q)·\frac{1+2\epsilon }{1-2\epsilon }+\frac{\mathrm{opt}(P,1)-\mathrm{opt}(P,k)}{(1-2\epsilon )\epsilon }\hfill \end{array}$$

Together with (2) this proves Lemma 2. □

Lemma 3.

Let S be a $(1,\epsilon )$-coreset for a weighted set P in ${\mathbb{R}}^d$. Let $Q\subseteq {\mathbb{R}}^d$ be a finite set. Then

$$(1-\epsilon )\underset{q\in Q}{\min }\mathrm{cost}(P,\left\{q\right\})\le \underset{q\in Q}{\min }\mathrm{cost}(S,\left\{q\right\})\le (1+\epsilon )\underset{q\in Q}{\min }\mathrm{cost}(P,\left\{q\right\})$$

(15)

Proof. 

Let $q_P\in Q$ be a center such that $\mathrm{cost}(P,\left\{q_P\right\})={\min }_{q\in Q}\mathrm{cost}(P,\left\{q\right\})$, and let $q_S\in Q$ be a center such that $\mathrm{cost}(S,\left\{q_S\right\})={\min }_{q\in Q}\mathrm{cost}(S,\left\{q\right\})$. The right side of (15) is bounded by

$$\underset{q\in Q}{\min }\mathrm{cost}(S,\left\{q\right\})=\mathrm{cost}(S,\left\{q_S\right\})\le \mathrm{cost}(S,\left\{q_P\right\})\le (1+\epsilon )\mathrm{cost}(P,\left\{q_P\right\})=(1+\epsilon )\underset{q\in Q}{\min }\mathrm{cost}(P,\left\{q\right\}),$$

where the first inequality is by the optimality of $q_S$, and the second inequality is since S is a coreset for P. Similarly, the left hand side of (15) is bounded by

$$\begin{array}{cc}\hfill (1-\epsilon )\underset{q\in Q}{\min }\mathrm{cost}(P,\left\{q\right\})=(1-\epsilon )\mathrm{cost}(P,\left\{q_P\right\})& \le (1-\epsilon )\mathrm{cost}(P,\left\{q_S\right\})\le (1-\epsilon )(1+\epsilon )\mathrm{cost}(S,\left\{q_S\right\})\hfill \\ & =(1-{\epsilon }^2)\underset{q\in Q}{\min }\mathrm{cost}(S,\left\{q\right\})\le \underset{q\in Q}{\min }\mathrm{cost}(S,\left\{q\right\}).\hfill \end{array}$$

where the last inequality follows from the assumption $\epsilon <1$. □

Lemma 4.

Let S be the output of a call to k$\text{-}\mathrm{M}\mathrm{EAN}\text{-}\mathrm{C}\mathrm{ORESET}(P,k,\epsilon )$. Then S is a $(k,15\epsilon )$-coreset for P.

Proof. 

By replacing P with $P_i$ in Lemma 1 for each $i\in \left[m\right]$ it follows that

$$\mathrm{cost}(P_i,Q)\le \underset{q\in Q}{\min }\mathrm{cost}(P_i,Q)\le \mathrm{cost}(P_i,Q)·\frac{1+2\epsilon }{1-2\epsilon }+\frac{\mathrm{opt}(P_i,1)-\mathrm{opt}(P_i,k)}{(1-2\epsilon )\epsilon }.$$

Summing the last inequality over each $P_i$ yields

$$\mathrm{cost}(P,Q)\le \sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,Q)\le \mathrm{cost}(P,Q)·\frac{1+2\epsilon }{1-2\epsilon }+\frac{1}{(1-2\epsilon )\epsilon }\sum _{i=1}^m\left(\mathrm{opt}(P_i,1)-\mathrm{opt}(P_i,k)\right).$$

(16)

Since $\left\{P_1,\cdots ,P_m\right\}$ is the partition of the m-means of P we have ${\sum }_{i=1}^m\mathrm{opt}(P_i,1)=\mathrm{opt}(P,m)$. By letting $Q_i$ be the m-means of $P_i$ we have

$$\sum _{i=1}^m\mathrm{opt}(P_i,k)=\sum _{i=1}^m\mathrm{cost}(P_i,Q_i)\ge \sum _{i=1}^m\mathrm{cost}(P_i,{\cup }_{j=1}^m{Q}_j)=\mathrm{cost}(P,{\cup }_{j=1}^m{Q}_j)\ge \mathrm{opt}(P,mk).$$

Hence,

$$\sum _{i=1}^m\left(\mathrm{opt}(P_i,1)-\mathrm{opt}(P_i,k)\right)\le \mathrm{opt}(P,m)-\mathrm{opt}(P,mk)\le {\epsilon }^2\mathrm{opt}(P,k)\le {\epsilon }^2\mathrm{cost}(P,Q),$$

where the second inequality is by Line 1 of the algorithm. Plugging the last inequality in (16) yields

$$\mathrm{cost}(P,Q)\le \sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,Q)\le \mathrm{cost}(P,Q)·\frac{1+3\epsilon }{1-2\epsilon }.$$

(17)

Using Lemma 3, for every $i\in \left[m\right]$

$$(1-\epsilon )\underset{q\in Q}{\min }\mathrm{cost}(P_i,\left\{q\right\})\le \underset{q\in Q}{\min }\mathrm{cost}(S_i,\left\{q\right\})\le (1+\epsilon )\underset{q\in Q}{\min }\mathrm{cost}(P_i,\left\{q\right\})$$

By summing over $i\in \left[m\right]$ we obtain

$$(1-\epsilon )\sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,\left\{q\right\})\le \sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(S_i,\left\{q\right\})\le (1+\epsilon )\sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,\left\{q\right\}).$$

By this and Lemma 1

$$(1-\epsilon )\sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,\left\{q\right\})\le \mathrm{cost}(S,Q)\le (1+\epsilon )\sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,\left\{q\right\}).$$

Plugging the last inequality in (17) yields

$$\begin{array}{cc}\hfill (1-\epsilon )\mathrm{cost}(P,Q)& \le (1-\epsilon )\sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,Q)\hfill \\ \hfill & \le \mathrm{cost}(S,Q)\hfill \\ \hfill & \le (1+\epsilon )\sum _{i=1}^m\underset{q\in Q}{\min }\mathrm{cost}(P_i,\left\{q\right\})\le (1+\epsilon )\mathrm{cost}(P,Q)·\frac{1+3\epsilon }{1-2\epsilon }\le (1+15\epsilon )\mathrm{cost}(P,Q).\hfill \end{array}$$

(18)

Hence, S is a $(k,15\epsilon )$ coreset for P. □

Lemma 5.

There is an integer $t<1+1/{\epsilon }^2$ such that

$$\mathrm{opt}(P,k^t)-\mathrm{opt}(P,k^{t+1})\le {\epsilon }^2·\mathrm{opt}(P,k).$$

(19)

Proof. 

Contradictively assume that (19) does not hold for every integer $i<1+1/{\epsilon }^2$. Hence,

$$\begin{array}{c}\hfill \mathrm{opt}(P,k)-\mathrm{opt}(P,k^{\lceil 1/{\epsilon }^2\rceil +1})=\sum _{i=1}^{\lceil 1/{\epsilon }^2\rceil }\left(\mathrm{opt}(P,k^i)-\mathrm{opt}(P,k^{i+1})\right)>\lceil 1/{\epsilon }^2\rceil ·{\epsilon }^2\mathrm{opt}(P,k)\ge \mathrm{opt}(P,k).\end{array}$$

Contradiction, since $\mathrm{opt}(P,k^{\lceil 1/{\epsilon }^2\rceil +1})\ge 0$. □

Using the mean of $P_i$ in Line 5 of the algorithm yields a $(1,\epsilon )$-coreset $S_i$ as shown in Lemma 1. The resulting coreset is not sparse, but gives the following result.

Theorem 2.

There is $m\le k^{1/{\epsilon }^2}$ such that the m-means of P is a $(k,15\epsilon )$-coreset for P.

Proof of Theorem 1.

We compute $S_i$ a $(1,\epsilon )$ mean coreset for 1-mean of $P_i$ at line 5 of Algorithm 1 by using variation of Frank-Wolfe [22] algorithm. It follows that $|S_i|=O(1/{\epsilon }^2)$ for each i, therefore the overall sparsity of S is $s\left(P\right)/{\epsilon }^2$. This and Lemma 4 concludes the proof. □

7. Comparison to Existing Approaches

In this section we provide experimental results of our main algorithm of coreset constructions. We compare the clustering with existing coresets and small/medium/large datasets. Unlike most of the coreset papers, we also run the algorithm on the distributed setting via a cloud as explained below.

7.1. Datasets

For our experimental results we use three well known datasets, and the English Wikipedia as follows.

MNIST handwritten digits [25]. The MNIST dataset consists of $n=\mathrm{60,000}$ grayscale images of handwritten digits. Each image is of size 28 × 28 pixels and was converted to a row vector row of $d=784$ dimensions.

Pendigits [26]. This dataset was downloaded from the UCI repository. It consists of 250 written letters by 44 humans. These humans were asked to write 250 digits in a random order inside boxes of 500 by 500 tablet pixel resolution. The tablet sends x and y tablet coordinates and pressure level values of the pen at fixed time intervals (sampling rate) of 100 milliseconds. Digits are represented as constant length feature vectors of size $d=16$ the number of digits in the dataset is $n=\mathrm{10,992}$.

NIPS dataset [27]. The OCR scanning of NIPS proceedings over 13 years. It has 15,000 pages and 1958 articles. For each author, there is a corresponding words counter vector, where the ith entry in the vector is the number of the times that the word used in one of the author’s submissions. There are overall $n=2865$ authors and $d=\mathrm{14,036}$ words in this corpus.

English Wikipedia [28]. Unlike previous datasets that were uploaded to memory and then compressed via streaming coresets, the English Wikipedia practically can not be uploaded completely to memory. The size of the dataset is 15GB after converting to a term-documents matrix via gensim [29]. It has 4M vectors, each of ${10}^5$ dimensions and an average of 200 non-zero entries, i.e., words per document.

7.2. The Experiment

We applied our coreset construction to boost the performance of Lloyd’s k-means heuristic as explained in Section 8 of previous work [6]. We compared the results with the current data summarization algorithms that can handle sparse data: uniform and importance sampling.

7.3. On the Small/Medium Datasets

We evaluate both the offline computation and the streaming data model. For offline computation we used the datasets above to produce coresets of size $100\le m\le 1500$, then computed k-means for $k=10,15,20,25$ till convergence. For the streaming data model, we divided each dataset into small subsets and computed coresets via the merge-and-reduce technique to construct a coreset tree as in Figure 1. Here, the coresets are smaller, of size $10\le m\le 500$, and the values for k-means are the same.

We computed the sum of squared distances to the original (full) set of points, from each resulting set of k centers that was computed from the coreset. These sets of centers are denoted by $C_1,C_2$ and $C_3$ for uniform, non uniform sampling and our coreset respectively. The “ground truth” or “optimal solution” $C^k$ was computed using k-means on the entire dataset until convergence. The empirical estimated error $\epsilon$ was then defined to be $ϵ=C_t/C_k-1$ for coreset number $t=1,2,3$. Note that, since Lloyd’s k-means is a heuristic, its performance on the reduced data might be better, i.e., $\epsilon <0$.

These experiments were run on a single common laptop machine with 2.2GHz quad-core Intel Core i7 CPU and 16GB of 1600MHz DDR3L RAM with 256GB PCIe-based onboard SSD.

7.4. On the Wikipedia Dataset

We compared the three discussed data summarization techniques, while each one was computed in parallel and in a distributed fashion on 16 EC2 virtual servers. We repeated computation for $k=16,32,64$ and 128, and coresets size in the range $[32,1024]$.

This experiment was executed via Amazon’s Web Services (“cloud”), using 16 EC2 virtual computation nodes of type c4.4xlarge, which 8 vCPU and 15GiB of RAM. We repeated distributed computation evaluating for coresets of sizes 256, 512, 1024 and 2048 points for k-means with $k=16,32,64,128$.

7.5. Results

The results of experiment for $k=15,20,25$ on small datasets for offline computation are depicted on Figure 2, where it’s evident that error of kmeans computation fed by our coreset algorithm results outperforms error of uniform and non-uniform sampling.

For streaming computation model our algorithm is able to provide results which are better than other two as could be explored in Figure 3. In addition, existing algorithms suffer from “cold start” as common in random sampling techniques: there is a slow convergence to the small error, compared to our deterministic algorithm that introduces small error already after a small sample size.

At Figure 4 presented results of the experiment on Wikipedia dataset for different values of $k=32,64,128$, as it could be easily observed proposed coreset algorithm provides good results of big sparse dataset and provides lower energy cost compared to uniform and non-uniform approaches.

Figure 5, Figure 6 and Figure 7 show the box-plot of error distribution for all the three coresets in the offline and streaming settings. Our algorithm shows a little variance across all experiments, its mean error is very close to its median error, indicating that it produces stable results.

Figure 8 shows the memory (RAM) footprint during the coreset construction based on synthetically generated random data. The oscillations corresponds to the number of coresets in the tree that each new subset needs to update. For example, the first point in a streaming tree is updated in $O\left(1\right)$, however the $2^i$th point for some $i\ge 1$ climbs up through $O\left(\log i\right)$ levels in the tree, so $O\left(\log i\right)$ coresets are merged.

8. Conclusions

We proved that any set of points in ${\mathbb{R}}^d$ has a $(k,\epsilon )$-coreset which consists of a weighted subset of the input points whose size is independent of n and d, and polynomial in $1/\epsilon$. Our algorithm carefully selects m such that the m-means of the input with appropriate weights (clusters’ size) yields such a coreset.

This allows us to finally compute coreset for sparse high dimensional data, in both the streaming and the distributed setting. As a practical example, we computed the first coreset for the full English Wikipedia. We hope that our open source code will allow researchers in the industry and academia to run these coresets on more databases such as images, speech or tweets.

The reduction to k-means allows us to use popular k-means heuristics (such as Lloyd-Max) and provable constant factor approximations (such as k-means++) in practice. Our experimental results on both a single machine and on the cloud shows that our coreset construction significantly improves over existing techniques, especially for small coresets, due to its deterministic approach.

We hope that this paper will also help the community to answer the following three open problems:

(i)

Can we simply compute the m-means for a specific value $m\in k^{O(1/\epsilon )}$ and obtain a $(k,\epsilon )$-coreset without using our algorithm?

(ii)

Can we compute such a coreset (subset of the input) whose size is $m\in {(k/\epsilon )}^{O\left(1\right)}$?

(iii)

Can we compute such a smaller coreset deterministically?