k-Means+++: Outliers-Resistant Clustering

Abstract

The k-means problem is to compute a set of k centers (points) that minimizes the sum of squared distances to a given set of n points in a metric space. Arguably, the most common algorithm to solve it is k-means++ which is easy to implement and provides a provably small approximation error in time that is linear in n. We generalize k-means++ to support outliers in two sense (simultaneously): (i) nonmetric spaces, e.g., M-estimators, where the distance $\mathrm{dist}(p,x)$ between a point p and a center x is replaced by $min\left\{\mathrm{dist}(p,x),c\right\}$ for an appropriate constant c that may depend on the scale of the input. (ii) k-means clustering with $m\ge 1$ outliers, i.e., where the m farthest points from any given k centers are excluded from the total sum of distances. This is by using a simple reduction to the $(k+m)$-means clustering (with no outliers).

1. Introduction

We first introduce the notion of clustering, the solution that is suggested by k-means++, and the generalization of the problem to support outliers in the input. We then describe our contribution in this context.

1.1. Clustering

For a given similarity measure, clustering is the problem of partitioning an input set of n objects into subsets, such that objects in the same group are more similar to each other than to objects in the other sets. As mentioned in [1], clustering problems arise in many different applications, including data mining and knowledge discovery [2], data compression and vector quantization [3], and pattern recognition and classification [4]. However, for most of its variants, it is an NP-Hard problem when the number k of clusters is part of the input, as elaborated and proved in [5,6]. In fact, the exponential dependency in k is unavoidable even for approximating the (regular) Euclidean k-means for clustering n points in the plane [7,8].

Hence, constant or near-constant (logarithmic in n or k) multiplicative factor approximations to the desired cost function were suggested over the years, whose running time is polynomial in both n and k. Arguably, the most common version in both academy and industry is the k-means cost function, where the input is a set of n points in a metric space, and the goal is to compute the set of k centers (points) that minimizes the sum of squared distances over each input point to its closest center.

The k-means++ method that was suggested independently by [9,10] is an extremely common and simple solution with probable approximation bounds. It provides an $\alpha \in O(logk)$ multiplicative approximation in time $O\left(dnk\right)$, e.g., for the Euclidean d-dimensional space. It was also shown in [10] that the approximation factor is $\alpha =O\left(1\right)$, if the input data is well separated in some formal sense. Recently, Ref. [11] provided an $\alpha \in O\left(1\right)$ approximation for points in the d-dimensional Euclidean space via exactly k centers in time complexity of $O(dn{k}^{2}loglogk)$.

The k-means++ algorithm is based on the intuition that the centroids should be well spread out. Hence, it samples k centers iteratively, via a distribution that is called $D^2$-sampling and is proportional to the distance of each input point to the centers that were already chosen. The first center is chosen uniformly at random from the input.

The k-means++ algorithm supports any distance to the power of $z\ge 1$ as explained in [9]. Deterministic and other versions for the sum of (nonsquared) distances were suggested in [12,13,14].

A more general approximations, called bicriteria or $(\alpha ,\beta )$ approximations, guarantee multiplicative factor $\alpha$-approximation but the number of used center (for approximating the optimal k centers) is $\beta k$ for some $\beta >1$. The factors $\alpha$ and $\beta$ might be depended on k and n, and different methods give different dependencies and running times. For example, Ref. [15] showed that sampling $O(klogk)$ different randomized centers yields an $O\left(1\right)$-approximation and leverage it to support streaming data. Similarly, it was proved in [16] that $D^2$-sampling $O\left(k\right)$ centers yields an $O\left(1\right)$ approximation.

The analysis of [17] explores the value of $\alpha$ as a function of $\beta$.

A coreset for the k-median/mean problem is a small weighted set (usually subset) of the input points that approximates the sum of distances or sum of squared distances from the original (big) set P to every given set of k centers, usually up to $(1+\epsilon )$ multiplicative factor. In particular, we may compute an $\alpha$-approximation on the coreset to obtain $\alpha (1+\epsilon )$-approximation for the original data.

For the special case of k-means in the Euclidean space we can replace d by $O(k/{\epsilon }^2)$, including for coresets constructions, as explained in [18]. Deterministic coresets for k-means of size independent of d were suggested in [19].

1.2. Seeding

As explained in [10], Lloyd [20,21,22] suggested a simple iterative heuristic that aims to minimize the clustering cost, assuming a solution to the case $k=1$ is known. It is a special case of the Expected Maximization (EM) heuristic for computing a local minimum. The algorithm is initialized with k random points (seeds, centroids). At each iteration, each of the input points is classified to its closest centroid. A new set of k centroids is constructed by taking the mean (or solving the problem for $k=1$, in the general case) of each of the current k clusters. This method is repeated until convergence or any given stopping condition.

Due to its simplicity, and the convergence to a local minimum [23], this method is very common; see [3,24,25,26,27,28,29] and references therein. The method has further improved in [1,30,31,32].

The drawback of this approach is that it converges to a local minimum—the one which is closest to the initial centers that had been chosen and may be arbitrarily far from the global minimum. There is also no upper bound for the convergence rate and number of iterations. Therefore, a lot of research has been done to choose good initial points, called “seeds” [33,34,35,36,37,38,39]. However, very few analytical guarantees were found to prove convergence.

A natural solution is to use provable approximation algorithms such as k-means++ above, and then apply Lloyd’s algorithm as a heuristic that hopefully improves the approximation in practice. Since Lloyd’s algorithm can only improve the initial solution, the provable upper bound on the approximation factor is still preserved.

1.3. Clustering with Outliers

In practice, data sets include some noise measurements which do not reflect a real part of the data. These are called outliers, and even a single outlier may completely change the optimal solution that is obtained without this outlier. One option to handle outliers is to change the distance function to a function that is more robust to outliers, such as M-estimators, e.g., where the distance $\mathrm{dist}(p,x)$ between a pair of points is replaced by $min\left\{\mathrm{dist}(p,x),c\right\}$ for some fixed $c>0$ that may depend on the scaling or spread of the data. Another option is to compute the set of k centers that minimizes the objective function, excluding the farthest m points from the candidate k centers. Here, $m\ge 1$ is a given parameter for the number of outliers. Of course, given the optimal k centers for this problem, the m outliers are simply the farthest m input points, and given these m outliers the optimal solution is the k-means for the rest of the points. However, the main challenge is to approximate the global optimum, i.e., compute the optimal centers and outliers simultaneously. Guessing m is less challenging since it is an integer between 1 and n, and the cost function is monotonic in m, which enables the usage of binary search.

As explained in [40], detecting the outliers themselves is also an NP-hard problem [41]. Intensive research has been done on this problem, as explained in [42], since the problem of outliers is known in numerous applications [43,44]). In the context of data mining, Ref. [45] proposed a definition of distance-based outlier, which is free of any distributional assumptions and it can be generalized to multidimensional datasets. Following [45], further variations have been proposed [46,47]. Consequently,  Ref. [48] introduced paradigm of local outlier factor (LOF). This paradigm has been extended in [43,49] in different directions.

As explained in [50], and following the discussion in [51,52] provided an algorithm based on Lagrange-relaxation technique. Several algorithms [51,53,54] were also developed. The work of [55] gives a factor of $O\left(1\right)$ and a running time of $O\left(n^m\right)$ for the problem of k-means with m outliers. Another heuristic was developed by [56]. Finally, [50] provided an $O\left(1\right)$-approximation in $O(k^2{(k+m)}^2{n}^{3}logn)$ time. In the context of k-means, Ref. [57] provided several algorithms of such constant factor approximation. However, the number of the points which approximate the outliers is much greater than m, and is dependent on the data, as well as the algorithm running time.

2. Our Contribution

A natural open question is: “can we generalize the k-means++ algorithm to handle outliers”?

This paper answers this question affirmably in three senses that may be combined together:

(i)

A small modification of k-means++ that supports M-estimators for handling outliers, with similar provable guarantees for both the running time and approximation factor. This family of functions includes most of the M-estimators, including nonconvex functions such as $M\left(x\right)=min\left\{x,c\right\}$ for some constant c, where $x=\mathrm{dist}(p,c)$ is the distance between an input point and its closest center c. In fact, our version in Algorithm 1 supports any pseudodistance function or $\rho$-metric that approximates the triangle inequality, as formalized in Definition 1 and Theorem 1.

(ii)

A generalization of this solution to the case of k-mean/median problem with m outliers that takes time $O\left(n\right)$ for constants k and m. To our knowledge, this is the first nontrivial approximation algorithm that takes time linear or even near-linear in n. The algorithm supports all the pseudodistance functions and $\rho$-metric spaces, including the above M-estimators. See Corollary 4.

(iii)

A Weak coreset of size $O(k+m)$ and a larger strong coreset for approximating the sum of distances to any k-centers ignoring their farthest m input points. For details and exact definition see next subsection and Theorem 2.

Novel reduction: from k-means to $(k+m)$-means. While the first result is a natural generalization of the original k-means++, the second result uses a simple but powerful general reduction from k-means with m outliers to $(k+m)$-means (without outliers), that we did not find in previous papers. More precisely, in Section 5, we prove that an approximation to the $(k+m)$-median with appropriate positive weight for each center (the size of its cluster), can be used to approximate the sum of distances from P to any k centers, excluding their farthest m points in P; see Corollary 3. This type of reductions may be considered as “coreset” and we suggest two types of them.

In the first coreset, the approximation error is additive, although the final result is a multiplicative constant factor. Nevertheless, the size of the suggested coreset is only $O(k+m)$, i.e., independent of both n and d, for constant $\rho$-metric as the Euclidean space; see Theorem 2 for details. In particular, applying exhaustive search (in time that is exponential in k) on this coreset implies an $O(logk+m)$-factor approximation for the k-median with m outliers, for any $(P,\rho )$-metric in $O\left(n\right)$ time when the parameters m and k are constants; see Corollary 4.

As stated in the previous section, the exponential dependency in k is unavoidable even for the (regular) k-means in the plane [7,8]. Nevertheless, constant factor approximations that take time that is polynomial in both k and m may be doable by applying more involved approximation algorithms for the k-median with m outliers on our small “coreset” which contains only $O(k+m)$ points. e.g., the polynomial time algorithm of Ke Chen [50].

Theorem 2 also suggests a second “traditional coreset” that yields $(1+\epsilon )$-approximation for the k-median with m outliers, i.e., that obtains $(1+\epsilon )$-approximation for the sum of distances from any set of k centers and their farthest m outliers in the original set P. The price is that we need $\alpha <\epsilon$ approximation to the $(k+m)$-median. As was proved in [58], this is doable in the Euclidean d-dimensional space by running k-means++ for ${(1/\epsilon )}^{O\left(d\right)}(k+m)logn$ iterations instead of only $O(k+m)$ iterations (number of resulting centers). It was also proved in [58] that the exponential dependency on d is unavoidable in the worst case. See Section 5 for details.

For the special case of k-means in the Euclidean space we can replace d by $O(k/{\epsilon }^2)$, including for the coresets constructions above, as explained in [18]. Deterministic version of our coresets for k-median with m outliers may be obtained via [19].

3. Triangular Calculus

The algorithms in this paper support a large family of distance and nondistance functions. To exactly define this family, and their dependency on both the approximation factors and running times of the algorithms, we need to introduce the notion of $\rho$-metric that generalizes the definition $(P,\mathrm{dist})$ of a metric space. For a point $p\in P$ and a set $X\subseteq P$, we define $f(p,X)={min}_{x\in X}f(p,x)$. The minimum or sum over an empty set is defined to be zero.

Definition 1

($\rho$-metric). Let $\rho \ge 1$, P be a finite set and $f:P^2\to [0,\infty )$ be a symmetric function such that $f(p,p)=0$ for every $p\in P$. The pair $(P,f)$ is a ρ-metric if for every $x,y,z\in P$ the following “approximated” triangle inequality holds:

$$f(x,z)\le \rho \left(f(x,y)+f(y,z)\right).$$

(1)

For example, metric space $(P,\mathrm{dist})$ is a $\rho$-metric for $\rho =1$. If we define $f(x,y)={\left(\mathrm{dist}(x,y)\right)}^2$, as in the k-means case, it is easy to prove (1) for $\rho =2$.

The approximated triangle inequality also holds for sets as follows.

Lemma 1.

(See 6.1 of [59]) Let $(P,f)$ be a ρ-metric. Then, for every pair of points $q,x\in P$ and a subset $X\subseteq P$ we have

$$\begin{array}{c}\hfill f(x,X)\le \rho \left(f\right(q,x)+f(q,X\left)\right).\end{array}$$

(2)

Proof. 

For every $p,q\in P$, and a center $x_q\in X$ that is closest to q, i.e., $f(q,X)=f(q,x_q)$, we have

$$\begin{array}{c}\hfill f(x,X)=\underset{p\in X}{min}f(x,p)\le f(x,x_q)\le \rho \left(f\right(q,x)+f(q,x_q\left)\right)=\rho \left(f\right(q,x)+f(q,X)).\end{array}$$

 □

Our generalization of the k-means++ algorithm for other distance functions needs only the above definition of $\rho$-metric. However, to improve the approximation bounds for the case of k-means with m outliers in Section 5, we introduce the following variant of the triangle inequality.

Definition 2

($(\rho ,\varphi ,\epsilon )$ metric). Let $(P,f)$ be a ρ-metric. For $\varphi ,\epsilon >0$, the pair $(P,f)$ is a $(\rho ,\varphi ,\epsilon )$-metric if for every $x,y,z\in P$ we have

$$f(x,z)-f(y,z)\le \varphi f(x,y)+\epsilon f(x,z).$$

(3)

For example, for a metric space $(P,\mathrm{dist}$) the inequality holds by the triangle inequality for $\varphi =1$ and every $\epsilon \ge 0$. For squared distance, we have $\varphi =O(1/\epsilon )$ for every $\epsilon >0$; see [18].

The generalization for sets is a bit more involved as follows.

Lemma 2.

Let$(P,f)$be a$(\rho ,\varphi ,\epsilon )$-metric. For every set$Z\subseteq P$we have

$$\mid f(x,Z)-f(y,Z)\mid \le (\varphi +\epsilon \rho )f(x,y)+\epsilon \rho \min \left\{f(x,Z),f(y,Z)\right\}.$$

Proof. 

Let $z_x,z_y\in Z$ such that $f(x,Z)=f(x,z_x)$ and $f(y,Z)=f(y,z_y)$. Without loss of generality, we assume $f(y,Z)\le f(x,Z)$. We have

$$\begin{array}{cc}\hfill \mid f(x,Z)-f(y,Z)\mid =f(x,Z)-f(y,Z)& =f(x,z_x)-f(y,z_y)\le f(x,z_y)-f(y,z_y)\hfill \\ \hfill & \le \varphi f(x,y)+\epsilon f(x,z_y),\hfill \end{array}$$

(4)

where the last inequality is by (3).

The last term is bounded by $f(x,z_y)\le f(y,z_y)+\varphi f(x,y)+\epsilon f(x,z_y)$ via Definition 2, but this bound is useless for the case $\epsilon >1$. Instead, we use (1) to obtain

$$f(x,z_y)\le \rho \left(f(x,y)+f(y,z_y)\right)=\rho f(x,y)+\rho f(y,Z).$$

Plugging the last inequality in (4) proves the case $f(y,Z)\le f(x,Z)$ as

$$\begin{array}{cc}\hfill \mid f(x,Z)-f(y,Z)\mid & \le \varphi f(x,y)+\epsilon \left(\rho f\right(x,y)+\rho f(y,Z\left)\right)\hfill \\ & =(\varphi +\epsilon \rho )f(x,y)+\epsilon \rho f(y,Z)=(\varphi +\epsilon \rho )f(x,y)+\epsilon \rho min\left\{f(y,Z),f(x,Z)\right\}.\hfill \end{array}$$

 □

Any $\rho$-metric is also a $(\rho ,\varphi ,\epsilon )$-metric for some other related constants as follows.

Lemma 3.

Let $(P,f)$ be a ρ-metric. Then $(P,f)$ is a $(\rho ,\varphi ,\epsilon )$-metric, where $\varphi =\rho$ and $\epsilon =\rho -1$.

Proof. 

Let $x,y,z\in P$. We need to prove that

$$f(x,z)-f(y,z)\le \varphi f(x,y)+\epsilon f(x,z).$$

Without loss of generality, $f(x,z)>f(y,z)$, otherwise the claim is trivial. We then have

$$f(x,z)-f(y,z)\le \rho \left(f(x,y)+f(y,z)\right)-f(y,z)=\rho f(x,y)+(\rho -1)f(y,z)\le \rho f(x,y)+(\rho -1)f(x,z),$$

where the first inequality is by the approximated triangle inequality in (1), and the second inequality is by the assumption $f(x,z)>f(y,z)$.  □

How can we prove that a given function $f:P^2\to [0,\infty )$ satisfies the condition of $\rho$-metric or $(\rho ,\varphi ,\epsilon )$-metric? If f is some function of a metric distance, say, $f\left(x\right)={\mathrm{dist}}^r\left(x\right)$ or most M-estimators functions such as $f\left(x\right)=min\left\{\mathrm{dist}\left(x\right),1\right\}$, this may be easy via the following lemma.

Lemma 4

(Log-Log Lipschitz condition). Let $g:[0,\infty )\to [0,\infty )$ be a monotonic nondecreasing function that satisfies the following (Log-Log Lipschitz) condition: there is $r>0$ such that for every $x>0$ and $\Delta >1$ we have

$$g(\Delta x)\le {\Delta }^{r}g\left(x\right).$$

(5)

Let $(P,\mathrm{dist})$ be a metric space, and $f:P^2\to [0,\infty )$ be a mapping from every $p,c\in P$ to $f(p,c)=g\left(\mathrm{dist}\right(p,c\left)\right)$. Then $(P,f)$ is a $(\rho ,\varphi ,\epsilon )$-metric where

(i) 

$\rho =\max \left\{2^{r-1},1\right\}$,

(ii) 

$\varphi ={\left(\frac{r-1}{\epsilon }\right)}^{r-1}$and$\epsilon \in (0,r-1)$, if $r>1$, and

(iii) 

$\varphi =1$and$\epsilon =0$, if $r\le 1$.

Proof. 

We denote $x=\mathrm{dist}(p,c)$, $y=\mathrm{dist}(q,c)$ and $z=\mathrm{dist}(p,q)$.

(i) We need to prove that $g\left(z\right)\le \rho \left(g\right(x)+g(y\left)\right)$. If $y=0$ then $q=c$, so

$$g\left(z\right)=g\left(\mathrm{dist}\right(p,q\left)\right)=g\left(\mathrm{dist}\right(p,c\left)\right)=g\left(x\right),$$

(6)

and Claim (i) holds for any $\rho \ge 1$. Similarly, Claim (i) holds for $x=0$ by symmetry. So we assume $x,y>0$. For every $b\in (0,1)$ we have

$$\begin{array}{ll}g\left(z\right)& =g\left(\mathrm{dist}\right(p,q\left)\right)\\ & \le g\left(\mathrm{dist}\right(p,c)+\mathrm{dist}(c,q\left)\right)=g(x+y)=bg(x+y)+(1-b)g(x+y)\\ \hfill & =bg\left(x·\frac{x+y}{x}\right)+(1-b)g\left(y·\frac{x+y}{y}\right)\end{array}$$

(7)

$$\begin{array}{ll}& \le bg\left(x\right){\left(\frac{x+y}{x}\right)}^r+(1-b)g\left(y\right){\left(\frac{x+y}{y}\right)}^r\\ & ={(x+y)}^r\left(\frac{bg\left(x\right)}{x^r}+\frac{(1-b)g\left(y\right)}{y^r}\right),\end{array}$$

(8)

where (7) is by the triangle inequality, and (8) is by substituting x and y respectively in (5). Substituting $b=x^r/(x^r+y^r)$ yields

$$g\left(z\right)\le (g\left(x\right)+g\left(y\right))\frac{{(x+y)}^r}{x^r+y^r}.$$

(9)

We now compute the maximum of the factor $h\left(x\right)=\frac{{(x+y)}^r}{x^r+y^r}$, whose derivative is zero when

$$r{(x+y)}^{r-1}(x^r+y^r)-{(x+y)}^r·r{x}^{r-1}=0,$$

i.e., when $x=y$. In this case $h\left(x\right)=2^{r-1}$. The other extremal values of h are $x=0$ and $x=\infty$ where $h\left(x\right)=1$. Hence, ${max}_{x\ge 0}h\left(x\right)=2^{r-1}$ for $r\ge 1$ and ${max}_{x\ge 0}h\left(x\right)=1$ for $r\in (0,1)$. Plugging these bounds in (9) yields Claim (i)

$$g\left(z\right)\le max\left\{2^{r-1},1\right\}(g\left(x\right)+g\left(y\right)).$$

(ii)–(iii) We prove that $g\left(x\right)-g\left(y\right)\le \varphi g\left(z\right)+\epsilon g\left(x\right)$. If $y>x$ then $g\left(x\right)-g\left(y\right)\le 0\le \varphi g\left(z\right)+\epsilon g\left(y\right)$.

If $x\le y$, we need to prove that $g\left(x\right)-g\left(y\right)\le \varphi g\left(x\right)+\epsilon g\left(x\right)$. If $y=0$ then $g\left(z\right)=g\left(x\right)$ by (6), and thus for every $\varphi \ge 1$ and $\epsilon >0$ we have

$$g\left(x\right)-g\left(y\right)=g\left(x\right)=g\left(z\right)\le \varphi g\left(z\right)+\epsilon g\left(y\right)$$

as desired. We thus need to prove the last inequality for $y>0$.

We assume

$$g\left(x\right)>\varphi g\left(z\right),$$

(10)

and $x>y$ (so $p\ne q$ and thus $z>0$), otherwise the claim trivially holds. Let $q=max\left\{r,1\right\}$. Hence,

$$g\left(x\right)=g(y·x/y)\le g\left(y\right)·{(x/y)}^r\le g\left(y\right)·{(x/y)}^q$$

where the first inequality is by (5) and the second is by the definition of q and the assumption $x>y\ge 0$. Rearranging gives $g\left(y\right)\ge g\left(x\right)·{(y/x)}^q$, so

$$g\left(x\right)-g\left(y\right)\le g\left(x\right)·(1-{(y/x)}^q).$$

(11)

We now first prove that

$$\begin{array}{cc}\hfill g\left(x\right)·(1-{(y/x)}^q)& \le \epsilon g\left(x\right)+\frac{\varphi }{q^q}·g\left(x\right){(1-{(y/x)}^q)}^q.\hfill \end{array}$$

(12)

Indeed, if $q=1$ then $r\le 1$, $\epsilon =0$, $\varphi =1$ and (12) trivially holds with equality. Otherwise $(q>1$), we let $p=\frac{q}{q-1}$ so that

$$\begin{array}{cc}\hfill g\left(x\right)·(1-{(y/x)}^q)& ={\left(p\epsilon g\left(x\right)\right)}^{1/p}·\left(\frac{{\left(g\left(x\right)\right)}^{1/q}(1-{(y/x)}^q)}{{\left(p\epsilon \right)}^{1/p}}\right)\hfill \\ \hfill & \le \epsilon g\left(x\right)+\frac{g\left(x\right){(1-{(y/x)}^q)}^q}{q{\left(p\epsilon \right)}^{q/p}}\hfill \end{array}$$

(13)

where the inequality is by Young’s inequality $ab\le \frac{a^p}{p}+\frac{b^q}{q}$ for every $a,b\ge 0$ and $p,q>0$ such that $1/p+1/q=1$. We then obtain (12) by substituting $\varphi ={(q-1)}^{q-1}/{\epsilon }^{q-1}$ so that

$$\frac{\varphi }{q^q}=\frac{{(q-1)}^{q-1}}{q{\left(q\epsilon \right)}^{q-1}}=\frac{1}{q{\left(p\epsilon \right)}^{q/p}}.$$

Next, we bound the rightmost expression of (12). We have $1-w^q\le q(1-w)$ for every $w\ge 0$, since the linear function $q(1-w)$ over $w\ge 0$ is tangent to the concave function $1-w^q$ at the point $w=1$, which is their only intersection point. By substituting $w=y/x$ we obtain

$$1-{(y/x)}^q\le q(1-y/x).$$

(14)

By the triangle inequality,

$$1-y/x=\frac{x-y}{x}\le \frac{z}{x}.$$

(15)

Combining (14) and (15) yields

$${(1-{(y/x)}^q)}^q\le {\left(q(1-y/x)\right)}^q\le q^q{(z/x)}^q.$$

(16)

Since $g\left(x\right)\ge \varphi g\left(z\right)\ge g\left(z\right)$ by (10) and the definition of $\varphi$, we have $x\ge z$, i.e., $(x/z)\ge 1$ by the monotonicity of g. Hence, $g\left(x\right)=g(z·x/z)\le g\left(z\right)·{(x/z)}^r\le g\left(z\right)·{(x/z)}^q$ by (5), so

$${(z/x)}^q\le g\left(z\right)/g\left(x\right).$$

(17)

By combining the last inequalities we obtain the desired result

$$\begin{array}{cc}\hfill g\left(x\right)-g\left(y\right)& \le g\left(x\right)·(1-{(y/x)}^q)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & \le \epsilon g\left(x\right)+\frac{\varphi g\left(x\right){(1-{(y/x)}^q)}^q}{q^q}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & \le \epsilon g\left(x\right)+\varphi g\left(x\right){(z/x)}^q\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & \le \epsilon g\left(x\right)+\varphi g\left(z\right),\hfill \end{array}$$

(21)

where (18) holds by (11), (19) is by (12), (20) holds by (16), and (21) is by (17).  □

4. k-Means++ for ρ-Metric

In this section we suggest a generalization of the k-means++ algorithm for every $\rho$-metric and not only distance to the power of $z\ge 1$ as claimed in the original version. In particular, we consider nonconvex M-estimator functions. Our goal is then to compute an $\alpha$-approximation to the k-median problem for $\alpha =O(logk)$.

Let $w:P\to (0,\infty )$ be called the weight function over P. For a nonempty set $Q\subseteq P$ we define

$$f(Q,X)=f_w(Q,X)=\sum _{p\in Q}w\left(p\right)f(p,X).$$

If Q is empty then $f(Q,X)=0$. For brevity, we denote $f(Q,p)=f(Q,\left\{p\right\})$, and $f^2(·)={\left(f(·)\right)}^2$. For an integer $k\ge 1$ we denote $\left[k\right]=\left\{1,\cdots ,k\right\}$.

Definition 3

(k-median for $\rho$-metric spaces). Let $(P,f)$ be a ρ-metric, and $k\in \left[n\right]$ be an integer. A set $X^{*}\subseteq P$ is a k-median of P if

$$f(P,X^{*})=\underset{X\subseteq P,\mid X\mid =k}{min}f(P,X),$$

and this optimum is denoted by $f^{*}(P,k):=f(P,X^{*})$. For $\alpha \ge 0$, a set $Y\subseteq P$ is an $\alpha$-approximation for the k-median of P if

$$f(P,X^{*})\le \alpha f^{*}(P,k).$$

Note that Y might have less or more than k centers.

The following lemma implies that sampling an input point, based on the distribution of w, gives a 2-approximation to the 1-mean.

Lemma 5.

For every nonempty set $Q\subseteq P$,

$$\sum _{x\in Q}w\left(x\right)f_w(Q,x)\le 2\rho f_w^{*}(Q,1)\sum _{x\in Q}w\left(x\right).$$

(22)

Proof. 

Let $p^{*}$ be the weighted median of Q, i.e., $f_w(Q,p^{*})=f_w^{*}(Q,1)$. By (1), for every $q,x\in Q$,

$$f(q,x)\le \rho \left(f(q,p^{*})+f(p^{*},x)\right).$$

Summing over every weighted $q\in Q$ yields

$$f_w(Q,x)\le \rho (f_w(Q,p^{*})+f(p^{*},x)\sum _{q\in Q}w\left(q\right))=\rho (f_w^{*}(Q,1)+f(x,p^{*})\sum _{q\in Q}w\left(q\right)).$$

Summing again over the weighted points of Q yields

$$\sum _{x\in Q}w\left(x\right)f_w(Q,x)\le \rho \sum _{x\in Q}w\left(x\right)(f_w^{*}(Q,1)+f(x,p^{*})\sum _{q\in Q}w\left(q\right))=2\rho f_w^{*}(Q,1)\sum _{x\in Q}w\left(x\right).$$

 □

We also need the following approximated triangle inequality for weighted sets.

Corollary 1

(Approximated triangle inequality). Let $x\in P$, and $Q,X\subseteq P$ be nonempty sets. Then

$$f(x,X)\sum _{q\in Q}w\left(q\right)\le \rho (f_w(Q,x)+f_w(Q,X)).$$

Proof. 

Summing (2) over every weighted $q\in Q$ yields

$$\begin{array}{cc}\hfill f(x,X)\sum _{q\in Q}w\left(q\right)& \le \rho (\sum _{q\in Q}w\left(q\right)f(q,x)+\sum _{q\in Q}w\left(q\right)f(q,X))\hfill \\ & =\rho \left(f_w\right(Q,x)+f_w(Q,X\left)\right).\hfill \end{array}$$

 □

The following lemma states that if we use sampling from P and hit a point in an uncovered cluster Q, then it is a 2-approximation for $f^{*}(P,1)$.

Lemma 6.

Let $Q,X\subseteq P$ such that $f_w(Q,X)>0$. Then

$$\frac{1}{f_w(Q,X)}\sum _{x\in Q}w\left(x\right)f(x,X)·f_w(Q,X\cup \left\{x\right\})\sum _{q\in Q}w\left(q\right)\le 2\rho \sum _{x\in Q}w\left(x\right)f_w(Q,x).$$

Proof. 

By Corollary 1, for every $x\in Q$,

$$f(x,X)\sum _{q\in Q}w\left(q\right)\le \rho (f_w(Q,x)+f_w(Q,X)).$$

Multiplying this by $\frac{f_w(Q,X\cup \left\{x\right\})}{f_w(Q,X)}$ yields

$$\begin{array}{cc}\hfill f(x,X)·\frac{f_w(Q,X\cup \left\{x\right\})}{f_w(Q,X)}\sum _{q\in Q}w\left(q\right)& \le \rho (f_w(Q,x)·\frac{f_w(Q,X\cup \left\{x\right\})}{f_w(Q,X)}+f_w(Q,X\cup \left\{x\right\}))\hfill \\ & \le \rho \left(f_w(Q,x)+f_w(Q,X\cup \left\{x\right\})\right)\le 2\rho f_w(Q,x).\hfill \end{array}$$

After summing over every weighted point $x\in Q$ we obtain

$$\sum _{x\in Q}w\left(x\right)f(x,X)·\frac{f_w(Q,X\cup \left\{x\right\})}{f_w(Q,X)}\sum _{q\in Q}w\left(q\right)\le 2\rho \sum _{x\in Q}w\left(x\right)f_w(Q,x).$$

 □

Finally, this lemma is the generalization of the original lemma of the k-means++ versions.

Lemma 7.

Let $t,u\ge 0$ be a pair of integers. If $u\in \left[k\right]$ and $t\in \left\{0,\cdots ,u\right\}$ then the following holds.

Let $\left\{P_1,\cdots ,P_k\right\}$ be a partition of P such that ${\sum }_{i=1}^{k}f*(P_i,1)=f*(P,k)$. Let $U={\bigcup }_{i=1}^u{P}_i$ denote the union of the first u sets. Let $X\subseteq P$ be a set that covers $P\backslash U$, i.e., $X\cap P_i\ne \varnothing$ for every integer $i\in \left[k\right]\backslash \left[u\right]$, and $X\cap U=\varnothing$. Let Y be the output of a call to KMeans++$(P,w,f,X,t)$; see Algorithm 1. Then

$$E\left[f\right(P,Y)]\le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_t+\frac{u-t}{u}·f(U,X),$$

(23)

where $H_t=\left\{\begin{array}{cc}{\sum }_{i=1}^t\frac{1}{i}\hfill & ift\ge 1\hfill \\ 1\hfill & ift=0\hfill \end{array}\right.$, and the randomness is over the t sampled points in $Y\backslash X$.

Algorithm 1: KMeans++($P,w,f,X,t$); see Theorem 1.

Proof. 

The proof is by the following induction on $t\ge 0$: (i) the base case $t=0$ (and any $u\ge 0$), and (ii) the inductive step $t\ge 1$ (and any $u\ge 0$).

(i) Base Case: $t=0$. We then have

$$\begin{array}{cc}\hfill E\left[f\right(P,Y\left)\right]& =E\left[f\right(P,X)]=f(P,X)=f(P\backslash U,X)+f(U,X)=f(P\backslash U,X)+\frac{u-t}{u}·f(U,X)\hfill \\ & \le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_t+\frac{u-t}{u}·f(U,X),\hfill \end{array}$$

where the first two equalities hold since $Y=X$ is not random, and the inequality holds since $t=0$, $H_0=1$ and $f*(U,u)\ge 0$. Hence, the lemma holds for $t=0$ and any $u\ge 0$.

(ii) Inductive step: $t\ge 1$. Let $y\in P$ denote the first sampled point that was inserted to $Y\backslash X$ during the execution of Line 5 in Algorithm 1. Let $X^{\prime }=X\cup \left\{y\right\}$. Let $j\in \left[k\right]$ such that $y\in P_j$, and $U^{\prime }=U\backslash P_j$ denote the remaining “uncovered” $u^{\prime }=\mid \left\{P_1,\cdots ,P_u\right\}\backslash P_j\mid$ clusters, i.e, $u^{\prime }\in \left\{u,u-1\right\}$. The distribution of Y conditioned on the known sample y is the same as the output of a call to KMeans++$(P,w,f,X^{\prime },t^{\prime })$ where $t^{\prime }=t-1$. Hence, we need to bound

$$\begin{array}{cc}\hfill E\left[f\right(P,Y\left)\right]& =\mathrm{Pr}(y\in P\backslash U)E\left[f\right(P,Y)\mid y\in P\backslash U]+\mathrm{Pr}(y\in U)E\left[f\right(P,Y)\mid y\in U].\hfill \end{array}$$

(24)

We will bound each of the last two terms by expressions that are independent of $X^{\prime }$ or $U^{\prime }$.

Bound on $E\left[f\right(P,Y)\mid y\in P\backslash U]$. Here $u^{\prime }=u\in \left[k\right]$, $U^{\prime }=U$ (independent of j), and by the inductive assumption that the lemma holds after replacing t with $t^{\prime }=t-1$, we obtain

$$\begin{array}{cc}\hfill E\left[f\right(P,Y)\mid y\in P\backslash U]& \le \left(f(P\backslash U^{\prime },X^{\prime })+4{\rho }^{2}f*(U^{\prime },u^{\prime })\right)H_{t^{\prime }}+\frac{u^{\prime }-t^{\prime }}{u^{\prime }}·f(U^{\prime },X^{\prime })\hfill \\ \hfill & =\left(f(P\backslash U,X^{\prime })+4{\rho }^{2}f*(U,u)\right)H_{t-1}+\frac{u-t+1}{u}·f(U,X^{\prime })\hfill \\ \hfill & \le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_{t-1}+\frac{u-t+1}{u}·f(U,X).\hfill \end{array}$$

(25)

Bound on $E\left[f\right(P,Y)\mid y\in U]$. In this case, $U^{\prime }=U\backslash P_j$ and $u^{\prime }=u-1\in \left\{0,\cdots ,k-1\right\}$. Hence,

$$\begin{array}{cc}\hfill E\left[f\right(P,Y)\mid y\in U]& =\sum _{m=1}^u\mathrm{Pr}(j=m)E\left[f\right(P,Y)\mid j=m]\hfill \\ \hfill & =\sum _{m=1}^u\mathrm{Pr}(j=m)\sum _{x\in P_m}\mathrm{Pr}(y=x)E\left[f\right(P,Y)\mid x=y].\hfill \end{array}$$

(26)

Put $m\in \left[u\right]$ and $x\in P_m$. We remove the above dependency of $E\left[f\right(P,Y)\mid y\in U]$ upon x and then m.

We have

$$\begin{array}{ll}E\left[f\right(P,Y)\mid y=x]\hfill & \le \left(f(P\backslash U^{\prime },X^{\prime })+4{\rho }^{2}f*(U^{\prime },u^{\prime })\right)H_{t-1}+\frac{u^{\prime }-t^{\prime }}{\max \left\{u^{\prime },1\right\}}·f(U^{\prime },X^{\prime })\hfill \\ & =\left(f(P\backslash U,X^{\prime })+f(P_m,X^{\prime })+4{\rho }^{2}f*(U,u)-4{\rho }^{2}f*(P_m,1)\right)H_{t-1}+\frac{u-t}{\max \left\{u^{\prime },1\right\}}·f(U^{\prime },X^{\prime })\hfill \\ & \le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_{t-1}+\frac{u-t}{\max \left\{u^{\prime },1\right\}}·f(U\backslash P_m,X)\hfill \\ & +\left(f(P_m,X^{\prime })-4{\rho }^{2}f*(P_m,1)\right)H_{t-1},\hfill \end{array}$$

(27)

where the first inequality holds by the inductive assumption if $u^{\prime }\ge 1$, or since $U^{\prime }=U\backslash P_j=\varnothing$ if $u^{\prime }=0$. The second inequality holds since $X\subseteq X^{\prime }=X\cup \left\{x\right\}$, and since $f*(U\backslash P_m,u-1)=f*(U,u)-f*(P_m,1)$.

Only the term $f(P_m,X^{\prime })=f(P_m,X\cup \left\{x\right\})$ depends on x and not only on m. Summing it over every possible $x\in P_m$ yields

$$\begin{array}{cc}\hfill \sum _{x\in P_m}\mathrm{Pr}(y=x)f_w(P_m,X^{\prime })& =\frac{1}{f_w(P_m,X)}\sum _{x\in P_m}w\left(x\right)f(x,X)f_w(P_m,X\cup \left\{x\right\})\hfill \\ & \le \frac{2\rho }{{\sum }_{q\in P_m}w\left(q\right)}\sum _{x\in P_m}w\left(x\right)f_w(P_m,x)\hfill \\ & \le \frac{2\rho }{{\sum }_{q\in P_m}w\left(q\right)}·2\rho f_w^{*}(P_m,1)\sum _{x\in P_m}w\left(x\right)\le 4{\rho }^{2}f*(P_m,1),\hfill \end{array}$$

where the inequalities follow by substituting $Q=P_m$ in Lemma 6 and Lemma 5, respectively. Hence, the expected value of (27) over x is nonpositive as

$$\sum _{x\in P_m}\mathrm{Pr}(y=x)\left(f(P_m,X^{\prime })-4{\rho }^{2}f*(P_m,1)\right)=-4{\rho }^{2}f*(P_m,1)+\sum _{x\in P_m}\mathrm{Pr}(y=x)f_w(P_m,X^{\prime })\le 0.$$

Combining this with (26) and then (27) yields a bound on $E\left[f\right(P,Y)\mid y=x]$ that is independent upon x,

$$\begin{array}{cc}\hfill E\left[f\right(P,Y)\mid y\in U]& =\sum _{m=1}^u\mathrm{Pr}(j=m)\sum _{x\in P_m}\mathrm{Pr}(y=x)E\left[f\right(P,Y)\mid y=x]\hfill \\ & \le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_{t-1}+\frac{u-t}{max\left\{u^{\prime },1\right\}}\sum _{m=1}^u\mathrm{Pr}(j=m)f(U\backslash P_m,X),\hfill \end{array}$$

(28)

It is left to remove the dependency on m, which occurs in the last term $f(U\backslash P_m,X)$ of (28). We have

$$\begin{array}{cc}\hfill \sum _{m=1}^u\mathrm{Pr}(j=m)f(U\backslash P_m,X)& =\sum _{m=1}^u\frac{f(P_m,X)}{f(P,X)}\left(f(U,X)-f(P_m,X)\right)\hfill \\ \hfill & =\frac{1}{f(P,X)}\left(f^2(U,X)-\sum _{m=1}^u{f}^2(P_m,X)\right).\hfill \end{array}$$

(29)

By Jensen’s (or power-mean) inequality, for every convex function $g:\mathbb{R}\to \mathbb{R}$, and a real vector $v=(v_1,\cdots ,v_u)$ we have $(1/u){\sum }_{m=1}^{u}g\left(v_m\right)\ge g\left((1/u){\sum }_{m=1}^u{v}_m\right)$. Specifically, for $g\left(z\right):=z^2$ and $v=\left(f(P_1,X),\cdots ,f(P_u,X)\right)$,

$$\sum _{m=1}^u\frac{1}{u}{f}^2(P_m,X)\ge {\left(\frac{1}{u}\sum _{m=1}^{u}f(P_m,X)\right)}^2.$$

Multiplying by u yields

$$\sum _{m=1}^u{f}^2(P_m,X)\ge \frac{f^2(U,X)}{u}.$$

Plugging this in (29) gives the desired bound on the term $f(U^{\prime },X)$,

$$\sum _{m=1}^u\mathrm{Pr}(j=m)f(U\backslash P_m,X)\le \left(1-\frac{1}{u}\right)\frac{f^2(U,X)}{f(P,X)}=\frac{u^{\prime }}{u}\frac{f^2(U,X)}{f(P,X)}\le \frac{max\left\{u^{\prime },1\right\}}{u}f(U,X),$$

where the last inequality holds since $f(U,X)\le f(P,X)$. Plugging the last inequality in (28) bounds $E\left[f\right(P,Y)\mid y\in U]$ by

$$E\left[f\right(P,Y)\mid y\in U]\le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_{t-1}+\frac{u-t}{u}f(U,X).$$

(30)

Bound on $E\left[f\right(P,Y\left)\right]$. Plugging (30) and (25) in (24) yields

$$\begin{array}{ll}E\left[f\right(P,Y\left)\right]& \le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_{t-1}\\ & +f(U,X)\left(\mathrm{Pr}(y\in P\backslash U)·\frac{u-t+1}{u}+\mathrm{Pr}(y\in U)·\frac{u-t}{u}\right).\end{array}$$

(31)

Firstly, we have

$$\mathrm{Pr}(y\in P\backslash U)·\frac{u-t+1}{u}+\mathrm{Pr}(y\in U)·\frac{u-t}{u}=\frac{u-t}{u}+\mathrm{Pr}(y\in P\backslash U)·\frac{1}{u}\le \frac{u-t}{u}+\mathrm{Pr}(y\in P\backslash U)·\frac{1}{t},$$

where the last inequality holds as $u\ge t$.

Secondly, since $U\subseteq P$,

$$f(U,X)\mathrm{Pr}(y\in P\backslash U)=f(U,X)\frac{f(P\backslash U,X)}{f(P,X)}\le f(P\backslash U,X).$$

Hence, we can bound (31) by

$$\begin{array}{cc}\hfill f(U,X)\left(\mathrm{Pr}(y\in P\backslash U)\frac{u-t+1}{u}+\mathrm{Pr}(y\in U)·\frac{u-t}{u}\right)& \le \frac{u-t}{u}·f(U,X)+f(U,X)\mathrm{Pr}(y\in P\backslash U)·\frac{1}{t}\hfill \\ & \le \frac{u-t}{u}·f(U,X)+\left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)·\frac{1}{t}.\hfill \end{array}$$

(32)

This proves the inductive step and bounds (31) by

$$\begin{array}{cc}\hfill E\left[f\right(P,Y\left)\right]& \le \left(f(P\backslash U,X)+4{\rho }^{2}f*(U,u)\right)H_t+\frac{u-t}{u}·f(U,X).\hfill \end{array}$$

 □

Corollary 2.

Let $\delta \in (0,1]$ and let $q_0$ be a point that is sampled at random from a nonempty set $Q\subseteq P$ such that $q_0=q$ with probability $\frac{w\left(q\right)}{{\displaystyle \sum _{q^{\prime }\in Q}w\left(q^{\prime }\right)}}$. Then with probability at least $1-\delta$,

$$f(Q,\left\{q_0\right\})\le \frac{2}{\delta }\rho f*(Q,1).$$

Proof. 

By Markov’s inequality, for every non-negative random variable G and $\delta >0$ we have

$$\begin{array}{cc}\hfill & Pr\{G<\frac{1}{\delta }E\left[G\right]\}\ge 1-\delta .\hfill \end{array}$$

(33)

Substituting $G=f(Q,\{q_0\left\}\right)$ yields

$$\begin{array}{c}\hfill {\displaystyle Pr\{f(Q,\left\{q_0\right\})<\frac{1}{\delta }\sum _{q_0\in Q}\frac{w\left(q_0\right)}{{\displaystyle \sum _{q\in Q}w\left(q\right)}}f(Q,\left\{q_0\right\})\}\ge 1-\delta .}\end{array}$$

(34)

By Lemma 5 we have

$$\begin{array}{c}\hfill {\displaystyle \sum _{q_0\in Q}\frac{w\left(q_0\right)}{{\displaystyle \sum _{q\in Q}w\left(q\right)}}f(Q,\left\{q_0\right\})\le 2\rho ·f*(Q,1).}\end{array}$$

(35)

Plugging (35) in (34) yields,

$$\begin{array}{cc}\hfill Pr\{f\{Q,\left\{q_0\right\})<\frac{2}{\delta }\rho ·f*(Q,1)\}& {\displaystyle \ge Pr\{f(Q,\left\{q_0\right\})<\frac{1}{\delta }\sum _{q_0\in Q}\frac{w\left(q_0\right)}{{\displaystyle \sum _{q\in Q}w\left(q\right)}}f(Q,\left\{q_0\right\})\}}\hfill \\ \hfill & \ge 1-\delta .\hfill \end{array}$$

 □

The following theorem is a particular case of Lemma 7. It proves that the output of KMeans++; see Algorithm 1, is a $O(logk)$-approximation of its optimum.

Theorem 1.

Let $(P,f)$ be a ρ-metric, and $k\ge 2$ be an integer; See Definition 1. Let $w:P\to (0,\infty )$, $\delta \in (0,1)$ and let Y be the output of a call to KMeans++$(P,w,f,\varnothing ,k)$; See Algorithm 1. Then $\mid Y\mid =k$, and with probability at least $1-\delta$,

$$f(P,Y)\le \frac{8{\rho }^2}{{\delta }^2}(1+lnk)f*(P,k).$$

Moreover, $\mid Y\mid$ can be computed in $O\left(ndk\right)$ time.

Proof. 

Let ${\delta }^{\prime }=\delta /2$ and let $\left\{P_1,\cdots ,P_k\right\}$ be an optimal partition of P, i.e., ${\sum }_{i=1}^{k}f*(P_i,1)=f*(P,k)$. Let $p_0$ be a point that is sampled at random from $P_k$ such that $p_0=p$ with probability ${\displaystyle \frac{w\left(p\right)}{{\sum }_{p^{\prime }\in P_k}w\left(p^{\prime }\right)}}$. Applying Lemma 7 with $u=t=k-1$ and $X=\left\{p_0\right\}$ yields,

$$\begin{array}{ll}E\left[f\right(P,Y\left)\right]& {\displaystyle \le \left(f(P_k,\left\{p_0\right\})+4{\rho }^{2}f*(P\backslash P_k,k-1)\right)·\sum _{i=1}^{k-1}\frac{1}{i}}\end{array}$$

(36)

$$\begin{array}{ll}& {\displaystyle \le (f(P_k,\{p_0\})+\frac{2{\rho }^2}{{\delta }^{\prime }}f*(P\backslash P_k,k-1))·\sum _{i=1}^{k-1}\frac{1}{i}}\end{array}$$

(37)

$$\begin{array}{ll}& {\displaystyle =(f(P_k,\left\{p_0\right\})+\frac{2{\rho }^2}{{\delta }^{\prime }}f*(P,k)-\frac{2{\rho }^2}{{\delta }^{\prime }}f*(P_k,1))·\sum _{i=1}^{k-1}\frac{1}{i},}\end{array}$$

(38)

where (38) holds by the definition of f* and $P_k$. By plugging $Q=P_k$ in Corollary 2, with probability at least $1-{\delta }^{\prime }$ over the randomness of $p_0$, we have

$$\begin{array}{c}f(P_k,\left\{p_0\right\})-\frac{2\rho }{{\delta }^{\prime }}f*(P_k,1)\le 0,\end{array}$$

(39)

and since $\rho \ge 1$, with probability at least $1-{\delta }^{\prime }$ we also have

$$\begin{array}{c}f(P_k,\left\{p_0\right\})-\frac{2{\rho }^2}{{\delta }^{\prime }}f*(P_k,1)\le 0.\end{array}$$

(40)

Plugging (40) in (38) yields that with probability at least $1-{\delta }^{\prime }$ over the randomness of $p_0$,

$$\begin{array}{cc}E\left[f\right(P,Y\left)\right]& {\displaystyle \le \frac{2{\rho }^2}{{\delta }^{\prime }}f*(P,k)·\sum _{i=1}^{k-1}\frac{1}{i}\le \frac{2{\rho }^2}{{\delta }^{\prime }}f*(P,k)·(1+\ln k).}\hfill \end{array}$$

(41)

Relating to the randomness of Y, by Markov’s inequality we have

$$\begin{array}{cc}\hfill & Pr\left\{f\right(P,Y)<\frac{1}{{\delta }^{\prime }}E\left[f\right(P,Y)\left]\right\}\ge 1-{\delta }^{\prime }.\hfill \end{array}$$

(42)

By (41) we have,

$$\begin{array}{c}\hfill Pr\left\{\frac{1}{{\delta }^{\prime }}E\left[f\right(P,Y)]\le \frac{2{\rho }^2}{{\delta }^{\prime 2}}(1+lnk)f*(P,k)\right\}\ge 1-{\delta }^{\prime }.\end{array}$$

(43)

Using the union bound on (42) and (43) we obtain

$$\begin{array}{cc}\hfill Pr\left\{f(P,Y)<\frac{2{\rho }^2}{{\delta }^{\prime 2}}(1+lnk)f*(P,k)\right\}& \ge 1-2{\delta }^{\prime },\hfill \end{array}$$

and thus

$$\begin{array}{cc}\hfill Pr\left\{f(P,Y)<\frac{8{\rho }^2}{{\delta }^2}(1+lnk)f*(P,k)\right\}& \ge 1-\delta .\hfill \end{array}$$

 □

5. k-Means with m Outliers

In this section we consider the problem of k-means with m outliers of a given set P, i.e., where the cost function for a given set X of k centers is $f(P_X,X)$ instead of $f(P,X)$, and $P_X$ is the subset of the closest $n-m$ points to the centers. Ties are broken arbitrarily. That is, $P\backslash P_X$ can be considered as the set of m outliers that are ignored in the summation of errors.

Definition 4

(k-median with m outliers). Let $(P,f)$ be a ρ-metric, $n=\mid P\mid$, and $k,m\in \left[n\right]$ be a pair of integers. For every subset $Q\subseteq P$ of points in P and a set $X\subseteq P$ of centers, denote by $Q_X$ the closest $n-m$ points to P. A set $X^{*}$ of k centers (points in P) is a k-median with m outliers of P if

$$f(P_{X^{*}},X^{*})\le \underset{X\subseteq P,\mid X\mid =k}{min}f(P_X,X).$$

For $\alpha \ge 0$, a set Y is an α-approximation to the k-median of P with m outliers if

$$f(P_Y,Y)\le \alpha \underset{X\subseteq P,\mid X\mid =k}{min}f(P_X,X).$$

Note that we allow Y to have $\mid Y\mid >k$ centers.

This is a harder and more general problem, since for $m=0$ we get the k-median problem from the previous sections.

We prove in this section that our generalized k-means++ can be served as a “weaker” type of coreset which admits an additive error that yields a constant factor approximation for the k-means with m outliers if $\rho$ is constant.

In fact, we prove that any approximated solution to the $(k+m)$-median of P implies such a coreset, but Algorithm 1 is both simple and general for any $\rho$-distance. Moreover, by taking more than $k+m$ centers, e.g., running Algorithm 1 additional iterations, we may reduce the approximation error $\alpha$ of the coreset to obtain “strong” regular coreset, i.e., that introduces a $(1+\epsilon )$-approximation for any given query set X of k centers. This is by having an $\alpha <\epsilon$ approximation for the $(k+m)$ median; see Theorem 2. Upper and lower bounds for the number $\mid X\mid$ of centers to obtain such $\alpha <\epsilon$ is the subject of [58]. The authors prove that a constant approximation to the $\mid X\mid =O({(1/\epsilon )}^{d}klogn)$-median of P yields such $\alpha =O\left(\epsilon \right)$-approximation to the k-median. This implies that running Algorithm 1 $O({(1/\epsilon )}^d(k+m)logn)$ iterations would yield a coreset that admits $(1+\epsilon )$-approximation error for any given set of k-centers with their farthest m outliers, as traditional coresets. Unfortunately, the lower bounds of [58] show that the exponential dependency in d is unavoidable. However, the counter example is extremely artificial. In real world data, we may simply run Algorithm 1 until the approximation error $f(P,\mid X\mid )$ is sufficiently small and hope this will happen after few $\mid X\mid$ iterations due to the structure of the data.

We state our result for the metric case and unweighted input. However, the proof essentially uses only the triangle inequality and its variants of Section 3. For simplicity of notation, we use the term multiset C instead of a weighted set $(C,w)$, where $C\subseteq P$, and $w:P\to [0,\infty )$ denote the number of copies of each item in C. The size $\mid C\mid$ of a multiset denotes its number of points (including duplicated points), unless stated otherwise.

Unlike the case of k-means/median, there are few solutions to k means with m outliers. Chen [50] provided a (multiplicative) constant factor approximation for the k-median with m outliers on any metric space (that satisfies the triangle inequality, i.e., with $\rho$ = 1) that runs in time $O(n^{3}log\left(n\right)d{k}^2{(k+m)}^2)$, i.e., polynomial in both k and m. Applying this solution on our suggested coreset as explained in Corollary 3 might reduce this dependency on n from cubic to linear, due to the fact that its construction time which is linear in n and also polynomial in $k+m$. In order to obtain a more general result for any $\rho$-distance, as in the previous section, we use a simple exhaustive search that takes time exponential in $k+m$ but still $O\left(n\right)$ for every constant $k,m$ and $\rho$.

Our solution also implies streaming, parallel algorithm for k-median with m outliers on distributed data. This is simply because many k-median algorithms exist for these computation models, and they can be used to construct the “coreset” in Theorem 2 after replacing k with $k+m$. An existing offline nonparallel solution for the k-median with m outliers, e.g., from the previous paragraph, may then be applied on the resulting coreset to extract the final approximation as explained in Corollary 4.

For every point $p\in P$, let $\mathrm{proj}(p,Y)\in Y$ denote its closest point in Y. For a subset Q of P, define $\mathrm{proj}(Q,Y)=\left\{\mathrm{proj}(p,Y)\mid p\in Q\right\}$ to be the union (multiset) of its $\mid Q\mid$ projection on Y. Ties are broken arbitrarily. Recall the definition of $P_X,C_X$ and $\alpha$-approximation from Definition 4. We now ready to prove the main technical result for the k-median with m outliers.

Theorem 2

(coreset for k-median with m outliers). Let $(P,f)$ be a $(\rho ,\varphi ,\epsilon )$ metric space, and let $n=\mid P\mid$. Let $k,m\in \left[n\right]$ such that $k+m<n$, and let Y be an α-approximation for the $(k+m)$-median of P (without outliers). Let $C:=\mathrm{proj}(P,Y)$. Then for every set $X\subseteq P$ of $\mid X\mid =k$ centers, we have

$$\begin{array}{cc}\hfill \mid f(P_X,X)-f(C_X,X)\mid & \le \left((\varphi +\epsilon \rho )\alpha +\epsilon \rho \right)f(P_X,X),\hfill \end{array}$$

(44)

and

$$f(P_X,X)\le (1+\epsilon \rho )f(C_X,X)+(\varphi +\epsilon \rho )\alpha f^{*}(P,k+m).$$

(45)

Proof. 

Let $X\subseteq P$ be a set of $\mid X\mid =k$ centers. The multiset $\mathrm{proj}(P_X,Y)$ contains $n-m$ points that are contained in the k centers of Y. However, we do not know how to approximate the number of copies of each center in $\mathrm{proj}(P_X,Y)$ without using $P_X$. One option is to guess $(1+\epsilon )$ approximation to the weight of each of these $k+m$ points, by observing that it is ${(1+\epsilon )}^i$ for some $i\in O(log(n)/\epsilon )$, and then using exhaustive search. However, the running time would be exponential in $k+m$ and the weights depend on the query X.

Instead, we observe that while we cannot compute $\mathrm{proj}(P_X,Y)$ via C, we have the upper bound $f(C_X,X)\le f(\mathrm{proj}(P_X,Y),X)$. It follows from the fact that

$$\mathrm{proj}(P_X,Y)\subseteq \mathrm{proj}(P,Y)=C,$$

and $\mid \mathrm{proj}(P_X,Y)\mid =n-m$, so

$$f(C_X,X)=\underset{Q\subseteq C,\mid Q\mid =n-m}{min}f(Q,X)\le f(\mathrm{proj}(P_X,Y),X).$$

(46)

We now bound the rightmost term.

For a single point $p\in P$ we have

$$\begin{array}{cc}\hfill \mid f(p,X)-f\left(\mathrm{proj}\right(p,Y),X)\mid & \le (\varphi +\epsilon \rho )f(p,\mathrm{proj}(p,Y))+\epsilon \rho min\left\{f(p,X),f\left(\mathrm{proj}\right(p,Y),X)\right\}\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill & =(\varphi +\epsilon \rho )f(p,Y)+\epsilon \rho min\left\{f(p,X),f\left(\mathrm{proj}\right(p,Y),X)\right\},\hfill \end{array}$$

(48)

where (47) holds by substituting $x=f(p,X)$ and $y=f\left(\mathrm{proj}\right(p,Y),X)$ (or vice vera) and $Z=X$ in Lemma 2, and (48) holds by the definition of $f(p,Y)$.

Summing (48) over every $p\in Q$ for some set $Q\subseteq P$ yields

$$\begin{array}{cc}\hfill \mid f(Q,X)-f\left(\mathrm{proj}\right(Q,Y),X)\mid & =\left|\sum _{p\in Q}\left(f(p,X)-f\left(\mathrm{proj}\right(p,Y),X)\right)\right|\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill & \le \sum _{p\in Q}\mid f(p,X)-f(\mathrm{proj}(p,Y),X)\mid \hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill & \le \sum _{p\in Q}\left((\varphi +\epsilon \rho )f(p,Y)+\epsilon \rho min\left\{f(p,X),f\left(\mathrm{proj}\right(p,Y),X)\right\}\right)\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill & \le (\varphi +\epsilon \rho )f(Q,Y)+\epsilon \rho min\left\{f(Q,X),f\left(\mathrm{proj}\right(Q,Y),X)\right\},\hfill \end{array}$$

(52)

where (49) holds by the definition of f, (50) holds by the triangle inequality, and (51) is by (48). The term $f(Q,Y)$ is bounded by

$$f(Q,Y)\le f(P,Y)\le \alpha f^{*}(P,k+m),$$

(53)

where the first inequality holds since $Q\subseteq P$, and the last inequality holds since Y is an $\alpha$-approximation for the $(k+m)$-median of P. Plugging (53) in (51) yields

$$\begin{array}{cc}\hfill \mid f(Q,X)-f\left(\mathrm{proj}\right(Q,Y),X)\mid & \le (\varphi +\epsilon \rho )f(Q,Y)+\epsilon \rho min\left\{f(Q,X),f\left(\mathrm{proj}\right(Q,Y),X)\right\}\hfill \\ \hfill & \le (\varphi +\epsilon \rho )\alpha f^{*}(P,k+m)+\epsilon \rho min\left\{f(Q,X),f\left(\mathrm{proj}\right(Q,Y),X)\right\}.\hfill \end{array}$$

(54)

The rest of the proof is by the following case analysis: (i) $f(C_X,X)\ge f(P_X,X)$, and (ii) $f(C_X,X)<f(P_X,X)$.

Case (i): $f(C_X,X)\ge f(P_X,X)$. The bound for this case is

$$\begin{array}{cc}\hfill f(C_X,X)-f(P_X,X)& \le f(\mathrm{proj}(P_X,Y),X)-f(P_X,X)\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill & \le (\varphi +\epsilon \rho )\alpha f^{*}(P,k+m)+\epsilon \rho f(P_X,X)\hfill \end{array}$$

(56)

where (55) holds by (46), and (56) holds by substituting $Q=P_X$ in (54). This bounds (44) for the case that $f(C_X,X)\ge f(P_X,X)$.

Case (ii):$f(C_X,X)<f(P_X,X)$. In this case, denote the $n-m$ corresponding points to $C_X$ in P by

$${\mathrm{proj}}^{-1}\left(C_X\right):=\left\{p\in P\mid \mathrm{proj}(p,Y)\in C_X\right\}.$$

Similarly to (46),

$${\mathrm{proj}}^{-1}\left(C_X\right)\subseteq \mathrm{proj}(P,Y)=C,$$

and $\mid {\mathrm{proj}}^{-1}\left(C_X\right)\mid =n-m$, so

$$f(P_X,X)=\underset{Q\subseteq P,\mid Q\mid =n-m}{min}f(Q,X)\le f({\mathrm{proj}}^{-1}\left(C_X\right),X).$$

(57)

Hence,

$$\begin{array}{cc}\hfill f(P_X,X)-f(C_X,X)& \le f({\mathrm{proj}}^{-1}\left(C_X\right),X)-f(C_X,X)\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill & \le (\varphi +\epsilon \rho )\alpha f^{*}(P,k+m)+\epsilon \rho f(C_X,X)\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & \le (\varphi +\epsilon \rho )\alpha f^{*}(P,k+m)+\epsilon \rho f(P_X,X),\hfill \end{array}$$

(60)

where (58) holds by (57), (59) is by substituting $Q={\mathrm{proj}}^{-1}\left(C_X\right)$ in (54), and (60) follows from the assumption $f(C_X,X)<f(P_X,X)$ of Case (ii).

Combining all together, using (56) and (60) we can bound (44) by

$$\begin{array}{cc}\hfill \mid f(P_X,X)-f(C_X,X)\mid & \le (\varphi +\epsilon \rho )\alpha f^{*}(P,k+m)+\epsilon \rho f(P_X,X)\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill & \le (\varphi +\epsilon \rho )\alpha f(P_X,X)+\epsilon \rho f(P_X,X)\hfill \end{array}$$

(62)

$$\begin{array}{cc}\hfill & \le \left((\varphi +\epsilon \rho )\alpha +\epsilon \rho \right)f(P_X,X),\hfill \end{array}$$

(63)

and (45) follows from (59), assuming $f(C_X,X)<f(P_X,X)$ (otherwise (45) trivially holds).  □

The motivation for (44) is to obtain a traditional coreset, in the sense of $(1+\epsilon )$-approximation to any given set of k centers. To this end, we need to have $\left(\right(\varphi +\epsilon \rho \left)\alpha \right)<\epsilon$. For the natural case where $\rho =O\left(1\right)$ this implies $\alpha <\epsilon$ approximation to the $(k+m)$-means $f^{*}(P,k+m)$ of P. This is doable for every input set P in the Euclidean space and many others as was proved in [58] but in the price of taking ${(1/\epsilon )}^{d}klogn$ centers.

This is why we also added the bound 45, which suffices to obtain a “weaker” coreset that yields only constant factor approximation to the k-means with m outliers, but using only $k+m$ centers, as explained in the following corollary.

Corollary 3

(From $(k+m)$-median to k-median with m outliers ). Let $(P,f)$ be a $(\rho ,\varphi ,\epsilon )$-metric, $n=\mid P\mid$, $\alpha ,\beta >0$ and $k,m\in \left[n\right]$ such that $k+m<n$. Suppose that, in $T(n,k+m)$ time, we can compute an α-approximation Y for the $(k+m)$-median of P (without outliers). Suppose also that we can compute in $t(\mid Y\mid )$ time, a β-approximation $X\subseteq C$ of the k-median with m outliers for any given multiset $C\subseteq Y$ of $\mid C\mid =n$ (possibly duplicated) points. Then, X is a

$$(1+\epsilon \rho )\beta \left(1+\varphi \alpha +\epsilon \rho \alpha +\epsilon \rho \right)+(\varphi +\epsilon \rho )\alpha$$

approximation for the k-median with m outliers of P and can be computed in $T(n,k+m)+t(\mid Y\mid )$ time.

Proof. 

We compute Y in $T(n,k+m)$ time and then project P onto Y to obtain a set $C:=\mathrm{proj}(P,Y)$ of $\mid Y\mid$ distinct points, as defined in the proof of Theorem 2. We then compute a $\beta$ approximation $X\subseteq C$ for the k-median with m outliers of P,

$$f(C_X,X)\le \beta \underset{Z\subseteq C,\mid Z\mid =k}{min}f(C_Z,Z),$$

(64)

in $t(\mid Y\mid )$ time. For a given set $X\subseteq P$ of centers and a subset $Q\subseteq P$ of points, we denote by $Q_X$ the closest $n-m$ points in Q to X. Let $X^{*}$ denote the k-median with m outliers of P. Hence,

$$\begin{array}{cc}\hfill f(P_X,X)& \le (1+\epsilon \rho )f(C_X,X)+(\varphi +\epsilon \rho )\alpha f^{*}(P,k+m)\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill & \le (1+\epsilon \rho )\beta f(C_{X^{*}},X^{*})+(\varphi +\epsilon \rho )\alpha f(P_{X^{*}},X^{*})\hfill \end{array}$$

(66)

$$\begin{array}{cc}\hfill & \le (1+\epsilon \rho )\beta \left(1+\varphi \alpha +\epsilon \rho \alpha +\epsilon \rho \right)f(P_{X^{*}},X^{*})+(\varphi +\epsilon \rho )\alpha f(P_{X^{*}},X^{*})\hfill \end{array}$$

(67)

$$\begin{array}{cc}\hfill & \le ((1+\epsilon \rho )\beta \left(1+\varphi \alpha +\epsilon \rho \alpha +\epsilon \rho \right)+(\varphi +\epsilon \rho )\alpha )f(P_{X^{*}},X^{*}),\hfill \end{array}$$

(68)

where (65) holds by (45), (66) holds by (64), and (67) holds by plugging $X^{*}$ in (44).  □

To get rid of the so many parameters in the last theorems, in the next corollary we assume that they are all constant and suggest a simple solution to the k-median with m outliers by running exhaustive search on our weaker coreset. The running time is exponential in k but this may be fixed by running the more involved polynomial-time approximation of Chen [50] for k-means with m outliers on our coreset.

Corollary 4.

Let $k,m\in \left[n\right]$ such that $k+m<n$, and let $\rho \ge 1$ be constants. Let $(P,f)$ be a ρ-metric. Then, a set $X\subseteq P$ can be computed in $O\left(n\right)$ time such that, with probability at least 0.99, X is a $O(ln(k+m\left)\right)$-approximation for the k-median with m outliers of P.

Proof. 

Given a set Q of $\mid Q\mid =n^{\prime }$ points, it is easy to compute its k-median with m outliers in $n^{O\left(k\right)}$ time as follows. We run exhaustive search over every subset Y of size $\mid Y\mid =k$ in Q. For each such subset Y, we compute its farthest m points Z in Q. The set Y that minimizes $f(Q\backslash Z,Y)$ is an optimal solution, since one of these sets of k centers is a k-median with m outliers of P. Since there are such $(\genfrac{}{}{0pt}{}{n^{\prime }}{k})={\left(n^{\prime }\right)}^{O\left(k\right)}$ subsets, and each check requires $O\left(nk\right)=O\left(n\right)$ time, the overall running time is $t\left(n^{\prime }\right)={\left(n^{\prime }\right)}^{O\left(k\right)}$.

Plugging $\delta =0.01$ in Theorem 1 implies that we can compute a set $Y\subseteq P$ of size $\mid Y\mid =k+m$ which is, with probability at least $1-\delta =0.99$, an $O(ln(k+m\left)\right)$-approximation to the $(k+m)$-median of P in time $T(n,k+m)=O\left(nd\right(k+m\left)\right)=O\left(n\right)$. By Lemma 3, $(P,f)$ is a $(\rho ,\varphi ,\epsilon )$-metric for $\varphi =\rho$, and $\epsilon =\rho -1$. Plugging $n^{\prime }=\mid Y\mid =k$ and $\beta =1$ in Corollary 3 then proves Corollary 4.   □

6. Conclusions

We proved that the k-means++ algorithm can be generalized to handle outliers by generalizing it to $\rho$-metric that supports M-estimators and also proved that k-means seeding can be used as a coreset for the k-means with m outliers. Open problems include generalizations of k-means++ for other shapes, such as k lines or k multidimensional subspaces, and using these approximations for developing coreset algorithms for these problems. Other directions include improving or generalizing the constant factor approximations for the original k-means++ and its variants in this paper, using recent improvement for the analysis of k-means++ and coresets for k-means clustering.