Coresets for the Average Case Error for Finite Query Sets

Abstract

Coreset is usually a small weighted subset of an input set of items, that provably approximates their loss function for a given set of queries (models, classifiers, hypothesis). That is, the maximum (worst-case) error over all queries is bounded. To obtain smaller coresets, we suggest a natural relaxation: coresets whose average error over the given set of queries is bounded. We provide both deterministic and randomized (generic) algorithms for computing such a coreset for any finite set of queries. Unlike most corresponding coresets for the worst-case error, the size of the coreset in this work is independent of both the input size and its Vapnik–Chervonenkis (VC) dimension. The main technique is to reduce the average-case coreset into the vector summarization problem, where the goal is to compute a weighted subset of the n input vectors which approximates their sum. We then suggest the first algorithm for computing this weighted subset in time that is linear in the input size, for $n\gg 1/\epsilon$, where $\epsilon$ is the approximation error, improving, e.g., both [ICML’17] and applications for principal component analysis (PCA) [NIPS’16]. Experimental results show significant and consistent improvement also in practice. Open source code is provided.

1. Introduction

In this paper, we assume that the input is a set P of items, called points. Usually, P is simply a finite set of n points in ${\mathbb{R}}^d$ or other metric space. In the context of PAC (probably approximately correct) learning [1], or empirical risk minimization [2] it represents the training set. In supervised learning every point in P may also include its label or class. We also assume a given function $w:P\to (0,\infty )$ called weights function that assigns a “weight” $w\left(p\right)>0$ for every point $p\in P$. The weights function represents a distribution of importance over the input points, where the natural choice is uniform distribution, i.e., $w\left(p\right)=1/\left|P\right|$ for every $p\in P$. We are also given a (possibly infinite) set X that is the set of queries [3] which represents candidate models or hypothesis, e.g., neural networks [4], SVMs [5] or a set of vectors in ${\mathbb{R}}^d$ with tuning parameters as in linear/ridge/lasso regression [6,7,8].

In machine learning and PAC-learning in particular, we often seek to compute the query that best describes our input data P for either prediction, classification, or clustering tasks. To this end, we define a loss function$f:P\times X\to \mathbb{R}$ that assigns a fitting cost $f(p,x)$ to every point $p\in P$ with respect to a query $x\in X$. For example, it may be a kernel function [9], a convex function [10], or an inner product. The tuple $(P,w,X,f)$ is called a query space and represents the input to our problem. In this paper, we wish to approximate the weighted sum of losses $f_w(P,x)={\sum }_{p\in P}w\left(p\right)f(p,x)$.

Methodology.

Our main tool for approximating the sum of losses above is called a coreset, which is a (small) weighted subset of the (potentially huge) input data, from which the desired sum of losses can be recovered in a very fast time, with guarantees on the small induced error; see Section 1.2. To compute such a coreset, we utilize the famous Frank–Wolfe algorithm. However, to compute a coreset in fast time, we first provide a scheme for provably boosting the running time of the Frank–Wolfe algorithm for our special case, without compromising its output accuracy; see Section 3.1. We then utilize the boosted version in order to compute a deterministic coreset in time faster than the state of the art.

1.1. Approximation Techniques for the Sum of Losses

Approximating the loss of a single query via uniform sampling. Suppose that we wish to approximate the mean $f(P,x)=\frac{1}{n}{\sum }_{p\in P}f(p,x)$ for a specific x in sub-linear time. Picking a random point p uniformly at random from P would give this result in expectation as $E\left[f(p,x)\right]={\sum }_{p\in P}f(p,x)/n=f(P,x)$. By Hoeffding inequality, the mean $f(S,x)=\frac{1}{\left|S\right|}{\sum }_{p\in S}f(p,x)$ of a uniform sample $S\subseteq P$ would approximate $f(P,x)$ with high probability. More precisely, for a given $\epsilon \in (0,1)$, if the size of the sample is $\left|S\right|\in O(M/{\epsilon }^2)$ where $M={max}_{p\in P}\left|f(p,x)\right|$ is the maximum absolute value of f, then with constant probability our approximation error is

$$er{r}_f\left(x\right)=|f(P,x)-f(S,x)|\le \epsilon .$$

$\epsilon$-Sample. Generally, we are interested in such data summarization S of P that approximates every query $x\in X$. An ε-sample is a pair $(S,u)$ where S is a subset of P (unlike, e.g., sketches [11]), and $u:S\to [0,\infty )$ is its weights function such that the weighted loss $f_u(S,x)=$${\sum }_{p\in S}u\left(p\right)f(p,x)$ of the (hopefully small) weighted subset S approximates the original weighted loss $f_w(P,x)$ [12], i.e.,

$$\forall x\in X:\left|f_w(P,x)-f_u(S,x)\right|\le \epsilon .$$

(1)

We usually assume that the input is normalized in the sense that w is a distribution, and $f:P\times X\to [-1,1]$. By defining the pair of vectors $f_w(P,X)={\left({\sum }_{p\in P}w\left(p\right)f(p,x)\right)}_{x\in X}$ and $f_u(S,X)={\left({\sum }_{p\in S}u\left(p\right)f(p,x)\right)}_{x\in X}$, we can define the error for a single x by $err\left(x\right)=|f_w(P,x)-f_u(S,x)|$, and then the error vector for the coreset $err\left(X\right)={\left(err\left(x\right)\right)}_{x\in X}$. We can rewrite (1) by

$${∥err\left(X\right)∥}_{\infty }={∥f_w(P,X)-f_u(S,X)∥}_{\infty }\le \epsilon .$$

(2)

PAC/DAC learning for approximating the sum of losses for multiple queries. Probably approximately correct (PAC) randomized constructions generalizes the Hoeffding inequality above from a single to multiple (usually infinite) queries and returns an $\epsilon$-sample for a given query space $(P,w,X,f)$ and $\delta \in (0,1)$, with probability at least $1-\delta$. Here, $\delta$ corresponds to the “probably” part, while “approximately correct” corresponds to $\epsilon$ in (2); see  [13,14]. Deterministic approximately correct (DAC) versions of PAC-learning suggest deterministic construction of $\epsilon$-samples, i.e., the probability of failure of the construction is $\delta =0$.

As common in machine learning and computer science in general, the main advantage of deterministic constructions is smaller bounds (in this case, on the size of the resulting $\epsilon$-sample), and their disadvantage is usually the slower construction time that may be unavoidable. When the query set X if finite, the Caratheodory theorem [15,16] suggests a deterministic algorithm that returns a 0-sample $(S,u)$ (i.e., $f_w(P,X)=f_u(S,X)$) of size $\left|S\right|\le \left|X\right|+1$. Deterministic constructions of $\epsilon$-sample are known for infinite sets of queries even when the VC-dimension is unbounded [17,18].

Sup-sampling: reducing the sample size via non-uniform sampling. As explained above, Hoeffding inequality implies an approximation of $f_w(P,x)$ by $f_u(S,x)$ where $u\left(p\right)=1/\left|S\right|$ and S is a random sample according to w whose size depends on $M\left(f\right)={max}_{p\in P}\left|f(p,x)\right|$. To reduce the sample size we may thus define $g(p,x)=\frac{f(p,x)}{\left|f\right(p,x\left)\right|}\in \left\{-1,1\right\}$, and $s\left(p\right)=\frac{w\left(p\right)\left|f\right(p,x\left)\right|}{{\sum }_{q\in P}w\left(q\right)\left|f(q,x)\right|}$. Now, $M\left(g\right)={max}_{p\in P}\left|g(p,x)\right|=1$, and by Hoeffding’s inequality, the error of approximating $g_s(P,x)$ via non-uniform random sample of size $1/{\epsilon }^2$ drawn from s, is $\epsilon$. Define $T={\sum }_{q\in P}w\left(q\right)\left|f(q,x)\right|$. Since $f_w(P,x)=T·g_s(P,x)$, approximating $g_s(P,x)$ up to $\epsilon$ error yields an error of $\epsilon T$ for $f_w(p,x)$. Therefore, the size is reduced from $M\left(f\right)/{\epsilon }^2$ to $T^2/{\epsilon }^2$ when $T^2\le M\left(f\right)$. Here, we sample $\left|S\right|=O(T^2/{\epsilon }^2)$ points from P according to the distribution s, and re-weight the sampled points by $u^{\prime }\left(p\right)=\frac{T}{\left|S\right|\left|f\right(p,x\left)\right|}$.

Unlike traditional PAC-learning, the sample now is non-uniform, and is proportional to $s\left(p\right)$, rather than w, as implied by Hoeffding inequality for non-uniform distributions. For sets of queries we generalize the definition for every $p\in P$ to $s\left(p\right)={sup}_{x\in X}\frac{w\left(p\right)\left|f\right(p,x\left)\right|}{{\sum }_{q\in P}w\left(q\right)\left|f(p,x)\right|}$ as in [19], which is especially useful for coresets below.

1.2. Coresets: A Data Summarization Technique for Approximating the Sum of Losses

Coreset for a given query space $(P,w,X,g)$, in this and many other papers, is a pair $(C,u)$ that is similar to $\epsilon$-sample in the sense that $C\subseteq P$ and $u:C\to [0,\infty )$ is a weights function. However, the additive error $\epsilon$ is now replaced with a multiplicative error $1\pm \epsilon$, i.e., for every $x\in X$ we require that, $|g_w(P,X)-g_u(C,X)|\le \epsilon ·g_w(P,X).$ Dividing by $g_w(P,X)$ and assuming $g_w(P,X)>0$, yields

$$\forall x\in X:|1-\frac{g_u(C,X)}{g_w(P,x)}|\le \epsilon .$$

(3)

Coresets are especially useful for learning big data since an off-line and possibly inefficient coreset construction for “small data" implies constructions that maintains coreset for streaming, dynamic (including deletions) and distributed data in parallel. This is via a simple and easy to implement framework that is sometimes called merge–reduce trees; see [20,21]. The fact that a coreset approximates every query (and not just the optimal one for some criterion) implies that we may solve hard optimization problems with non-trivial and non-convex constraints by running a possibly inefficient algorithm such as exhaustive search on the coreset, or running existing heuristics numerous times on the small coreset instead of once on the original data. Similarly, parameter tuning or cross validation can be applied on a coreset that is computed once for the original data as explained in [22].

An $\epsilon$-coreset for a query space $(P,w,X,g)$ is simply an $\epsilon$-sample for the query space $(P,w,X,f)$, after defining $f(p,x):=\frac{g(p,x)}{g_w(P,x)}$, as explained, e.g., in [19]. By defining the error for a single x by $er{r}^{\prime }\left(x\right)=|1-g_u(C,x)/g_w(P,x)|=|f_w(P,x)-f_u(C,x)|=er{r}_f\left(x\right)$, we obtain an error vector for the coreset $er{r}^{\prime }\left(X\right)={\left(er{r}^{\prime }\left(x\right)\right)}_{x\in X}$. We can then rewrite (3) as in (2):

$${∥er{r}^{\prime }\left(X\right)∥}_{\infty }\le \epsilon .$$

In the case of coresets, the ${\displaystyle \underset{x\in X}{sup}w\left(p\right)f(p,x)=\underset{x\in X}{sup}\frac{w\left(p\right)g(p,x)}{g_w(P,x)}}$ of a point $p\in P$ is called sensitivity [14], leverage score (in ${\ell }_2$ approximations) [23], Lewis weights (in ${\ell }_p$ approximations), or simply importance [24].

1.3. Problem Statement: Average Case Analysis for Data Summarization

Average case analysis (e.g., [25]) was suggested about a decade ago as an alternative to the (sometimes infamous) worst-case analysis of algorithms in theoretical computer science. The idea is to replace the analysis for the worst-case input by the average input (in some sense). Inspired by this idea, a natural variant of (2) and its above implications is an $\epsilon$-sample that approximates well the average query. We suggest to define an $(\epsilon ,∥·∥)$-sample as

$$∥err\left(X\right)∥=∥f_w(P,X)-f_u(S,X)∥\le \epsilon ,$$

(4)

which generalizes (2) from ${∥·∥}_{\infty }$ to any norm, such as the ${\ell }_z$ norm ${∥err\left(X\right)∥}_z$. For example, for the ${\ell }_2$, MSE or Frobenius norm, we obtain

$$\begin{array}{c}\hfill \sqrt{{\sum }_{x\in X}{\left(f_w(P,x)-f_u(S,x)\right)}^2}\le \epsilon .\end{array}$$

(5)

A generalization of the Hoeffding Inequality from 1963 with tight bounds was suggested relatively recently for the ${\ell }_z$ norm for any $z\ge 2$ and many other norms [26,27]. Here we assume a single query ($\left|X\right|=1$), a distribution weights function, and a bound on ${sup}_{p\in P}\left|f(p,X)\right|$ that determines the size of the sample, as in Hoeffding inequality.

A less obvious question, which is the subject of this paper, is how to compute deterministic $\epsilon$-samples that satisfies (4), for norms other than the infinity norm. While the Caratheodory theorem suggests deterministic constructions of 0-samples (for any error norm) as explained above, our goal is to obtain coreset whose size is smaller or independent of $\left|X\right|$.

The next question is how to generalize the idea of sup-sampling, i.e., where the function f is unbounded, for the case of norms other than ${∥·∥}_{\infty }$. Our main motivation for doing so is to obtain new and smaller coresets by combining the notion of $\epsilon$-sample and sup-sampling or sensitivity as explained above for the ${∥·∥}_{\infty }$ case. That is, we wish a coreset for a given query space, that would bound the non-${\ell }_{\infty }$ norm error

$$∥{\left(1-\frac{g_u(C,x)}{g_w(P,x)}\right)}_{x\in X}∥=∥er{r}^{\prime }\left(X\right)∥\le \epsilon .$$

To summarize, our questions are: How can we smooth the error function and approximate the “average” query via: (i) Deterministic $\epsilon$-samples (for DAC-learning )? (ii) Coresets (via sensitivities/sup sampling for non-infinity norms)?

1.4. Our Contribution

We answer affirmably these questions by suggesting $\epsilon$-samples and coresets for the average query. We focus on the case $z=2$, i.e., the Frobenius norm, and finite query set X and hope that this would inspire the research and applications of other norms and general sets. For suggestions in this direction and future work see Section 5.2. The main results of this paper are the following constructions of an $(\epsilon ,{∥·∥}_2)$-sample $(S,u)$ for any given finite query space $(P,w,X,f)$ as defined in (5):

(i)

Deterministic construction that returns a coreset of size $\left|S\right|\in O(1/{\epsilon }^2)$ in time $O\left(min\left\{nd/{\epsilon }^2,nd+dlog{\left(n\right)}^2/{\epsilon }^4\right\}\right)$; see Theorem 2 and Corollary 4.

(ii)

Randomized construction that returns such a coreset (of size $\left|S\right|\in O(1/{\epsilon }^2)$) with probability at least $1-\delta$ in sub-linear time $O\left(d\left(log{\left(\frac{1}{\delta }\right)}^2+\frac{log\left(\frac{1}{\delta }\right)}{{\epsilon }^2}\right)\right)$; see Lemma 5.

Algorithm. This result is of independent interest for faster and sparser convex optimization. To our knowledge, this is also the first application of sensitivity outside the coreset regime.

1.5. Overview and Organization

The rest of the paper in organized as follows. First in Section 2, we list the applications of our proposed methods, such as a faster coreset construction algorithm for least mean squares solver. We also compare our results to the state of the art to justify our practical contribution.

In Section 3, we first give our notations and relevant mathematical definitions, we explain the relation between the problem of computing an $(\epsilon ,{∥·∥}_2)$-sample (average-case coreset) to the problem of computing a vector summarization coreset, where the goal (of the vector summarization coreset problem) is to compute a weighted subset of the n input vectors which approximates their sum. Here, we suggest a coreset for this problem of size $O(1/\epsilon )$ in $O(nd/\epsilon )$ time; see Theorem 2 and Algorithm 2. Then, in Section 3.1 we show how to improve the running time of this result and compute a coreset of the same size in $O(nd+d{log}^2\left(n\right)/{\epsilon }^2)$ time; see Corollary 4 and Algorithm 3. In addition, we suggest a non-deterministic coreset of the same size but in time that is independent of the number of points n; see Lemma 5 and Algorithm 4.

In Section 4, we explain how our vector summarization coreset results can be used to improve all the previously mentioned applications (from Section 2). In Section 5 we conduct various experiments on real world datasets, where we apply different coreset construction algorithms presented in this paper to a variety of applications, in order to boost their running time, or reduce their memory storage. We also compare our results to many competing methods. Finally, we conclude our paper and discuss future work at Section 5.2. Due to space limitations and simplicity of the reading, the proofs of the claims are placed in the Appendix A, Appendix B, Appendix C, Appendix D, Appendix E, Appendix F, Appendix G, Appendix H, Appendix I.

2. On the Applications of Our Method and the Current State of the Art

In what follows, we will present some of the applications of our theoretical contributions as well as discussing the current state of the art coreset/sketch methods in terms of running time for each of application. Figure 1 summarizes the main applications of our result.

(i)

Vector summarization: the goal is to maintain the sum of a (possibly infinite) stream of vectors in ${\mathbb{R}}^d$, up to an additive error of $\epsilon$ multiplied by their variance. This is a generalization of frequent items/directions [28].

As explained in [29], the main real-world application is extractions and compactly representing groups and activity summaries of users from underlying data exchanges. For example, GPS traces in mobile networks can be exploited to identify meetings, and exchanges of information in social networks sheds light on the formation of groups of friends. Our algorithm tackles these application by providing provable solution to the heavy hitters problem in proximity matrices. The heavy hitters problem can be used to extract and represent in a compact way friend groups and activity summaries of users from underlying data exchanges.

We propose a deterministic algorithm which reduces each subset of n vectors into $O(1/\epsilon )$ weighted vectors in $O(nd+dlog{\left(n\right)}^2/{\epsilon }^2)$ time, improving upon the $nd/\epsilon$ of [29] (which is the current state of the art in terms of running time), for a sufficiently large n; see Corollary 4, and Figures 2 and 3. We also provide a non-deterministic coreset construction in Lemma 5. The merge-and-reduce tree can then be used to support streaming, distributed or dynamic data.

(ii)

Kernel Density Estimates (KDE): by replacing $\epsilon$ with ${\epsilon }^2$ for the vector summarization, we obtain fast construction of an $\epsilon$-coreset for KDE of Euclidean kernels [17]; see more details in Section 4. Kernel density estimate is a technique for estimating a probability density function (continuous distribution) from a finite set of points to better analyse the studied probability distribution than when using a traditional [30,31].

(iii)

1-mean problem: a coreset for 1-mean which approximates the sum of squared distances over a set of n points to any given center (point) in ${\mathbb{R}}^d$. This problem arises in facility location problems (e.g., to compute the optimal location for placing an antenna such that all the customers are satisfied). Our deterministic construction computes such a weighted subset of size $O(1/{\epsilon }^2)$ in $O(min\left\{nd/{\epsilon }^2,nd+dlog{\left(n\right)}^2/{\epsilon }^4\right\})$ time. Previous results of [19,32,33,34] suggested coresets for such problem. Unlike our results, these works are either non-deterministic, the coreset is not a subset of the input, or the size of the coreset is linear in d.

(iv)

Coreset for LMS solvers and dimensionality reduction: for example, a deterministic construction for singular value decomposition (SVD) that gets a matrix $A\in {\mathbb{R}}^{n\times d}$ and returns a weighted subset of $k^2/{\epsilon }^2$ rows, such that their weighted distance to any k-dimensional non-affine (or affine in the case of PCA) subspace approximates the distance of the original points to this subspace. The SVD and PCA are very common algorithms (see [35]), and can be used for noise reduction, data visualization, cluster analysis, or as an intermediate step to facilitate other analyses. Thus, improving them might be helpful for a wide range of real-world applications. In this paper, we propose a deterministic coreset construction that takes $O(n{d}^2+d^2{k}^{4}log{\left(n\right)}^2/{\epsilon }^4)$ time, improving upon the state of the art result of [35] which requires $O(n{d}^2{k}^2/{\epsilon }^2)$ time; see Table 1. Many non-deterministic coresets constructions were suggested for those problems, the construction techniques apply non-uniform sampling [36,37,38], Monte-Carlo sampling [39], and leverage score sampling [23,40,41,42,43,44,45].

Table 1. Known deterministic subset coresets for LMS solvers. Our result has the fastest running time for sufficiently large n and d.

Error	Size	Time	Citation	Notes
$\epsilon$	$O(k^2/{\epsilon }^2)$	$O(n{d}^2{k}^2/{\epsilon }^2)$	[35]	N/A
$\epsilon$	$O(d/{\epsilon }^2)$	$poly(n,d,\epsilon )$	[46]	inefficient for large n
0	$O\left(d^2\right)$	$O(n{d}^2+log\left(n\right)poly\left(d\right))$	[22]	inefficient for large d
$\epsilon$	$O(k/{\epsilon }^2)$	$poly(n,d,k,\epsilon )$	[47]	inefficient for large n
$\epsilon$	$O(k^2/{\epsilon }^2)$	$O(n{d}^2+log{\left(n\right)}^2{d}^2{k}^4/{\epsilon }^4)$	🟉	N/A

3. Vector Summarization Coreset

Notation 1.

We denote by $\left[n\right]=\left\{1,\cdots ,n\right\}$. For a vector $v\in {\mathbb{R}}^d$, the 0-norm is denoted by ${∥v∥}_0$ and is equal to the number of non-zero entries in v. We denote by $e_{\left(i\right)}$ the ith standard basis vector in ${\mathbb{R}}^n$ and by $\mathbf{0}$ the vector ${(0,\cdots ,0)}^T\in {\mathbb{R}}^n$. A vector $w\in {[0,1]}^n$ is called a distribution vector if all its entries are non-negative and sums up to one. For a matrix $A\in {\mathbb{R}}^{m\times n}$ and $i\in \left[m\right],j\in \left[n\right]$ we denote by $A_{i,j}$ the jth entry of the ith row of A. A weighted set is a pair $(Q,m)$ where $Q=\left\{q_1,\cdots ,q_n\right\}\subseteq {\mathbb{R}}^d$ is a set of npoints, and $m={(m_1,\cdots ,m_n)}^T\in {\mathbb{R}}^n$ is a weights vector that assigns every $q_i\in Q$ a weight $m_i\in \mathbb{R}$. A matrix $A\in {\mathbb{R}}^{d^{\prime }\times d}$ is orthogonal if $A^{T}A=I\in {\mathbb{R}}^{d\times d}$.

Adaptations. To adapt to the notation of the following sections and the query space $(P,w,X,f)$ to the techniques that we use, we restate (4) as follows. Previously, we denote the queries $X=\left\{x_1,\cdots ,x_d\right\}$, and the input set by $P=\left\{p_1,\cdots ,p_n\right\}$. Now, each input point $p_i$ in the input set P corresponds to a point $q_i=\left(f(p_i,x_1),\cdots ,f(p_i,x_d)\right)\in {\mathbb{R}}^d$, i.e., each entry of $q_i$ equals to $f(p_i,x)$ for a different query x. Throughout the rest of the paper, for technical reason and simplicity, we might alternate between the weights function notation and a weights vector notation. In such cases, the weights function $w:P\to [0,\infty )$ and weight $w\left(q_i\right)$ of $q_i$, $i\in \left[m\right]$ are replaced by a vector of weights $m=(m_1,\cdots ,m_n)\in {[0,\infty )}^n$ and $m_i$, respectively, and vice versa. In such cases, the $\epsilon$-sample is represented by a sparse vector $u\in [0,\infty )$ where $S=\left\{p_i\in P\mid u_i>0,i\in \left[n\right]\right\}$ is the chosen subset of P.

Hence, $f_w(P,X)={\sum }_{p\in P}w\left(p\right)\left(f(p,x_1),\cdots ,f(p,x_d)\right)={\sum }_{i=1}^n{m}_i{q}_i$, and $f_u(S,X)={\sum }_{i=1}^n{u}_i{q}_i$.

From $(\epsilon ,{∥·∥}_2)$-samples to $\epsilon$-coresets. We now define an $\epsilon$-coreset for vector summarization, which is a re-weighting of the input weighted set $(Q,m)$ by a new weights vector u, such that the squared norm of the difference between the weighted means of $(Q,u)$ and $(Q,m)$ is small. This relates to Section 1.3, where an $(\sqrt{\epsilon },{∥·∥}_2)$-sample there (in Section 1.3) is an $\epsilon$-coreset for the vector summarization here.

Definition 1

(vector summarization $\epsilon$-coreset). Let $(Q,m)$ and $(Q,u)$ be two weighted sets of n points in ${\mathbb{R}}^d$, and let $\epsilon \in [0,1)$. Let $\mu ={\sum }_{i=1}^n\frac{m_i}{{∥m∥}_1}{q}_i$, ${\sigma }^2={\sum }_{i=1}^n\frac{m_i}{{∥m∥}_1}{∥q_i-\mu ∥}^2$, and $\tilde{\mu }={\sum }_{i=1}^n\frac{u_i}{{∥u∥}_1}{q}_i$. Then $(Q,u)$ is avector summarization $\epsilon$-coresetfor $(Q,m)$ if ${∥\tilde{\mu }-\mu ∥}_2^2\le \epsilon {\sigma }^2$.

Analysis flow. In what follows we (first) assume that the points of our input set P lie inside the unit ball (${\forall }_{p\in P}:∥p∥\le 1$). For such an input set, we present a construction of a variant of a vector summarization coreset, where the error is $\epsilon$ and does not depend on the variance of the input. This construction is based on the Frank–Wolfe algorithm [48]; see Theorem 1 and Algorithm 1. This is by reducing the problem to the problem of maximizing a concave function $f\left(x\right)$ over every vector in the unit simplex. Such problems can be solved approximately by a simple greedy algorithm known as the Frank–Wolfe algorithm.

Algorithm 1:FRANK–WOLFE$(f,K)$; Algorithm 1.1 of [48]

1: Input: A concave function $f:{\mathbb{R}}^n\to \mathbb{R}$, and the number of iterations K. 2: Output: A vector $x\in {\mathbb{R}}^n$ that satisfies Theorem 1. 3: $x_{\left(0\right)}:=$ the unit n-simplex vertex with largest f value. 4: for$k\in \left\{0,\cdots ,K\right\}$do 5: $i^{\prime }:={arg\; max}_i\nabla f{\left(x_{\left(k\right)}\right)}_i$ 6: ${\displaystyle {\alpha }^{\prime }:=\underset{{\alpha }_{[0,1]}}{arg\; max}f\left(x_{\left(k\right)}+\alpha (e_{\left(i^{\prime }\right)}-x_{\left(k\right)})\right)}$ 7: $x_{(k+1)}:=x_{\left(k\right)}+{\alpha }^{\prime }(e_{\left(i^{\prime }\right)}-x_{\left(k\right)})$ 8: end for 9: Return $x_{(k+1)}$

We then present a proper coreset construction in Algorithm 2 and Theorem 2 for a general input set Q in ${\mathbb{R}}^d$. This algorithm is based on a reduction to the simpler case of points inside the unit ball; see Figure 1 for illustration. This reduction is inspired by the sup-sampling (see Section 1), there (in Section 1) the functions are normalized (to obtain values in $[-1,1]$) and reweighted (to obtain a non-biased estimator), then the bounds were easily obtained using the Hoeffding inequality. Here, we apply different normalizations and reweightings, and instead of the non-deterministic Hoeffding inequality, we suggest a deterministic version using the Frank–Wolfe algorithm. Our new suggested normalizations (and reweightings) allow us to generalize the result to many more applications as in Section 4.

Figure 1. Illustration of Algorithm 2, its normalization of the input, its main applications (red boxes) and their plugged parameters. Algorithm 2 utilizes and boosts the run-time of the Frank–Wolfe algorithm for those applications; see Section 1.4.

For brevity purposes, all proofs of the technical results can be found at the Appendix A, Appendix B, Appendix C, Appendix D, Appendix E, Appendix F, Appendix G, Appendix H, Appendix I.

Theorem 1

(Coreset for points in the unit ball). Let $P=\{p_1,\cdots ,p_n\}$ be a set of n points in ${\mathbb{R}}^d$ such that $∥p_i∥\le 1$ for every $i\in \left[n\right]$. Let $\epsilon \in (0,1)$ and $w={(w_1,\cdots ,w_n)}^T\in {[0,1]}^n$ be a distribution vector. For every $x={(x_1,\cdots ,x_n)}^T\in {\mathbb{R}}^n$, define $f\left(x\right)=-{∥{\sum }_{i=1}^n(w_i-x_i)p_i∥}^2$. Let $\tilde{u}$ be the output of a call to $Frank-Wolfe(f,⌈\frac{8}{\epsilon }⌉)$; see Algorithm 1. Then:

(i) 

$\tilde{u}$ is a distribution vector with ${∥\tilde{u}∥}_0\le ⌈\frac{8}{\epsilon }⌉$,

(ii) 

${∥{\sum }_{i=1}^n(w_i-\widetilde{u_i})p_i∥}^2\le \epsilon$, and

(iii) 

$\widetilde{u_i}$ is computed in $O\left(\frac{nd}{\epsilon }\right)$ time.

We now show how to obtain a vector summarization $\epsilon$-coreset of size $O(1/\epsilon )$ in $O\left(\frac{nd}{\epsilon }\right)$ time for any set $P\subseteq {\mathbb{R}}^d$.

Theorem 2

(Vector summarization coreset). Let $(Q,m)$ be a weighted set of n points in ${\mathbb{R}}^d$, $\epsilon \in (0,1)$, and let u be the output of a call to $CoreSet(Q,m,\frac{\epsilon }{16})$; see Algorithm 2. Then, $u\in {\mathbb{R}}^n$ is a vector with ${∥u∥}_0\le \frac{128}{\epsilon }$ non-zero entries that is computed in $O\left(\frac{nd}{\epsilon }\right)$ time, and $(Q,u)$ is a vector summarization ε-coreset for $(Q,m)$.

Algorithm 2:CORESET$(Q,m,\epsilon )$

1: Input: A weigthed set $(Q,m)$ of $n\ge 2$ points in ${\mathbb{R}}^d$ and an error parameter $\epsilon \in (0,1)$. 2: Output: A weight vector $u\in {[0,\infty )}^n$ with $O(1/\epsilon )$ non-zero entries that satisfies Theorem 2. 3: $w:=\frac{m}{{∥m∥}_1}$ 4: ${\mu }_w:={\sum }_{i=1}^n{w}_i{q}_i$ 5: ${\sigma }_w:=\sqrt{{\sum }_{i=1}^n{w}_i{∥q_i-\mu ∥}^2}$ 6: for every $i\in \{1,\cdots ,n\}$ do 7: $p_i:=\frac{q_i-\mu }{\sigma }$ 8: ${\displaystyle p_i^{\prime }:=\frac{{(p_i^T\mid 1)}^T}{{∥(p_i^T\mid 1)∥}^2}}$       {Notice: $∥p_i^{\prime }∥\le 1$.} 9: ${\displaystyle w_i^{\prime }:=\frac{w_i{∥(p_i^T\mid 1)∥}^2}{2}}$ 10: end for 11: Compute a sparse vector $u^{\prime }$ with $O(1/\epsilon )$ non-zero entries, such that ${∥{\sum }_{i=1}^n(w_i^{\prime }-u_i^{\prime })p_i^{\prime }∥}^2\le \epsilon$ {E.g., using Algorithm 1 (see Theorem 1).} 12: ${\displaystyle u_i:={∥m∥}_1·\frac{2u_i^{\prime }}{{∥(p_i^T\mid 1)∥}^2}}$    for every $i\in \left\{1,\cdots ,n\right\}$ 13: Returnu

3.1. Boosting the Coreset’s Construction Running Time

In this section, we present Algorithm 3, which aims to boost the running time of Algorithm 1 from the previous section; see Theorem 3. The main idea behind this new boosted algorithm is as follows: instead of running the Frank–Wolfe algorithm on a (full) set of input data, it can be more efficient to partition the input into a constant number k of equal-sized chunks, pick some representative for each chunk (its mean), run the Frank–Wolfe algorithm only on the set of representatives (the set of means) to obtain back a subset of those representative, and then continue recursively only with the chunks whose representative was chosen by the algorithm. Although the Frank–Wolfe algorithm is now applied multiple times (rather than once), each of those runs is much more efficient since only the small set of representatives is considered.

This immediately implies a faster construction time of vector summarization $\epsilon$-coresets for general input sets; see Corollary 4 and Figure 1 for illustration.

Theorem 3

(Faster coreset for points in the unit ball). Let P be a set of n points in ${\mathbb{R}}^d$ such that $∥p∥\le 1$ for every $p\in P$. Let $w:P\to (0,1)$ be a weights function such that ${\sum }_{p\in P}w\left(p\right)=1$, $\epsilon \in (0,1)$, and let $(C,u)$ be the output of a call to $Fast-FW-CoreSet(P,w,\epsilon )$; see Algorithm 3. Then

(i) 

$\left|C\right|\le 8/\epsilon$ and ${\sum }_{p\in C}u\left(p\right)=1$,

(ii) 

${∥{\sum }_{p\in P}w\left(p\right)p-{\sum }_{p\in C}u\left(p\right)p∥}^2\le 2\epsilon$, and

(iii) 

$(C,u)$ is computed in $O\left(nd+\frac{d·log{\left(n\right)}^2}{{\epsilon }^2}\right)$ time.

Corollary 4

(Faster vector summarization coreset). Let $(Q,m)$ be a weighted set of n points in ${\mathbb{R}}^d$, and let $\epsilon \in (0,1)$. Then in $O\left(nd+d·\frac{log{\left(n\right)}^2}{{\epsilon }^2}\right)$ time, we can compute a vector $u={(u_1,\cdots ,u_n)}^T\in {\mathbb{R}}^n$, such that u has ${∥u∥}_0\le 128/\epsilon$ non-zero entries and $(Q,u)$ is a vector summarization $\left(2\epsilon \right)$-coreset for $(Q,m)$.

Algorithm 3:FAST-FW-CORESET$(P,w,\epsilon )$

1: Input: A weighed set $(P,w)$ of $n\ge 2$ points in ${\mathbb{R}}^d$ and an error parameter $\epsilon \in (0,1)$. 2: Output: A pair $(C,u)$ that satisfies Theorem 3 3: $k:=\frac{2log\left(n\right)}{\epsilon }$ 4: if$\left|P\right|\le k$then 5: Return: A vector summarization $\epsilon$-corset for $(P,w)$ using Theorem 1. 6: end if 7: $\left\{P_1,\cdots ,P_k\right\}:=$ a partition of P into k disjoint subsets, each contains at most $⌈n/k⌉$ points. 8: for every $i\in \left\{1,\cdots ,k\right\}$ do 9: ${\displaystyle {\mu }_i:=\frac{1}{{\sum }_{q\in P_i}w\left(q\right)}·\sum _{p\in P_i}w\left(p\right)·p}$ {The weighted mean of $P_i$} 10: $w^{\prime }\left({\mu }_i\right):={\sum }_{p\in P_i}w\left(p\right)$ 11: end for 12: $(\tilde{\mu },\tilde{u}):=$ a vector summarization $\left(\frac{\epsilon }{log\left(n\right)}\right)$-corset for the weighted set $(\left\{{\mu }_1,\cdots ,{\mu }_k\right\},w^{\prime })$ using Theorem 1. 13: ${\displaystyle C:=\bigcup _{{\mu }_i\in \tilde{\mu }}{P}_i}$ {C is the union over all subsets $P_i$ whose mean ${\mu }_i$ was chosen in $\tilde{\mu }$.} 14: for every ${\mu }_i\in \tilde{\mu }$ and $p\in P_i$ do 15: ${\displaystyle u\left(p\right):=\frac{\tilde{u}\left({\mu }_i\right)w\left(p\right)}{{\sum }_{q\in P_i}w\left(q\right)}}$ 16: end for 17: $(C,u):=Fast-FW-CoreSet(C,u,\epsilon )$ 18: Return:$(C,u)$

In what follows, we show how to compute a vector summarization coreset with high probability in a time that is sublinear in the input size $\left|Q\right|=n$. This is based on the geometric median trick, that suggests the following procedure: (i) sample $k>1$ sets $\left\{S_1,\cdots ,S_k\right\}$ of the same (small) size from the original input set Q, (ii) for each such sampled set $S_i$ ($i\in \left[k\right]$), compute its mean ${\overline{s}}_i$, and finally, (iii) compute and return the geometric median of those means $\overline{s}=\left\{{\overline{s}}_1,\cdots ,{\overline{s}}_k\right\}$. This geometric median is guaranteed to approximate the mean of the original input set Q.

We show that there is no need to compute this geometric median, as it is a difficult computational task. We prove that there exists a set $S_{i^{\ast }}$ from the sampled subsets such that its mean ${\overline{s}}_{i^{\ast }}$ is very close to this geometric median, with high probability. Thus, ${\overline{s}}_{i^{\ast }}$ is a good approximation to the desired mean of the original input set. Furthermore, we show that ${\overline{s}}_{i^{\ast }}$ is simply the point in $\overline{s}$ that minimizes its sum of (non-squared) distances to this set $\overline{s}$, i.e., $i^{\ast }\in {arg\; min}_{j\in \left[k\right]}{\sum }_{i=1}^k{∥{\overline{s}}_i-{\overline{s}}_j∥}_2$. An exhaustive search over the points of $\overline{s}$ can thus recover ${\overline{s}}_{i^{\ast }}$. The corresponding set $S_{i^{\ast }}$ is the resulted vector summarization coreset; see Lemma 5 and Algorithm 4.

Lemma 5

(Fast probabilistic vector summarization coreset). Let Q be a set of n points in ${\mathbb{R}}^d$, $\mu =\frac{1}{n}{\sum }_{p\in P}q$, and ${\sigma }^2=\frac{1}{n}{\sum }_{p\in P}{∥q-\mu ∥}^2$. Let $\epsilon \in (0,1)$, $\delta \in (0,0.9]$, and let $S\subseteq {\mathbb{R}}^d$ be the output of a call to $Prob-Weak-Coreset(Q,\epsilon ,\delta )$; see Algorithm 4. Then:

(i) 

$S\subseteq Q$ and $\left|S\right|=\frac{4}{\epsilon }$,

(ii) 

with probability at least $1-3\delta$ we have ${∥\frac{1}{\left|S\right|}{\sum }_{p\in S}p-\mu ∥}^2\le 33·\epsilon {\sigma }^2$, and

(iii) 

S is computed in $O\left(dlog{\left(\frac{1}{\delta }\right)}^2+\frac{dlog\left(\frac{1}{\delta }\right)}{\epsilon }\right)$ time.

Algorithm 4:PROB-WEAK-CORESET$(Q,\epsilon ,\delta )$

1: Input: A set Q of $n\ge 2$ points in ${\mathbb{R}}^d$, $\epsilon \in (0,1)$, and $\delta \in (0,1)$. 2: Output: A subset $S\subseteq P$ that satisfies Lemma 5. 3: $k:=3.5log\left(\frac{1}{\delta }\right)+1$. 4: $S:=$ an i.i.d sample of size $\frac{4k}{\epsilon }$. 5: $\left\{S_1,\cdots ,S_k\right\}:=$ a partition of S into k disjoint subsets, each contains $\frac{4}{\epsilon }$ points. 6: ${\overline{s}}_i:=$ the mean of the i’th subset $S_i$ for $i\in \left[k\right]$. 7: $i^{\ast }:={arg\; min}_{j\in \left[k\right]}{\sum }_{i=1}^k{∥{\overline{s}}_i-{\overline{s}}_j∥}_2$. 8: Return$S_{i^{\ast }}$

4. Applications

Coreset for 1-mean. A 1-mean $\epsilon$-coreset for $(Q,m)$ is a weighted set $(Q,u)$ such that for every $x\in {\mathbb{R}}^d$, the sum of squared distances from x to either $(Q,m)$ or $(Q,u)$, is approximately the same. To maintain the above property, we prove that it suffices for $(Q,u)$ to satisfy the following: the mean, the variance, and the sum of weights of $(Q,u)$ should approximate the mean, the variance, and the sum of weights of $(Q,m)$, respectively, up to an additive error that depends linearly on $\epsilon$. Then note that when plugging ${\epsilon }^2$ (rather than $\epsilon$) as input to Algorithm 2, the output is guaranteed to satisfy the above 3 properties, by construction of u.

The following theorem computes a 1-mean $\epsilon$-coreset.

Theorem 6.

Let $(Q,m)$ be a weighted set of n points in ${\mathbb{R}}^d$, $\epsilon \in (0,1)$. Then in

$$O\left(min\left\{nd+d·\frac{log{\left(n\right)}^2}{{\epsilon }^4},\frac{nd}{{\epsilon }^2}\right\}\right)$$

time we can compute a vector $u={(u_1,\cdots ,u_n)}^T\in {\mathbb{R}}^n$, where ${∥u∥}_0\le \frac{128}{{\epsilon }^2}$, such that:

$$\forall x\in {\mathbb{R}}^d:\left|\sum _{i=1}^n(m_i-u_i){∥q_i-x∥}^2\right|\le \epsilon \sum _{i=1}^n{m}_i{∥q_i-x∥}^2.$$

Coreset for KDE. Given two sets of points Q and $Q^{\prime }$, and a kernel $\mathbb{K}:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}$ that is defined by the kernel map $\varphi$, the maximal difference

$$\underset{x\in {\mathbb{R}}^d}{sup}\left|\sum _{q\in Q}\frac{\mathbb{K}(x,q)}{\left|Q\right|}-\sum _{q^{\prime }\in Q^{\prime }}\frac{\mathbb{K}(x,q^{\prime })}{|Q^{\prime }|}\right|$$

between the kernel costs of Q and $Q^{\prime }$ is upper bounded by ${∥{\mu }_{\widehat{Q}}-{\mu }_{\widehat{Q^{\prime }}}∥}_2$, where ${\mu }_{\widehat{Q}}$ and ${\mu }_{\widehat{Q^{\prime }}}$ are the means of $\widehat{Q}=\left\{\varphi \left(q\right)\mid q\in Q\right\}$ and ${\widehat{Q}}^{\prime }=\left\{\varphi \left(q\right)\mid q\in Q^{\prime }\right\}$, respectively, [49]. Given $\widehat{Q}$, we can compute a vector summarization ${\epsilon }^2$-coreset ${\widehat{Q}}^{\prime }$, which satisfies that ${∥{\mu }_{\widehat{Q}}-{\mu }_{\widehat{Q^{\prime }}}∥}_2^2\le {\epsilon }^2$. By the above argument, this is also an $\epsilon$-KDE coreset.

Coreset for dimensionality reduction and LMS solvers. An $\epsilon$-coreset for the k-SVD (k-PCA) problem of Q is a small weighted subset of Q that approximates the sum of squared distances from the points in Q to every non-affine (affine) k-dimensional subspace of ${\mathbb{R}}^d$, up to a multiplicative factor of $1\pm \epsilon$; see Corollary 7. Coreset for LMS solvers is the special case of $k=d-1$.

In [35], it is shown how to leverage an ${(\epsilon /k)}^2$-coreset for the vector summarization problem in order to compute an $\epsilon$-coreset for k-SVD. In [45], it is shown how to compute a coreset for k-PCA via a coreset for k-SVD, by simply adding another entry with some value $r\in \mathbb{R}$ to each vector of the input. Algorithm 5 combines both the above reductions, along with a computation of a vector summarization ${(\epsilon /k)}^2$-coreset to compute the desired coreset for dimensionality reduction (both k-SVD and k-PCA). To compute the vector summarization coreset we utilize our new algorithms from the previous sections, which are faster than the state of the art algorithms.

Algorithm 5:DIM-CORESET$(A,k,\epsilon )$

1: Input: A matrix $A\in {\mathbb{R}}^{n\times d}$, an integer $k\in \left[d\right]$, and an error parameter $\epsilon \in (0,1)$. 2: Output: A diagonal matrix $W\in {\mathbb{R}}^{n\times n}$ that satisfies Corollary 7. 3: ${\displaystyle r:=1+\underset{i\in \left[n\right]}{max}\frac{4{∥a_i∥}^2}{{\epsilon }^4}}$ {where $a_i$ is the ith row of A} 4: $U,\Sigma ,V$: = the full SVD of $[A\mid {(r,\cdots ,r)}^T]\in {\mathbb{R}}^{n\times (d+1)}$ 5: $v_i:=\left(U_{i,1},\cdots ,U_{i,k},\frac{U_{i,k+1:d}{\Sigma }_{k+1:d,k+1:d}}{{∥{\Sigma }_{k+1:d,k+1:d}∥}_F}\right)$ for every $i\in \left[n\right]$ 6: $\widetilde{v_i}$:= the row stacking of $v_i{v}_i^T\in {\mathbb{R}}^{d\times d}$ for every $i\in \left[n\right]$ 7: $(\left\{\widetilde{v_1},\cdots ,\widetilde{v_n}\right\},u)$: = a vector summarization ${\left(\frac{\epsilon }{5k}\right)}^2$-coreset for $(\left\{\widetilde{v_1},\cdots ,\widetilde{v_n}\right\},(1,\cdots ,1))$. 8: $W:=$ a diagonal matrix in ${\mathbb{R}}^{n\times n}$, where $W_{i,i}=\sqrt{u_i}$, $\forall i\in \left[n\right]$. 9: ReturnW

Corollary 7

(Coreset for dimensionality reduction). Let Q be a set of n points in ${\mathbb{R}}^d$, and let $A\in {\mathbb{R}}^{n\times d}$ be a corresponding matrix containing the points of Q in its rows. Let $\epsilon \in (0,\frac{1}{2})$ be an error parameter, $k\in \left[d\right]$ be an integer, and W be the output of a call to $DIM-CoreSet(A,k,\epsilon )$. Then:

(i) 

W is a diagonal matrix with $O\left(\frac{k^2}{{\epsilon }^2}\right)$ non-zero entries,

(ii) 

W is computed in $O\left(min\left\{n{d}^2+\frac{d^{2}log{\left(n\right)}^2{k}^4}{{\epsilon }^4},\frac{n{d}^2{k}^2}{{\epsilon }^2}\right\}\right)$ time, and

(iii) 

there is a constant c, such that for every $\ell \in {\mathbb{R}}^d$ and an orthogonal $X\in {\mathbb{R}}^{d\times (d-k)}$ we have

$$|1-\frac{{∥W(A-\ell )X∥}_F^2}{{∥(A-\ell )X∥}_F^2}|\le c\epsilon .$$

Here, $A-\ell$ is the subtraction of ℓ from every row of A.

Where do our methods fit in? Theoretically speaking, the 1-mean problem (also known as the arithmetic mean problem), is a widely used tool for reporting central tendencies in the field of statistics, as it is also used in machine learning. As for the practical aspect of such problem, it can be either used to obtain an estimation of the mathematical expectation of signal strength in a area [50], or as an imputation technique used to fill in missing values, e.g., in the context of filling in missing values of heart monitor sensor data [51]. Note that a variant of this problem is widely used in the context of deep learning, namely, the moving averages. Algorithms 3 and 4 can boost such methods when given large-scale datasets. In addition, our algorithms extend also to SVD, PCA, and LMS where these methods are known for their usages and efficiencies in discovering a low dimensional representation of high dimensional data. From a practical point of view, SVD showed promising results when dealing with on calibration of a star sensor on-orbit calibration [52], denoising a 4-dimensional computed tomography of the brain in stroke patients [53], removal of cardiac interference from trunk electromyogram [54], among many other applications.

We propose a summarization technique (see Algorithm 5) that aims to compute an approximation towards the SVD factorization of large-scale datasets where applying the SVD factorization on the dataset is not possible due to insufficient memory or long computational time.

5. Experimental Results

We now apply different coreset construction algorithms presented in this paper to a variety of applications, in order to boost their running time, or reduce their memory storage. We note that a complete open source code is provided [55].

Software/Hardware. The algorithms were implemented in Python $3.6$ [56] using “Numpy” [57]. Tests were conducted on a PC with Intel i9-7960X CPU @2.80 GHz x 32 and 128 Gb RAM.

We compare the following algorithms: (To simply distinguish between our algorithms and the competing ones in the graphs, observe that the labels of our algorithms starts with the prefix “Our-”, while the competing methods do not.)

(i)

Uniform: Uniform random sample of the input Q, which requires sublinear time to compute.

(ii)

Sensitivity-sum: Random sampling based on the “sensitivity” for the vector summarization problem [58]. Sensitivity sampling is a widely known technique [19], which guarantees that a subsample of sufficient size approximates the input well. The sensitivity of a point $q\in Q$ is $\frac{1}{n}+\frac{{∥q∥}^2}{{\sum }_{q^{\prime }\in Q}{∥q^{\prime }∥}^2}$. This algorithm takes $O\left(nd\right)$ time.

(ii)

ICML17: The vector summarization coreset construction algorithm from [29] (see Algorithm 2 there), which runs in $O(nd/\epsilon )$ time.

(iv)

Our-rand-sum: Our coreset construction from Lemma 5, which requires $O\left(dlog{\left(\frac{1}{\delta }\right)}^2+\frac{dlog\left(\frac{1}{\delta }\right)}{\epsilon }\right)$ time.

(v)

Our-slow-sum: Our coreset construction from Corollary 2, which requires $O(nd/\epsilon )$ time.

(vi)

Our-fast-sum: Our coreset construction from Corollary 4, which requires $O(nd+dlog{\left(n\right)}^2/{\epsilon }^2)$ time.

(vii)

Sensitivity-svd: Similar to Sensitivity-sum above, however, now the sensitivity is computed by projecting the rows of the input matrix A on the optimal k-subspace (or an approximation of it) that minimizes its sum of squared distances to the rows of A, and then computing the sensitivity of each row i in the projected matrix $A^{\prime }$ as ${∥u_i∥}^2$, where $u_i$ is the ith row the matrix U from the SVD of $A^{\prime }=UD{V}^T$; see [37]. This takes $O\left(ndk\right)$ time.

(viii)

NIPS16: The coreset construction algorithm from [35] (see Algorithm 2 there) which requires $O(n{d}^2{k}^2/{\epsilon }^2)$ time.

(ix)

Our-slow-svd: Corollary 7 offers a coreset construction for SVD using Algorithm 5, which utilizes Algorithm 2. However, Algorithm 2 either utilizes Algorithm 1 (see Theorem 2) or Algorithm 3 (see Theorem 4). Our-slow-svd applies the former option, which requires $O(n{d}^2{k}^2/{\epsilon }^2))$ time.

(x)

Our-fast-svd: Corollary 7 offers a coreset construction for SVD using Algorithm 5, which utilizes Algorithm 2. However, Algorithm 2 either utilizes Algorithm 1 (see Theorem 2) or Algorithm 3 (see Theorem 4). Our-fast-svd uses the latter option, which requires $O(n{d}^2+d^{2}log{\left(n\right)}^2{k}^4/{\epsilon }^4)$ time.

Datasets. We used the following datasets from the UCI ML library [59]:

(i)

New York City Taxi Data [60]. The data covers the taxi operations at New York City. We used the data describing $n=14.7M$ trip fares at the year of 2013. We used the $d=6$ numerical features (real numbers).

(ii)

US Census Data (1990) [61]. The dataset contains $n=2.4M$ entries. We used the entire $d=68$ real-valued attributes of the dataset.

(iii)

Buzz in social media Data Set [62]. It contains $n=0.5M$ examples of buzz events from two different social networks: Twitter, and Tom’s Hardware. We used the entire $d=77$ real-valued attributes.

(iv)

Gas Sensors for Home Activity Monitoring Data Set [63]. This dataset has $n=919,438$ recordings of a gas sensor array composed of 8 MOX gas sensors, and a temperature and humidity sensor. We used the last $d=10$ real-valued attributes of the dataset.

Discussion regarding the chosen datasets. The Buzz in social media data set is widely used in the context of Principal Component Regression (or, PCR in short), that is used for estimating the unknown regression coefficients in a standard linear regression model. The goal of PCR in the context of this dataset, is to predict popularity of a certain topic on Twitter over a period. It is known that the solution of the PCR problem can be approximated using the known SVD decomposition problem. Our techniques enable us to benefit from the coreset advantages, e.g., to boost the PCR approximated solution (PCA) while using low memory, and supporting the streaming model by maintaining a coreset for the data (tweets) seen so far; each time a new point (tweet) is received, it is added to current stored coreset in memory. Once the stored coreset is large enough, our compression (coreset construction algorithm) is applied. This procedure is repeated until the stream of points is empty.

The New York City taxi data contains information about the locations of passengers as well as the locations of their destinations. Thus, the goal is to find a location which is close to the most wanted destinations. This problem can be formulated as a facility location problem, which can be reduced to an instance of the 1-mean problem. Hence, since our methods admit faster solution as well as provable approximation for the facility location problem, we can leverage our coreset to speed up the computations using this dataset.

Finally, regarding the remaining datasets, PCA has been widely used either for low-dimensional embedding or, e.g., to compute the arithmetic mean. By using our methods, we can boost the PCA while admitting an approximated solution.

The experiments.

(i)

Vector summarization:The goal is to approximate the mean of a huge input set, using only a small weighted subset of the input. The empirical approximation error is defined as ${∥\mu -{\mu }_s∥}^2$, where $\mu$ is the mean of the full data and ${\mu }_s$ is the mean of the weighted subset computed via each compared algorithm; see Figure 2 and Figure 3.

In Figure 2, we report the empirical approximation error ${∥\mu -{\mu }_s∥}^2$ as a function of the subset (coreset) size, for each of the datasets (i)–(ii), while in Figure 3 we report the overall computational time for computing the subset (coreset) and for solving the 1-mean problem on the coreset, as a function of the subset size.

(ii)

k-SVD:The goal is to compute the optimal k-dimensional non-affine subspaces of a given input set. We can either compute the optimal subspace using the original (full) input set, or using a weighted subset (coreset) of the input. We denote by $S^{\ast }$ and $S^{\prime }$ the optimal subspace when computed either using the full data or using the subset at hand, respectively. The empirical approximation error is defined as the ratio $|(c^{\ast }-c^{\prime })/c^{\ast }|$, where $c^{\ast }$ and $c^{\prime }$ are the sum of squared distances between the points of original input set to $S^{\ast }$ and $S^{\prime }$, respectively; see Figure 4, Figure 5 and Figure 6. Intuitively, this ratio represents the relative SSD error of recovering an optimal k-dimensional non-affine subspace on the compression, rather than using the full data.

In Figure 4 we report the empirical error $|(c^{\ast }-c^{\prime })/c^{\ast }|$ as a function of the coreset size. In Figure 5 we report the overall computational time in took to compute the coreset and to recover the optimal subspace using the coreset, as a function of the coreset size. In both figures we have three subfigures, each one for a different chosen value of k (the dimension of the subspace). Finally, in Figure 6 the x axis is the size of the dataset (which we compress to a subset of size 150), while the y-axis is the approximation error on the left hand side graph, and on the right hand side it is the overall computational time it took to compute the coreset and to recover the optimal subspace using the coreset.

Figure 2. Experimental results for vector summarization. The x axis is the size of the subset (coreset), while the y axis is the approximation error ${∥\mu -{\mu }_s∥}^2$. The difference between the two graphs is the chosen dataset.

Figure 3. Experimental results for vector summarization. The x axis is the size of the subset (coreset), while the y axis is the overall time took to compute the coreset and to solve the problem on it. The difference between the two graphs is the chosen dataset.

Figure 4. Experimental results for k-SVD, we used Dataset (iii). The x axis is the size of the subset (coreset), while the y axis is the approximation error $\epsilon$. The difference between the 3 graphs is the chosen low dimension k.

Figure 5. Experimental results for k-SVD, we used Dataset (iii). The x axis is the size of the subset (coreset), while the y axis is the overall time took to compute the coreset and to solve the problem on it. The difference between the 3 graphs is the chosen low dimension k.

Figure 6. Experimental results for k-SVD, we used Dataset (iii). The x axis is the size of the dataset which we compress to subsample of size 150, while the y-axis is the approximation error in the left hand side graph, and in the right hand side it is the overall time took to compute the coreset and to solve the problem on it.

5.1. Discussion

Vector summarization experiment:As predicted by the theory and as demonstrated in Figure 2 and Figure 3, our fast and deterministic algorithm Our-fast-sum (the red line in the figures) achieves either the same or smaller approximation errors in most cases compared to the deterministic alternatives Our-slow-sum (orange line) and ICML17 (green line), while being up to $\times 10$ times faster. Hence, when we seek a fast time deterministic solution for computing a coreset for the vector summarization problem, our algorithm Our-fast-sum is the favorable choice.

Compared to the randomized alternatives, Our-fast-sum is obviously slower, but achieves an error more than 3 orders of magnitude smaller. However, our fast and randomized algorithm Our-rand-sum (brown line) constantly achieves better results compared to the other randomized alternatives; It yields approximation error up to $\times 50$ smaller, while maintaining the same computational time. This is demonstrated on both datasets. Hence, our compression can be used to speed up tasks, e.g., computing the PCA or PCR, as described above.

k-SVD experiment: Here, in Figure 4, Figure 5 and Figure 6 we witness a similar phenomena, where our fast and deterministic algorithm Our-fast-svd achieves the same or smaller approximation errors compared to the deterministic alternatives Our-slow-svd and NIPS16, respectively, while being up to $\times 4$ times faster. Compared to the randomized alternatives, Our-fast-svd is slower as predicted, but achieves an error up to 2 orders of magnitude smaller. This is demonstrated for increasing sample sizes (as in Figure 4 and Figure 5), for increasing dataset size (as in Figure 6), and for various values of k (see Figure 4, Figure 5 and Figure 6).

5.2. Conclusions and Future Work

This paper generalizes the definition of $\epsilon$-sample and coreset from the worst case error over every query to average ${\ell }_2$ error. We then showed a reduction from the problem of computing such coresets to the vector summarization coreset construction problem. Here, we suggest deterministic and randomized algorithms for computing such coresets, the deterministic version takes $O(min\left\{nd/\epsilon ,nd+dlog{\left(n\right)}^2/{\epsilon }^2\right\})$, and the randomized $O(dlog{\left(\frac{1}{\delta }\right)}^2+\frac{dlog\left(\frac{1}{\delta }\right)}{\epsilon })$. Finally, we showed how to leverage an $\left({\epsilon }^2\right)$-coreset for the vector summarization problem in order to compute an $\epsilon$-coreset for the 1-mean problem, and similarly for k-SVD and k-PCA problem via computing an ${(\epsilon /k)}^2$ vector summarization coreset after some reprocessing on the data.

Open problems include generalizing these results for other types of norms, or other functions such as M-estimators that are robust to outliers. We hope that the source code and the promising experimental results would encourage also practitioners to use these new types of approximations. Normalization via this new sensitivity type reduced the bounds on the number of iterations of the Frank–Wolfe algorithm by orders of magnitude. We believe that it can be used more generally for provably faster convex optimization, independently of coresets or $\epsilon$-samples. We leave this for future research.

Appendix A.1. Reduction to Normalized Weighted Set

In this section, we argue that in order to compute a vector summarization $\epsilon$-coreset for an input weighted set $(Q,m)$, it suffices to compute a vector summarization $\epsilon$-coreset for its corresponding normalized (and much simpler) weighted set $(P,w)$ as in Definition A1; see Corollary A1. However, first, in Observation A1, we show how to compute a corresponding normalized weighted set $(P,w)$ for any input weighted set $(Q,m)$.

Observation A1.

Let $Q=\{q_1,\cdots ,q_n\}$ be a set of $n\ge 2$ points in ${\mathbb{R}}^d$, $m\in {(0,\infty )}^n$, $w\in {(0,1]}^n$ be a distribution vector such that $w=\frac{m}{{∥m∥}_1}$, $\mu ={\sum }_{i=1}^n{w}_i{q}_i$ and $\sigma =\sqrt{{\sum }_{i=1}^n{w}_i{∥q_i-\mu ∥}^2}$. Let $P=\left\{p_1,\cdots ,p_n\right\}$ be a set of n points in ${\mathbb{R}}^d$, such that for every $j\in \left[n\right]$ we have $p_j=\frac{q_j-\mu }{\sigma }$. Then, $(P,w)$ is the corresponding normalized weighted set of $(Q,m)$, i.e., (i)–(iii) hold as follows:

(i)

${\sum }_{i=1}^n{w}_i=1$,

(ii)

${\sum }_{i=1}^n{w}_i{p}_i=\mathbf{0}$, and

(iii)

${\sum }_{i=1}^n{w}_i{∥p_i∥}^2=1$.

Proof. 

$$\sum _{i=1}^n{w}_i=1,\mathrm{immediately}\mathrm{holds}\mathrm{by}\mathrm{the}\mathrm{definition}\mathrm{of}w.$$

$$\begin{array}{cc}\hfill \sum _{i=1}^n{w}_i{p}_i& =\sum _{i=1}^n{w}_i·\frac{q_i-\mu }{\sigma }=\frac{1}{\sigma }\left(\sum _{i=1}^n{w}_i{q}_i-\sum _{i=1}^n{w}_i\mu \right)\hfill \\ & =\frac{1}{\sigma }\left(\mu -\sum _{i=1}^n{w}_i\mu \right)=\frac{1}{\sigma }\mu \left(1-\sum _{i=1}^n{w}_i\right)=0,\hfill \end{array}$$

where the first equality holds by the definition of $p_i$, the third holds by the definition of $\mu$, and the last is since w is a distribution vector.

$$\begin{array}{cc}\hfill \sum _{i=1}^n{w}_i{∥p_i∥}^2& =\sum _{i=1}^n{w}_i{∥\frac{q_i-\mu }{\sigma }∥}^2=\frac{1}{{\sigma }^2}\sum _{i=1}^n{w}_i{∥q_i-\mu ∥}^2\hfill \\ & =\frac{{\sum }_{i=1}^n{w}_i{∥q_i-\mu ∥}^2}{{\sum }_{i=1}^n{w}_i{∥q_i-\mu ∥}^2}=1,\hfill \end{array}$$

where the first and third equality hold by the definition of $p_i$ and $\sigma$, respectively. □

Corollary A1.

Let $(Q,m)$ be a weighted set, and let $(P,w)$ be its corresponding normalized weighted set as computed in Observation A1. Let $(P,u)$ be a vector summarization ε-coreset for $(P,w)$ and let $u^{\prime }={∥m∥}_1·u$. Then $(Q,u^{\prime })$ is a vector summarization ε-coreset for $(Q,m)$.

Proof. 

Put $x\in {\mathbb{R}}^d$ and let $y=\frac{x-\mu }{\sigma }$. Now, for every $j\in \left[n\right]$, we have that

$$\begin{array}{cc}\hfill {∥q_j-x∥}^2& ={∥\sigma p_j+\mu -(\sigma y+\mu )∥}^2\hfill \\ \hfill & ={∥\sigma p_j-\sigma y∥}^2\hfill \\ \hfill & ={\sigma }^2\left|\right|p_j-y{\left|\right|}^2,\hfill \end{array}$$

(A1)

where the first equality is by the definition of y and $p_j$.

Let $(P,u)$ be a vector summarization $\epsilon$-coreset for $(P,w)$. We prove that $(Q,u^{\prime })$ is a vector summarization $\epsilon$-coreset for $(Q,m)$. We observe the following

$$\begin{array}{cc}\hfill & {∥\sum _{i=1}^n\frac{m_i}{{∥m∥}_1}{q}_i-\sum _{i=1}^n\frac{u_i^{\prime }}{{∥u^{\prime }∥}_1}{q}_i∥}^2={∥\sum _{i=1}^n{w}_i{q}_i-\sum _{i=1}^n\frac{u_i}{{∥u∥}_1}{q}_i∥}^2\hfill \\ & ={∥\sum _{i=1}^n(w_i-\frac{u_i}{{∥u∥}_1})(p_i\sigma +\mu )∥}^2\hfill \\ & ={∥\sum _{i=1}^n(w_i-\frac{u_i}{{∥u∥}_1})p_i\sigma +\mu \sum _{i=1}^n(w_i-\frac{u_i}{{∥u∥}_1})∥}^2\hfill \\ & ={∥\sum _{i=1}^n(w_i-\frac{u_i}{{∥u∥}_1})\left(p_i\sigma \right)∥}^2\le \epsilon {\sigma }^2\hfill \end{array}$$

(A2)

where the first equality holds since $w=\frac{m}{{∥m∥}_1}$ and $\frac{u^{\prime }}{{∥u^{\prime }∥}_1}=\frac{mu}{m{∥u∥}_1}=\frac{u_i}{{∥u∥}_1}$, the second holds by (A1), and the last inequality holds since $(P,u)$ is a vector summarization $\epsilon$-coreset for $(P,w)$. □

Appendix A.2. Vector Summarization Problem Reduction

Given a normalized weighted set $(P,w)$ as in Definition A1, in the following lemma we prove that a weighted set $(P,u)$ is a vector summarization $\epsilon$-coreset for the normalized weighted set $(P,w)$ if and only if the squared ${\ell }_2$ norm of the weighted mean of $(P,u)$ is smaller than $\epsilon$.

Lemma A2.

Let $(P,w)$ be a normalized weighted set of n points in ${\mathbb{R}}^d$, $\epsilon \in (0,1)$, and $u\in {\mathbb{R}}^n$ be a weight vector. Let $\overline{p}={\sum }_{i=1}^n\frac{w_i}{{∥w∥}_1}{p}_i$, $\overline{s}={\sum }_{i=1}^n\frac{u_i}{{∥u∥}_1}{p}_i$, and ${\sigma }^2={∥p_i-\overline{p}∥}^2$. Then, $(P,u)$ is a vector summarization ε-coreset for $(P,w)$, i.e., ${∥\overline{p}-\overline{s}∥}^2\le \epsilon {\sum }_{i=1}^n{w}_i{\sigma }^2$ if and only if ${∥\overline{s}∥}^2\le \epsilon$.

Proof. 

The proof holds since $(P,w)$ is a normalized wighted set, i.e., $\overline{p}=0$, and ${\sigma }^2=1$. □